Skip to content

Blog#

Executing experiments with Ray

Ray is an open-source unified framework for scaling AI and Python applications. You can use Ray as an execution backend to run MLtraq experiments using its native joblib integration.

The example shows how to set up a local Ray cluster, running an experiment with four workers. The 1000 runs of the experiment are allocated to the 4 workers, 250 each. The contents of the parameter "execution.backend_params" are passed as arguments to ray.init(...).

How can I install Ray?

You can install Ray with: 

pip install "ray[default]"

Executing experiments with Ray

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
from ray import get_runtime_context

from mltraq import Run, create_experiment, options


def get_worker_id(run: Run):
    run.fields.worker_id = get_runtime_context().get_worker_id()


with options().ctx(
    {
        "execution.backend": "ray",
        "execution.backend_params": {
            "logging_level": "FATAL",
            "include_dashboard": False,
            "num_cpus": 4,
        },
    }
):
    e = create_experiment().add_runs(i=range(1000)).execute(get_worker_id)

print(e.runs.df().worker_id.value_counts())
Output
worker_id
c0e0b37ade2ef740207fa962678de2451fe1d0a7d6d812e04e89771c    253
43c709068c1607c78f6454e6f2f7b0df12b5a87f3586680016c9f9e8    251
c95aba5f24c5b06c5a6ed9b116c78fae4e4b7d73718f41041c2e4bf0    248
678267493ca345bbb84834ab5728dced3b7d5aa65fc9a801d38cf7d0    248
Name: count, dtype: int64