Skip to content

Track sequences#

How can I track parameters, time series, metrics and more?#

In this experiment, we demonstrate how to track constants, complex objects, as well as rich multidimensional time series. The sequence tracks also timestamps in the timestamp column, omitted to avoid unnecessary changes whenever we updated the docs. Reloading experiments, access to tracking data and resumed computation is also included.

Example on tracking

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import numpy as np
from mltraq import Run, Sequence, create_session

# Create a new session, bound to an in-memory SQLite database by default.
session = create_session()
print(session)
print("--\n")

# Create a new experiment "example" with two runs and grid parameter "X".
experiment = session.create_experiment("example").add_runs(X=[1, 2])
print(experiment)
print("--\n")


def fibonacci(n):
    """
    Return n-th value in Fibonacci series.
    """
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)


# Define a step function to execute on the runs.
def step(run: Run):
    """
    This step tracks:
     1. the number of times we call it in `N`
     2. A sequential log of Fibonacci numbers generated at each execution in `sequence`
     3. A constant scalar in `score`
     4. A constant Numpy array in `predictions`
    """

    # Dictionary `run.fields` can use it to track a rich set of object types, including:
    # dict, list, set, tuple, int, float, str, and Numpy arrays,
    # Pandas series/frames, PyArrow tables.
    run.fields.X = run.params.X
    run.fields.N = run.fields.get("N", 0)
    run.fields.N += 1
    run.fields.sequence = run.fields.get("sequence", Sequence())
    run.fields.sequence.append(
        n=run.fields.N, fibonacci=fibonacci(run.fields.N)
    )
    run.fields.score = 1
    run.fields.predictions = np.array([1, 2, 3])


# Execute the experiment twice on the two runs, running each time the step function 3 times.
# Functions are chained together, creating a pipeline to execute on each run.
experiment.execute([step] * 3, args_field="args")
experiment.execute([step] * 3, args_field="args")

# Persist the experiment to database.
experiment.persist()

# Reload the experiment from database.
experiment = session.load_experiment("example")

# Execute the step one more time, reaching 7 evaluations per run.
experiment.execute([step], args_field="args")

# Print the tracked sequence in one of the executed runs.
print("Contents of sequence in a single run:")
print(experiment.runs.first().fields.sequence.df()[["n", "fibonacci"]])
print("--\n")

print("Contents of X, N and score on all runs:")
print(experiment.runs.df()[["X", "N", "score"]])
Output
Session(db="sqlite:///:memory:", experiments(0)=[])
--

Experiment(name="example", runs.count=2, id="d65df69e-1175-44a5-be2f-2232765703b8")
--

Contents of sequence in a single run:
     n  fibonacci
0  1.0        1.0
1  2.0        1.0
2  3.0        2.0
3  4.0        3.0
4  5.0        5.0
5  6.0        8.0
6  7.0       13.0
--

Contents of X, N and score on all runs:
   X  N  score
0  2  7      1
1  1  7      1