Tracking Runs with Ivory

First create several runs for demonstration.

import ivory

client = ivory.create_client("examples")
run = client.create_run('torch', fold=2)
run.start()

[3] 2020-06-20 15:23:52 (1.21s) python3 (20.2s)

[I 200620 15:23:52 tracker:48] A new experiment created with name: 'torch'
[epoch#0] loss=21.27 val_loss=6.933 lr=0.001 best
[epoch#1] loss=7.318 val_loss=5.955 lr=0.001 best
[epoch#2] loss=6.176 val_loss=5.054 lr=0.001 best
[epoch#3] loss=5.281 val_loss=4.591 lr=0.001 best
[epoch#4] loss=4.411 val_loss=3.402 lr=0.001 best
[epoch#5] loss=3.327 val_loss=2.703 lr=0.001 best
[epoch#6] loss=2.485 val_loss=1.81 lr=0.001 best
[epoch#7] loss=1.838 val_loss=1.274 lr=0.001 best
[epoch#8] loss=1.276 val_loss=0.9247 lr=0.001 best
[epoch#9] loss=0.8944 val_loss=0.6791 lr=0.001 best
run = client.create_run('torch', fold=3)
run.start('both')

[4] 2020-06-20 15:23:53 (1.26s) python3 (21.5s)

[epoch#0] loss=21.51 val_loss=9.308 lr=0.001 best
[epoch#1] loss=7.742 val_loss=9.016 lr=0.001 best
[epoch#2] loss=7.198 val_loss=7.626 lr=0.001 best
[epoch#3] loss=6.383 val_loss=6.84 lr=0.001 best
[epoch#4] loss=5.829 val_loss=6.014 lr=0.001 best
[epoch#5] loss=5.034 val_loss=5.606 lr=0.001 best
[epoch#6] loss=4.281 val_loss=4.397 lr=0.001 best
[epoch#7] loss=3.631 val_loss=3.608 lr=0.001 best
[epoch#8] loss=2.765 val_loss=2.891 lr=0.001 best
[epoch#9] loss=2.236 val_loss=2.267 lr=0.001 best
task = client.create_task('torch')
runs = task.product(fold=range(3), verbose=0)
for run in runs:
    pass
    # Do something

[5] 2020-06-20 15:23:54 (447ms) python3 (21.9s)

[run#2] fold=0
[run#3] fold=1
[run#4] fold=2
task = client.create_task('torch')
runs = task.product(n_splits=[3, 4], verbose=0)
for run in runs:
    pass
    # Do something

[6] 2020-06-20 15:23:55 (322ms) python3 (22.3s)

[run#5] n_splits=3
[run#6] n_splits=4
task = client.create_task('torch')
runs = task.chain(lr=[1e-4, 1e-3], batch_size=[16, 32], verbose=0)
for run in runs:
    pass
    # Do something

[7] 2020-06-20 15:23:55 (641ms) python3 (22.9s)

[run#7] lr=0.0001
[run#8] lr=0.001
[run#9] batch_size=16 lr=0.0001
[run#10] batch_size=32 lr=0.0001

Tracking Interface

Search functions

Client.search_run_ids() makes an iterator that returns Run IDs of runs.

# A helper function
def print_run_info(run_ids):
    for run_id in run_ids:
        print(run_id[:5], client.get_run_name(run_id))

run_ids = client.search_run_ids('torch')
print_run_info(run_ids)

[8] 2020-06-20 15:23:56 (100ms) python3 (23.0s)

cb865 run#10
ee268 run#9
86d77 run#8
1e101 run#7
14c20 task#2
cc4d9 run#6
4543c run#5
01d94 task#1
facee run#4
635c5 run#3
732d1 run#2
eb3a8 task#0
84a48 run#1
ce78f run#0

You can filtering runs by passing keyword arguments.

run_ids = client.search_run_ids('torch', lr=1e-4, batch_size=32)
print_run_info(run_ids)

[9] 2020-06-20 15:23:56 (211ms) python3 (23.2s)

cb865 run#10

Client.search_nested_run_ids() makes an iterator that returns Run IDs of runs that have a parent run. Optionally, you can filter runs.

run_ids = client.search_nested_run_ids('torch')
print_run_info(run_ids)

[10] 2020-06-20 15:23:56 (71.0ms) python3 (23.3s)

cb865 run#10
ee268 run#9
86d77 run#8
1e101 run#7
cc4d9 run#6
4543c run#5
facee run#4
635c5 run#3
732d1 run#2

Note that the run#0 isn't returned because it was created by Client.create_run() directly.

Client.search_parent_run_ids() makes an iterator that returns Run IDs of runs that have nested runs. In this case, parent runs are three tasks we made above.

run_ids = client.search_parent_run_ids('torch')
print_run_info(run_ids)

[11] 2020-06-20 15:23:56 (46.0ms) python3 (23.3s)

14c20 task#2
01d94 task#1
eb3a8 task#0

Get functions

Client.get_run_id() returns a Run ID of runs you select by run name.

run_id = client.get_run_id('torch', run=0)
print_run_info([run_id])

[12] 2020-06-20 15:23:56 (39.0ms) python3 (23.4s)

ce78f run#0

Client.get_run_ids() makes an iterator that returns Run IDs of runs you select by run names.

run_ids = client.get_run_ids('torch', task=range(1, 3))
print_run_info(run_ids)

[13] 2020-06-20 15:23:56 (81.7ms) python3 (23.4s)

01d94 task#1
14c20 task#2

Client.get_nested_run_ids() makes an iterator that returns Run IDs of runs that have a parent you select by run names.

run_ids = client.get_nested_run_ids('torch', task=range(2))
print_run_info(run_ids)

[14] 2020-06-20 15:23:56 (152ms) python3 (23.6s)

facee run#4
635c5 run#3
732d1 run#2
cc4d9 run#6
4543c run#5

Client.get_parent_run_id() returns a Run ID of a run that is refered by a nested run.

run_id = client.get_parent_run_id('torch', run=5)
print_run_info([run_id])

[15] 2020-06-20 15:23:56 (41.0ms) python3 (23.6s)

01d94 task#1

Set function

Sometimes, you may want to change a parent for nested runs. Use Client.set_parent_run_id().

run_ids = client.get_nested_run_ids('torch', task=2)
print_run_info(run_ids)

[16] 2020-06-20 15:23:56 (84.0ms) python3 (23.7s)

cb865 run#10
ee268 run#9
86d77 run#8
1e101 run#7
client.set_parent_run_id('torch', run=(0, 2, 3), task=2)
run_ids = client.get_nested_run_ids('torch', task=2)
print_run_info(run_ids)

[17] 2020-06-20 15:23:57 (222ms) python3 (23.9s)

cb865 run#10
ee268 run#9
86d77 run#8
1e101 run#7
635c5 run#3
732d1 run#2
ce78f run#0

Next Step

Once you got Run ID(s), you can load a run, a member of a run, or collect results of multiple runs for an ensemble. See the quickstart.