Tracking UI

Ivory uses MLFlow Tracking for the workflow tracking and model saving. For this feature, the Client instace has to have a Tracker instance.

First create several runs for demonstration.

import ivory

client = ivory.create_client("examples")
run = client.create_run('torch')
run.start('both')

[3] 2020-06-20 15:24:22 (1.26s) python3 (50.4s)

[I 200620 15:24:22 tracker:48] A new experiment created with name: 'torch'
[epoch#0] loss=19.56 val_loss=7.696 lr=0.001 best
[epoch#1] loss=7.47 val_loss=6.247 lr=0.001 best
[epoch#2] loss=6.753 val_loss=6.538 lr=0.001
[epoch#3] loss=5.735 val_loss=4.631 lr=0.001 best
[epoch#4] loss=4.799 val_loss=3.765 lr=0.001 best
[epoch#5] loss=4.023 val_loss=2.892 lr=0.001 best
[epoch#6] loss=2.992 val_loss=2.08 lr=0.001 best
[epoch#7] loss=2.187 val_loss=1.411 lr=0.001 best
[epoch#8] loss=1.551 val_loss=1.592 lr=0.001
[epoch#9] loss=1.024 val_loss=0.9122 lr=0.001 best
task = client.create_task('torch')
runs = task.product(fold=range(3), verbose=0)
for run in runs:
    run.start('both')

[4] 2020-06-20 15:24:23 (4.06s) python3 (54.4s)

[run#1] fold=0
[run#2] fold=1
[run#3] fold=2
task = client.create_task('torch')
runs = task.chain(lr=[1e-4, 1e-3], batch_size=[16, 32], verbose=0)
for run in runs:
    run.start('both')

[5] 2020-06-20 15:24:27 (5.18s) python3 (59.6s)

[run#4] lr=0.0001
[run#5] lr=0.001
[run#6] batch_size=16 lr=0.001
[run#7] batch_size=32 lr=0.001
from ivory.utils.range import Range

study = client.create_study('torch', lr=Range(1e-5, 1e-3, log=True))
study.optimize(n_trials=5, verbose=0)

[6] 2020-06-20 15:24:33 (7.22s) python3 (1min7s)

[I 2020-06-20 15:24:33,098] A new study created with name: torch.lr.study#0
[run#8] lr=0.0005617
[I 2020-06-20 15:24:34,544] Finished trial#0 with value: 4.446846199035645 with parameters: {'lr': 0.000561694309300103}. Best is trial#0 with value: 4.446846199035645.
[run#9] lr=2.332e-05
[I 2020-06-20 15:24:35,950] Finished trial#1 with value: 14.365483665466309 with parameters: {'lr': 2.3319404904620393e-05}. Best is trial#0 with value: 4.446846199035645.
[run#10] lr=0.0001296
[I 2020-06-20 15:24:37,379] Finished trial#2 with value: 7.584907138347626 with parameters: {'lr': 0.00012963450909320425}. Best is trial#0 with value: 4.446846199035645.
[run#11] lr=4.874e-05
[I 2020-06-20 15:24:38,817] Finished trial#3 with value: 6.94333416223526 with parameters: {'lr': 4.874482477479321e-05}. Best is trial#0 with value: 4.446846199035645.
[run#12] lr=0.0005352
[I 2020-06-20 15:24:40,234] Finished trial#4 with value: 4.568214583396911 with parameters: {'lr': 0.0005352429859687107}. Best is trial#0 with value: 4.446846199035645.
<optuna.study.Study at 0x142121106c8>

Tracking UI

Optionally, you can update missing parameters:

client.update_params('torch')

[7] 2020-06-20 15:24:40 (357ms) python3 (1min7s)

In a terminal, move to the working directory (examples), then run

$ ivory ui

You can view the UI using URL http://localhost:5000 in your browser.

Table 1 A collection of runs. Parameters, metrics, tags are logged.

png

You can compare the training results among runs.

png

Figure 1 Comparison of training curves

See also the official MLFlow documentation.