Ivory

Multiple Runs

Task

Ivory implements a special run class Task that controls multiple nested runs. Task is useful for parameter search or cross validation.

import ivory

client = ivory.create_client("examples")  # Set the working directory
task = client.create_task('torch')  # Or, experiment.create_task()
task

[3] 2020-06-20 15:23:48 (45.0ms) python3 (15.7s)

[I 200620 15:23:48 tracker:48] A new experiment created with name: 'torch'

Task(id='ff42564ad6bf4384a93355441e17e3ae', name='task#0', num_instances=3)

The Task class has two functions to generate multiple runs: Task.prodcut() and Task.chain(). These two function have the same functionality as itertools of Python starndard library.

Product

The Task.prodcut() makes an iterator that returns runs from cartesian product of input parameters.

task = client.create_task('torch')
# verbose=0: No progress bar.
runs = task.product(fold=range(2), factor=[0.5, 0.7], verbose=0)
runs

[4] 2020-06-20 15:23:48 (38.0ms) python3 (15.7s)

<generator object Task.product at 0x0000014201493248>

for run in runs:
    pass  # Do somthing, for example, run.start()

[5] 2020-06-20 15:23:48 (514ms) python3 (16.3s)

[run#0] fold=0 factor=0.5
[run#1] fold=0 factor=0.7
[run#2] fold=1 factor=0.5
[run#3] fold=1 factor=0.7

You can specify other parameters that don't change during iteration.

task = client.create_task('torch')
runs = task.product(fold=range(2), factor=[0.5, 0.7], lr=1e-4, verbose=0)
for run in runs:
    pass  # Do somthing, for example, run.start()

[6] 2020-06-20 15:23:49 (607ms) python3 (16.9s)

[run#4] lr=0.0001 fold=0 factor=0.5
[run#5] lr=0.0001 fold=0 factor=0.7
[run#6] lr=0.0001 fold=1 factor=0.5
[run#7] lr=0.0001 fold=1 factor=0.7

Chain

The Task.chain() makes an iterator that returns runs from the first input paramter until it is exhausted, then proceeds to the next parameter, until all of the parameters are exhausted. Other parameters have default values if they don't be specified by additional key-value pairs.

task = client.create_task('torch')
runs = task.chain(
    fold=range(2),
    factor=[0.5, 0.7],
    lr=[1e-4, 1e-3],
    batch_size=32,
    use_best_param=False,
    verbose=0)
runs

[7] 2020-06-20 15:23:50 (65.0ms) python3 (16.9s)

<generator object Task.chain at 0x0000014201493148>

for run in runs:
    pass  # Do somthing, for example, run.start()

[8] 2020-06-20 15:23:50 (908ms) python3 (17.8s)

[run#8] batch_size=32 fold=0
[run#9] batch_size=32 fold=1
[run#10] batch_size=32 factor=0.5
[run#11] batch_size=32 factor=0.7
[run#12] batch_size=32 lr=0.0001
[run#13] batch_size=32 lr=0.001

The use_best_param keyword argument is useful for dynamic updating of parameters. If True (default), the parameter that got the best score is used during the following iterations.

task = client.create_task('torch')
runs = task.chain(
    fold=range(2),
    factor=[0.5, 0.7],
    lr=[1e-4, 1e-3],
    use_best_param=True,
    verbose=0)
for run in runs:
    pass  # Do somthing, for example, run.start()
    # We do nothing, so the first values are used.

[9] 2020-06-20 15:23:51 (1.07s) python3 (18.9s)

[run#14] fold=0
[run#15] fold=1
[run#16] factor=0.5 fold=0
[run#17] factor=0.7 fold=0
[run#18] lr=0.0001 fold=0 factor=0.5
[run#19] lr=0.001 fold=0 factor=0.5

Range

Ivory provides the ivory.utils.range.Range class for parameter ranging. This class can be used as the standard range, but more flexible, especially for the float type.

from ivory.utils.range import Range

list(Range(6))  # The stop value is included.

[10] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)

[0, 1, 2, 3, 4, 5, 6]

list(Range(3, 6))  # Start and stop.

[11] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[3, 4, 5, 6]

list(Range(3, 10, 2))  # Step size.

[12] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[3, 5, 7, 9]

list(Range(3, 10, num=4))  # Sampling size.

[13] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[3, 5, 8, 10]

list(Range(0.0, 1.0, 0.25))  # float type.

[14] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[0.0, 0.25, 0.5, 0.75, 1.0]

list(Range(0.0, 1.0, num=5))  # Sampling size

[15] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[0.0, 0.25, 0.5, 0.75, 1.0]

list(Range(1e-3, 1e2, num=6, log=True))  # Log scale

[16] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[0.001, 0.01, 0.1, 1.0, 10.0, 100.0]

A Range instance can be created from a string.

list(Range('3-7'))  # <start>-<stop>

[17] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)

[3, 4, 5, 6, 7]

list(Range('3-7-2')) # <start>-<stop>-<step>

[18] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)

[3, 5, 7]

list(Range('0.0-1.0:5')) # <start>-<stop>:<num>

[19] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)

[0.0, 0.25, 0.5, 0.75, 1.0]

list(Range('1e-3_1e2:6.log'))  # '_' instead of '-', log scale

[20] 2020-06-20 15:23:52 (4.00ms) python3 (19.0s)

[0.001, 0.01, 0.1, 1.0, 10.0, 100.0]