Multiple Runs
Task
Ivory implements a special run class Task
that controls multiple nested runs. Task
is useful for parameter search or cross validation.
import ivory
client = ivory.create_client("examples") # Set the working directory
task = client.create_task('torch') # Or, experiment.create_task()
task
[3] 2020-06-20 15:23:48 (45.0ms) python3 (15.7s)
[I 200620 15:23:48 tracker:48] A new experiment created with name: 'torch'
Task(id='ff42564ad6bf4384a93355441e17e3ae', name='task#0', num_instances=3)
The Task
class has two functions to generate multiple runs: Task.prodcut()
and Task.chain()
. These two function have the same functionality as itertools
of Python starndard library.
Product
The Task.prodcut()
makes an iterator that returns runs from cartesian product of input parameters.
task = client.create_task('torch')
# verbose=0: No progress bar.
runs = task.product(fold=range(2), factor=[0.5, 0.7], verbose=0)
runs
[4] 2020-06-20 15:23:48 (38.0ms) python3 (15.7s)
<generator object Task.product at 0x0000014201493248>
for run in runs:
pass # Do somthing, for example, run.start()
[5] 2020-06-20 15:23:48 (514ms) python3 (16.3s)
[run#0] fold=0 factor=0.5
[run#1] fold=0 factor=0.7
[run#2] fold=1 factor=0.5
[run#3] fold=1 factor=0.7
You can specify other parameters that don't change during iteration.
task = client.create_task('torch')
runs = task.product(fold=range(2), factor=[0.5, 0.7], lr=1e-4, verbose=0)
for run in runs:
pass # Do somthing, for example, run.start()
[6] 2020-06-20 15:23:49 (607ms) python3 (16.9s)
[run#4] lr=0.0001 fold=0 factor=0.5
[run#5] lr=0.0001 fold=0 factor=0.7
[run#6] lr=0.0001 fold=1 factor=0.5
[run#7] lr=0.0001 fold=1 factor=0.7
Chain
The Task.chain()
makes an iterator that returns runs from the first input paramter until it is exhausted, then proceeds to the next parameter, until all of the parameters are exhausted. Other parameters have default values if they don't be specified by additional key-value pairs.
task = client.create_task('torch')
runs = task.chain(
fold=range(2),
factor=[0.5, 0.7],
lr=[1e-4, 1e-3],
batch_size=32,
use_best_param=False,
verbose=0)
runs
[7] 2020-06-20 15:23:50 (65.0ms) python3 (16.9s)
<generator object Task.chain at 0x0000014201493148>
for run in runs:
pass # Do somthing, for example, run.start()
[8] 2020-06-20 15:23:50 (908ms) python3 (17.8s)
[run#8] batch_size=32 fold=0
[run#9] batch_size=32 fold=1
[run#10] batch_size=32 factor=0.5
[run#11] batch_size=32 factor=0.7
[run#12] batch_size=32 lr=0.0001
[run#13] batch_size=32 lr=0.001
The use_best_param
keyword argument is useful for dynamic updating of parameters. If True
(default), the parameter that got the best score is used during the following iterations.
task = client.create_task('torch')
runs = task.chain(
fold=range(2),
factor=[0.5, 0.7],
lr=[1e-4, 1e-3],
use_best_param=True,
verbose=0)
for run in runs:
pass # Do somthing, for example, run.start()
# We do nothing, so the first values are used.
[9] 2020-06-20 15:23:51 (1.07s) python3 (18.9s)
[run#14] fold=0
[run#15] fold=1
[run#16] factor=0.5 fold=0
[run#17] factor=0.7 fold=0
[run#18] lr=0.0001 fold=0 factor=0.5
[run#19] lr=0.001 fold=0 factor=0.5
Range
Ivory provides the ivory.utils.range.Range
class for parameter ranging. This
class can be used as the standard range
, but more flexible, especially for the float type.
from ivory.utils.range import Range
list(Range(6)) # The stop value is included.
[10] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)
[0, 1, 2, 3, 4, 5, 6]
list(Range(3, 6)) # Start and stop.
[11] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[3, 4, 5, 6]
list(Range(3, 10, 2)) # Step size.
[12] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[3, 5, 7, 9]
list(Range(3, 10, num=4)) # Sampling size.
[13] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[3, 5, 8, 10]
list(Range(0.0, 1.0, 0.25)) # float type.
[14] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[0.0, 0.25, 0.5, 0.75, 1.0]
list(Range(0.0, 1.0, num=5)) # Sampling size
[15] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[0.0, 0.25, 0.5, 0.75, 1.0]
list(Range(1e-3, 1e2, num=6, log=True)) # Log scale
[16] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
A Range
instance can be created from a string.
list(Range('3-7')) # <start>-<stop>
[17] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)
[3, 4, 5, 6, 7]
list(Range('3-7-2')) # <start>-<stop>-<step>
[18] 2020-06-20 15:23:52 (5.00ms) python3 (18.9s)
[3, 5, 7]
list(Range('0.0-1.0:5')) # <start>-<stop>:<num>
[19] 2020-06-20 15:23:52 (4.00ms) python3 (18.9s)
[0.0, 0.25, 0.5, 0.75, 1.0]
list(Range('1e-3_1e2:6.log')) # '_' instead of '-', log scale
[20] 2020-06-20 15:23:52 (4.00ms) python3 (19.0s)
[0.001, 0.01, 0.1, 1.0, 10.0, 100.0]