Hyperparameter Tuning

Suggest Function

To optimize a set of hyperparameters, define a suggest function. Here are example functions.

File 9 rectangle/suggest.py

def suggest_lr(trial, min=1e-5, max=1e-3):
    trial.suggest_loguniform("lr", min, max)


def suggest_hidden_sizes(trial, max_num_layers, min_size=10, max_size=30):
    num_layers = trial.suggest_int("num_layers", 2, max_num_layers)
    for k in range(num_layers):
        trial.suggest_int(f"hidden_sizes:{k}", min_size, max_size)

A suggest function must take a trial (an instance of Trial) as the first argument but you can add arbitrary arguments if you need. For more details about what the Trial can do, see the offical Optuna documentation.

Note

In the suggest_hidden_sizes(), we use 0-indexed colon-notation, because Optuna doesn't suggest a list itself but its element.

These suggest functions don't return any parameters. The only work of suggest functions is to make the Trial instance suggest parameters. Suggested parameters are stored in the Trial instance, so that nothing is needed from suggest functions.

Note that an objective function in Optuna has only one trial argument, so that we have to use the functools.partial() to make a pure suggest function.

from functools import partial
from rectangle.suggest import suggest_lr, suggest_hidden_sizes

lr = partial(suggest_lr, min=1e-5, max=1e-2)
hidden_sizes = partial(suggest_hidden_sizes, max_num_layers=3)

[3] 2020-06-20 15:23:58 (6.00ms) python3 (24.9s)

Study

Ivory implements a special run class Study that controls hyperparameter tuning using Optuna.

import ivory

client = ivory.create_client("examples")  # Set the working directory
study_lr = client.create_study('torch', lr=lr)
study_hs = client.create_study('torch', hidden_sizes=hidden_sizes)
study_lr

[4] 2020-06-20 15:23:58 (91.0ms) python3 (25.0s)

[I 200620 15:23:58 tracker:48] A new experiment created with name: 'torch'
Study(id='48460cd925134cbeb5033cfb2e91ede7', name='study#0', num_instances=5)

In the Client.create_study(), you can pass a keyword argument in which the key is a suggest name and the value is a pure suggest function.

Objective

The ivory.core.objective.Objective class provides objective functions that return a score to minimize or maximize. But you don't need to know about the Objective class in details. Ivory builds an objective function from a suggest function and sends it to Optuna so that Optuna can optimize the parameters.

A Study instance has an Objective instance.

study_lr.objective

[5] 2020-06-20 15:23:58 (4.00ms) python3 (25.0s)

Objective(['lr'])
study_hs.objective

[6] 2020-06-20 15:23:58 (4.00ms) python3 (25.0s)

Objective(['hidden_sizes'])

Optimization

Then "optimize" the learning rate and hidden sizes just for fun.

optuna_study_lr = study_lr.optimize(n_trials=3, fold=3, epochs=3)

[7] 2020-06-20 15:23:58 (1.94s) python3 (26.9s)

[I 2020-06-20 15:23:58,368] A new study created with name: torch.lr.study#0
[run#0] lr=2.473e-05 fold=3 epochs=3
[epoch#0] loss=84.9 val_loss=81.53 lr=2.473e-05 best
[epoch#1] loss=75.93 val_loss=72.07 lr=2.473e-05 best
[epoch#2] loss=65.7 val_loss=61.3 lr=2.473e-05 best
[I 2020-06-20 15:23:59,020] Finished trial#0 with value: 61.3017183303833 with parameters: {'lr': 2.4732972291277312e-05}. Best is trial#0 with value: 61.3017183303833.
[run#1] lr=0.004571 fold=3 epochs=3
[epoch#0] loss=13.09 val_loss=5.294 lr=0.004571 best
[epoch#1] loss=7.122 val_loss=2.763 lr=0.004571 best
[epoch#2] loss=5.946 val_loss=1.33 lr=0.004571 best
[I 2020-06-20 15:23:59,638] Finished trial#1 with value: 1.3301505535840987 with parameters: {'lr': 0.0045710953332283675}. Best is trial#1 with value: 1.3301505535840987.
[run#2] lr=0.0003579 fold=3 epochs=3
[epoch#0] loss=81.91 val_loss=37.03 lr=0.0003579 best
[epoch#1] loss=12.24 val_loss=9.582 lr=0.0003579 best
[epoch#2] loss=7.788 val_loss=9.338 lr=0.0003579 best
[I 2020-06-20 15:24:00,266] Finished trial#2 with value: 9.337713980674744 with parameters: {'lr': 0.00035786096146409664}. Best is trial#1 with value: 1.3301505535840987.
optuna_study_hs = study_hs.optimize(n_trials=3, epochs=3)

[8] 2020-06-20 15:24:00 (2.02s) python3 (29.0s)

[I 2020-06-20 15:24:00,298] A new study created with name: torch.hidden_sizes.study#1
[run#3] hidden_sizes:0=16 hidden_sizes:1=28 hidden_sizes:2=23 num_layers=3 epochs=3
[epoch#0] loss=37.55 val_loss=5.88 lr=0.001 best
[epoch#1] loss=6.224 val_loss=5.166 lr=0.001 best
[epoch#2] loss=5.125 val_loss=3.951 lr=0.001 best
[I 2020-06-20 15:24:00,974] Finished trial#0 with value: 3.9506407141685487 with parameters: {'hidden_sizes:0': 16, 'hidden_sizes:1': 28, 'hidden_sizes:2': 23, 'num_layers': 3}. Best is trial#0 with value: 3.9506407141685487.
[run#4] hidden_sizes:0=24 hidden_sizes:1=24 num_layers=2 epochs=3
[epoch#0] loss=23.84 val_loss=6.409 lr=0.001 best
[epoch#1] loss=6.523 val_loss=5.263 lr=0.001 best
[epoch#2] loss=5.656 val_loss=4.537 lr=0.001 best
[I 2020-06-20 15:24:01,622] Finished trial#1 with value: 4.536559498310089 with parameters: {'hidden_sizes:0': 24, 'hidden_sizes:1': 24, 'num_layers': 2}. Best is trial#0 with value: 3.9506407141685487.
[run#5] hidden_sizes:0=22 hidden_sizes:1=25 hidden_sizes:2=27 num_layers=3 epochs=3
[epoch#0] loss=26.73 val_loss=8.15 lr=0.001 best
[epoch#1] loss=7.776 val_loss=6.177 lr=0.001 best
[epoch#2] loss=6.573 val_loss=5.906 lr=0.001 best
[I 2020-06-20 15:24:02,293] Finished trial#2 with value: 5.905864572525024 with parameters: {'hidden_sizes:0': 22, 'hidden_sizes:1': 25, 'hidden_sizes:2': 27, 'num_layers': 3}. Best is trial#0 with value: 3.9506407141685487.

Note

By cliking an icon () in the above cells, you can see the Optuna's log.

The returned value of the Study.optimize() is an Optuna's Study instance (not Ivory's one).

optuna_study_lr

[9] 2020-06-20 15:24:02 (4.00ms) python3 (29.0s)

<optuna.study.Study at 0x1420ecf1548>

The Study instance is named after the experiment name, suggest name, and run name.

optuna_study_lr.study_name

[10] 2020-06-20 15:24:02 (4.00ms) python3 (29.0s)

'torch.lr.study#0'

In user attributes that Optuna's Study and Trial instances provide, Run ID is saved.

optuna_study_lr.user_attrs

[11] 2020-06-20 15:24:02 (6.00ms) python3 (29.0s)

{'run_id': '48460cd925134cbeb5033cfb2e91ede7'}
optuna_study_lr.trials[0].user_attrs

[12] 2020-06-20 15:24:02 (6.00ms) python3 (29.0s)

{'run_id': '123700db98b144bb8f0f8a5a59b502a9'}

On the other hand, MLFlow Tracking's run (not Ivory's one) has a tag to refer Optuna's study and trial.

mlflow_client = client.tracker.client
mlflow_client

[13] 2020-06-20 15:24:02 (4.00ms) python3 (29.0s)

<mlflow.tracking.client.MlflowClient at 0x1420144b988>
run_id = optuna_study_lr.user_attrs['run_id']
run = mlflow_client.get_run(run_id)  
run.data.tags['study_name']

[14] 2020-06-20 15:24:02 (8.00ms) python3 (29.0s)

'torch.lr.study#0'
run_id = optuna_study_lr.trials[0].user_attrs['run_id']
run = mlflow_client.get_run(run_id)  
run.data.tags['trial_number']

[15] 2020-06-20 15:24:02 (10.0ms) python3 (29.0s)

'0'

You may have a question. How does Optuna optimize the parameters without any score? The answer is the Monitor instance. An Objective instance gets the monitoring score from run.monitor and sends it to Optuna so that Optuna can determine the next suggestion. All you need is to make your Run instance have a Monitor instance. Check the YAML parameter file:

File 10 torch.yml

library: torch
datasets:
  data:
    class: rectangle.data.Data
    n_splits: 4
  dataset:
  fold: 0
model:
  class: rectangle.torch.Model
  hidden_sizes: [20, 30]
optimizer:
  class: torch.optim.SGD
  params: $.model.parameters()
  lr: 1e-3
scheduler:
  class: torch.optim.lr_scheduler.ReduceLROnPlateau
  optimizer: $
  factor: 0.5
  patience: 4
results:
metrics:
monitor:
  metric: val_loss
early_stopping:
  patience: 10
trainer:
  loss: mse
  batch_size: 10
  epochs: 10
  shuffle: true
  verbose: 2

The Monitor instance monitors val_loss (actually this is the default value, so that you can delete this line) and the default mode is min (smaller is better). If your monitor is accuracy, for example, set the monitor like this:

monitor:
  metric: accuracy
  mode: max

Parametric Optimization

Again read the suggest functions.

File 11 rectangle/suggest.py

def suggest_lr(trial, min=1e-5, max=1e-3):
    trial.suggest_loguniform("lr", min, max)


def suggest_hidden_sizes(trial, max_num_layers, min_size=10, max_size=30):
    num_layers = trial.suggest_int("num_layers", 2, max_num_layers)
    for k in range(num_layers):
        trial.suggest_int(f"hidden_sizes:{k}", min_size, max_size)

The suggest_hidden_sizes() has some logic but the code of the suggest_lr() is too simple to define a function. You may not want to write such a function. Ivory can do that for you. You can pass key-iterable pairs to the client.create_study() instead of key-callable pairs.

tuple, range, Range

A tuple, range, or Range instance represents parameter range.

study = client.create_study('torch', lr=(1e-3, 1e-2))
_ = study.optimize(n_trials=5, epochs=1, verbose=0)

[16] 2020-06-20 15:24:02 (2.18s) python3 (31.2s)

[I 2020-06-20 15:24:02,434] A new study created with name: torch.lr.study#2
[run#6] lr=0.004785 epochs=1
[I 2020-06-20 15:24:02,870] Finished trial#0 with value: 7.505557787418366 with parameters: {'lr': 0.0047846720934690504}. Best is trial#0 with value: 7.505557787418366.
[run#7] lr=0.007948 epochs=1
[I 2020-06-20 15:24:03,276] Finished trial#1 with value: 16.9930198431015 with parameters: {'lr': 0.007948219494249426}. Best is trial#0 with value: 7.505557787418366.
[run#8] lr=0.008386 epochs=1
[I 2020-06-20 15:24:03,701] Finished trial#2 with value: 21.089563941955568 with parameters: {'lr': 0.008385933151314614}. Best is trial#0 with value: 7.505557787418366.
[run#9] lr=0.009467 epochs=1
[I 2020-06-20 15:24:04,116] Finished trial#3 with value: 24.760809683799742 with parameters: {'lr': 0.009466755734626534}. Best is trial#0 with value: 7.505557787418366.
[run#10] lr=0.006797 epochs=1
[I 2020-06-20 15:24:04,534] Finished trial#4 with value: 63.910634422302245 with parameters: {'lr': 0.006797194491767582}. Best is trial#0 with value: 7.505557787418366.

In the above cell, lr=Range(1e-3, 1e-2) also works. For integer parameters, you can use normal range as well as tuple or Range.

params = {'hidden_sizes.0': range(10, 21)}  # Range(10, 20), or (10, 20)
study = client.create_study('torch', params)
_ = study.optimize(n_trials=5, epochs=1, verbose=0)

[17] 2020-06-20 15:24:04 (2.24s) python3 (33.4s)

[I 2020-06-20 15:24:04,641] A new study created with name: torch.hidden_sizes.0.study#3
[run#11] hidden_sizes.0=10 epochs=1
[I 2020-06-20 15:24:05,062] Finished trial#0 with value: 7.2038188695907595 with parameters: {'hidden_sizes.0': 10}. Best is trial#0 with value: 7.2038188695907595.
[run#12] hidden_sizes.0=19 epochs=1
[I 2020-06-20 15:24:05,489] Finished trial#1 with value: 7.636005270481109 with parameters: {'hidden_sizes.0': 19}. Best is trial#0 with value: 7.2038188695907595.
[run#13] hidden_sizes.0=11 epochs=1
[I 2020-06-20 15:24:05,913] Finished trial#2 with value: 7.774815452098847 with parameters: {'hidden_sizes.0': 11}. Best is trial#0 with value: 7.2038188695907595.
[run#14] hidden_sizes.0=19 epochs=1
[I 2020-06-20 15:24:06,341] Finished trial#3 with value: 6.721940863132477 with parameters: {'hidden_sizes.0': 19}. Best is trial#3 with value: 6.721940863132477.
[run#15] hidden_sizes.0=11 epochs=1
[I 2020-06-20 15:24:06,777] Finished trial#4 with value: 7.97547961473465 with parameters: {'hidden_sizes.0': 11}. Best is trial#3 with value: 6.721940863132477.

You can specify a step

params = {'hidden_sizes.0': range(10, 21, 3)}
study = client.create_study('torch', params)
_ = study.optimize(n_trials=5, epochs=1, verbose=0)

[18] 2020-06-20 15:24:06 (2.34s) python3 (35.8s)

[I 2020-06-20 15:24:06,903] A new study created with name: torch.hidden_sizes.0.study#4
[run#16] hidden_sizes.0=19 epochs=1
[I 2020-06-20 15:24:07,351] Finished trial#0 with value: 6.568264007568359 with parameters: {'hidden_sizes.0': 19}. Best is trial#0 with value: 6.568264007568359.
[run#17] hidden_sizes.0=10 epochs=1
[I 2020-06-20 15:24:07,789] Finished trial#1 with value: 8.529261600971221 with parameters: {'hidden_sizes.0': 10}. Best is trial#0 with value: 6.568264007568359.
[run#18] hidden_sizes.0=16 epochs=1
[I 2020-06-20 15:24:08,230] Finished trial#2 with value: 7.528908741474152 with parameters: {'hidden_sizes.0': 16}. Best is trial#0 with value: 6.568264007568359.
[run#19] hidden_sizes.0=19 epochs=1
[I 2020-06-20 15:24:08,667] Finished trial#3 with value: 7.347911989688873 with parameters: {'hidden_sizes.0': 19}. Best is trial#0 with value: 6.568264007568359.
[run#20] hidden_sizes.0=10 epochs=1
[I 2020-06-20 15:24:09,116] Finished trial#4 with value: 6.824846786260605 with parameters: {'hidden_sizes.0': 10}. Best is trial#0 with value: 6.568264007568359.

If you need sampling in log scale, use Range with log=True.

from ivory.utils.range import Range

study = client.create_study('torch', lr=Range(1e-3, 1e-2, log=True))
_ = study.optimize(n_trials=5, epochs=1, verbose=0)

[19] 2020-06-20 15:24:09 (2.45s) python3 (38.2s)

[I 2020-06-20 15:24:09,246] A new study created with name: torch.lr.study#5
[run#21] lr=0.006629 epochs=1
[I 2020-06-20 15:24:09,725] Finished trial#0 with value: 23.470214128494263 with parameters: {'lr': 0.0066287900160612}. Best is trial#0 with value: 23.470214128494263.
[run#22] lr=0.003144 epochs=1
[I 2020-06-20 15:24:10,181] Finished trial#1 with value: 6.392648601531983 with parameters: {'lr': 0.0031441144268029536}. Best is trial#1 with value: 6.392648601531983.
[run#23] lr=0.005022 epochs=1
[I 2020-06-20 15:24:10,651] Finished trial#2 with value: 4.59294992685318 with parameters: {'lr': 0.005022326394376906}. Best is trial#2 with value: 4.59294992685318.
[run#24] lr=0.003322 epochs=1
[I 2020-06-20 15:24:11,107] Finished trial#3 with value: 5.0058059871196745 with parameters: {'lr': 0.00332243814575827}. Best is trial#2 with value: 4.59294992685318.
[run#25] lr=0.001459 epochs=1
[I 2020-06-20 15:24:11,567] Finished trial#4 with value: 5.597119307518005 with parameters: {'lr': 0.0014591650124987697}. Best is trial#2 with value: 4.59294992685318.

list

A list represents parameter choice.

params = {'hidden_sizes.0': [10, 20, 30]}
study = client.create_study('torch', params)
_ = study.optimize(n_trials=5, epochs=1, verbose=0)

[20] 2020-06-20 15:24:11 (2.54s) python3 (40.7s)

[I 2020-06-20 15:24:11,709] A new study created with name: torch.hidden_sizes.0.study#6
[run#26] hidden_sizes.0=30 epochs=1
[I 2020-06-20 15:24:12,212] Finished trial#0 with value: 6.970041018724442 with parameters: {'hidden_sizes.0': 30}. Best is trial#0 with value: 6.970041018724442.
[run#27] hidden_sizes.0=20 epochs=1
[I 2020-06-20 15:24:12,679] Finished trial#1 with value: 6.162596261501312 with parameters: {'hidden_sizes.0': 20}. Best is trial#1 with value: 6.162596261501312.
[run#28] hidden_sizes.0=30 epochs=1
[I 2020-06-20 15:24:13,161] Finished trial#2 with value: 7.544045042991638 with parameters: {'hidden_sizes.0': 30}. Best is trial#1 with value: 6.162596261501312.
[run#29] hidden_sizes.0=30 epochs=1
[I 2020-06-20 15:24:13,631] Finished trial#3 with value: 8.475840830802918 with parameters: {'hidden_sizes.0': 30}. Best is trial#1 with value: 6.162596261501312.
[run#30] hidden_sizes.0=30 epochs=1
[I 2020-06-20 15:24:14,105] Finished trial#4 with value: 6.80062427520752 with parameters: {'hidden_sizes.0': 30}. Best is trial#1 with value: 6.162596261501312.

Product

If a key and value are tuples, the entry means cartesian product of suggest functions like Task.product().

params = {('hidden_sizes', 'lr'): (hidden_sizes, Range(1e-4, 1e-3))}
study = client.create_study('torch', params)
optuna_study = study.optimize(n_trials=10, epochs=1, verbose=0)

[21] 2020-06-20 15:24:14 (5.46s) python3 (46.2s)

[I 2020-06-20 15:24:14,276] A new study created with name: torch.hidden_sizes.lr.study#7
[run#31] hidden_sizes:0=23 hidden_sizes:1=26 hidden_sizes:2=30 lr=0.000285 num_layers=3 epochs=1
[I 2020-06-20 15:24:14,829] Finished trial#0 with value: 73.63323173522949 with parameters: {'hidden_sizes:0': 23, 'hidden_sizes:1': 26, 'hidden_sizes:2': 30, 'lr': 0.00028500875002022773, 'num_layers': 3}. Best is trial#0 with value: 73.63323173522949.
[run#32] hidden_sizes:0=15 hidden_sizes:1=23 lr=0.0003589 num_layers=2 epochs=1
[I 2020-06-20 15:24:15,349] Finished trial#1 with value: 8.391229891777039 with parameters: {'hidden_sizes:0': 15, 'hidden_sizes:1': 23, 'lr': 0.00035887166285874963, 'num_layers': 2}. Best is trial#1 with value: 8.391229891777039.
[run#33] hidden_sizes:0=10 hidden_sizes:1=16 lr=0.0009096 num_layers=2 epochs=1
[I 2020-06-20 15:24:15,838] Finished trial#2 with value: 6.040180587768555 with parameters: {'hidden_sizes:0': 10, 'hidden_sizes:1': 16, 'lr': 0.0009095927923320399, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.
[run#34] hidden_sizes:0=16 hidden_sizes:1=27 lr=0.0003067 num_layers=2 epochs=1
[I 2020-06-20 15:24:16,357] Finished trial#3 with value: 9.55086771249771 with parameters: {'hidden_sizes:0': 16, 'hidden_sizes:1': 27, 'lr': 0.0003067305802012419, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.
[run#35] hidden_sizes:0=21 hidden_sizes:1=13 lr=0.0005063 num_layers=2 epochs=1
[I 2020-06-20 15:24:16,879] Finished trial#4 with value: 7.796209609508514 with parameters: {'hidden_sizes:0': 21, 'hidden_sizes:1': 13, 'lr': 0.0005063167017779712, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.
[run#36] hidden_sizes:0=13 hidden_sizes:1=14 hidden_sizes:2=11 lr=0.0007649 num_layers=3 epochs=1
[I 2020-06-20 15:24:17,384] Finished trial#5 with value: 6.721876096725464 with parameters: {'hidden_sizes:0': 13, 'hidden_sizes:1': 14, 'hidden_sizes:2': 11, 'lr': 0.0007648866552945914, 'num_layers': 3}. Best is trial#2 with value: 6.040180587768555.
[run#37] hidden_sizes:0=12 hidden_sizes:1=28 lr=0.0003215 num_layers=2 epochs=1
[I 2020-06-20 15:24:17,921] Finished trial#6 with value: 10.44865968823433 with parameters: {'hidden_sizes:0': 12, 'hidden_sizes:1': 28, 'lr': 0.00032149466296796944, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.
[run#38] hidden_sizes:0=23 hidden_sizes:1=20 lr=0.0006336 num_layers=2 epochs=1
[I 2020-06-20 15:24:18,457] Finished trial#7 with value: 7.703984320163727 with parameters: {'hidden_sizes:0': 23, 'hidden_sizes:1': 20, 'lr': 0.0006336399565950085, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.
[run#39] hidden_sizes:0=17 hidden_sizes:1=27 hidden_sizes:2=22 lr=0.0006164 num_layers=3 epochs=1
[I 2020-06-20 15:24:19,014] Finished trial#8 with value: 9.060218453407288 with parameters: {'hidden_sizes:0': 17, 'hidden_sizes:1': 27, 'hidden_sizes:2': 22, 'lr': 0.0006163710231144914, 'num_layers': 3}. Best is trial#2 with value: 6.040180587768555.
[run#40] hidden_sizes:0=15 hidden_sizes:1=26 lr=0.0001284 num_layers=2 epochs=1
[I 2020-06-20 15:24:19,563] Finished trial#9 with value: 59.055799674987796 with parameters: {'hidden_sizes:0': 15, 'hidden_sizes:1': 26, 'lr': 0.0001283989499444509, 'num_layers': 2}. Best is trial#2 with value: 6.040180587768555.

Note

You can mix suggest funtions and parametric optimization.

Note

You may feel that "params = {'hidden_sizes.1': hidden_sizes, 'lr': Range(1e-4, 1e-3)}" is better, but the above style is intentional.

In parametric optimization, the name of Optuna's Study instance is dot-joint style:

optuna_study.study_name

[22] 2020-06-20 15:24:19 (4.00ms) python3 (46.2s)

'torch.hidden_sizes.lr.study#7'

Study from YAML file

As a normal Run, a Study instance also can be created from a YAML file. Pass an extra keyword argument to the client.create_experiment(). The key is the instance name (in this case study) and value is a YAML file name without its extension.

experiment = client.create_experiment('torch', study='study')
experiment

[23] 2020-06-20 15:24:19 (11.0ms) python3 (46.2s)

Experiment(id='1', name='torch', num_instances=1)

Here is the contents of study.yml file.

File 12 study.yml

objective:
  lr: rectangle.suggest.suggest_lr
  hidden_sizes:
    def: rectangle.suggest.suggest_hidden_sizes
    max_num_layers: 3
    min_size: __default__
    max_size: __default__

Suggest functions should be callable, hidden_sizes uses a def key to create a callable. On the other hand, lr is just one line. If a suggest funtion can be called without additional arguments, you can omit the def key. Using this experiment, we can create Study instances with a suggest function.

study_lr = client.create_study('torch', 'lr')
study_lr.objective

[24] 2020-06-20 15:24:19 (174ms) python3 (46.4s)

Objective(['lr'])
study_hs = client.create_study('torch', 'hidden_sizes')
study_hs.objective

[25] 2020-06-20 15:24:19 (178ms) python3 (46.6s)

Objective(['hidden_sizes'])
study_hs.objective.hidden_sizes

[26] 2020-06-20 15:24:19 (3.00ms) python3 (46.6s)

functools.partial(<function suggest_hidden_sizes at 0x000001420C0508B8>, max_num_layers=3, min_size=10, max_size=30)

For min_size and max_size, default values are inspected from the signature.

study_lr.optimize(n_trials=3, epochs=3, verbose=0)

[27] 2020-06-20 15:24:19 (2.32s) python3 (48.9s)

[I 2020-06-20 15:24:19,976] A new study created with name: torch.lr.study#8
[run#41] lr=8.41e-05 epochs=3
[I 2020-06-20 15:24:20,729] Finished trial#0 with value: 9.582427203655243 with parameters: {'lr': 8.409509940148645e-05}. Best is trial#0 with value: 9.582427203655243.
[run#42] lr=3.765e-05 epochs=3
[I 2020-06-20 15:24:21,499] Finished trial#1 with value: 73.85470199584961 with parameters: {'lr': 3.7651669961268316e-05}. Best is trial#0 with value: 9.582427203655243.
[run#43] lr=1.924e-05 epochs=3
[I 2020-06-20 15:24:22,268] Finished trial#2 with value: 93.58401222229004 with parameters: {'lr': 1.924316867807553e-05}. Best is trial#0 with value: 9.582427203655243.
<optuna.study.Study at 0x14212124e48>

Pruning

Optuna provides the pruning functionality. Ivory can uses this feature seamlessly.

Here is updated contents of study.yml file.

File 13 study.yml

tuner:
  pruner:
    class: optuna.pruners.MedianPruner
objective:
  lr: rectangle.suggest.suggest_lr
  hidden_sizes:
    def: rectangle.suggest.suggest_hidden_sizes
    max_num_layers: 3
    min_size: __default__
    max_size: __default__

The Tuner instance has Optuna's MedianPruner. (Off course, you can use other pruners.) A Study instance give an ivory.callbacks.Pruning instance to a run when the run is created, then with Ivory's callback system, the Pruning instance communicates with Optuna in order to determine the step of pruning.

Note

Pruning is supported for PyTorch and TensorFlow now.