MODULE

IVORY.CORE.CLIENT

This module provides the Ivory Client class that is one of the main classes of Ivory library.

To create an Client instance:

import ivory

client = ivory.create_client()

Here, the current directory becomes the working directory in which experiment YAML files exist. If you want to refer other directory, use:

client = ivory.create_client('path/to/working_directory')
Classes
Functions
class

ivory.core.client.Client(params=None, **objects)

The Ivory Client class.

Parameters
  • params (optional) Parameter dictionary to create this instance.
  • **objects
Attributes
  • dict
  • experiments (dict(str: Experiment))
  • id (str) Instance ID given by MLFlow Tracking.
  • name (str) Instance name.
  • params (dict, optional) Parameter dictionary that is used to to create this instance.
  • source_name (str) Name of the YAML parameter file that is used to create this instance.
  • tracker (Tracker) A Tracker instance for tracking run process.
  • tuner (Tuner) A Tuner instance for hyperparameter tuning.
Methods
  • create_experiment(name, *args, **kwargs) (Experiment) Creates an Experiment according to the YAML file specified by name.</>
  • create_run(name, args, **kwargs) (Run) Creates a Run.</>
  • create_study(name, args, run_number, **suggests) (Study) Creates a Study instance for hyperparameter tuning.</>
  • create_task(name, run_number) (Task) Creates a Task instance for multiple runs.</>
  • get_nested_run_ids(name, **kwargs) (str) Returns an iterator that yields nested Run IDs of parent runs.</>
  • get_parent_run_id(name, **kwargs) (str) Returns a parent Run ID of a nested run.</>
  • get_run_id(name, **kwargs) (str) Returns a Run ID.</>
  • get_run_ids(name, **kwargs) (str) Returns an iterator that yields Run IDs.</>
  • get_run_name(run_id) (str) Returns a run name (run#XXX, task#XXX, etc.) for Run ID.</>
  • get_run_name_tuple(run_id) (str, int) Returns a run name as a tuple of (run class name, run number).</>
  • load_instance(run_id, instance_name, mode) (any) Returns a member of a Run created using parameters loaded from MLFlow Tracking.</>
  • load_params(run_id) (dict(str: any)) Returns a parameter dictionary loaded from MLFlow Tracking.</>
  • load_results(run_ids, callback, reduction, verbose) (Results) Loads results from multiple runs and concatenates them.</>
  • load_run(run_id, mode) (Run) Returns a Run instance created using parameters loaded from MLFlow Tracking.</>
  • load_run_by_name(name, mode, **kwargs) (Run) Returns a Run instance created using parameters loaded from MLFlow Tracking.</>
  • remove_deleted_runs(name) (int) Removes deleted runs from a local file system.</>
  • search_nested_run_ids(name, **query) (str) Returns an iterator that yields matching nested Run IDs.</>
  • search_parent_run_ids(name, **query) (str) Returns an iterator that yields matching parent Run IDs.</>
  • search_run_ids(name, run_name, parent_run_id, parent_only, nested_only, exclude_parent, best_score_limit, **query) (str) Returns an iterator that yields matching Run IDs.</>
  • set_parent_run_id(name, **kwargs) Sets parent Run ID to runs.</>
  • set_terminated(name, status, **kwargs) Sets runs' status to terminated.</>
  • set_terminated_all(name) Sets all runs' status to terminated.</>
method

create_experiment(name, *args, **kwargs)Experiment

Creates an Experiment according to the YAML file specified by name.

Parameters
  • name (str) Experiment name.
  • *args Additional parameter files.
  • **kwargs Additional parameter files.

A YAML file named <name>.yml or <name>.yaml should exist under the working directory.

Any additionanl parameter files are added through *args and/or **kwargs.

Examples

Positional argument style:

experiment = client.create_experiment('example', 'study')

In this case, study.yml is like this, including the instance name study:

study:
  tuner:
    pruner:
    class: optuna.pruners.MedianPruner
  objective:
    lr: example.suggest_lr

Keyword argument style:

experiment = client.create_experiment('example', study='study')

In this case, study.yml is like this, omitting the instance name study:

tuner:
  pruner:
    class: optuna.pruners.MedianPruner
objective:
  lr: example.suggest_lr
method

create_run(name, args=None, **kwargs)Run

Creates a Run.

Parameters
  • name (str) Experiment name.
  • args (dict, optional) Parameter dictionary to update the default values of Experiment.
  • **kwargs Additional parameters.
Examples

To update a fold number:

run = client.create_run('example', fold=3)

If a parameter name includes dots:

run = client.create_run('example', {'model.class': 'your.new.Model'})
method

create_task(name, run_number=None)Task

Creates a Task instance for multiple runs.

Parameters
  • name (str) Experiment name.
  • run_number (int, optional) If specified, load an existing task instead of creating a new one.
See Also

Multiple Runs in Tutorial

method

create_study(name, args=None, run_number=None, **suggests)Study

Creates a Study instance for hyperparameter tuning.

Parameters
  • name (str) Experiment name.
  • args (str or dict) Suggest name (str) or parametric optimization (dict).
  • run_number (int, optional) If specified, load an existing study instead of creating a new one.
  • **suggests Parametric optimization.
Examples

To use a suggest function:

study = client.create_study('example', 'lr')

For parametric optimization:

study = client.create_study('example', lr=(1e-5, 1e-3))

If a parameter name includes dots:

study = client.create_study('example', {'hidden_sizes.0': range(5, 20)})
See Also
method

get_run_id(name, **kwargs) → str

Returns a Run ID.

Parameters
  • name (str) Experiment name.
  • **kwargs
Examples

To get a Run ID of run#4.

client.get_run_id('example', run=4)

To get a Run ID of task#10.

client.get_run_id('example', task=10)
generator

get_run_ids(name, **kwargs) → str

Returns an iterator that yields Run IDs.

Parameters
  • name (str) Experiment name.
  • **kwargs
Examples

To get an iterator that yields Run IDs for Runs.

client.get_run_id('example', run=[1, 2, 3])

To get an iterator that yields Run IDs for Tasks.

client.get_run_id('example', task=range(3, 8))
method

get_parent_run_id(name, **kwargs) → str

Returns a parent Run ID of a nested run.

Parameters
  • name (str) Experiment name.
  • **kwargs
Examples

To get a prarent Run ID of run#5.

client.get_parent_run_id('example', run=5)
generator

get_nested_run_ids(name, **kwargs) → str

Returns an iterator that yields nested Run IDs of parent runs.

Parameters
  • name (str) Experiment name.
  • **kwargs
Examples

To get an iterator that yields Run IDs of runs whose parent is task#2.

client.get_nested_run_ids('example', task=2)

Multiple parents can be specified.

client.get_nested_run_ids('example', task=range(3, 8))
method

set_parent_run_id(name, **kwargs)

Sets parent Run ID to runs.

Parameters
  • name (str) Experiment name.
  • **kwargs
Examples

To set task#2 as a parant for run#4.

client.set_parent_run_id('example', task=2, run=4)

Multiple nested runs can be specified.

client.set_parent_run_id('example', task=2, run=range(3))
method

get_run_name(run_id) → str

Returns a run name (run#XXX, task#XXX, etc.) for Run ID.

Parameters
  • run_id (str) Run ID
method

get_run_name_tuple(run_id) → (str, int)

Returns a run name as a tuple of (run class name, run number).

Parameters
  • run_id (str) Run ID
generator

search_run_ids(name='', run_name='', parent_run_id='', parent_only=False, nested_only=False, exclude_parent=False, best_score_limit=None, **query) → str

Returns an iterator that yields matching Run IDs.

Parameters
  • name (str, optional) Experiment name pattern for filtering.
  • run_name (str, optional) Run name pattern for filtering.
  • parent_run_id (str or iterable of str) If specified, search from runs that have the parent id(s).
  • parent_only (bool, optional) If True, search from parent runs.
  • nested_only (bool, optional) If True, search from nested runs.
  • exclude_parent (bool, optional) If True, skip parent runs.
  • best_score_limit (float, optional) Yields runs with the best score better than this value.
  • **query Key-value pairs for filtering.
generator

search_parent_run_ids(name='', **query) → str

Returns an iterator that yields matching parent Run IDs.

Parameters
  • name (str, optional) Experiment name pattern for filtering.
  • **query Key-value pairs for filtering.
generator

search_nested_run_ids(name='', **query) → str

Returns an iterator that yields matching nested Run IDs.

Parameters
  • name (str, optional) Experiment name pattern for filtering.
  • **query Key-value pairs for filtering.
method

set_terminated(name, status=None, **kwargs)

Sets runs' status to terminated.

Parameters
  • name (str)
  • status (str, optional) A string value of mlflow.entities.RunStatus. Defaults to “FINISHED”.
  • **kwargs
Examples

To terminate a run:

client.set_terminated('example', run=5)

To kill multiple runs:

client.set_terminated('example', 'KILLED', run=[3, 5, 7])
method

set_terminated_all(name='')

Sets all runs' status to terminated.

Parameters
Examples

To terminate all of the runs of the example experiment:

client.set_terminated_all('example')

To terminate all of the runs globally:

client.set_terminated_all()
method

load_params(run_id) → dict(str: any)

Returns a parameter dictionary loaded from MLFlow Tracking.

Parameters
  • run_id (str) Run ID for a run to be loaded.
method

load_run(run_id, mode='test')Run

Returns a Run instance created using parameters loaded from MLFlow Tracking.

Parameters
  • run_id (str) Run ID for a run to be loaded.
  • mode (str, optional) Mode name: 'current', 'best', or 'test'. Default is ''test''.
method

load_run_by_name(name, mode='test', **kwargs)Run

Returns a Run instance created using parameters loaded from MLFlow Tracking.

Parameters
  • name (str) Experiment name pattern for filtering.
  • mode (str, optional) Mode name: 'current', 'best', or 'test'.
  • **kwargs
Examples

To load run#4 of the example experiment.

client.load_run_by_name('example', run=4)
method

load_instance(run_id, instance_name, mode='test') → any

Returns a member of a Run created using parameters loaded from MLFlow Tracking.

Parameters
  • run_id (str) Run ID for a run to be loaded.
  • instance_name (str) Instance name.
  • mode (str, optional) Mode name: 'current', 'best', or 'test'.
method

load_results(run_ids, callback=None, reduction='none', verbose=True)

Loads results from multiple runs and concatenates them.

Parameters
  • run_ids (Union(str, iterable of str)) Multiple run ids to load.
  • callback (callable) Callback function for each run. This function must take a (index, output, target) and return a tuple with the same signature.
  • reduction (str, optional)
  • verbose (bool, optional) If True, tqdm progress bar is displayed.
Returns (Results)

A concatenated results instance.

method

remove_deleted_runs(name='')

Removes deleted runs from a local file system.

Parameters
  • name (str, optional) Experiment name pattern for filtering.
Returns (int)

Number of removed runs.

function

ivory.core.client.create_client(directory='', name='client', tracker=True, tuner=True)

Creates an Ivory Client instance.

Parameters
  • directory (str, optional) A working directory. If a YAML file specified by the name parameter exists, the file is loaded to configure the client. In addition, this directory is automatically inserted to sys.path.
  • name (str, optional) A YAML config file name.
  • tracker (bool, optional) If True, the client instance has a tracker.
  • tuner (bool, optional) If True, the client instance has a tuner.
Returns (Client)

An created client.

Note

If tracker is True (default value), a mlruns directory is made under the working directory by MLFlow Tracking.