Library Comparison

So far, we have used PyTorch in this tutorial, but Ivory can perform machine learning with other libraries.

Base Parameter File

Before examples, we write two base or template parameter files, which are extended by other parameter files later.

File 14 A base parameter YAML file for various libraries (data.yml)

datasets:
  data:
    class: rectangle.data.Data
    n_splits: 4
  dataset:
  fold: 0

File 15 A base parameter YAML file for various libraries (base.yml)

extends: data
model:
  hidden_sizes: [20, 30]
optimizer:
  lr: 1e-3
results:
metrics:
monitor:
  metric: val_loss
trainer:
  loss: mse
  batch_size: 5
  epochs: 5
  shuffle: false
  verbose: 2

In base.yml, the first line "extends: data" means that the file extends (or includes, in this case) data.yml.

Neural Network Libraries

In this section we compare three neural network libraries (TensorFlow, NNabla, and PyTorch), and show that using different libraries on the same problem is straightforward.

import tensorflow
import nnabla
import torch
print(tensorflow.__version__)
print(nnabla.__version__)
print(torch.__version__)

[3] 2020-06-20 15:23:40 (1.22s) python3 (8.70s)

2020-06-20 15:23:41,736 [nnabla][INFO]: Initializing CPU extension...
C:\Users\daizu\miniconda3\envs\daizu\lib\site-packages\nnabla\function_bases.py:58: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  spec.args[1:], spec.varargs, spec.keywords, defaults)
2.1.0
1.7.0
1.5.0+cu101

First define models:

File 16 A Model definition in TensorFlow (rectangle/tensorflow.py)

from tensorflow import keras
from tensorflow.keras.layers import Dense


def create_model(hidden_sizes):
    layers = [Dense(hidden_sizes[0], activation="relu", input_shape=[2])]
    for hidden_size in hidden_sizes[1:]:
        layers.append(Dense(hidden_size, activation="relu"))
    layers.append(Dense(1))
    return keras.Sequential(layers)

File 17 A Model definition in NNabla (rectangle/nnabla.py)

import nnabla as nn
import nnabla.functions as F
import nnabla.parametric_functions as PF

import ivory.nnabla.model


class Model(ivory.nnabla.model.Model):
    def __init__(self, hidden_sizes):
        super().__init__()
        self.hidden_sizes = hidden_sizes

    def forward(self, x):
        for k, hidden_size in enumerate(self.hidden_sizes):
            with nn.parameter_scope(f"layer{k}"):
                x = F.relu(PF.affine(x, hidden_size))
        with nn.parameter_scope(f"layer{k+1}"):
            x = PF.affine(x, 1)
        return x

File 18 A Model definition in PyTorch (rectangle/torch.py)

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self, hidden_sizes):
        super().__init__()
        layers = []
        for in_features, out_features in zip([2] + hidden_sizes, hidden_sizes + [1]):
            layers.append(nn.Linear(in_features, out_features))
        self.layers = nn.ModuleList(layers)

    def forward(self, x):
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))
        return self.layers[-1](x)

For simplicity, the TensorFlow model is defined by using the keras.Sequential(), so that we call the create_model() to get the model.

Next, write parameter YAML files:

File 19 A parameter YAML file for TensorFlow (tensorflow.yml)

library: tensorflow
extends: base
model:
  call: rectangle.tensorflow.create_model
optimizer:
  class: tensorflow.keras.optimizers.SGD

File 20 A parameter YAML file for NNabla (nnabla.yml)

library: nnabla
extends: base
model:
  class: rectangle.nnabla.Model
optimizer:
  class: nnabla.solvers.Sgd

File 21 A parameter YAML fine for PyTorch (torch2.yml)

library: torch
extends: base
model:
  class: rectangle.torch.Model
optimizer:
  class: torch.optim.SGD
  _: $.model.parameters()

These YAML files are very similar. The only difference is that, in PyTorch, an optimizer needs model parameters at the time of instantiation.

Note

The model for TensorFlow is a function. A new call key is used. (But you can stil use class, or call for a class, vice versa, because both a class and function are callable.)

Next, create three runs.

import ivory

client = ivory.create_client("examples")
run_tf = client.create_run('tensorflow')
run_nn = client.create_run('nnabla')
run_torch = client.create_run('torch2')

[4] 2020-06-20 15:23:41 (1.64s) python3 (10.3s)

[I 200620 15:23:41 tracker:48] A new experiment created with name: 'tensorflow'
C:\Users\daizu\miniconda3\envs\daizu\lib\site-packages\nnabla\parametric_functions.py:98: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  defaults + (None,))
C:\Users\daizu\miniconda3\envs\daizu\lib\site-packages\nnabla\solvers.py:18: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  from .solver import *
[I 200620 15:23:43 tracker:48] A new experiment created with name: 'nnabla'
[I 200620 15:23:43 tracker:48] A new experiment created with name: 'torch2'

For comparison, equalize initial parameters.

import torch

# These three lines are only needed for this example.
run, trainer = run_nn, run_nn.trainer
run.model.build(trainer.loss, run.datasets.train, trainer.batch_size)
run.optimizer.set_parameters(run.model.parameters())

ws_tf = run_tf.model.weights
ws_nn = run_nn.model.parameters().values()
ws_torch = run_torch.model.parameters()
for w_tf, w_nn, w_torch in zip(ws_tf, ws_nn, ws_torch):
    w_nn.data.data = w_tf.numpy()
    w_torch.data = torch.tensor(w_tf.numpy().T)

[5] 2020-06-20 15:23:43 (14.0ms) python3 (10.4s)

Then, start the runs.

run_tf.start('both')  # Slower due to usage of GPU for a simple network.

[6] 2020-06-20 15:23:43 (1.89s) python3 (12.2s)

C:\Users\daizu\miniconda3\envs\daizu\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:1389: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  if isinstance(sample_weight_mode, collections.Mapping):
C:\Users\daizu\miniconda3\envs\daizu\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py:544: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  if isinstance(inputs, collections.Sequence):
[epoch#0] loss=10.74 val_loss=5.403 best
[epoch#1] loss=5.6 val_loss=3.937 best
[epoch#2] loss=4.053 val_loss=2.613 best
[epoch#3] loss=2.655 val_loss=1.571 best
[epoch#4] loss=1.63 val_loss=0.9675 best
run_nn.start('both')

[7] 2020-06-20 15:23:45 (842ms) python3 (13.1s)

2020-06-20 15:23:45,382 [nnabla][INFO]: DataSource with shuffle(False)
2020-06-20 15:23:45,383 [nnabla][INFO]: Using DataIterator
2020-06-20 15:23:45,385 [nnabla][INFO]: DataSource with shuffle(False)
2020-06-20 15:23:45,386 [nnabla][INFO]: Using DataIterator
2020-06-20 15:23:45,387 [nnabla][INFO]: DataSource with shuffle(False)
2020-06-20 15:23:45,388 [nnabla][INFO]: Using DataIterator
[epoch#0] loss=10.74 val_loss=5.403 best
[epoch#1] loss=5.6 val_loss=3.937 best
[epoch#2] loss=4.053 val_loss=2.613 best
[epoch#3] loss=2.655 val_loss=1.571 best
[epoch#4] loss=1.63 val_loss=0.9675 best
run_torch.start('both')

[8] 2020-06-20 15:23:46 (812ms) python3 (13.9s)

[epoch#0] loss=10.74 val_loss=5.403 lr=0.001 best
[epoch#1] loss=5.6 val_loss=3.937 lr=0.001 best
[epoch#2] loss=4.053 val_loss=2.613 lr=0.001 best
[epoch#3] loss=2.655 val_loss=1.571 lr=0.001 best
[epoch#4] loss=1.63 val_loss=0.9675 lr=0.001 best

Metrics during training are almost same. Visualize the results:

import matplotlib.pyplot as plt

# A helper function
def plot(run):
    dataset = run.results.val
    plt.scatter(dataset.target.reshape(-1), dataset.output.reshape(-1))
    plt.xlim(0, 25)
    plt.ylim(0, 25)
    plt.xlabel('Target values')
    plt.ylabel('Predicted values')

for run in [run_tf, run_nn, run_torch]:
    plot(run)

[9] 2020-06-20 15:23:47 (95.1ms) python3 (14.0s)

image/png

Actual outputs are like below:

x = run_tf.datasets.val[:5][1]
run_tf.model.predict(x)

[10] 2020-06-20 15:23:47 (21.0ms) python3 (14.0s)

array([[9.168745 ],
       [9.302884 ],
       [8.557519 ],
       [6.005666 ],
       [3.7306597]], dtype=float32)
x = run_nn.datasets.val[:5][1]
run_nn.model(x)

[11] 2020-06-20 15:23:47 (4.00ms) python3 (14.0s)

array([[9.168744 ],
       [9.302884 ],
       [8.557518 ],
       [6.005667 ],
       [3.7306597]], dtype=float32)
x = run_torch.datasets.val[:5][1]
run_torch.model(torch.tensor(x))

[12] 2020-06-20 15:23:47 (6.00ms) python3 (14.0s)

tensor([[9.1687],
        [9.3029],
        [8.5575],
        [6.0057],
        [3.7307]], grad_fn=<AddmmBackward>)

You can ensemble these results, although this is meaningless in this example.

from ivory.callbacks.results import concatenate

results = concatenate(run.results for run in [run_tf, run_nn, run_torch])
index = results.val.index.argsort()
results.val.output[index[:15]]

[13] 2020-06-20 15:23:47 (5.00ms) python3 (14.0s)

array([[9.168744 ],
       [9.168744 ],
       [9.168745 ],
       [9.302884 ],
       [9.302884 ],
       [9.302884 ],
       [8.557519 ],
       [8.557518 ],
       [8.557519 ],
       [6.0056663],
       [6.005667 ],
       [6.0056663],
       [3.7306597],
       [3.7306602],
       [3.7306592]], dtype=float32)
reduced_results = results.mean()
reduced_results.val.output[:5]

[14] 2020-06-20 15:23:47 (11.0ms) python3 (14.0s)

array([[9.168744 ],
       [9.302884 ],
       [8.557519 ],
       [6.0056667],
       [3.7306597]], dtype=float32)

Scikit-learn

Ivory can optimize various scikit-learn's estimators. Before showing some examples, we need reshape the target array.

File 22 A base parameter YAML file for various estimators (data2.yml)

extends: data
datasets:
  dataset:
    transform: rectangle.data.transform

The dataset has a transform argument. This function reshapes the target array to match the shape for scikit-learn estimators (1D from 2D).

Code 3 rectangle.data.transform()

TypeError: module, class, method, function, traceback, frame, or code object was expected, got list
TypeError                                 Traceback (most recent call last)
<ipython-input-166-2e957216a06d> in <module>
     28         source = f"@dataclass{args}\n{source}"
     29     return source
---> 30 getsource(_)

<ipython-input-166-2e957216a06d> in getsource(obj)
      4     else:
      5         is_dataclass = False
----> 6     source = inspect.getsource(obj)
      7     defaults = [('init', True), ('repr', True), ('eq', True), ('order', False),
      8                 ('unsafe_hash', False), ('frozen', False)]

~\miniconda3\envs\daizu\lib\inspect.py in getsource(object)
    971     or code object.  The source code is returned as a single string.  An
    972     OSError is raised if the source code cannot be retrieved."""
--> 973     lines, lnum = getsourcelines(object)
    974     return ''.join(lines)
    975 

~\miniconda3\envs\daizu\lib\inspect.py in getsourcelines(object)
    953     raised if the source code cannot be retrieved."""
    954     object = unwrap(object)
--> 955     lines, lnum = findsource(object)
    956 
    957     if istraceback(object):

~\miniconda3\envs\daizu\lib\inspect.py in findsource(object)
    766     is raised if the source code cannot be retrieved."""
    767 
--> 768     file = getsourcefile(object)
    769     if file:
    770         # Invalidate cache if needed.

~\miniconda3\envs\daizu\lib\inspect.py in getsourcefile(object)
    682     Return None if no way can be identified to get the source.
    683     """
--> 684     filename = getfile(object)
    685     all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]
    686     all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]

~\miniconda3\envs\daizu\lib\inspect.py in getfile(object)
    664     raise TypeError('module, class, method, function, traceback, frame, or '
    665                     'code object was expected, got {}'.format(
--> 666                     type(object).__name__))
    667 
    668 def getmodulename(path):

RandomForestRegressor

File 23 A parameter YAML file for RandomForestRegressor (rfr.yml)

library: sklearn
extends: data2
estimator:
  model: sklearn.ensemble.RandomForestRegressor
  n_estimators: 5
  max_depth: 3
results:
metrics:

There are nothing difference to start a run.

run = client.create_run('rfr')
run.start()

[17] 2020-06-20 15:23:47 (380ms) python3 (14.6s)

[I 200620 15:23:47 tracker:48] A new experiment created with name: 'rfr'
[run#0] mse=2.635

Because RandomForestRegressor estimator has a criterion attribute, the metrics are automatically calculated. Take a look at the outputs.

plot(run)

[18] 2020-06-20 15:23:47 (76.0ms) python3 (14.7s)

image/png

Ridge

File 24 A parameter YAML file for Ridge (ridge.yml)

library: sklearn
extends: data2
estimator:
  model: sklearn.linear_model.Ridge
results:
metrics:
  mse:
  mse_2: rectangle.metrics.mean_squared_error

Because Ridge estimator has no criterion attribute, you have to specify metrics if you need. A mse key has empty (None) value. In this case, the default function (sklearn.metrics.mean_squared_error()) is chosen. On the other hand, mse_2's value is a custom function's name:

Code 4 rectangle.metrics.mean_squared_error()

TypeError: module, class, method, function, traceback, frame, or code object was expected, got list
TypeError                                 Traceback (most recent call last)
<ipython-input-172-2e957216a06d> in <module>
     28         source = f"@dataclass{args}\n{source}"
     29     return source
---> 30 getsource(_)

<ipython-input-172-2e957216a06d> in getsource(obj)
      4     else:
      5         is_dataclass = False
----> 6     source = inspect.getsource(obj)
      7     defaults = [('init', True), ('repr', True), ('eq', True), ('order', False),
      8                 ('unsafe_hash', False), ('frozen', False)]

~\miniconda3\envs\daizu\lib\inspect.py in getsource(object)
    971     or code object.  The source code is returned as a single string.  An
    972     OSError is raised if the source code cannot be retrieved."""
--> 973     lines, lnum = getsourcelines(object)
    974     return ''.join(lines)
    975 

~\miniconda3\envs\daizu\lib\inspect.py in getsourcelines(object)
    953     raised if the source code cannot be retrieved."""
    954     object = unwrap(object)
--> 955     lines, lnum = findsource(object)
    956 
    957     if istraceback(object):

~\miniconda3\envs\daizu\lib\inspect.py in findsource(object)
    766     is raised if the source code cannot be retrieved."""
    767 
--> 768     file = getsourcefile(object)
    769     if file:
    770         # Invalidate cache if needed.

~\miniconda3\envs\daizu\lib\inspect.py in getsourcefile(object)
    682     Return None if no way can be identified to get the source.
    683     """
--> 684     filename = getfile(object)
    685     all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]
    686     all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]

~\miniconda3\envs\daizu\lib\inspect.py in getfile(object)
    664     raise TypeError('module, class, method, function, traceback, frame, or '
    665                     'code object was expected, got {}'.format(
--> 666                     type(object).__name__))
    667 
    668 def getmodulename(path):

This functionality allows us to add arbitrary metrics as long as they can be calculated with true and pred arrays .

run = client.create_run('ridge')
run.start()  # Both metrics would give the same values.

[21] 2020-06-20 15:23:47 (374ms) python3 (15.1s)

[I 200620 15:23:48 tracker:48] A new experiment created with name: 'ridge'
[run#0] mse=1.77 mse_2=1.77
plot(run)

[22] 2020-06-20 15:23:48 (80.7ms) python3 (15.2s)

image/png

LightGBM

For LightGBM, Ivory implements two estimators:

  • ivory.lightgbm.estimator.Regressor
  • ivory.lightgbm.estimator.Classifier

File 25 A parameter YAML file for LightGBM (lgb.yml)

extends: data2
estimator:
  class: ivory.lightgbm.estimator.Regressor
  boosting_type: gbdt
  num_leaves: 10
  learning_rate: 0.1
  max_depth: 4
  num_boost_round: 10
  verbose_eval: 2
results:
metrics:
  mse:
run = client.create_run('lgb')
run.start()

[23] 2020-06-20 15:23:48 (308ms) python3 (15.5s)

[I 200620 15:23:48 tracker:48] A new experiment created with name: 'lgb'
[2]    training's l2: 16.6352  valid_1's l2: 15.7235
[4] training's l2: 11.526   valid_1's l2: 11.0195
[6] training's l2: 8.00956  valid_1's l2: 7.87861
[8] training's l2: 5.5925   valid_1's l2: 5.66989
[10]    training's l2: 3.92488  valid_1's l2: 4.09496
[run#0] mse=4.095
plot(run)

[24] 2020-06-20 15:23:48 (86.0ms) python3 (15.6s)

image/png