Model Structure

Model

We have prepared a Datasets instance for PyTorch. Now define a MLP model that works with this Datasets.

The model is defined in rectangle/torch.py

File 6 rectangle/torch.py

import torch.nn as nn
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self, hidden_sizes):
        super().__init__()
        layers = []
        for in_features, out_features in zip([2] + hidden_sizes, hidden_sizes + [1]):
            layers.append(nn.Linear(in_features, out_features))
        self.layers = nn.ModuleList(layers)

    def forward(self, x):
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))
        return self.layers[-1](x)

We again use Ivory's instance creation system.

import yaml

# A helper function.
def create(doc, name, **kwargs):
    params = yaml.safe_load(doc)
    return create_instance(params, name, **kwargs)

doc = """
library: torch
datasets:
  data:
    class: rectangle.data.Data
    n_splits: 5
  dataset:
  fold: 0
model:
  class: rectangle.torch.Model
  hidden_sizes: [3, 4, 5]
"""
datasets = create(doc, 'datasets')
model = create(doc, 'model')
model

[2] 2020-06-20 15:23:48 (9.00ms) python3 (15.6s)

Model(
  (layers): ModuleList(
    (0): Linear(in_features=2, out_features=3, bias=True)
    (1): Linear(in_features=3, out_features=4, bias=True)
    (2): Linear(in_features=4, out_features=5, bias=True)
    (3): Linear(in_features=5, out_features=1, bias=True)
  )
)

We can uses this model as usual.

import torch

index, input, target = datasets.train[:5]
input

[3] 2020-06-20 15:23:48 (4.00ms) python3 (15.6s)

array([[2.0772183, 4.9461417],
       [1.8550245, 4.6405816],
       [4.6277347, 4.6336703],
       [2.3827288, 1.1293393],
       [2.826326 , 2.8065689]], dtype=float32)
model(torch.tensor(input))

[4] 2020-06-20 15:23:48 (6.00ms) python3 (15.6s)

tensor([[0.0749],
        [0.0749],
        [0.0619],
        [0.0629],
        [0.0596]], grad_fn=<AddmmBackward>)

Optimizer

To train a model, we need an optimizer. For example

import torch.optim

optimizer = torch.optim.SGD(params=model.parameters(), lr=1e-3)
optimizer

[5] 2020-06-20 15:23:48 (4.00ms) python3 (15.6s)

SGD (
Parameter Group 0
    dampening: 0
    lr: 0.001
    momentum: 0
    nesterov: False
    weight_decay: 0
)

Now try to describe this optimizer in a dictionary style. However, the first argument params is neigher a simple literal nor an other instance. It is an iterable of learnable parameters obtained from a model. Ivory provides "$-notation" to tackle this problem.

doc = """
optimizer:
  class: torch.optim.SGD
  params: $.model.parameters()
  lr: 0.001
"""
optimizer = create(doc, 'optimizer', globals={'model': model})
optimizer

[6] 2020-06-20 15:23:48 (5.00ms) python3 (15.6s)

SGD (
Parameter Group 0
    dampening: 0
    lr: 0.001
    momentum: 0
    nesterov: False
    weight_decay: 0
)

A "$" is a starting point to refer other instance stored in the globals dictionary. In this case, $.model is replaced by the model instance in globals, then .parameters() invokes a call of Model.parameters().

Scheduler

A scheduler controls the learning rate of an optimizer.

doc = """
scheduler:
  class: torch.optim.lr_scheduler.ReduceLROnPlateau
  optimizer: $
  factor: 0.5
  patience: 4
"""
scheduler = create(doc, 'scheduler', globals={'optimizer': optimizer})
scheduler

[7] 2020-06-20 15:23:48 (5.00ms) python3 (15.6s)

<torch.optim.lr_scheduler.ReduceLROnPlateau at 0x1420126fd88>

If a $-notation has no suffix, the value becomes its key itself. The following two examples are equivalent:

optimizer: $
optimizer: $.optimizer

Now we have had both data and a model.