Ivory
MODULE

IVORY.CORE.DATA

</>

Ivory uses four classes for data presentation: Data, Dataset, Datasets, and DataLoaders.

Basically, you only need to define a class that is a subclass of Data and use original Dataset and Datasets. An example parameter YAML file is:

datasets:
  data:
    class: your.Data  # a subclass of ivory.core.data.Data
  dataset:
  fold: 0

But if you need, you can define your Dataset and/or Datasets.

datasets:
  class: your.Datasets  # a subclass of ivory.core.data.Datasets
  data:
    class: your.Data  # a subclass of ivory.core.data.Data
  dataset:
    def: your.Dataset  # a subclass of ivory.core.data.Dataset
  fold: 0

The DataLoaders is used internally by ivory.torch.trainer.Trainer or ivory.nnabla.trainer.Trainer classes to yield a minibatch in training loop.

Note

Use a 'def' key for dataset instead of 'class'. See Tutorial

Classes
  • Data — Base class to provide data to a Dataset instance.</>
  • Dataset — Dataset class represents a set of data for a mode and fold.</>
  • Datasets — Dataset class represents a collection of Dataset for a fold.</>
  • DataLoaders — DataLoaders class represents a collection of DataLoader.</>
dataclass

ivory.core.data.Data()

</>

Base class to provide data to a Dataset instance.

To make a subclass, you need to assign the following attributes in the Data.init():

  • index: Index of samples.
  • input: Input data.
  • target: Target data.
  • fold: Fold number.
Methods
  • get(index) (tuple) — Returns a tuple of (index, input, target) according to the index.</>
  • get_index(mode, fold) (ndarray) — Returns index according to the mode and fold.</>
  • get_input(index) — Returns input data.</>
  • get_target(index) — Returns target data.</>
  • init() — Initializes index, input, target, and fold attributes.</>
method

init()

</>

Initializes index, input, target, and fold attributes.

The fold number of test data must be -1.

Examples

For regression

def init(self):
    self.index = np.range(100)
    self.input = np.random.randn(100, 5)
    self.target = np.random.randn(100)
    self.fold = np.random.randint(5)
    self.fold[80:] = -1

For classification

def init(self):
    self.index = np.range(100)
    self.input = np.random.randn(100, 5)
    self.target = np.random.randint(100, 10)
    self.fold = np.random.randint(5)
    self.fold[80:] = -1
method

get_index(mode, fold) → ndarray

</>

Returns index according to the mode and fold.

Parameters
  • mode (str) — Mode name: 'train', 'val', or 'test'.
  • fold (int) — Fold number.
method

get_input(index)

</>

Returns input data.

By default, this function returns self.input[index]. You can override this behavior in a subclass.

Parameters
  • index (int or 1D-array) — Index.
method

get_target(index)

</>

Returns target data.

By default, this function returns self.target[index]. You can override this behavior in a subclass.

Parameters
  • index (int or 1D-array) — Index.
method

get(index) → tuple

</>

Returns a tuple of (index, input, target) according to the index.

Parameters
  • index (int or 1D-array) — Index.
dataclass

ivory.core.data.Dataset(data, mode, fold, transform=None)

</>

Dataset class represents a set of data for a mode and fold.

Parameters
  • data (Data) — Data instance that provides data to Dataset instance.
  • mode (str) — Mode name: 'train', 'val', or 'test'.
  • fold (int) — Fold number.
  • transform (callable, optional) — Callable to transform the data.

The transform must take 2 or 3 arguments: (mode, input, optional target) and return a tuple of (input, optional target).

Attributes
  • data (Data) — Data instance that provides data to Dataset instance.
  • fold (int) — Fold number.
  • mode (str) — Mode name: 'train', 'val', or 'test'.
  • transform (callable, optional) — Callable to transform the data.
Methods
  • get(index) (tuple) — Returns a tuple of (index, input, target) according to the index.</>
  • init() — Called at initialization. You can add any process in a subclass.</>
  • sample(n, frac) (tuple) — Returns a tuple of (index, input, target) randomly sampled.</>
method

init()

</>

Called at initialization. You can add any process in a subclass.

method

get(index=None) → tuple

</>

Returns a tuple of (index, input, target) according to the index.

If index is None, reutrns all of the data.

Parameters
  • index (int or 1D-array, optional) — Index.
method

sample(n=0, frac=0.0) → tuple

</>

Returns a tuple of (index, input, target) randomly sampled.

Parameters
  • n (int, optional) — Size of sampling.
  • frac (float, optional) — Ratio of sampling.
dataclass

ivory.core.data.Datasets(data, dataset, fold)

</>
Bases
ivory.core.collections.Dict

Dataset class represents a collection of Dataset for a fold.

Parameters
  • data (Data) — Data instance that provides data to Dataset instance.
  • dataset (callable) — Dataset factory.
  • fold (int) — Fold number.
Attributes
  • data (Data) — Data instance that provides data to Dataset instance.
  • dataset (callable) — Dataset factory.
  • fold (int) — Fold number.
  • test (Dataset) — Test dataset.
  • train (Dataset) — Train dataset.
  • val (Dataset) — Validation dataset.
class

ivory.core.data.DataLoaders(datasets, batch_size, shuffle)

</>
Bases
ivory.core.collections.Dict

DataLoaders class represents a collection of DataLoader.

Parameters
  • datasets (Datasets) — Datasets instance.
  • batch_size (int) — Batch_size
  • shuffle (bool) — If True, train dataset is shuffled. Validation and test dataset are not shuffled regardless of this value.
Attributes
  • test (Dataset) — Test dataset.
  • train (Dataset) — Train dataset.
  • val (Dataset) — Validation dataset.