Configs¶

The configurations provide an API to easily manage hyper-parameters and other configurable parameters of the experiments. The configuration of each experiment run are stored. These can be viewed on Dashboard.

%%capture
!pip install labml

import torch
from torch import nn

from labml import tracker, monit, experiment, logger
from labml.configs import BaseConfigs, option, calculate, hyperparams, aggregate

Define a configuration class¶

class DeviceConfigs(BaseConfigs):
    use_cuda: bool = True
    cuda_device: int = 0

    device: any

Calculated configurations¶

@option(DeviceConfigs.device)
def cuda(c: DeviceConfigs):
    is_cuda = c.use_cuda and torch.cuda.is_available()
    if not is_cuda:
        return torch.device("cpu")
    else:
        if c.cuda_device < torch.cuda.device_count():
            return torch.device(f"cuda:{c.cuda_device}")
        else:
            logger.log(f"Cuda device index {c.cuda_device} higher than "
                       f"device count {torch.cuda.device_count()}", Text.warning)
            return torch.device(f"cuda:{torch.cuda.device_count() - 1}")

Inheriting and re-using configuration classes¶

Configs classes can be inherited. This lets you separate configs into modules instead of passing monolithic config object.

You can even inherit a entire experiment setups and make a few modifications.

class Configs(DeviceConfigs):
    model_size: int = 1024
    input_size: int = 10
    output_size: int = 10

    model: any = 'two_hidden_layer'
    epochs = 10
    steps_per_epcoch = 1024
    total_steps: int
    variant: str

Defining configurations options¶

You can specify multiple config calculator functions. You pick which one to use by its name.

class OneHiddenLayerModule(nn.Module):
    def __init__(self, input_size: int, model_size: int, output_size: int):
        super().__init__()
        self.input_fc = nn.Linear(input_size, model_size)
        self.output_fc = nn.Linear(model_size, output_size)

    def forward(x: torch.Tensor):
        x = F.relu(self.input_fc(x))
        return self.output_fc(x)

# This is just for illustration purposes, ideally you should have a configuration
# for number of hidden layers.
# A real world example would be different architectures, like a dense network vs a CNN
class TwoHiddenLayerModule(nn.Module):
    def __init__(self, input_size: int, model_size: int, output_size: int):
        super().__init__()
        self.input_fc = nn.Linear(input_size, model_size)
        self.middle_fc = nn.Linear(model_size, model_size)
        self.output_fc = nn.Linear(model_size, output_size)

    def forward(x: torch.Tensor):
        x = F.relu(self.input_fc(x))
        x = F.relu(self.middle_fc(x))
        return self.output_fc(x)


@option(Configs.model)
def one_hidden_layer(c: Configs):
    return OneHiddenLayerModule(c.input_size, c.model_size, c.output_size)

@option(Configs.model)
def two_hidden_layer(c: Configs):
    return TwoHiddenLayerModule(c.input_size, c.model_size, c.output_size)

Note that the configurations calculators pass only the needed parameters and not the whole config object. The library forces you to do that.

However, you can directly set the model as an option, with __init__ accepting Configs as a parameter, it is not a usage pattern we encourage.

Calculating with predefined functions of lambdas¶

You can also compute configs with lambda functions or predefined functions

_ = calculate(Configs.total_steps, [Configs.epochs, Configs.steps_per_epcoch], lambda e, s: e * s)

Aggregates¶

You can use aggregates to setup configs that depend on each other. For example, this is useful when you have LSTM based model and a CNN based model, and different data loaders for each architecture.

Here we use variant to set both model and number of epochs.

aggregate(Configs.variant, 'small', (Configs.model, 'one_hidden_layer'), (Configs.epochs, 10))
aggregate(Configs.variant, 'large', (Configs.model, 'two_hidden_layer'), (Configs.epochs, 100))

Hyper-parameters¶

LabML will identify any parameter you explicily specify outside the declaration of the class as hyper-parameters (in this case variant because we set it with conf.variant = 'large'. You can also specify hyper-parameters manually.

The hyper-parameters will be highlighted among other configs in logs and in dashboard. These will also be logged in to Tensorboard.

hyperparams(Configs.epochs)
hyperparams(Configs.total_steps, is_hyperparam=False)

Running the experiment¶

Here’s how you run an experiment with the configurations.

conf = Configs()
conf.variant = 'large'
experiment.create(name='test_configs')
experiment.configs(conf)
logger.inspect(model=conf.model)

Prepare model...[DONE]     16.92ms
model: TwoHiddenLayerModule(\n  (input_fc): Linear(in_features=10, out_features=1024, bia ...

experiment.start()

test_configs: 66482396392b11eba0aeacde48001122
    [dirty]: "link nn"

<labml.internal.experiment.watcher.ExperimentWatcher at 0x7f93a76ceb50>


--------------------------------------------------
LABML WARNING
LabML App Warning: empty_token: Please create a valid token at https://web.lab-ml.com.
Click on the experiment link to monitor the experiment and add it to your experiments list.
--------------------------------------------------

Monitor experiment at https://web.lab-ml.com/run?run_uuid=66482396392b11eba0aeacde48001122