Experiment

Here is a tutorial on Google Colab that shows how to use the experiment module

Create Experiment

labml.experiment.create(*, uuid: Optional[str] = None, name: Optional[str] = None, python_file: Optional[str] = None, comment: Optional[str] = None, writers: Optional[Set[str]] = None, ignore_callers: Optional[Set[str]] = None, tags: Optional[Set[str]] = None, disable_screen: bool = False)

Create an experiment

Keyword Arguments

• name (str, optional) – name of the experiment

• python_file (str, optional) – path of the Python file that created the experiment

• comment (str, optional) – a short description of the experiment

• writers (Set[str], optional) – list of writers to write stat to. Defaults to {'tensorboard', 'sqlite', 'web_api'}.

• ignore_callers – (Set[str], optional): list of files to ignore when automatically determining python_file

• tags (Set[str], optional) – Set of tags for experiment

labml.experiment.record(*, name: Optional[str] = None, comment: Optional[str] = None, writers: Optional[Set[str]] = None, tags: Optional[Set[str]] = None, exp_conf: Optional[Dict[str, any]] = None, lab_conf: Optional[Dict[str, any]] = None, token: Optional[str] = None, disable_screen: bool = False)

This combines create(), configs() and start().

Keyword Arguments

• name (str, optional) – name of the experiment

• comment (str, optional) – a short description of the experiment

• writers (Set[str], optional) – list of writers to write stat to. Defaults to {'tensorboard', 'sqlite', 'web_api'}.

• tags (Set[str], optional) – Set of tags for experiment

• exp_conf (Dict[str, any], optional) – a dictionary of experiment configurations

• lab_conf (Dict[str, any], optional) – a dictionary of configurations for LabML. Use this if you want to change default configurations such as web_api, and data_path.

• token (str, optional) – a shortcut to provide LabML mobile app token (or url - web_api) instead of including it in lab_conf. You can set this with labml.lab.configure(), or with a configuration file for the entire project.

labml.experiment.evaluate()

This should be used for evaluation of a saved experiment. This will not record anything.

labml.experiment.start()

Starts the experiment. Run it using with statement and it will monitor and report, experiment completion and exceptions.

labml.experiment.get_uuid()

Returns the UUID of the current experiment run

Checkpoints

labml.experiment.load(run_uuid: str, checkpoint: Optional[int] = None)

Loads a the run from a previous checkpoint. You need to separately call experiment.start to start the experiment.

Parameters

• run_uuid (str) – experiment will start from a saved state in the run with UUID run_uuid

• checkpoint (str, optional) – if provided the experiment will start from given checkpoint. Otherwise it will start from the last checkpoint.

labml.experiment.load_models(models: List[str], run_uuid: str, checkpoint: Optional[int] = None)

Loads and starts the run from a previous checkpoint.

Parameters

• models (List[str]) – List of names of models to be loaded

• run_uuid (str) – experiment will start from a saved state in the run with UUID run_uuid

• checkpoint (str, optional) – if provided the experiment will start from given checkpoint. Otherwise it will start from the last checkpoint.

labml.experiment.save_checkpoint()

Saves model checkpoints

labml.experiment.add_pytorch_models(**kwargs: torch.nn.Module)

labml.experiment.add_pytorch_models(models: Dict[str, torch.nn.Module])

Set variables for saving and loading

Parameters

models (Dict[str, torch.nn.Module]) – a dictionary of torch modules used in the experiment. These will be saved with labml.experiment.save_checkpoint() and loaded with labml.experiment.load().

labml.experiment.add_sklearn_models(models: Dict[str, any])

Warning

This is still experimental.

Set variables for saving and loading

Parameters

models (Dict[str, any]) – a dictionary of SKLearn models These will be saved with labml.experiment.save_checkpoint() and loaded with labml.experiment.load().

labml.experiment.add_model_savers(savers: Dict[str, ModelSaver])

Add custom model savers

Parameters

savers (Dict[str, ModelSaver]) – a dictionary of labml.experiment.ModelSaver. These will be saved with labml.experiment.save_checkpoint() and loaded with labml.experiment.load().

class labml.experiment.ModelSaver

An abstract class defining model saver/loader.

The implementation should keep a reference to the model and load and save the model parameters.

save(checkpoint_path: Path) → any

Saves the model in the given checkpoint path

Parameters

checkpoint_path (pathlib.Path) – The path to save the model at

Returns any meta info, such as the individual filename(s)

load(checkpoint_path: Path, info: any) → None

Loads the model from the given checkpoint path

Parameters

• checkpoint_path (pathlib.Path) – The path to load the model from

• info (any) – The returned meta data when saving

Configurations & Hyper-parameters

labml.experiment.configs(conf_dict: Dict[str, any])

labml.experiment.configs(conf_dict: Dict[str, any], conf_override: Dict[str, any])

labml.experiment.configs(conf: BaseConfigs)

labml.experiment.configs(conf: BaseConfigs, conf_override: Dict[str, any])

Calculate configurations

This has multiple overloads

labml.experiment.configs(conf_dict: Dict[str, any])

labml.experiment.configs(conf_dict: Dict[str, any], conf_override: Dict[str, any])

labml.experiment.configs(conf: BaseConfigs)

labml.experiment.configs(conf: BaseConfigs, run_order: List[Union[List[str], str]])

labml.experiment.configs(conf: BaseConfigs, *run_order: str)

labml.experiment.configs(conf: BaseConfigs, conf_override: Dict[str, any])

labml.experiment.configs(conf: BaseConfigs, conf_override: Dict[str, any], run_order: List[Union[List[str], str]])

labml.experiment.configs(conf: BaseConfigs, conf_override: Dict[str, any], *run_order: str)

Parameters

• conf (BaseConfigs, optional) – configurations object

• conf_dict (Dict[str, any], optional) – a dictionary of configs

• conf_override (Dict[str, any], optional) – a dictionary of configs to be overridden

labml.experiment.load_configs(run_uuid: str, *, is_only_hyperparam: bool = True)

Load configs of a previous run

Parameters

run_uuid (str) – if provided the experiment will start from a saved state in the run with UUID run_uuid

Keyword Arguments

is_only_hyperparam (bool, optional) – if True all only the hyper parameters are returned

Bundle checkpoints

labml.experiment.save_bundle(path: Path, run_uuid: str, checkpoint: Optional[int] = None, *, data_files: Optional[List[str]] = None)

Create a .tar.gz file with the configs and checkpoints that can be distributed and loaded easily.

Parameters

• path (Path) – .tar.gz file path

• run_uuid (str) – experiment run to bundle

• checkpoint (str, optional) – if provided the given checkpoint will be bundled. Otherwise it will bundle the last checkpoint.

• data_files – List of data files (relative to labml.lab.get_data_path()) to be bundled.

labml.experiment.load_bundle(path: Path, *, url: Optional[str] = None) → Tuple[str, int]

Extract a bundle into experiments folder and returns the run_uuid and checkpoint.

Parameters

• path (Path) – .tar.gz file path

• url (str) – url to download the .tar.gz file from

Distirbuted training

labml.experiment.distributed(rank: int, world_size: int)

Set the rank and world_size of the current process in an distributed training setup.

Parameters

• rank (int) – rank of the current process

• world_size (int) – number of training processes

Utilities

labml.experiment.save_numpy(name: str, array: ndarray)

Saves a single numpy array. This is used to save processed data.