Optimization

Parameters

XPM Configxpmir.letor.optim.ParameterOptimizer(*, optimizer, scheduler, module, filter)

Bases: experimaestro.core.objects.Config

Associates an optimizer with a list of parameters to optimize

optimizer: xpmir.letor.optim.Optimizer

The optimizer

scheduler: xpmir.letor.schedulers.Scheduler

The optional scheduler

module: xpmir.letor.optim.Module

The module from which parameters should be extracted

filter: xpmir.letor.optim.ParameterFilter = xpmir.letor.optim.ParameterFilter()

How parameters should be selected for this (by default, use them all)

XPM Configxpmir.letor.optim.Optimizer

Bases: experimaestro.core.objects.Config

XPM Configxpmir.letor.optim.ParameterFilter

Bases: experimaestro.core.objects.Config

XPM Configxpmir.letor.optim.Module

Bases: experimaestro.core.objects.Config, TorchModule

A module contains parameters

Batching

XPM Configxpmir.letor.batchers.Batcher

Bases: experimaestro.core.objects.Config

Responsible for micro-batching when the batch does not fit in memory

The base class just does nothing (no adaptation)

XPM Configxpmir.letor.batchers.PowerAdaptativeBatcher

Bases: xpmir.letor.batchers.Batcher

Starts with the provided batch size, and then divides in 2, 3, etc. until there is no more OOM

Devices

The devices configuration allow to select both the device to use for computation and the way to use it (i.e. multi-gpu settings).

XPM Configxpmir.letor.devices.Device

Bases: experimaestro.core.objects.Config

Device to use, as well as specific option (e.g. parallelism)

XPM Configxpmir.letor.devices.CudaDevice(*, gpu_determ, cpu_fallback, distributed)

Bases: xpmir.letor.devices.Device

CUDA device

gpu_determ: bool = False

Sets the deterministic

cpu_fallback: bool = False

Fallback to CPU if no GPU is available

distributed: bool = False

Use distributed computation (DDP) when distributed is True and the number of GPUs greater than 1

Schedulers

XPM Configxpmir.letor.schedulers.CosineWithWarmup(*, num_warmup_steps, num_cycles)

Bases: xpmir.letor.schedulers.Scheduler

Cosine schedule with warmup

Uses the implementation of the transformer library

https://huggingface.co/docs/transformers/main_classes/optimizer_schedules#transformers.get_cosine_schedule_with_warmup

num_warmup_steps: int

Number of warmup steps

num_cycles: float = 0.5

Number of cycles

num_cycles: float = 0.5

Number of cycles

num_warmup_steps: int

Number of warmup steps

XPM Configxpmir.letor.schedulers.LinearWithWarmup(*, num_warmup_steps, min_factor)

Bases: xpmir.letor.schedulers.Scheduler

Linear warmup followed by decay

num_warmup_steps: int

Number of warmup steps

min_factor: float = 0.0

Minimum multiplicative factor

min_factor: float = 0.0

Minimum multiplicative factor

num_warmup_steps: int

Number of warmup steps

XPM Configxpmir.letor.schedulers.Scheduler

Bases: experimaestro.core.objects.Config

Base class for all optimizers schedulers