Optimization
Parameters
- XPM Configxpmir.letor.optim.ParameterOptimizer(*, optimizer, scheduler, module, filter)
Bases:
experimaestro.core.objects.Config
Associates an optimizer with a list of parameters to optimize
- optimizer: xpmir.letor.optim.Optimizer
The optimizer
- scheduler: xpmir.letor.schedulers.Scheduler
The optional scheduler
- module: xpmir.letor.optim.Module
The module from which parameters should be extracted
- filter: xpmir.letor.optim.ParameterFilter = xpmir.letor.optim.ParameterFilter()
How parameters should be selected for this (by default, use them all)
- XPM Configxpmir.letor.optim.Optimizer
Bases:
experimaestro.core.objects.Config
- XPM Configxpmir.letor.optim.ParameterFilter
Bases:
experimaestro.core.objects.Config
- XPM Configxpmir.letor.optim.Module
Bases:
experimaestro.core.objects.Config
,TorchModule
A module contains parameters
Batching
- XPM Configxpmir.letor.batchers.Batcher
Bases:
experimaestro.core.objects.Config
Responsible for micro-batching when the batch does not fit in memory
The base class just does nothing (no adaptation)
- XPM Configxpmir.letor.batchers.PowerAdaptativeBatcher
Bases:
xpmir.letor.batchers.Batcher
Starts with the provided batch size, and then divides in 2, 3, etc. until there is no more OOM
Devices
The devices configuration allow to select both the device to use for computation and the way to use it (i.e. multi-gpu settings).
- XPM Configxpmir.letor.devices.Device
Bases:
experimaestro.core.objects.Config
Device to use, as well as specific option (e.g. parallelism)
- XPM Configxpmir.letor.devices.CudaDevice(*, gpu_determ, cpu_fallback, distributed)
Bases:
xpmir.letor.devices.Device
CUDA device
- gpu_determ: bool = False
Sets the deterministic
- cpu_fallback: bool = False
Fallback to CPU if no GPU is available
- distributed: bool = False
Use distributed computation (DDP) when distributed is True and the number of GPUs greater than 1
Schedulers
- XPM Configxpmir.letor.schedulers.CosineWithWarmup(*, num_warmup_steps, num_cycles)
Bases:
xpmir.letor.schedulers.Scheduler
Cosine schedule with warmup
Uses the implementation of the transformer library
- num_warmup_steps: int
Number of warmup steps
- num_cycles: float = 0.5
Number of cycles
- num_cycles: float = 0.5
Number of cycles
- num_warmup_steps: int
Number of warmup steps
- XPM Configxpmir.letor.schedulers.LinearWithWarmup(*, num_warmup_steps, min_factor)
Bases:
xpmir.letor.schedulers.Scheduler
Linear warmup followed by decay
- num_warmup_steps: int
Number of warmup steps
- min_factor: float = 0.0
Minimum multiplicative factor
- min_factor: float = 0.0
Minimum multiplicative factor
- num_warmup_steps: int
Number of warmup steps
- XPM Configxpmir.letor.schedulers.Scheduler
Bases:
experimaestro.core.objects.Config
Base class for all optimizers schedulers