Masked Language Model


XPM Configxpmir.mlm.samplers.MLMSampler(*, datasets)[source]

Bases: Sampler

Submit type: xpmir.mlm.samplers.MLMSampler

Sample texts from various sources

This sampler can be used for Masked Language Modeling to sample from several datasets.

datasets: List[]

Lists of datasets to sample from


XPM Configxpmir.mlm.trainer.MLMTrainer(*, hooks, model, batcher, sampler, batch_size, lossfn)[source]

Bases: LossTrainer

Submit type: xpmir.mlm.trainer.MLMTrainer

Trainer for Masked Language Modeling

hooks: List[xpmir.learning.context.TrainingHook] = []

Hooks for this trainer: this includes the losses, but can be adapted for other uses The specific list of hooks depends on the specific trainer

model: xpmir.learning.optim.Module

If the model to optimize is different from the model passsed to Learn, this parameter can be used – initialization is still expected to be done at the learner level

batcher: xpmir.learning.batchers.Batcher = xpmir.learning.batchers.Batcher.XPMValue()

How to batch samples together

sampler: xpmir.learning.base.Sampler
batch_size: int = 16

Number of samples per batch

lossfn: xpmir.mlm.trainer.MLMLoss = xpmir.mlm.trainer.CrossEntropyLoss.XPMValue(weight=1.0)


XPM Configxpmir.mlm.trainer.MLMLoss(*, weight)[source]

Bases: Config

Submit type: xpmir.mlm.trainer.MLMLoss

weight: float = 1.0
XPM Configxpmir.mlm.trainer.CrossEntropyLoss(*, weight)[source]

Bases: MLMLoss

Submit type: xpmir.mlm.trainer.CrossEntropyLoss

Computes cross-entropy

Uses a CE with logits if the scorer output type is not a probability

weight: float = 1.0