Running IR experiments

The module xpmir.experiments contain code factorizing boilerplate for launching experiments

For instance, one can define a standard IR experiments that learns (with tensorboard), evaluates a model on a different metrics and upload it to HuggingFace.

Example

An experiment.py file:

from xpmir.experiments.ir import PaperResults, ir_experiment, ExperimentHelper
from xpmir.papers import configuration

@configuration
class Configuration:
    #: Default learning rate
    learning_rate: float = 1e-3

@ir_experiment()
def run(
    helper: ExperimentHelper, cfg: Configuration
) -> PaperResults:
    ...

    return PaperResults(
        models={"my-model@RR10": outputs.listeners[validation.id]["RR@10"]},
        evaluations=tests,
        tb_logs={"my-model@RR10": learner.logpath},
    )

With full.yaml located in the same folder as experiment.py

file: experiment
learning_rate: 1e-4

The experiment can be started with

Common handling

class xpmir.experiments.cli.ExperimentCallable(*args, **kwargs)

Bases: Protocol

class xpmir.experiments.cli.ExperimentHelper(callable: ExperimentCallable)

Bases: object

Helper for experiments

xpmir.experiments.cli.load(yaml_file: Path)

Loads a YAML file, and parents one if they exist

IR experiment

class xpmir.experiments.ir.IRExperimentHelper(callable: ExperimentCallable)

Bases: ExperimentHelper

xpmir.experiments.ir.ir_experiment(*args, **kwargs)

Wraps an experiment into an IR experiment

1. Upload to github (if requested)

2. Print the results