Running IR experiments
The module experimaestro.experiments contain code factorizing boilerplate for launching experiments, which is specialized in xpmir with specific experiment helpers.
For instance, one can define a standard IR experiments that learns (with tensorboard), evaluates a model on a different metrics and upload it to HuggingFace.
With full.yaml located in the same folder as experiment.py
file: experiment
learning_rate: 1e-4
The experiment can be started with
experimaestro run-experiment --run-mode normal full.yaml
Learning experiment
Generic learning experiments can be conducted with the :py:module:`xpmir.experiments.learning` module that allows to easily use a Tensorboard service.
from experimaestro.experiments import configuration
from xpmir.experiments.learning import PaperResults, learning_experiment, LearningExperimentHelper
@configuration
class Configuration:
#: Default learning rate
learning_rate: float = 1e-3
@learning_experiment()
def run(
helper: LearningExperimentHelper, cfg: Configuration
) -> PaperResults:
...
IR experiment
Example
from experimaestro.experiments import configuration
from xpmir.experiments.ir import PaperResults, ir_experiment, IRExperimentHelper
@configuration
class Configuration:
#: Default learning rate
learning_rate: float = 1e-3
@ir_experiment()
def run(
helper: IRExperimentHelper, cfg: Configuration
) -> PaperResults:
...
return PaperResults(
models={"my-model@RR10": outputs.listeners[validation.id]["RR@10"]},
evaluations=tests,
tb_logs={"my-model@RR10": learner.logpath},
)
API
- class xpmir.experiments.ir.IRExperimentHelper(callable: ExperimentCallable)[source]
Bases:
LearningExperimentHelper
Helper for IR experiments
- callable: ExperimentCallable
Run function
- property tensorboard_service
Returns a tensorboard service
- xpmir.experiments.ir.ir_experiment(*args, **kwargs)
Uses an IR experiment helper that provides
1. Tensorboard service (from Learning) 1. Upload to HuggingFace 1. Printing the evaluation results