Evaluation

XPM Taskxpmir.evaluation.Evaluate(*, measures, dataset, retriever)

Bases: xpmir.evaluation.BaseEvaluation, experimaestro.core.objects.Task

Evaluate a retriever

measures: List[xpmir.measures.Measure] = [Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure]]: List of metrics

aggregated: Pathgenerated: Path for aggregated results

detailed: Pathgenerated: Path for detailed results

dataset: datamaestro_text.data.ir.Adhoc: The dataset for retrieval

retriever: xpmir.rankers.Retriever: The retriever to evaluate

Metrics

Metrics are backed up by the module ir_measures

XPM Configxpmir.measures.Measure(*, identifier, rel, cutoff)

Bases: datamaestro_text.data.ir.Measure

Mirrors the ir_measures metric object

identifier: str: main identifier

rel: int = 1: minimum relevance score to be considered relevant (inclusive)

cutoff: int: Cutoff value

List of defined measures

xpmir.measures.AP = Config[xpmir.measures.measure]: Average precision metric

xpmir.measures.P = Config[xpmir.measures.measure]: Precision at rank

xpmir.measures.RR = Config[xpmir.measures.measure]: Reciprocical rank

xpmir.measures.nDCG = Config[xpmir.measures.measure]: Normalized Discounted Cumulated Gain

Measures can be used with the @ operator. Exemple:

from xpmir.measures import AP, P, nDCG, RR
from xpmir.evaluation import Evaluate

Evaluate(measures=[AP, P@20, nDCG, nDCG@10, nDCG@20, RR, RR@10], ...)