Evaluation

Evaluation

XPM Taskxpmir.evaluation.Evaluate(*, measures, dataset, retriever)

Bases: xpmir.evaluation.BaseEvaluation, experimaestro.core.objects.Task

Evaluate a retriever

measures: List[xpmir.measures.Measure] = [Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure], Config[xpmir.measures.measure]]

List of metrics

aggregated: Pathgenerated

Path for aggregated results

detailed: Pathgenerated

Path for detailed results

dataset: datamaestro_text.data.ir.Adhoc

The dataset for retrieval

retriever: xpmir.rankers.Retriever

The retriever to evaluate

Metrics

Metrics are backed up by the module ir_measures

XPM Configxpmir.measures.Measure(*, identifier, rel, cutoff)

Bases: datamaestro_text.data.ir.Measure

Mirrors the ir_measures metric object

identifier: str

main identifier

rel: int = 1

minimum relevance score to be considered relevant (inclusive)

cutoff: int

Cutoff value

List of defined measures

xpmir.measures.AP = Config[xpmir.measures.measure]

Average precision metric

xpmir.measures.P = Config[xpmir.measures.measure]

Precision at rank

xpmir.measures.RR = Config[xpmir.measures.measure]

Reciprocical rank

xpmir.measures.nDCG = Config[xpmir.measures.measure]

Normalized Discounted Cumulated Gain

Measures can be used with the @ operator. Exemple:

from xpmir.measures import AP, P, nDCG, RR
from xpmir.evaluation import Evaluate

Evaluate(measures=[AP, P@20, nDCG, nDCG@10, nDCG@20, RR, RR@10], ...)