Evaluation

Metrics

Metrics are backed up by the module ir_measures

XPM Configxpmir.measures.Measure(*, identifier, rel, cutoff)[source]

Bases: Measure

Submit type: xpmir.measures.Measure

Mirrors the ir_measures metric object

identifier: str: main identifier

rel: int = 1: minimum relevance score to be considered relevant (inclusive)

cutoff: int: Cutoff value

List of defined measures

xpmir.measures.AP = Config[xpmir.measures.measure]: Average precision metric

xpmir.measures.P = Config[xpmir.measures.measure]: Precision at rank

xpmir.measures.R = Config[xpmir.measures.measure]: Recall at rank

xpmir.measures.RR = Config[xpmir.measures.measure]: Reciprocical rank

xpmir.measures.Success = Config[xpmir.measures.measure]: 1 if a document with at least rel relevance is found in the first cutoff documents, else 0.

xpmir.measures.nDCG = Config[xpmir.measures.measure]: Normalized Discounted Cumulated Gain

Measures can be used with the @ operator. Exemple:

from xpmir.measures import AP, P, nDCG, RR
from xpmir.evaluation import Evaluate

Evaluate(measures=[AP, P@20, nDCG, nDCG@10, nDCG@20, RR, RR@10], ...)