- TREC 2019
Deep Learning track web page
(See track web page for links to test and training corpora.)
NIST judgments for the Document Ranking task
NIST judgments for the Passage Ranking task
Note: Documents were judged on a four-poiunt scale of Not Relevant (0), Relevant (1),
Highly Relevant (2) and Perfect (3). Levels 1--3 are considered to be relevant for
measures that use binary relevance judgments.
Passages were judged on a four-point scale of Not Relevant (0), Related (1),
Highly Relevant (2), and Perfect (3), where 'Related' is actually NOT
Relevant---it means that the passage was on the same general topic, but did not
answer the question. Thus, for Passage Ranking task runs (only), to
compute evaluation measures that use binary relevance judgments using trec_eval,
you either need to use trec_eval's -l option [trec_eval -l 2 qrelsfile runfile]
or modify the qrels file to change all 1 judgments to 0.