2019 Deep Learning Track
Return to the TREC home page TREC home Return to the TREC Data home page Data home          National Institute of Standards and Technology Home Page
TREC 2019
Deep Learning track web page
(See track web page for links to test and training corpora.)

NIST judgments for the Document Ranking task
NIST judgments for the Passage Ranking task


Note: Documents were judged on a four-point scale of Not Relevant (0), Relevant (1), Highly Relevant (2) and Perfect (3). Levels 1--3 are considered to be relevant for measures that use binary relevance judgments. Passages were judged on a four-point scale of Not Relevant (0), Related (1), Highly Relevant (2), and Perfect (3), where 'Related' is actually NOT Relevant---it means that the passage was on the same general topic, but did not answer the question. Thus, for Passage Ranking task runs (only), to compute evaluation measures that use binary relevance judgments using trec_eval, you either need to use trec_eval's -l option [trec_eval -l 2 qrelsfile runfile] or modify the qrels file to change all 1 judgments to 0.

Last updated: Friday, 17-Jun-2022 08:35:04 EDT
Date created: February 19, 2020
trec@nist.gov