Topics

Relevance judgments for document retrieval with conditions for assessment in second field:

• 1: Fully manual assessment (same as in previous years of TREC)
• 2: Exclude docs that have UMBRELA judgment < 1; otherwise, fully manual assessment
• 3: Exclude docs that have UMBRELA judgment < 1; assessor is informed that UMBRELA judgment is "Related"
• 4: Exclude docs that have UMBRELA judgment < 1; assessor is informed that UMBRELA judgment is "Highly Relevant"
• 5: Exclude docs that have UMBRELA judgment < 1; assessor is informed that UMBRELA judgment is "Perfectly Relevant"

UMBRELA passage relevance judgments

Nugget assessments (up to two runs per team):

• Partial nugget assessments (November 8, 2024)
• Final nugget assessments (December 18, 2024)

Citation/support assessments (up to two runs per team, one citation per sentence):

• final.citation_judgments_with_prediction.20241025.jsonl assessor shown prediction
• final.citation_judgments_without_prediction.20241025.jsonl assessor not shown prediction
• final.citation_judgments_Webassess.20241031.jsonl assessor not shown prediction (may include assessment of more than one citation per sentence, but these extra assessments are ignored for official scores)