Web Ad Hoc Task
Topics retro-fitted around actual MSNSearch queries
- created by NIST assessors
- “title” field contains actual query except spelling was corrected
- other parts describe assessor’s interpretation
3-way judgments by topic author
- not relevant, relevant, highly relevant
Evaluation by trec_eval
- official scores use all relevant
- emphasis on early precision