QA List Task
Instance-finding task
- 25 questions that specified a target number of instances to retrieve
- Name 4 U.S. cities that have a “Shubert” theater
- What are 9 novels written by John Updike?
- response is an unordered set of the target number of instances
- an instance is a single [docid, answer-string] pair
- questions constructed by NIST assessors
- target chosen such that collection had at least that number of instances but > 1 doc required
- single document may have > 1 instance