This directory contains the anonymized responses and assessments for each run for the list and "other" questions in the main task of the QA track of TREC 14. Each original run tag for each question type (list, other) has been mapped to a new tag, "RUN-X", where X is a random number. Responses/assessments for the list questions are in: list/RUN-X.list_judged Responses/assessments for the "other" questions are in: other/RUN-X.otherjudged A list assessment file contains lines of the form qid tag docid correctness-value distinct? string where all the fields except correctness-value and distinct? are what was submitted to NIST. Correctness- value takes on the values -1, 1, 2, 3 for Wrong, Right, Unsupported and ineXact and are assigned as for factoid questions. distinct? is a binary value. It is 1 only if the corresponding string was judged Right and it is the one occurrence of this answer in the set. If multiple equivalent Right answers are returned for a list question, an arbitrary instance is marked distinct, and the others remain unmarked. The numerator for recall and precision of the list responses is the number of answers for which distinct? is one. An "other" question assessment file contains two parts per question. The first part repeats what was submitted to NIST with the exception that an "item number" is added. An item number is simply the count of the strings submitted to NIST. Thus the format of this part is qid tag item-number docno string The second part lists the nuggets that were assigned per item: qid tag item-number nugget-number where nugget number is the number of the nugget as listed in the "other_answers.txt" file posted to the web site. For example the line 52.6 example 12 6 means that the sixth nugget of question 52.6 ("over 10,000 Burger Kings worldwide") was found in the 12th item in the response submitted to NIST. If a nugget appeared multiple times in a response, it was marked only once with the assessor picking the item he or she thought was the "best" match (for an unknown and arbitrary definition of best). A single item might match multiple nuggets, in which case the item is repeated in the second part. For example: 52.6 example 5 2 52.6 example 5 8 Means that the fifth item in the response for question 52.6 contained nuggets 2 ("Diageo is parent company"), and 8 ("Diageo to spinoff BK"). If no nuggets were found in the response, then the second part will be empty.