Relevance Judgments for TREC 2004 Genomics Track Ad Hoc Task Bill Hersh 9/27/04 The accompanying files contain the relevance judgments for the 50 topics of the ad hoc task. The MEDLINE documents were generated by pooling the top 75 ranking documents from each topic for the top priority run for each of the 27 particpating groups. A total of 48,753 documents were judged, with an average of 975 (range 476-1450) per topic. Records were judged as not relevant (NR), possibly relevant (PR), or definitely relevant (DR). Every topic had at least one PR document, while three had no DR documents. For the official results of TREC 2004, we will consider any document judged as PR or DR to be relevant. In other words, the qrels file for trec_eval will have all documents judged to be PR or DR. The table below shows the number of DR, PR, P+DR, NR, and total documents for each topic. The final rows contain the minimum, mean, and maximum for each topic. We are currently carrying out an analysis of the two different judges, who are also in the process of carrying out some judgments in duplicate. There are two files of relevance judgments: trecgen04-judgments.txt - a file of all relevance judgments done, based on the format: where = 1 (DR), 2 (PR), or 3 (NR) trecgen04-qrels.txt - a file of relevant documents, based on the format: <0> The qrels file is used for input to the trec_eval program to generate recall-precision results. The trec_eval program considers any document with a judgment greater than 0 to be relevant, which in the trecgen04-qrels.txt file is all documents. Those wanting to experiment with DR-only relevance judgments can do so, although only 47 topics can be used (since 3 have no DR documents). Topic DR PR P+DR NR Total 1 38 41 79 800 879 2 40 61 101 1163 1264 3 149 32 181 1008 1189 4 12 18 30 1140 1170 5 5 19 24 1147 1171 6 41 53 94 693 787 7 56 59 115 615 730 8 76 85 161 777 938 9 103 12 115 478 593 10 3 1 4 1122 1126 11 87 24 111 631 742 12 166 90 256 554 810 13 5 19 24 1094 1118 14 13 8 21 927 948 15 50 40 90 1021 1111 16 94 53 147 931 1078 17 2 1 3 1147 1150 18 0 1 1 1391 1392 19 0 1 1 1134 1135 20 55 61 116 698 814 21 26 54 80 596 676 22 125 85 210 875 1085 23 137 21 158 757 915 24 7 19 26 926 952 25 6 26 32 1110 1142 26 35 12 47 745 792 27 19 10 29 726 755 28 6 7 13 823 836 29 33 10 43 713 756 30 101 64 165 917 1082 31 0 138 138 739 877 32 441 55 496 611 1107 33 30 34 64 748 812 34 1 30 31 747 778 35 253 18 271 446 717 36 164 90 254 422 676 37 138 11 149 327 476 38 334 89 423 742 1165 39 146 171 317 1033 1350 40 134 143 277 891 1168 41 333 249 582 298 880 42 191 506 697 308 1005 43 25 170 195 544 739 44 485 164 649 575 1224 45 108 48 156 983 1139 46 111 86 197 545 742 47 81 284 365 1085 1450 48 53 102 155 966 1121 49 32 41 73 1027 1100 50 79 223 302 789 1091 Total 4629 3639 8268 40485 48753 Min 0 1 1 298 476 Mean 92.58 72.78 165.36 809.7 975.06 Max 485 506 697 1391 1450