APPENDIX A This appendix contains tables of results for all the TREC-1 participants except the TIPSThR panel, whose tables appear in Appendix B. The tables in Appendix A and Appendix B show various measures of the perfor- mance on the adhoc and routing tasks. The adhoc results come first, followed by the routing results, with the tables in the same order as the presentation order of the papers. The definitions of the evaluation measures are given in the Overview, section 4, and readers unfamiliar with these measures should read that section first. Care should be taken in comparing the tables across systems. These measures show performance only, with no measure of user or system effort. Additionally, because of misunderstandings about the query categories, some results may reflect manual adjustments of the queries even though the results are not formally categorized as feedback results. The tables contain four major boxes of statistics and two graphs. Box 1-- Summary Statistics line 1-- unique run identifier, data subset, and query construction method used Data subset full (disks 1 and 2 for adhoc, disk 2 for routing) category B (the official subset of data, 1/4 of the data using the Wall Street Journal articles) disk 1 only disk 2 only wsj, disk 1 only ~all Street Journal from disk 1) wsj, disk 2 only (Wall Street Journal from disk 2) Query construction method automatic (method 1) manual (method 2) feedback (method 3, frozen evaluation used) line 2-- Number of topics included in averages. line 3-- Total number of documents retrieved over all topics. Here, "retrieved" means having a rank less 200. line 4-- Total number of relevant documents for all topics in the collection (whether retrieved or not). line 5-- Total number of relevant retrieved documents for this run. Box 2-- Recall Level Averages lines 1-11-- The average over all topics of the precision at each of the 11 recall points given. Note that this is interpolated precision: e.g., for a particular topic, if the precision at 0.50 recall is greater than the precision at OAO recall, then the precision at 0.50 recall is used for both the 0.50 and 0A0 recall levels. line 12-- The average precision based on the 11 recall points in lines 1-11. line 13-- The average precision based on 3 intermediate recall points (0.2, 0.5, and 0.8). Box 3-- Document Level Averages lines 1-5-- The average recall and precision after the given number of documents have been retrieved. 373 Box 4-- ROC Averages lines 1-10-- The average prohability of detection (recall) at a fixed probability of false alarrn rate (fallout). Ten equally spaced false alarm points in the high precision end of the curve were used, and for each topic, the highest detection value in the region surrounding that false alarm point is selected. The table shows the averages of these points across all topics. Graph 1-- Recall-Precision Curve This is a plot of the data shown in Box 2. Graph 2-- Norrnal Deviate - Detection-False Alarm This is a plot of the data shown in Box 4, but plotted on probability scales. See Overview, section 42 for a reference explaining these plots. 374 Lilts - City University, London Summary Statistics Run number citya 1-full, automatic Recall-Precision Curve 1 Num~queries 50 Total number of docu~nents over all queries Retrieved 9989 0.8 - Relevant 16400 Rel~ret 2722 0.6 - ecall Level Averages Document Level Averages Precision Lii Precision Recall Precision 0.4 [) 0.7089 at 5 docs 0.0122 0.4960 [) 0.3855 at 15 docs 0.0298 0.4600 0.2 [) 0.1727 at 30 docs 0.0504 0.4207 [) 0.0594 at 100 docs 0.1276 0.3462 A A [) 0.0089 at 200 docs 0.1892 0.2722 0 [) 0.0000 0 0.2 0.4 0.6 ROC Averages Recall [) 0.0000 False Alarm Detection [) 0.0000 0.00000 0.0379 NORMAL DEVIATE - Detection-False Alarm [) 0.0000 0.00001 0.0555 0.001 0.02 0.16 C 5 0.84 0.98 0. 9 [) 0.0000 0.00002 0.0734 3 [) 0.0000 0.00003 0.0847 ~r~cision for all pointsl ½<½ 0.00004 0.0985 2 ~vg 0.1214 0.00005 0.1086 ~e precision for 3 0.00006 0.1200 Lediate points 0.00007 0.1291 0.50, 0.80) 0.00008 0.1377 0 ~vg 0.0576 0.00009 0.1442 -1 -~ -2 -3 I II -3 -2 -1 0 2 3 ;u'ts - City University, London Summary Statistics ] Run number citymi-full, manual 1 Recall- Precision Curve Num~queries 50 1 Total number of documents over all queries Retrieved 9898 0.8. Relevant 16400 Rel~ret 3042 0.6 - b~ecall Level Averages Document Level_Averages Precision all Precision Recall Precision 0.4 0 0.7813 at 5 docs 0.0129 0.5760 0 0.4481 at 15 docs 0.0328 0.5000 0.2 0 0.2399 at 30 docs 0.0593 0.4573 0 0.0938 at 100 docs 0.1514 0.3818 0 0.0506 at 200 docs 0.2224 0.3042 0 0 0.0425 0 0.0166 ROC Averages False Alarm Detection 0 0.0154 0 0.0147 0.00000 0.0525 0 0.0133 0.00001 0.0735 0 0.0000 0.00002 0.0940 0.00003 0.1097 for all points 0.00004 0.1243 Avg 0.1560 J 0.00005 0.1360 ge precision for 3 0.00006 0.1456 iediate points 0.00007 0.1565 0.00008 0.1631 I 0.00009 0.1715 0.50, 0.80) ~vg 0.0990 0 0.2 0.4 0.6 0. Recall NORMAL DEVIATE - Detection-False Alarm 0. `01 0.02 3 0 -1 -2 -3 0.16 ( 5 C ). I I I I 0.84 0.98 0.1 -3 -2 -1 0 2 Lilts - City University, London Summary Statistics Run number citym2-full, feedback 1 Num~queries 50 Total number of documents over all queries Retrieved 10829 0.8 Relevant 16400 Rel~ret 3431 0.6 ecall Level Averages Document Level Averages Precision LIl Precision Recall Precision 0.4 [) 0.7569 at 5 docs 0.0126 0.5800 [) 0.5072 at 15 docs 0.0353 0.5333 0.2 () 0.3262 at 30 docs 0.0654 0.5193 0 0.1442 at 100 docs 0.1662 0.4304 0 0.1016 at 200 docs 0.2407 0.3300 0 0 0.0667 ROC Averages 0 0.0418 False Alarm Detection 0 0.0174 0.00000 0.0795 0 0.0170 0.00001 0.1028 0 0.0143 0.00002 0.1245 0 0.0000 0.00003 0.1377 ge precision for all points 0.00004 0.1556 Avg 0.1812 0.00005 0.1684 ge precision for 3 0.00006 0.1766 iediate points 0.00007 0.1878 0.50, 0.80) 0.00008 0.1954 ~vg 0.1366 0.00009 0.2047 Recall-Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm o ol 0.02 3 2 1 0 -1 -2 -3 .0 _________________ 0.16 0.5 0.84 0.98 0.( -3 -2 -1 0 1 2 Its - University of Pittsburgh Summary Statistics Run number upitt3~disk2 only, automatic 1 Num4ueries 50 Total number of documents over all queries Retrieved 5383 0.8 Relevant 5923 Rel~et 1249 0.6 Precision `call Level Averages Document Level Averages [1 Precision ____________ Recall Precision 0.4 0.5622 at 5 docs 0.0192 0.3280 0.2723 at 15 docs 0.0509 0.2867 0.2 0.2025 at 30 docs 0.0849 0.2533 0.1382 at 100 docs 0.1753 0.1844 0.0708 at 200 docs 0.2206 0.1249 0 0.0309 ROC Averages 0.0279 False Alarm Detection 0.0155 0.00000 0.0303 0.0136 0.00001 0.0426 0.0000 0.00002 0.0582 0.0000 0.00003 0.0675 ;~ecision for all points 0.00004 0.0745 ~vg 0.1213 0.00005 0.0819 ,`e precision for 3 0.00006 0.0881 ~diate points 0.00007 0.0912 ).50, 0.80) 0.00008 0.0946 vg 0.0824 0.00009 0.1017 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 3 2 1 0 -1 0.5 0.84 0.98 0.9~. ½½' -2 iw-. -3II -3 -2 -1 0 2 3 Ilts - Cornell University L Summary Statistics Run number crnla2-full, automatic 1 Num~queries 50 Total number of documents over all queries 0.8 Retrieved 10000 Relevant 16400 Rel~ret 3345 0.6 - ecall Level Averages Precision 0 0.7367 0 0.4506 0 0.2803 0 0.1945 0 0.1102 0 0.0714 0 0.0357 0 0.0229 0 0.0095 0 0.0000 0 0.0000 L~ecision for all points 1 Avg 0.1738 tge precision for 3 nediate points I 0.50, 0.80) Avg 0.1204 Precision Document Level Averages 0.4 Recall Precision at 5 docs 0.0123 0.5120 at 15 docs 0.0337 0.4773 0.2 at 30 docs 0.0613 0.4400 at 100 docs 0.1661 0.3972 0 at 200 docs 0.2572 0.3345 Recall-Precision Curve 0 0.2 0.4 0.6 ROC Averages Recall False Alarm Detection 0.00000 0.0624 0.00001 0.0945 0.00002 0.1143 0.00003 0.1340 0.00004 0.1498 0.00005 0.1608 0.00006 0.1737 0.00007 0.1854 0.00008 0.1940 0.00009 0.2025 NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 2 1 0 -1 -2 0.16 0.5 0.84 0.98 0. 7 -3 I I I .3 -2 -1 0 2 ults - Cornell University Summary Statistics Run number crnlp2-full, automatic 1 Num~queries 50 Total number of documents over all queries Retrieved 10000 0.8 Relevant 16400 Rel~ret 3793 0.6 Lecall Level Averages all Precision 0 0.7803 0 0.5389 `0 0.3507 0 0.2134 :0 0.1366 0 0.0917 0 0.0724 0.0229 `0 0.0187 0 0.0098 0 0.0000 ~e precision for all points Avg 0.2032 ~e precision for 3 riediate points 0.50, 0.80) ~vg 0.1537 Precision Document Level Averages Recall Precision 0.4 at 5 docs 0.0145 0.5560 at 15 docs 0.0412 0.5400 0.2 at 30 docs 0.0764 0.5187 at 100 docs 0.1931 0.4414 at 200 docs 0.3001 0.3793 0 Recall-Precision Curve 0 0.2 0.4 0.6 0. ROC Averages Recall False Alarm Detection 0.00000 0.0759 NORMAL DEVIATE - Detection-False Alarm 0.00001 0.1047 0.001 0.02 0.16 0.5 0.84 0.98 0. 0.00002 0.1335 3 0.00003 0.1528 0.00004 0.1727 0.00005 0.1893 ½½' 0.00006 0.2051 0.00007 0.2244 0.00008 0.2362 0 0.00009 0.2483 -1 -2 -3 ~It -3 -2 -1 0 2 Lilts University of California, Berkeley Summary Statistics Run number Brkly2-full, automatic I Num~queries 50 Total number of documents over all queries Retrieved 10000 Relevant 16400 Rel~ret 3394 ecall Level Averages Precision 0 0.7093 0 0.4362 0 0.2583 0 0.1333 0 0.0754 0 0.0408 0 0.0045 0 0.0000 0 0.0000 0 0.0000 0 0.0000 ge precision for all points Avg 0.1507 ] `ge precision for 3 ~ediate points ½\ 0.50, 0.80) ~vg 0.0997 0.8 - 0.6 Precision 0.4 0.2 0 Document Level Averages Recall Precision at 5 docs 0.0091 0.4400 at 15 docs 0.0288 0.4573 at 30 docs 0.0545 0.4533 at 100 docs 0.1592 0.4080 at 200 docs 0.2550 0.3394 ROC Averages False Alarm Detection 0.00000 0.0403 0.00001 0.0664 0.00002 0.0894 0.00003 0.1064 0.00004 0.1281 0.00005 0.1418 0.00006 0.1568 0.00007 0.1717 0.00008 0.1824 0.00009 0.1938 Recall-Precision Curve A 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 0.5 0.84 0.98 0. 3 2 1 0 -1 -2 -3 7 -~ -3 -2 -1 0 2 ;ults - Universitaet Dortmnnd Summary Statistics J Run number fuhra2-full, automatic 1 Num~queries 50 1 Total number of documents over all queries Retrieved 10000 0.8 Relevant 16400 Rel~ret 3553 0.6 ~call Level Averages Document Level Averages Precision all Precision Recall Precision 0.4 0.7966 at 5 docs 0.0138 0.5800 0 0.4968 at 15 docs 0.0363 0.5173 0.2 0.3204 at 30 docs 0.0675 0.5127 0.1961 at 100 docs 0.1697 0.4232 0.1302 at 200 docs 0.2666 0.3553 0 0.0864 ROC Averages 0.0527 False Alarm Detection 0 0.0376 0.00000 0.0718 0 0.0128 0.00001 0.1040 0 0.0072 0.00002 0.1295 0 0.0000 0.00003 0.1460 Lge precision for all points 0.00004 0.1628 Avg 0.1943 I 0.00005 0.1768 ~e precision for 3 0.00006 0.1895 riediate points 0.00007 0.2027 0.50, 0.80) 0.00008 0.2114 ~vg 0.1399 0.00009 0.2214 Recall-Precision Curve 0 0.2 0.4 0.6 0. Recall 5 NORMAL DEVIATE - Detection-False Alarm Ol 0.02 0.16 ( 3 2 1 0 -1 -2 0.84 0.98 0. ½<½ -3 __________________ I I -3 -2 -1 0 2 Its - Universitaet Dortmund Summary Statistics Run number fuhrp2-fulI, automatic Num~queries 50 ] 1 1 Total number of documents over all queries 0.8 Retrieved 10000 Relevant 16400 Rel~ret 3613 0.6 - Recall- Precision Curve call Level Averages Precision 0.7236 0.5083 0.3641 0.2350 0.1722 0.1116 0.0508 0.0479 0.0234 0.0228 0.0000 ;e precision for a~ll points ~vg 0.2054 I e precision for 3 ediate points ).50, 0.80) 0.1664 Precision Document Level Averages 0.4 Recall Precision at 5 docs 0.0142 0.5440 at 15 docs 0.0393 0.5293 0.2 at 30 docs 0.0740 0.5140 at 100 docs 0.1950 0.4468 at 200 docs 0.2829 0.3613 0 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0841 0.00001 0.1194 0.00002 0.1477 0.00003 0.1678 0.00004 0.1865 0.00005 0.2027 0.00006 0.2140 0.00007 0.2254 0.00008 0.2346 0.00009 0.2439 NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 3 2 1 0 0.5 0.84 0.98 0.9'. ½½' I I -1 -2 -3 I I I I -3 -2 -1 0 2 3 ults - University of Illinois at Chicago Summary Statistics J Run number danyul-full, automatic Recall-Precision Curve Num~queries 50 1 Total number of documents over all queries 1 Retrieved 95881 0.8 Relevant 16400 ReLret 1514 0.6 ecall Level Averages Lll Precision [) 0.4710 [) 0.1652 [) 0.0497 [) 0.0059 [) 0.0000 0.0000 [) 0.0000 ) 0.0000 ) 0.0000 ) 0.0000 ) 0.0000 ~recision for all poi ntsl ~vg 0.0629 ] ~e precision for 3 ediate points 0.50, 0.80) ~vg 0.0166 Document Level Averages Precision Recall Precision 0.4 at 5 docs 0.0065 0.2760 at 15 docs 0.0138 0.2253 0.2 at 30 docs 0.0259 0.2347 at 100 docs 0.0578 0.1818 A A A at 200 docs 0.0961 0.1514 0 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0191 0.00001 0.0297 0.00002 0.0350 0.00003 0.0386 0.00004 0.0423 0.00005 0.0458 0.00006 0.0493 0.00007 0.0525 0.00008 0.0563 0.00009 0.0621. NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 0.16 0.5 0.84 0.98 0.9 2 1 7 0 ½ -1 -2 -3 W[ I I -3 -2' -1 0 1 2 3 ilts - Belicore Summary Statistics Run number IsiacOl-full, automatic Num~queries 50 Total number of documents over all queries Retrieved 9995 Relevant 16400 { Rel~ret 2786 ~call Level Averages Lil Precision 0 0.6864 0 0.3808 0 0.1926 0 0.0924 0 0.0391 0 0.0104 0 0.0000 0 0.0000 0 0.0000 0 0.0000 `0 0.0000 ,~~ecision for all points Avg 0.1274 ~e precision for 3 riediate points 0.50, 0.80) Avg 0.0677 1 Document Level Averages 1 0.8 - 0.6 Precision 0.4 Recall Precision at 5 docs 0.0096 0.4640 at 15 docs 0.0243 0.4107 0.2 at 30 docs 0.0458 0.3967 at 100 docs 0.1247 0.3446 0 at 200 docs 0.1983 0.2786 Recall- Precision Curve 0 0.2 0.4 0.6 ROC Averages Recall False Alarm Detection 0.00000 0.0393 0.00001 0.0559 0 3 0.00002 0.0729 0.00003 0.0867 2 0.00004 0.0988 0.00005 0.1092 1 0.00006 0.1196 0.00007 0.1283. 0.00008 0.1357 0 0.00009 0.1442 -1 2 .3 NORMAL DEVIATE - Detection-False Alarm .001 0.02 0.16 0.5 0.84 0.98 0. I I I -3 -2 -1 0 1 2 ;u'ts - Beilcore [ Summ&y Statistics Run number IsiazO 1-full, automatic Recall-Precision Curve Num~queries 50 1 Total number of documents over all queries Retrieved 9993 0.8 - Relevant 16400 Rel~ret 2469 0.6 ~call Level Averages all Precision 0 0.6448 0 0.3367 .0 0.1510 0 0.0524 :0 0.0248 0 0.0000 0 0.0000 `0 0.0000 .0 0.0000 0 0.0000 `0 0.0000 ge precision for all points Avg 0.1100 j ge precision for 3 ~ediate points 0.50, 0.80) ~vg 0.0503 Document Level Averages Precision Recall Precision 0.4 at 5 docs 0.0085 0.4320 at 15 docs 0.0229 0.3840 0.2 at 30 docs 0.0391 0.3473 at 100 docs 0.1063 0.3010 at 200 docs 0.1745 0.2469 0 A A 0 0.2 0.4 0.6 0. ROC Averages False Alarm Detection 0.00000 0.0311 0.00001 0.0438 0.00002 0.0581 0.00003 0.0683 0.00004 0.0809 0.00005 0.0908 0.00006 0.0983 0.00007 0.1055 0.00008 0.1139 0.00009 0.1242 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 2 1 0 -1 -2 0.16 0.5 0.84 0.98 0. 7 ½ -3 I I I -3 -2 -1 0 1 2 Lts - Queens College, CUNY Summary Statistics Run number pircs1~ategory B, automatic 1 1 N um~queries 23 Total number of documents over all queries 0.8 Retrieved 4370 Relevant 3238 Rel~ret 1237 0.6 Precision call Level Averages Document Level Averages 0.4 1 Precision Recall Precision 0.7481 at 5 docs 0.0321 0.4522 0.5259 at 15 docs 0.0938 0.4928 0.2 0.4328 at 30 docs 0.1649 0.4507 0.3071 at 100 docs 0.3595 0.3570 0.2707 at 200 docs 0.5018 0.2689 0 0.2173 ROC Averages 0.1333 False Alarm Detection 0.0105 0.00000 0.0454 0.0000 0.00001 0.0505 0.0000 0.00002 0.1007 0.0000 0.00003 0.1111 e~ecision for all points 1 0.00004 0.1418 Lvg 0.2405 J 0.00005 0.1588 e precision for 3 0.00006 0.1771 Ajate points 0.00007 0.1890 0.00008 0.1797 0.00009 0.2151 1 .50, 0.80) vg 0.2167 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 0.16 0.5 0.84 0.98 0.99 2 ½½' 1 0 -1 -2 w--4, -3 I I I I -3 -2 -1 0 2 3 Lits - Queens College, CUNY L Summary Statistics 1 F Run number pircs2~ategory B, manual Num~queries 25 Total number of documents over all queries Retrieved 5000 0.8~ Relevant 3318 Rel~ret 1414 0.6 Recall-Precision Curve call Level Averages 11 Precision 0.7998 0.6742 0.5626 0.4322 I 0.3106 I 0.2783 0.2037 0.1580 0.0681 0.0396 p 0.0133 e precision for all points ~vg 0.3219 ;e precision for 3 ~diate points ).50, 0.80) vg 0.3030 Document Level Averages Precision ____________ Recall Precision 0.4 at 5 docs 0.0665 0.6320 at 15 docs 0.1340 0.5307 0.2 - at 30 docs 0.2156 0.4787 at 100 docs 0.4245 0.3680 at 200 docs 0.5876 0.2828 0 ROC Averages Recall False Alarm Detection 0.00000 0.1084 0.00001 0.1273 0.00002 0.1565 0.00003 0.1829 0.00004 0.2107 0.00005 0.2313 0.00006 0.2436 0.00007 0.2574 0.00008 0.2840 0.00009 0.2969 0 0.2 0.4 0.6 0.8 NORMAL DEVIATE - Detection-False Alarm o ol 0.02 3 2 1 0 .1 -2 -3 0.16 C 5 0.84 0.98 0.9 .0 . I I ________I -3 -2 -1 0 1 2 3 `its - Queens College, CUNY L Summ&y Statistics1 Recall-Precision Curve Run number pircs3~ategory B, 1 Num4ueries feedbac2k5 Total number of documents over all queries 0.8< Retrieved 5230 1 Relevant 3318 Rel~ret 1526 0.6 Precision Document Level Averages 0.4 ecall Level Averages Recall Precision Lll Precisi6n at 5 docs 0.0725 0.6400 0.7970 at 15 docs 0.1325 0.5467 0.2 [) 0.6978 at 30 docs 0.2090 0.4920 [) 0.5873 at 100 docs 0.4559 0.3920 I- 0 D 0.4398 at 200 docs 0.6163 0.2970 0 0.2 0.4 0.6 () 0.3595 ROC Averages Recall 0 0.2982 False Alarm Detection 0 0.2460 NORMAL DEVIATE - Detection-False Alarm 0.00000 0.1409 0 0.1700 0. 001 0.02 0.16 0.5 0.84 0.98 0.1 0.00002 0.1911 3 0 0.0898 0.00004 0.2198 0 0.0489 0.00006 0.2751 0 0.0136 0.00008 0.2806 .~ecision for all points 1 0.00010 0.2931 Avg 0.3407 0.00012 0.3500 ge precision for 3 0.00014 0.3654 ~diate points 0.00016 0.3668 0.50, 0.80) 0.00018 0.4119 ~vg 0.3251 0.00020 0.3941 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 iults - Queens College, CUNY Summary Statistics ] Run number pircs4-category B, ~Num~queries feedbac2k5 1 Total number of documents over all queries Retrieved 5230 0.8 Relevant 3318 Rel~et 1582 0.6 Document Level Averages Precision ecall Level Averages Recall Precision 0.4 Precision at 5 docs 0.0695 0.6240 0 0.8242 at 15 docs 0.1389 0.5707 () 0.6935 at 30 docs 0.2272 0.5187 0.2 () 0.6105 at 100 docs 0.4774 0.4152 [) 0.4726 at 200 docs 0.6339 0.3074 0 [) 0.3672 [) 0.3418 ROC Averages ) 0.2869 False Alarm Detection ) 0.1983 0.00000 0.1608 ) 0.1206 0.00002 0.2035 ) 0.0609 0.00004 0.2576 ____ 0.0207 0.00006 0.2715 0.00008 0.3202 ~ precision for all pointsl 0.00010 0.3380 ~vg 0.3634 ] 0.00012 0.3645 ~e precision for 3 0.00014 0.3857 ediate points 0.00016 0.3791 0.00018 0.4337 1 0.00020 0.4211 ).50, 0.80) .vg 0.3576 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 ( 5 0.84 0.98 0.9 3 2 1 0 .1 -2 -3 7£ I I I -3 -2 -1 0 2 3 ilts - New York University I Summ&y Statistics Run number nyuir1~ategory B, 1 Num~quenes automat2i5c I Total number of documents over all queries Retrieved 4986 0.8 Relevant 3318 Rel~et 1161 0.6 ~all Level Averages .11 Precision 0.7830 0.6216 0.4891 0.3815 0.2350 0.1871 0.1447 p 0.0705 0.0229 0.0000 ____ 0.0000 ~recision for all points ~vg 0.2669 ~ precision for 3 ediate points ~.50, 0.80) 0.2330 Document Level Averages Precision Recall Precision 0.4 at 5 docs 0.0713 0.5680 at 15 docs 0.1326 0.5200 0.2 at 30 docs 0.1868 0.4320 at 100 docs 0.3350 0.3140 at 200 docs 0.4294 0.2322 0 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.1363 NORMAL DEVIATE - Detection-False Alarm 0.00001 0.1770 0.001 0.02 0.16 ( 5 0.84 0.98 0 C 0.00002 0.1698 3 0.00003 0.2026 ½<½ 0.00004 0.1911 2 0.00005 0.2292 0.00006 0.2109 1 0.00007 0.2149 0.00008 0.2491 0 0.00009 0.2578 -1 -2 -3 W½ I I -3 .2 -1 0 2 3 Lilts - New York University Summary Statistics Run number nyuir2-category B, automatic Num~queries 25 1~'otai- number of documents over all queries Retrieved 4990 Relevant 3318 Rel~et 958 ~all Level Averages Precision 0.7168 0.4858 0.3722 0.2032 p 0.1382 p 0.0599 p 0.0419 0.0058 0.0030 0.0000 0.0000 eprecision for all points Lvg 0.1843 e precision for 3 ~iate points .50, 0.80) vg 0.1450 Document Level Averages ____________ Recall Precision at 5 docs 0.0444 0.5200 at 15 docs 0.0929 0.4267 at 30 docs 0.1424 0.3627 at 100 docs 0.2550 0.2684 at 200 docs 0.3466 0.1916 ROC Averages False Alarm Detection 0.00000 0.0754 0.00001 0.0983 0.00002 0.1269 0.00003 0.1227 0.00004 0.1305 0.00005 0.1531 0.00006 0.1639 0.00007 0.1668 0.00008 0.1717 0.00009 0.1549 Recall-Precision Curve 1 0.8 0.6 - Precision 0.4 0.2 0 A 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 3 2 1 0 -i -1 -2 ( 5 0.84 0.98 0 C ½<½ -3 ½~ -3 -2 -1 0 2 3 ilts New York University Summary Statistics Run number nyuir3~ategory B, automatic 1 Num~queries 25 Total number of documents over all queries Retrieved 4989 0.8 Relevant 3318 Rel~et 1039 0.6 Precision ecall Level Averages Document Level Averages dl Precision Recall Precision 0.4 ) 0.7368 at 5 docs 0.0571 0.5360 ) 0.5592 at 15 docs 0.1297 0.4907 0.2 ) 0.4660 at 30 docs 0.1734 0.4147 ) 0.3235 at 100 docs 0.2918 0.2916 ) 0.2099 at 200 docs 0.3908 0.2078 0 ) 0.1385 ROC Averages [) 0.0801 False Alarm Detection [) 0.0560 0.00000 0.1289 0.0056 0.00001 0.1575 [) 0.0000 0.00002 0.1761 [) 0.0000 0.00003 0.1594 ~e precision for all points ~ 0.00004 0.1880 Avg 0.23411 0.00005 0.1966 ~e precision for 3 0.00006 0.2254 iediate points 0.00007 0.2033 0.50, 0.80) 0.00008 0.2169 ~vg 0.2034 0.00009 0.2424 Recall-Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 ( 3 2 1 0 -1 ½½' 5 0.84 0.98 0.~ -2 -~ -3 I I I I -3 -2 -1 0 2 ~ults - OCLC Online Computer Library Center, Inc. L L Summary Statistics number Num~queries50 Total number of documents over all queries Retrieved 10000 Relevant 2172 Rel~et 1274 Lecall Level Averages all Precision 0 0.5744 0 0.4535 0 0.3841 0 0.3443 0 0.2679 0 0.2202 () 0.1561 [) 0.1083 [) 0.0623 [) 0.0231 ) 0.0114 1,e precision for all points ~vg 0.2369 v,e precision for 3 ediate points [).50, 0.80) vg 0.2222 Document Level Averages Recall Precision at 5 docs 0.0726 0.3600 at 15 docs 0.1514 0.3093 at 30 docs 0.2541 0.2753 at 100 docs 0.4897 0.2018 at 200 docs 0.5930 0.1274 ROC Averages False Alarm Detection 0.00000 0.0479 0.00001 0.0538 0.00002 0.0907 0.00003 0.0909 0.00004 0.1104 0.00005 0.1161 0.00006 0.1313 0.00007 0.1315 0.00008 0.1327 0.00009 0.1419 Recall-Precision Curve 1 0.8 - 0.6 Precision 0.4 0.2 - I I 0 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 2 0 -1 0.16 0.5 0.84 0.98 0.9 ½<½ -2 ~_ -3I -3 -2 -1 0 2 3 Lits - CITRI, Royal Melbourne Institute of Technology [ Summ&y Statistics 1 Run number citri2-full, automatic 1 Num~queries 50 Total number of documents over all queries Retrieved 10000 Relevant 16400 Rel~ret 2348 ~all Level Averages 11 Precision F- 0.6496 0.3515 0.1189 0.0460 0.0274 ) 0.0081 ) 0.0075 ) 0.0000 ) 0.0000 ) 0.0000 ) 0.0000 ~e precision for all points~ ~vg 0.1099 ] ~ precision for 3 iediate points 0.50, 0.80) ~vg 0.0423 Document Level Averages Recall Precision at 5 docs 0.0102 0.4440 at 15 docs 0.0280 0.4293 at 30 docs 0.0483 0.3900 at 100 docs 0.1165 0.3198 at 200 docs 0.1668 0.2348 ROC Averages False Alarm Detection 0.00000 0.0404 0.00001 0.0559 0.00002 0.0686 0.00003 0.0810 0.00004 0.0899 0.00005 0.0979 0.00006 0.1056 0.00007 0.1137 0.00008 0.1192 0.00009 0.1239 0.8 - 0.6 Precision 0.4 0.2 0 Recall- Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.16 0 0.84 0.98 0. 0 t I I I I 0. Ol 0.02 3 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 ~ults - CITRI, Royal Melbourne Institute of Technology L t Summary Statistics Run number citri3-full, automatic Recall-Precision Curve Num~queries 50 1 Total number of documents over all queries Retrieved 10000 Relevant 16400 Rel~et 2011 ~call Level Averages Document Level Averages Recall Precision at 5 docs 0.0109 0.5000 at 15 docs 0.0289 0.4453 at 30 docs 0.0553 0.4320 at 100 docs 0.1162 0.3080 at 200 docs 0.1431 0.2011 all Precision 0 0.7436 0 0.2834 0 0.1325 0 0.0742 () 0.0192 () 0.0147 [) 0.0120 [) 0.0000 ) 0.0000 ) 0.0000 ) 0.0000 ;e precision for all points ~vg 0.1163 r,eprecisionfor3l ediate points ).50, 0.80) I vg 0.0491 0.8 - 0.6 Precision 0.4 0.2 0 0 0.2 0.4 0.6 ROC Averages False Alarm Detection Recall 0.00000 0.0480 NORMAL DEVIATE - Detection-False Alarm 0.00001 0.0675 0.00002 0.0824 0 `C Ol 0.02 0.16 0.5 0.84 0.98 0.~ 3 0.00003 0.0931 0.00004 0.1019 0.00005 0.1083 0.00006 0.1140 0.00007 0.1200 0.00008 0.1241 0.00009 0.1269 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 3 Its - Carnegie Mellon University L Summary Statistics Run number clarta-full, manual 1 1 Num~queries 50 Total number of documents over all queries 1 0.8 Retrieved 10000 1 Relevant 16400 Rel~ret 2110 J 0.6 Precision ~all Level Averages Document Level Averages 11 Precision Recall Precision 0.4 0.6138 at 5 docs 0.0084 0.3920 p 0.2634 at 15 docs 0.0219 0.3520 0.2 p 0.1445 at 30 docs 0.0420 0.3467 0.0810 at 100 docs 0.1051 0.2732 0.0479 at 200 does 0.1570 0.2110 0 0.0152 ROC Averages 0.0052 False Alarm Detection 0.0000 0.00000 0.0316 0.0000 0.00001 0.0460 0.0000 0.00002 0.0619 ____ 0.0000 0.00003 0.0736 r,~ecision for all points I 0.00004 0.0810 ~vg 0.1065 0.00005 0.0901 `,e precision for 3 0.00006 0.0975 ediate points 0.00007 0.1068 0.50, 0.80) 0.00008 0.1133 Lvg 0.0533 0.00009 0.1201 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm U I I 0. Ol 0.02 3 2 1 0 -1 -2 -3 0.16 0.5 0.84 0.98 0.9 -3 -2 -1 0 2 *~ Lilts - Carnegie Mellon University [ Summ&y Statistics Run number clart~full, manual 1 Recall-Precision Curve Num4ueries 50 J 1 t Total number of documents over all queries I Retrieved 10000 1 0.8 Relevant 16400 Rel~ret 3409 ~all Level Averages 11 Precision 0.9010 0.5497 0.3609 0.1997 0.1304 0.0514 0.0423 0.0169 0.0000 0.0000 0.0000 precision for all points vg 0.2048 precision for 3 diate points *!50~0.80)0.1374 ] I 0.6 Document Level Averages Precision ____________ Recall Precision 0.4 at 5 docs 0.0179 0.7560 at 15 docs 0.0455 0.6533 at 30 docs 0.0804 0.6020 0.2 at 100 docs 0.1834 0.4570 at 200 docs 0.2567 0.3409 0 0 0.2 0.4 ROC Averages False Alarm Detection 0.00000 0.0976 0.00001 0.1235 0.00002 0.1471 0.00003 0.1655 0-00004 0.1772 0.00005 0.1868 0.00006 0.1964 0.00007 0.2063 0.00008 0.2136 0.00009 0.2220 Recall 0.6 0.8 NORMAL DEVIATE - Detection-False Alarm a ~. I I I I 0. ~O1 3 2 1 0 -1 -2 .3 0.02 0.16 5 0.84 0.98 0.99 ( C 0 3 -2 -1 0 2 3 Lilts - Conquest Software, Inc. Summ&y Statistics Run number CnQstl-full, automatic I Num4ueries 50 Total number of documents over all queries Retrieved 10000 Relevant 16400 Rel~ret 2779 ~call Level Averages all Precision 0.6593 LO 0.3368 0.2395 0.0984 0.0862 50 0.0720 60 0.0568 70 0.0331 80 0.0164 90 ~079 00 0.0000 `age precision for all points t Avg 0.1461 age precision for 3 mediate points [),0.50, 0.80) Avg 0.1093 Document Level Averages Recall Precision at 5 docs 0.0097 0.4160 at 15 docs 0.0276 0.3907 at 30 docs 0.0531 0.3800 at 100 docs 0.1467 0.3408 at 200 docs 0.2232 0.2779 ROC Averages False Alarm Detection 0.00000 0.0364 0.00001 0.0682 0.00002 0.0901 0.00003 0.1097 0.00004 0.1270 0.00005 0.1398 0.00006 0.1506 0.00007 0.1641 0.00008 0.1723 0.00009 0.1801 Recall~Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0. Recall 0.001 0.02 0.16 0.5 0.84 0.98 C 3 2 1 0 1 -2 NORMAL DEVIATE - Detection-False Alarm -3 3 -2 -1 0 1 2 ~u1ts Conquest Software, Inc. Summary Statistics [ Run number CnQst2-full, manual Num~queries 50 1 { Total number of documents over all queries Retrieved 10000 0.8~ Relevant 16400 Rel~ret 3700 0.6 Lecall Level Averages Document Level Averages Precision all Precision Recall Precision 0.4 0.8047 at 5 docs 0.0126 0.5520 0 0.5346 at 15 docs 0.0410 0.5653 0 0.3268 at 30 docs 0.0733 0.5267 0.2 0 0.2078 at 100 docs 0.2041 0.4730 0 0.1621 at 200 docs 0.2960 0.3700 0 0 0.1192 0 0.0812 ROC Averages 0 0.0644 False Alarm Detection 0 0.0216 0.00000 0.0787 () 0.0187 0.00001 0.1148 ____ 0.0000 0.00002 0.1500 0.00003 0.1740 ~ precision for all points 1 0.00004 0.1949 ~vg 0.2128 0.00005 0.2086 ~e precision for 3 0.00006 0.2222 ediate points 0.00007 0.2375 0.50, 0.80) 0.00008 0.2490 Lvg 0.1559 0.00009 0.2571 Recall-Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 3 2 0 -1 -2 `01 0.02 0.16 ( -- 1 5 0.84 0.98 0.~ ½<½ -3 I I I -3 -2 -1 0 2 3 Lilts - GE Research and Development Center I Summary Statistics Run number gecrdl-full, manual 1 Num~queries 50 Total number of documents over all queries Retrieved 7365 Relevant 16400 Rel~ret 3363 Precision ecall Level Averages Document Level Averages Precision Recall Precision 0.4 0 0.8682 at 5 docs 0.0164 0.6640 0 0.5258 at 15 docs 0.0413 0.6107 0.2 0 0.3154 at 30 docs 0.0725 0.5593 0 0.1562 at 100 docs 0.1813 0.4508 0 0.1196 at 200 docs 0.2452 0.3363 0 0 0.0522 ROC Averages 0 0.0395 False Alarm Detection 0 0.0112 0.00000 0.0824 0 0.0000 0.00001 0.1065 0 0.0000 0.00002 0.1352 0 0.0000 0.00003 0.1554 ,~ecision for all points 1 0.00004 0.1698 Avg 0.1898 0.00005 0.1831 ge precision for 3 0.00006 0.1933 ~diate points 0.00007 0.2002 0.50, 0.80) 0.00008 0.2089 ~vg 0.1226 0.00009 0.2166 ] 0.8 1 0.6 Recall-Precision Curve 0 0.2 0.4 0.6 0. Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 0.5 3 2 1 0 -1 0.84 0.98 0. <½' -77 -2 -3 I I I I .3 -2 -1 0 1 2 ~sults - GE Research and Development Center ] Run number gecrd2-full, manual Num~queries 50 1 [ Total number of documents over all queries Retrieved 8146 0.8 Relevant 16400 f Rel~ret 3689 0.6 Recall Level Averages Document Level Averages Precision ~all Precision Recall Precision 0.4 DO 0.8769 at 5 docs 0.0158 0.6480 10 0.5646 at 15 docs 0.0439 0.6253 20 0.3367 at 30 docs 0.0768 0.5753 0.2 30 0.1644 at 100 docs 0.1852 0.4628 0.1013 at 200 docs 0.2712 0.3689 0 0.0723 0.0495 ROC Averages 0.0041 False Alarm Detection 0 0.0000 0.00000 0.0807 0 0.0000 0.00001 0.1120 0 0.0000 0.00002 0.1383 0.00003 0.1544 ge precision for all points 0.00004 0.1698 Avg 0.1973 0.00005 0.1802 ge precision for 3 0.00006 0.1912 lediate points 0.00007 0.2024 0.50, 0.80) 0.00008 0.2168 ~vg 0.1363 0.00009 0.2249 Recall-Precision Curve 0 0.2 0.4 0.6 0. Recall NORMAL DEVIATE - Detection-Falac Alarm 0.001 0.02 0.16 0.5 0.84 0.98 0. 3 2 0 I I I -3 -2 -1 0 2 3 -1 -2 -3 Its - VPI & SU Summary Statistics Run number vpidt2-wsj, diski only, manual 1 Num~queries 50 Total number of documents over all queries 0.8 Retrieved 10000 Relevant 4056 Rel~ret 1371 0.6 - Recall-Precision Curve call Level Averages I Precision 0.5153 0.2841 0.1898 0.1285 0.0988 0.0746 0.0527 0.0338 0.0271 0.0222 0.0222 e precision for all points Lvg 0.1317 e precision for 3 diate points ~.50, 0.80) vg 0.0972 Precision Document Level Averages Recall Precision 0.4 at 5 docs 0.0476 0.3160 at 15 docs 0.0640 0.2147 0.2 - at 30 docs 0.0843 0.1760 at 100 docs 0.2050 0.1246 at 200 docs 0.4481 0.1371 0 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0391 0.00001 0.0452 0.00002 0.0529 0.00003 0.0522 0.00004 0.0595 0.00005 0.0616 0.00006 0.0607 0.00007 0.0633 0.00008 0.0677 0.00009 0.0661 NORMAL DEVIATE - Detection-False Alarm a I I I 0.01 3 2 1 0 -1 -2 -3 0.02 0.16 0.5 0.84 0.98 0.9~ -3 -2 -1 0 2 3 ults - GTE Laboratories number f Num~queries50j 1 Total number of documents over all queries Retrieved 10000 0.8 - Relevant 4056 t Rel~et 1561 0.6 ecall Level Averages Document Level Averages Precision Precision Recall Precision 0.4 0.6492 at 5 docs 0.0769 0.4040 0.4468 at 15 docs 0.1282 0.3387 0.3278 at 30 docs 0.1764 0.2887 0.2 0.2484 at 100 docs 0.3653 0.2252 0.1916 at 200 docs 0.4633 0.1561 0 0.1455 0.1159 ROC Averages 0.0963 False Alarm Detection 0.0474 0.00000 0.0472 0.0166 0.00001 0.0570 0.0055 0.00002 0.0821 0.00003 0.1110 for all points 0.00004 0.1185 vg~~83 0.00005 0.1454 precision 0.00006 0.1332 diate points 0.00007 0.1446 0.00008 0.1704 ig~0.~ 0.00009 0.1598 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 0.5 0.54 0.98 0.9~ 3 2 1 0 -1 ½<½ -2 -3 ½w< II -3 -2 -1 0 1 2 3 C C 0 its - GTE Laboratories Summary Statistics Recall-Precision Curve Run number fairs2-wsj, disk2 only, automatic 1 Num4ueries 50 Total number of documents over all queries 0.8 Retrieved 10000 Relevant 2172 Reljet1188I 0.6 Precision ~all Level Averages Document Level Averages 0.4 Precision Recall Precision 0.5538 at 5 docs 0.0592 0.3440 0.4134 at 15 docs 0.1500 0.2947 0.2 0.3582 at 30 docs 0.2355 0.2673 ) 0.3148 at 100 docs 0.4410 0.1876 I I 0 ) 0.2551 at 200 docs 0.5362 0.1188 0 0.2 0.4 0.6 0.8 ) 0.1907 ROC Averages Recall ) 0.1427 False Alarm Detection 0.1086 NORMAL DEVIATE - Detection-False Alarm 0.00000 0.0622 [) 0.0603 0. 0 Ol 0.02 0.16 0.5 0.84 0.98 0.9 0.00001 0.0758 3 () 0.0312 0.00002 0.0873 [) 0.0092 0.00003 0.0873 2 ge precision for all points 0.00004 0.0964 0.2216 0.00005 0.1198 1 ge precision for 3 0.00006 0.1350 ~ediate points 0.00007 0.1350 0 0.50, 0.80) 0.00008 0.1421 ~vg 0.2030 0.00009 0.1470 -1 -2 -3 I I 3 -2 -1 0 2 ults - PRC, Inc. Summary Statistics J Run number prceol- 1 Recall-Precision Curve Num~queries 50 j 1 Total number of documents over all queries Retrieved 100001 0.8 - Relevant 4056 Rel~ret 1666 J 0.6 ecall Level Averages Document Level Averages Precision Lll Precision Recall Precision 0.4 0.6598 at 5 docs 0.0379 0.4040 ) 0.4382 at 15 docs 0.1193 0.3653 ) 0.3451 at 30 docs 0.1714 0.3033 0.2 ) 0.2536 at 100 docs 0.3438 0.2322 0.1753 at 200 docs 0.4557 0.1666 0 0.1400 0.1016 ROC Averages 0.0489 False Alarm Detection 0.0185 0.00000 0.0388 0.0015 0.00001 0.0534 0.0015 0.00002 0.0582 0.00003 0.0690 eprecision for all points 1 0.00004 0.0753 ~vg 0.1985 0.00005 0.0943 ;e precision for 3 0.00006 0.0901 ~diate points 0.00007 0.1012 0.00008 0.1033 1 0.00009 0.1143 ~.50, 0.80) vg 0.1679 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0 Ol 0.02 0.16 3 2 1 0 -1 -2 -3 C 0.5 0.84 0.98 0 -3 -2 -1 0 1 2 3 Its - Siemens Corporate Research, Inc. [ Summ&y Statistics L Run number siemsl-full, automatic 1 1 Num~quenes 50 Total number of documents over all queries 0.8 Retrieved 9992 Relevant 16400 Rel~et 3393 0.6 Precision ~all Level Averages Document Level Averages 0.4 11 Precision Recall Precision 0.7524 at 5 docs 0.0131 0.5200 0.4670 at 15 docs 0.0344 0.4827 0.2 0.3242 at 30 docs 0.0650 0.4687 0.1973 at 100 docs 0.1687 0.4100 0 0.1179 at 200 docs 0.2564 0.3393 0.0672 ROC Averages 0.0241 False Alarm Detection 0.0000 0.00000 0.0660 ) 0.0000 0.00001 0.0959 ) 0.0000 0.00002 0.1194 ) 0.0000 0.00003 0.1369 ~e~cision for all points~ 0.00004 0.1558 ~vg 0.1773 J 0.00005 0.1675 ~ precision for 3 0.00006 0.1775 Lediate points 0.00007 0.1897 0.00008 0.1971 0.00009 0.2065 I 0.50, 0.80) ~vg 0.1305 Recall-Precision Curve A 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 0.5 0.84 0.98 0.9 3 2 1 0 -1 ½<½' -2 -~ -3 -3 -2 -1 0 2 `suits - City University, London L Summary Statistics Run number cityri-full, automatic Recall-Precision Curve Num~queries 50 1 L J 4'otal number of documents over all queries Retrieved 10000 0.8 - Relevant 15459 L Rel~ret 3286 ¼\ 0.6 ecall Level Averages Document Level Averages Precision 11 Precision Recall Precision 0.4 0.7676 at 5 docs 0.0113 0.5480 0.5016 at 15 docs 0.0352 0.5453 0.2853 0.2 at 30 docs 0.0646 0.5093 0.1646 at 100 docs 0.1653 0.4376 0.0522 at 200 docs 0.2391 0.3286 0 0.0105 p 0.0045 ROC Averages 0.0000 False Alarm Detection 0.0000 0.00000 0.0607 0.0000 0.00001 0.0857 0.0000 0.00002 0.1103 0.00003 0.1268 e precision for all points] 0.00004 0.1422 Lvg 0.1624 ] 0.00005 0.1547 e precision for 3 0.00006 0.1655 `diate points 0.00007 0.1752 0.00008 0.1834 } 0.00009 0.1911 `.50, 0.80) vg 0.0986 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 3 0 -1 5 0.84 0~98 0.9 -2 -3 W½11 -3 -2 -1 0 2 3 ults - University of Pittsburgh L Summary Statistics Run number upitt3-full, automatic 1 1 Num4ueries 50 Total number of documents over all queries 0.8 Retrieved 7611 Relevant 15459 Rel~ret 1277 0.6 call Level Averages 1 Precision 0.5285 0.1806 0.0511 0.0000 0.0000 0.0000 0.0000 p 0.0000 0.0000 0.0000 0.0000 r~ecision for all points ~vg 0.0691 ] ~ precision for 3 ediate points [).50, 0.80) Lvg 0.0170 Precision Document Level Averages 0.4 Recall Precision at 5 docs 0.0074 0.3520 at 15 docs 0.0179 0.3013 0.2 at 30 docs 0.0307 0.2727 at 100 docs 0.0737 0.2102 0 at 200 docs 0.0889 0.1277 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Marm Detection 0.00000 0.0261 0.00001 0.0331 0.00002 0.0392 0.00003 0.0436 0.00004 0.0492 0.00005 0.0540 0.00006 0.0600 0.00007 0.0635 0.00008 0.0673 0.00009 0.0710 NORMAL DEVIATE - Detection-False Alarm 0. U ol 0.02 3 2 0 -1 -2 0.16 0.5 I - ½½' 0.84 0.98 0.91 ½ 7 .3 -2 -1 0 2 ~ I t .3 ;ults - Cornell University [ Summary Statistics J [Run number crnla2-full, automatic 1 [ Num~queries 50 1 Total number of documents over all queries Retrieved 10000 0.8 Relevant 15459 Rel~et 3399 0.6 call Level Averages Document Level Averages Precision 1 Precision Recall Precision 0.4 0.7798 at 5 docs 0.0116 0.5640 0.5162 at 15 docs 0.0349 0.5333 0.3278 0.2 at 30 docs 0.0646 0.5113 0.1823 at 100 docs 0.1690 0.4452 0.0647 at 200 docs 0.2491 0.3399 0 0.0248 0.0066 ROC Averages False Alarm Detection 0.0000 0.0000 0.00000 0.0679 0.0000 0.00001 0.0901 0.0000 0.00002 0.1120 0.00003 0.1313 precision for all points 0.00004 0.1447 vg 0.1729 0.00005 0.1583 precision for 3 0.00006 0.1712 ~iate points 0.00007 0.1828 50, 0.80) 0.00008 0.1926 g 0.1175 0.00009 0.2018 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0,101 3 2 1 0 -1 -2 -3 C 9 I I I 0.84 0.98 0 ~C 0 0 0 0 0 0. 0. -3 -2 -1 0 1 2 3 0.02 0.16 0.5 ults - Universitaet Dortmund Summ&y Statistics ] Run number fuhra2-full, automatic 1 1 Num~queries 50 Total number of documents over all queries 0.8 Retrieved 10000 Relevant 15459 Rel~ret 4221 0.6 Precision ~all Level Averages Document Level Averages 0.4 11 Precision Recall Precision 0.7630 at 5 docs 0.0124 0.6040 0.6153 at 15 docs 0.0378 0.6000 0.2 0.4279 at 30 docs 0.0760 0.5947 0.3089 at 100 docs 0.2088 0.5328 0.1686 at 200 docs 0.3121 0.4221 0 0.0516 ROC Averages 0.0203 False Alarm Detection 0.0036 0.00000 0.0926 0.0000 0.00001 0.1286 ) 0.0000 0.00002 0.1573 ) 0.0000 0.00003 0.1796 ~recision for all points] 0.00004 0.2013 ~vg 0.2145 J 0.00005 0.2174 iediate points 0.00006 0.2300 ~e precision for 3 0.00007 0.2457 0.50, 0.80) 0.00008 0.2563 ~vg 0.1598 0.00009 0.2660 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm U . 9 I I 0.01 0.02 3 2 1 0 -1 -2 -3 0.16 a 0.84 0.98 0. -3 -2 -1 0 1 2 `suits - University of Illinois at Chicago Summary Statistics Run number danyul-full, automatic 1 Num~queries 50 Total number of documents over all queries Retrieved 9999 Relevant 15459 Rel~ret 1253 ecall Level Averages Precision 0.4415 0.1044 0.0154 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 ;e precision for all points 0.0510 ,~ precision for 3 ~diate points ).50, 0.80) vg 0.0051 I J Document Level Averages Recall Precision at 5 docs 0.0041 0.2480 at 15 docs 0.0111 0.2240 at 30 docs 0.0192 0.1907 at 100 docs 0.0500 0.1536 at 200 docs 0.0804 0.1253 ROC Averages False Alarm Detection 0.00000 0.0116 0.00001 0.0195 0.00002 0.0256 0.00003 0.0264 0.00004 0.0322 0.00005 0.0371 0.00006 0.0406 0.00007 0.0451 0.00008 0.0482 0.00009 0.0507 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 A 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE. Detection-False Alarm 0.001 0.02 0.16 ( 5 0.84 0.98 0.9' 3 0 -1 -2 -3 `7! I -3 -2 -1 0 2 3 Lilts - Beilcore Summary Statistics 1 Recall-Precision Curve Run number IsircOl-full, automatic 1 Num4ueries 50 j Total number of documents over all queries 1 0.8 - Retrieved j 10000 1 Relevant 15459 Rel~ret 2234 0.6 Precision call Level Averages Document Level Averages 0.4 1 Precision Recall Precision 0.6323 at 5 docs 0.0106 0.4160 0.3061 at 15 docs 0.0286 0.3893 0.2 0.1901 at 30 docs 0.0506 0.3573 0.0883 at 100 docs 0.1273 0.3024 0 0.0370 at 200 docs 0.1847 0.2234 0.0077 ROC Averages 0.0000 False Alarm Detection 0.0000 0.00000 0.0418 0.0000 0.00001 0.0513 0.0000 0.00002 0.0719 0.0000 0.00003 0.0828 e precision for all points 0.00004 0.0977 ~vg 0.1147 I 0.00005 0.1069 e precision for 3 0.00006 0.1170 ediate points 0.00007 0.1259 ).50, 0.80) 0.00008 0.1312 vg 0.0659 0.00009 0.1389 _ A- 0.8 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 0.5 0.84 0.98 0.9~ 3 I I 2 1 0 -1 -2 -3 ½½1 -3 -2 -1 0 2 3 ults - Belicore [ Summary Statistics [ Run number lsirc02-full, automatic 1 Num~queries 50 j [ Total number of documents over all queries Retrieved 10000 Relevant 15459 Rel~ret 1837 all Level Averages I Precision 0.6033 0.2134 0.1125 0.0533 0.0072 0.0039 0.0000 0.0000 0.0000 0.0000 0.0000 precision for all points vg 0.0903 precision for 3 ~iate points 50, 0.80) g 0.0388 Document Level Averages Recall Precision at 5 docs 0.0080 0.3920 at 15 docs 0.0213 0.3360 at 30 docs 0.0399 0.3067 at 100 docs 0.0910 0.2448 at 200 docs 0.1388 0.1837 ROC Averages False Alarm Detection 0.00000 0.0279 0.00001 0.0437 0.00002 0.0504 0.00003 0.0599 0.00004 0.0668 0.00005 0.0739 0.00006 0.0796 0.00007 0.0872. 0.00008 0.0919 0.00009 0.0970 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 A 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE. Detection.False Alarm 0.001 0.02 0.16 3 0.5 0.84 0.98 0 i9c 0 2 y 0 1 0 0 ½ -1 0 -2 0 .3 ~I II -3 -2 -1 0 1 2 3 0 `sults - Queens College, CUNY Summary Statistics Recall-Precision Curve Run number pircsl-category B, automatic 1 Num~queries 25 Total number of documents over all queries Retrieved g 5000 0.8 Relevant 3766 Rel~ret 1471 0.6 Precision ~call Level Averages Document Level Averages Precision Recall Precision 0.4 0 0.7918 at 5 docs 0.0475 0.6080 0 0.6256 at 15 docs 0.1448 0.5973 0.2 0 0.5378 at 30 docs 0.2280 0.5213 0 0.4373 at 100 docs 0.4434 0.3980 0 0.3648 at 200 docs 0.5764 0.2942 0 0 0.2 0.4 0.6 0. 0 0.3395 Recall ROC Averages 0 0.2814 False Alarm Detection 0 0.1627 NORMAL DEVIATB - Detection-False Alarm 0.00000 0.0974 0 0.1132 0.00001 0.1370 0.001 0.02 0.16 ( 5 0.84 0.98 0. 0 0.0804 3 0.00002 0.1767 0 0.0391 0.00003 0.1966 ge precision for all points 1 0.00004 0.2096 2 Avg 0.3430 0.00005 0.2426 y `ge precision for 3 0.00006 0.2476 1 ~ediate points 0.00007 0.2694 0.50, 0.80) 0.00008 0.2910 0 ~vg 0.3302 0.00009 0.2947 -1 -2 -3 I I I -3 -2 -1 0 2 suits - Queens College, CUNY Summary Statistics Run number pircs2-category B, 1 Num~queries manu2al] Total number of documents over all queries Retrieved 5000 0.8 Relevant 3766 Rel~ret 1539 0.6 ~call Level Averages il Precision 0.8428 0.6482 0.5581 0.4842 0.4171 0.3422 0.3079 0.2008 0.1220 0.0905 0.0404 ~r~ecision for all points ~vg 0.3686 r,e precision for 3 ediate points ).50, 0.80) .vg 0.3408 Document Level_Averages Precision Recall Precision 0.4 at 5 docs 0.0476 0.6400 at 15 docs 0.1408 0.6053 0.2 at 30 docs 0.2352 0.5400 at 100 docs 0.4649 0.4236 at 200 docs 0.5878 0.3078 0 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.1106 NORMAL DEVIATE - Detection-False Alarm 0.00001 0.1384 0.001 0.02 0.16 0.5 0.84 0.98 0 0.00002 0.1974 3 0.00003 0.1914 0.00004 0.2318 2 0.00005 0.2701 0.00006 0.2985 0.00007 0.3051. 0.00008 0.3175 ½ 0 0.00009 0.3184 -1 -2 -3 I I I -3 -2 -1 0 1 2 3 `suits - New York University Summary Statistics Run number nyuirl-category B, automatic 1 Num~queries 25 Total number of documents over all queries Retrieved 5000 0.8 Relevant 3766 Rel~et 834 0.6, Precision ~call Level Averages Document Level Averages all Precision Recall Precision 0.4 0 0.5691 at 5 docs 0.0338 0.3360 0 0.3618 at 15 docs 0.0671 0.2960 0.2 0 0.1846 at 30 docs 0.0979 0.2787 0 0.1314 at 100 docs 0.2117 0.2276 0 0.1065 at 200 docs 0.2988 0.1668 0 0 0.0481 ROC Averages 0 0.0295 False Alarm Detection 0 0.0104 0.00000 0.0397 0 0.0104 0.00001 0.0537 0 0.0000 0.00002 0.0666 0 0.0000 0.00003 0.0698 ge precision for all points 0.00004 0.0762 Avg 0.1320 0.00005 0.0801 ge precision for 3 0.00006 0.0872 riediate points 0.00007 0.0899 0.50, 0.80) 0.00008 0.0963 ~vg 0.0811 0.00009 0.1008 Recall-Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 0.16 3 2 1 0 I I -1 -2 C 0.84 0.98 0. y -3 I I I -3 -2 -1 0 1 2 suits University of Central Florida I [ Summary Statistics Run number UCFQA1-category B, automatic Recall-Precision Curve Num~queries 25 1 Total number of documents over all queries Retrieved 5000 0.8 Relevant 3766 Rel~et 704 0.6 ~all Level Averages 11 Precision 0.5248 0.2190 0.1293 0.0689 p 0.0389 p 0.0069 0.0000 p 0.0000 I 0.0000 0.0000 p 0.0000 ;e precision for all points] ~vg 0.0898 ,e precision for 3 diate points ~.50, 0.80) vg 0.0454 Document Level Averages Precision Recall Precision 0.4 at 5 docs 0.0116 0.2560 at 15 docs 0.0328 0.2080 0.2 at 30 docs 0.0559 0.1933 at 100 docs 0.1661 0.1860 A A at 200 docs 0.2446 0.1408 0 0 0.2 0.4 0.6 0.8 ROC Averages False Alarm Detection 0.00000 0.0138 0.00001 0.0223 0.00002 0.0261 0.00003 0.0282 0.00004 0.0324 0.00005 0.0367 0.00006 0.0453 0.00007 0.0433. 0.00008 0.0500 0.00009 0.0518 Recall NORMAL DEVIA~E Detection-False Alarm 0 p01 3 2 1 0 -1 -2 -3 .0 _________________ 0.02 0.16 C 5 0.84 0.98 0. -3 -2 -1 0 2 3 suits - Advanced Decision Systems I [ Summary Statistics Run number adsba1~ategory B, automatic 1 Num4ueries 25 Total number of documents over all queries 0.8 Retrieved 4698 Relevant 3766 Rel~et 572 0.6 Precision ecall Level Averages Document Level Averages 0.4 Lll Precision Recall Precision 0.3889 at 5 docs 0.0118 0.1680 () 0.1508 at 15 docs 0.0220 0.1467 0.2 () 0.1210 at 30 docs 0.0442 0.1493 0 0.0957 at 100 docs 0.1137 0.1244 0 0.0623 at 200 docs 0.1771 0.1144 0 0 0.0307 ROC Averages 0 0.0301 False Alarm Detection 0 0.0051 0.00000 0.0268 0 0.0000 0.00001 0.0329 0 0.0000 0.00002 0.0225 0 0.0000 0.00003 0.0459 ge precision for all points ] 0.00004 0.0306 Avg 0.0804 ] 0.00005 0.0324 ge precision for 3 0.00006 0.0499 ~ediate points 0.00007 0.0517 0.00008 0.0545 0.00009 0.0561 } 0.50, 0.80) ~vg 0.0506 Recall-Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE Detection-False Alarm 0.001 0.02 3 0.16 0.5 0.84 0.98 0.~ 2 ½½I 1 0 -1 -iw -2 -3 I I I I -3 -2 -1 0 2 suits - Advanced Decision Systems Summary Statistics Run number adsba2category B, automatic Num~queries 25 `total number of documents over all queries R~eved4712 Relevant 3766 Rel~et 559 `call Level Averages Precision 0.3378 0.1223 0.0661 0.0358 0.0338 0.0186 0.0116 0.0116 0.0000 0.0000 0.0000 precision for all points vg 0.0580 I precision for 3 diate points .50, 0.80) 0.0282 J ] 1 Document Level Averages _____________ Recall Precision at 5 docs 0.0075 0.1200 at 15 docs 0.0174 0.1173 at 30 docs 0.0367 0.1253 at 100 docs 0.0931 0.1084 at 200 docs 0.1778 0.1118 ROC Averages False Alarm Detection 0.00000 0.0086 0.00001 0.0102 0.00002 0.0174 0.00003 0.0181 0.00004 0.0238 0.00005 0.0258 0.00006 0.0277 0.00007 0.0193 0.00008 0.0261 0.00009 0.0326 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 A 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm 0. ~1 0.02 0.16 3 2 1 0 C C 0.5 0.84 -1 0.98 0 ~9 -2 C -3 C -3 -2 -1 0 1 2 3 hsults - Australian Computing and Communications Institute Summary Statistics Recall-Precision Curve Run number manual 1 50 Total number 0.8 - Retrieved 8531 Relevant 15459 IRel~ret3l68 0.6 ~call Level Averages all Precision 0.6690 LO 0.4610 0.2867 0.1662 0.0839 0.0179 0.0164 0.0000 0.0000 0.0000 [)0 0.0000 ~ge precision for all points Avg 0.1547 J age precision for 3 rnediate points 1 0.50, 0.80) Avg 0.1015 Document Level Averages Recall p~recision at 5 docs 0.0093 0.4720 at 15 docs 0.0291 0.4787 at 30 docs 0.0570 0.4713 at 100 docs 0.1474 0.4084 at 200 docs 0.2184 0.3168 Precision 0.4 0.2 - 0 A 0 0.2 0.4 0.6 0. ROC Averages Recall False Alarm Detection 0.00000 0.0557 0.00001 0.0844 0.00002 0.1084 0.00003 0.1237 0.00004 0.1388 0.00005 0.1495 0.00006 0.1599 0.00007 0.1687 0.00008 0.1768 0.00009 0.1844 NORMAL DEVIATE - Detection-False Alarm I I 0.5 0.84 0.98 0 0.001 0.02 3 0.16 y 2 1 ½ 0 -1 -2 -w .3 I __________________ -3 -2 -1 0 2 ~sults - Australian Computing and Communications Institute Summary Statistics ] Run number cpgcn2-full, automatic 1 1 Num~queries 50 Total number of documents over all queries Retrieved 9468 0.8 - Relevant 15459 Rel~ret 2907 0.6 Precision ~call Level Averages Document Level Averages all Precision _____________ Recall Precision 0.4 0 0.6706 at 5 docs 0.0125 0.4600 0 0.4330 at 15 docs 0.0330 0.4680 0.2 0 0.2312 at 30 docs 0.0537 0.4260 0 0.1320 at 100 docs 0.1389 0.3750 0 0.0371 at 200 docs 0.2107 0.2907 0 0 0.0242 ROC Averages 0 0.0237 False Alarm Detection 0 0.0000 0.00000 0.0660 0 0.0000 0.00001 0.0808 `0 0.0000 0.00002 0.0964 0 0.0000 0.00003 0.1080 ge precision for all points 1 0.00004 0.1176 Avg 0.1411 0.00005 0.1273 ~e precision for 3 0.00006 0.1347 riediate points 0.00007 0.1438 0.50, 0.80) 0.00008 0.1512 ~vg 0.0852 0.00009 0.1602 Recall- Precision Curve 0 0.2 0.4 0.6 0. Recall NORMAL DEVIATE - Detection-False Alarm 0 C `01 0.02 0.16 0.5 0.84 0.98 0. 3 0 -1 -2 -3 __________________ I I -3 -2 -1 0 1 2 `ults - Carnegie Mellon University Summary Statistics Run number clarta-full, manual 1 Num~queries 50 Total number of documents over all queries 0.8 Retrieved I 10000 Relevant 15459 Rel~ret 2139 0.6 Precision ~all Level Averages Document Level_Averages 11 Precision Recall Precision 0.4 0.4736 at 5 docs 0.0059 0.2560 p 0.2712 at 15 docs 0.0205 0.3120 0.2 0.1033 at 30 docs 0.0387 0.3173 0.0394 at 100 docs 0.1002 0.2756 0.0170 at 200 docs 0.1499 0.2139 0 0.0000 ROC Averages 0.0000 False Alarm Detection 0.0000 0.00000 0.0267 0.0000 0.00001 0.0391 p 0.0000 0.00002 0.0531 0.0000 0.00003 0.0620 precision for all points 0.00004 0.0733 ~vg 0.0822 0.00005 0.0809 ~,e precision for 3 0.00006 0.0883 ediate points 0.00007 0.0969 0.50, 0.80) 0.00008 0.1036 ~vg 0.0344 0.00009 0.1095 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE- Detection-False Alarm 0. ~1 0.02 3 2 1 0 -1 -2 -3 0.16 ( C ). I I 5 0.84 0.98 0.~ -3 -2 -1 0 1 2 ~ults - Carnegie Mellon University Summary Statistics Run number clartb-full, manual Num~queries 50 Total number of documents over all queries Retrieved 10000 Relevant 15459 Rel~ret 3427 `call Level Averages Precision 0.7479 0.5245 0.3443 0.2125 0.0680 0.0259 0.0000 0.0000 0.0000 0.0000 0.0000 e~precision for all points vg 0.1748 e precision for 3 `diate points .50, 0.80) vg 0.1234 Document Level Averages Recall Precision at 5 docs 0.0110 0.5120 at 15 docs 0.0360 0.5320 at 30 docs 0.0675 0.5140 at 100 docs 0.1778 0.4560 at 200 docs 0.2521 0.3427 ROC Averages False Alarm Detection 0.00000 0.0717 0.00001 0.0990 0.00002 0.1251 0.00003 0.1450 0.00004 0.1617 0.00005 0.1747 0.00006 0.1863 0.00007 0.1981 0.00008 0.2052 0.00009 0.2132 Recall- Precision Curve 1 0.8 0.6 - Precision 0.4 0.2 I I A A 0 0 0.2 0.4 0.6 0.8 Recall NORMAL DEVIATE - Detection-False Alarm Ol 0.02 0.16 3. 5 0.84 0.98 0 3 ½<½ 2 1 0 -1 -2 -3 1711 -3 -2 -1 0 2 3 suits - GE Research and Development Center Summary Statistics Run number gecrdl-full, manual 1 Num~queries 50 Total number of documents over all queries 0.8 Retrieved 6911 Relevant 15459 Rel~ret 3308 0.6 Precision ecall Level Averages Document Level Averages tll Precision _____________ Recall Precision 0.4 () 0.8044 at 5 docs 0.0133 0.6080 () 0.5286 at 15 docs 0.0359 0.5840 0.2 0 0.3427 at 30 docs 0.0720 0.5747 () 0.2006 at 100 docs 0.1713 0.4486 0 0.0703 at 200 docs 0.2377 0.3308 0 0 0.0275 ROC Averages 0 0.0171 False Alarm Detection 0 0.0000 0.00000 0.0674 0 0.0000 0.00001 0.1082 0 0.0000 0.00002 0.1394 0 0.0000 0.00003 0.1558 ge precision for all points 0.00004 0.1720 Avg 0.1810 0.00005 0.1859 ge precision for 3 0.00006 0.1912 ~ediate points 0.00007 0.1976 0.50, 0.80) 0.00008 0.2040 ~vg 0.1234 0.00009 0.2105 Recall-Precision Curve ¼--- A 0 0.2 0.4 0.6 Recall NORMAL DEVIATE - Detection-False Alarin 0.001 0.02 3 I I 0.16 0.5 0.84 0.98 0 2 ½½' 1 0 -1 -~ -2 -3II -3 -2 -1 0 2 ults - GE Research and Development Center I Summ&y Statistic s] Run number gecrd2-full, manual 1 Num~queries 50 ] Total number of documents over all queries Retrieved 7820 Relevant 15459 Rel~ret 3581 call Level Averages Precision 0.8637 0.5725 0.3519 0.2218 0.0869 0.0299 ) 0.0157 ) 0.0000 ) 0.0000 ) 0.0000 [) 0.0000 ~ precision for all points Avg 0.1948 ge precision for 3 lediate points 0.50, 0.80) ~vg 0.1273 Document Level Averages Recall Precision at 5 docs 0.0161 0.6880 at 15 docs 0.0417 0.6387 at 30 docs 0.0752 0.5953 at 100 docs 0.1832 0.4672 at 200 docs 0.2633 0.3581 0.8 0.6 Precision 0.4 0.2 0 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0773 0.00001 0.1087 0.00002 0.1414 0.00003 0.1629 0.00004 0.1847 0.00005 0.1966 0.00006 0.2044 0.00007 0.2129 0.00008 0.2174 0.00009 0.2262 0.001 0.02 0.16 ( 3 2- 1- 0 NORMAL DEVIATE - Detection-False Alarm I. 5 0.84 0.98 0. -1 -2 -~ -3 -3 -2 -1 0 1 2 hsults - TRW Systems and Development Division Summary Statistics Run number trwl-full, manual 1 Num~queries 50 Total number of documents over~all queries Retrieved 6830 0.8 Relevant 15459 Rel~ret 3206 0.6 Precision Lecall Level Averages Document Level Averages all Precision ____________ Recall Precision 0.4 `0 0.8281 at 5 docs 0.0174 0.6160 0 0.5248 at 15 docs 0.0388 0.5547 0.2 0 0.3157 at 30 docs 0.0720 0.5453 `0 0.1816 at 100 docs 0.1678 0.4294 :0 0.0764 at 200 docs 0.2332 0.3206 0 0 0.0474 ROC Averages i'0 0.0198 False Alarm Detection `0 0.0000 0.00000 0.0762 `0 0.0000 0.00001 0.1095 0 0.0000 0.00002 0.1364 0 0.0000 0.00003 0.1504 Lge precision for all p0 0.00004 0.1661 Avg 0.1813 ints] 0.00005 0.1783 Lge precision for 3 0.00006 0.1863 riediate points 0.00007 0.1960 0.50, 0.80) 0.00008 0.2010 Avg 0.1210 0.00009 0.2045 Recall-Precision Curve 0 0.2 0.4 0.6 0. Recall NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 2 1 0 -1 -2 .3 I I 0.16 C 0.84 0.98 0. y -w I I I I .3 -2 -1 0 2 suits - GTE Laboratories Summary Statistics Run number fairsi-category B, automatic 1 Num~queries 25 Total number of documents over all queries Retrieved 5000 0.8 Relevant 3766 Rel~ret 1124 0.6 Precision ecall Level Averages Document Level Averages Precision Recall Precision 0.4 () 0.7339 at 5 docs 0.0370 0.4560 () 0.4871 at 15 docs 0.0918 0.4480 0.2 0 0.3661 at 30 docs 0.1514 0.4200 0 0.2942 at 100 docs 0.3166 0.3136 0 0.2256 at 200 docs 0.4206 0.2248 0 0 0.1707 ROC Averages 0 0.0639 False Alarm Detection 0 0.0352 0.00000 0.0641 0 0.0208 0.00001 0.0853 0 0.0128 0.00002 0.1094 0 0.0000 0.00003 0.1166 ge precision for all points 0.00004 0.1299 Avg 0.2191 0.00005 0.1419 ge precision for 3 0.00006 0.1599 iediate points 0.00007 0.1532 0.50, 0.80) 0.00008 0.1732 ~vg 0.1858 0.00009 0.1834 Recall- Precision Curve 0 0.2 0.4 0.6 Recall NORMAL DEVIATE- Detection-False Alarm 0.001 0.02 3 0.16 I I y 2 1 ½ 0 -1 5 0.84 0.98 0.~ -w -2 -3 3 -2 -1 0 1 2 ~ ~u1ts - PARA Group Summary Statistics Recall-Precision Curve Run number parazi-full, manual 1 1 N um~queries 50 Total number of documents over all queries Retrieved 8823 0.8 Relevant 15459 Rel~ret 3503 0.6 ~all Level Averages 11 Precision p 0.7604 p 0.5288 p 0.3483 0.2336 0.0647 0.0302 0.0187 0.0000 0.0000 0.0000 ____ 0.0000 ;e precision for all points I 0.1804 ~ precision for 3 ediate points ).50, 0.80) .vg 0.1262 Precision Document Level Averages Recall Precision 0.4 at 5 docs 0.0141 0.5720 at 15 docs 0.0356 0.5467 0.2 at 30 docs 0.0693 0.5353 at 100 docs 0.1825 0.4744 I I A at 200 docs 0.2553 0.3503 0 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0799 0.00001 0.1129 0.00002 0.1358 0.00003 0.1497 0.00004 0.1670 0.00005 0.1788 0.00006 0.1928 0.00007 0.2058 0.00008 0.2131 0.00009 0.2214 NORMAL DEVIATE - Detection-False Alarm 0.001 0.02 3 2 1 0 -1 -2 0.16 0.5 0.84 0.98 0. ½<½ -3 ~I II -3 -2 -1 0 2 3