APPENDIX A This appendix contains tables of results for all the ~II~EC-2 participants. The tables in Appendix A show varions measures of the performance on the adlioc and routing tasks. The acihoc results corne first, followed by the routing results, with the tables in the same order as the presentation order of the papers. The definitions of the evaluation measures are given in the Overview, section 4, and readers unfamiliar with these measures should read that section first. Care should be taken in comparing the tables across systerns. These measures show performance only, with no measure of user or system effort. Each table contains four major boxes of statistics and three graphs. Box 1-- Summary Statistics line 1-- unique run identifier, data subset, and query construction method used Data subset flill (disks 1 and 2 for adhoc, disk 3 for routing) category B (the official subset of data, 1/4 of the data using the Wall Street Journal articles for adhoc and the San 3ose Mercury News articles for routing) Query construction method automatic manual feedback (frozen evaluation used) line 2-- Number of topics included in averages. line 3-- Total number of documents retrieved over all topics. Here, "retrieved" means having a rank less 1001. line 4-- Total number of relevant documents for all topics in the collection (whether retrieved or not). line 5-- Total number of relevant retrieved documents for this run. Box 2-- Recall Level Averages lines 1-11-- The average over all topics of the precision at each of the 11 recall points given. Note that this is interpolated precision: e.g., for a particular topic, if the precision at 0.50 recall is greater than the precision at 0.40 recall, then the precision at 0.50 recall is used for both the 0.50 and 0A0 recall levels. line 12-- The average precision as calculated in a non-interpolated manner (see section 4 of the Overview for details on this calculation). Box 3-- Document Level Averages lines 1-9-- The average recall and precision after the given number of documents have been retrieved. line 10-- the R precision. This is a new evaluation measure being tried that averages the precisions found for each topic at the document level of R, where R is the number of relevant documents for that topic. A-i Box 4-- Fallout Level Averages lihes 1-11-- The average recall (probability of detection) at a fixed fallout (probability of false al&m rate). Ten equally spaeed fallout points in the high precislon end of the curve were used, and for each topic, the highest recall value in the region surrounding that fallout point is selected. The table shows the averages of these points across all topics. Graph 1-- Recall-Precision Curve This is a plot of the data shown in Box 2. Graph 2-- Fallout-Recall Curve This is a plot of the data shown in Box 4. Graph 3-- Normal Deviate - Fallout-Recall This is a plot of the data shown in Box 4, but plotted on probability scales. A-2 ults - City University, London L Summary Statistics 1 Recall-Precision Curve Run Number cityau-full1 automatic 1 Num of Queries 50 j Total number of documents over all queries 1 0.8 - Retrieved: 50000 Relevant: 10785 Rel~ret: 6412 j 0.6 - ecall Level_Averages Recall Precision 0.00 0.7148 0.10 0.4224 0.20 0.3528 0.30 0.2983 0.40 0.2623 0.50 0.2191 0.60 0.1835 0.10 0.1299 0.80 0.0931 0.90 0.0421 1.00 0.0000 `age precision over all ant docs interpolated 0.2272 Document Level Averages Precision At 5 docs 0.5000 At 10 docs 0.4640 At 15 docs 0.4667 At 20 docs 0.4570 At 30 docs 0.4340 At 100 docs 0.3512 At 200 docs 0.2970 At 500 docs 0.1966 At 1000 docs 0.1282 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2849 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0.6 - t Level Averages NORMAL DEVIATE - Fallout-Recall Recall )~0 0.3344 0.4481 0.5113 0.5761 BO 0.6171 60 0.6438 00 0.6438 0.84 0.98 0 0.4 0.2 0.98 2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I 0.001 -3 -2 -1 0 1 2 3 ~sIiIts - City University, London Summary Statistics [ Run Number citymf-fu II, feedback Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 51221 0.8 Relevant: 10785 Rel~ret: 6409 0.6 - Recall Level Averages Recall Precision 0.00 0.7328 0.10 0.4944 0.20 0.4180 0.30 0.3552 0.40 0.2809 0.50 0.2205 0.60 0.1584 0.70 0.0901 0.80 0.0482 0.90 0.0184 1.00 0.0000 rage precision over all ~ant docs interpolated 0.2324 it Level Averages Recall ~ 0.3417 20 0.4289 ~0 0.4898 50 0.5366 30 0.5682 ~0 0.5932 ~0 0.6012 ~ 0.6012 50 0.6012 30 0.6012 )0 0.6012 Document Level Averages Precision Precision 0.4 At S docs 0.4920 At 10 docs 0.5040 At 15 docs 0.4827 0.2 At 20 docs 0.4750 At 30 docs 0.4680 0 At 100 docs 0.4004 At 200 docs 0.3162 At 500 docs 0.1979 At 1000 docs 0.1275 R-Precision (precision after R 1 docs retrieved (where R is of relevant docu- the number ments)) Exact 0.2971 NORMAL DEVIATE - Fallout-Recall i01 0.02 0.16 -3 -2 -1 0 1 2 3 2 0.98 Recall 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 suits - Rut~ers University Summary Statistics Run Number rutcombx-full, partial topics, manual 1 Num of Queries 25 Total number of documents over all queries Retrieved: 25000 Relevant: 4119 Rel~ret: 2378 ~ecall Level_Averages Recall Precision 0.00 0.1145 0.10 0.4611 0.20 0.3894 0.30 0.3325 0.40 0.2841 0.50 0.2001 0.60 0.1502 0.10 0.1057 0.80 0.0101 0.90 0.0345 1.00 0.0104 rage precision over all ~ant docs -interpolated 0.2284 it Level Averages ~t Recall DO 0.3413 20 0.4453 40 0.5131 60 0.5584 80 0.5901 DO 0.6090 20 0.6202 40 0.6202 60 0.6202 80 0.6202 DO 0.6202 Document Level Averages Precision At 5 docs 0.4120 At 10 docs 0.4600 At 15 docs 0.4640 At 20 docs 0.4520 At 30 docs 0.4280 At 100 docs 0.3396 At 200 docs 0.2432 At 500 docs 0.1489 At 1000 docs 0.0951 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2185 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.84 0.98 0 0.4 0.2 2 0.98 1 0.84 0 0.5 0 Recall-Precision Curve 1 0.8 - 0.6 - Precision 0.4 0.2 I I I I 0 1 0.8 0.6 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I 0.001 -3 -2 -1 0 1 2 3 ~sults - Rutgers University Summary Statistics Run Number rutfmed-full, partial topics, manual Recall-Precision Curve Num of Queries 25 1 Total number of documents over all queries Retrieved: 24001 0.8 Relevant: 4119 Rel~ret: 2314 Recall Level_Averages Recall Precision 0.00 0.6054 0.10 0.4215 0.20 0.3574 0.30 0.2726 0.40 0.2152 0.50 0.1459 0.60 0.1067 0.70 0.0753 0.80 0.0496 0.90 0.0204 1.00 0.0000 `rage precision over all vant docs -interpolated 0.1852 Document Level Averages Precision At 5 docs 0.3920 At 10 docs 0.3880 At 15 docs 0.3787 At 20 docs 0.3860 At 30 docs 0.3733 At 100 docs 0.2892 At 200 docs 0.2256 At 500 docs 0.1418 At 1000 docs 0.0926 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2464 0.6 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I it Level Averages ut Recall ~ 0.3238 20 0.4356 40 0.4861 60 0.5370 80 0.5625 00 0.5863 20 0.5933 40 0.5933 60 0.5933 80 0.5933 00 0.5933 NORMAL DEVIATE - Fallout-Recall 0.6 Recall )01 0.02 0.98 0 0.4 0.2 - 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ,ults - Cornell University Summary Statistics Run Number crnlL2-full, automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8224 ecall Level_Averages Recall Precision 0.00 0.7756 0.10 0.5381 0.20 0.4732 0.30 0.4321 0.40 0.3959 0.50 0.3585 0.60 0.2991 0.70 0.2357 0.80 0.1705 0.90 0.0904 1.00 0.0148 age precision over all ant docs interpolated 0.3258 Level Averages t Recall Po 0.4533 0 0.5983 *0 0.6796 10 0.7287 0 0.7634 0 0.7838 .0 0.7864 0 0.7864 10 0.7864 0 0.7864 0 0.7864 Document Level Averages Precision At 5 docs 0.5640 At 10 docs 0.5460 At 15 docs 0.5373 At 20 docs 0.5210 At 30 docs 0.4933 At 100 docs 0.4406 At 200 docs 0.3852 At 500 docs 0.2600 At 1000 docs 0.1645 R~Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3641 NORMAL DEVIATE - Fallout-Recall Ol 0.02 0.98 0 0.4 0.2 2 0.98 1 0.84 0 0.5 0 Recall-Precision Curve 1 0.8, 0.6 Precision 0.4 0.2 I I 0 1 0.8 0.6 Recall 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 suits - Cornell University Summary Statistics ] Run Number crnlV2-full1 automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8018 ~ecall Level Averages Recall Precision 0.00 0.7644 0.10 0.5209 0.20 0.4739 0.30 0.4388 0.40 0.3971 0.50 0.3544 0.60 0.2950 0.70 0.2241 0.80 0.1356 0.90 0.0661 1.00 0.0076 ~ge precision over all 1 ~nt docs nterpolated 0.3163 J Level Averages Recall D 0.4516 0.5815 0.6605 ) 0.7148 0.7514 ) 0.7675 ) 0.7713 ) 0.7713 ) 0.7713 Document Level Averages ________________ Precision At 5 docs 0.5480 At 10 docs 0.5220 At 15 docs 0.5187 At 20 docs 0.5130 At 30 docs 0.5013 At 100 docs 0.4434 At 200 docs 0.3789 At 500 docs 0.2529 At 1000 docs 0.1604 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3640 0.6 NORMAL DEVIATE - Fallout-Recall 101 0.02 0.84 0.9g 0 )~999 0.4 2 0.98 0.2 0.84 0 0.5 0 Recall Recall-Precision Curve 1 Precision 0.4 0.2 0 L I I I 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ililts - University of California, Berkeley Summary Statistics Run Number Brkly3-full1 automatic 1 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10185 Rel~ret: 1166 0.6 Recall-Precision Curve ecall Level Averages Recall Precision 0.00 0.8461 0.10 0.5630 0.20 0.5016 0.30 0.4423 0.40 0.3946 0.50 0.3491 0.60 0.2963 0.10 0.2126 0.80 0.1490 0.90 0.0543 1.00 0 0040 age precision over all ant docs interpolated 0 3210 t Level Averages it Recall 70 0.4522 ?0 0.5186 10 0.6602 )0 0.1125 30 0.1411 )0 0.1611 ?0 0.1635 10 0.1635 ,0 0.1635 30 0.1635 )0. 0.1635 Document Level Averages Precision At S docs 0.6000 At 10 docs 0.5160 At 15 docs 0.5581 At 20 docs 0.5400 At 30 docs 0.5281 At 100 docs 0.4568 At 200 docs 0.3195 At 500 docs 0.2485 At 1000 docs 0.1553 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- Precision 0.4 0.2 0 1 ments)) 0.8 Exact 0.3691 0.6 NORMAL DEVIATE - Fallout-Recall Recall Ol 0.02 0.84 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ~"1ts - Universitaet Dortmiind Summary Statistics ] Run Number dortl2-fu II, automatic of Queries 50 Total number Retrieved: 50000 Relevant: 10785 jRel~ret:7774 ecall Level_Averages Recall Precision 0.00 0.7324 0.10 0.5410 0.20 0.4939 0.30 0.4370 0.40 0.3850 0.50 0.3324 0.60 0.2821 0.70 0.2133 0.80 0.1340 0.90 0.0707 1.00 0.0063 ge precision over all 1 nt docs ~terpolated 0.3151 ] Level Averages Recall 0.4380 0.5641 0.6401 0.6920 0.7313 0.7519 0.7557 0.7557 0.7557 0.7557 0.7557 Document Level Averages ________________ Precision At 5 docs 0.5120 At 10 docs 0.5280 At 15 docs 0.5213 At 20 docs 0.5210 At 30 docs 0.5120 At 100 docs 0.4562 At 200 docs 0.3781 At 500 docs 0.2439 At 1000 docs 0.1555 L R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3604 NORMAL DEVIATE - Fallout-Recall Ol 0.02 0.84 0.98 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 I I I Recall 1 0.8 0.6 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I -1 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 -2 -3 0.02 0.001 -3 -2 -1 0 1 2 3 Lilts - Universitaet Dortmund [ Summary Statistics Run Number dortQ2-full, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8259 `call Level Averages ~ecall Precision 0.00 0.7673 0.10 0.5518 0.20 0.5060 0.30 0.4572 0.40 0.4114 0.50 0.3635 0.60 0.3048 0.70 0.2388 0.80 0.1565 0.90 0.0777 1.00 0.0086 ge precision over all nt docs iterpolated 0.3340 1 Document Level Averages Precision At 5 docs 0.5560 At 10 docs 0.5440 At 15 docs 0.5427 At 20 docs 0.5440 At 30 docs 0.5213 At 100 docs 0.4626 At 200 docs 0.3959 At 500 docs 0.2607 At 1000 docs 0.1652 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3722 1 0.8 0.6 Precision 0.4 0.2 0 1 0.8 0.6 Level Averages NORMAL DEVIATE - Fallout-Recall Recall Recall ol 0.02 0.98 0 )~999 0.4 0.4625 0.5959 0.6731 0.7216 0.7643 0.7850 0.7908 0.7908 0.7908 0.7908 0.7908 2 0.98 0.2 1 0.84 0 0.5 0 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I 0.001 -3 -2 -1 0 1 2 3 ~su1ts - University of Massachusetts at Amherq.t [ Summary Statistics Run Number INQOOl-full, automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8281 ~ecall Level Averages Recall Precision 0.00 0.8275 0~10 0.5979 0.20 0.5458 0.30 0.4832 0.40 0.4369 0.50 0.3758 0.60 0.3181 0.70 0.2382 0.80 0.1634 0.90 0.0825 1.00 0.0094 `age precision over all I `ant docs interpolated 0.3556 t Level Averages t Recall 0 0.4769 .0 0.6107 .0 0.6913 0 0.7427 `0 0.7713 0 0.7890 0 0.1933 0 0.7933 0 0.7933 0 0.7933 0 0.7933 Document Level Averages ________________ Precision At 5 docs 0.6160 At 10 docs 0.5960 At 15 docs 0.5880 At 20 docs 0.5840 At 30 docs 0.5733 At 100 docs 0.4934 At 200 docs 0.4099 At 500 docs 0.2665 At 1000 docs 0.1656 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3946 0.6 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.84 0.98 0 )~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 Recall Recall-Precision Curve 1 0.8' 0.6 Precision 0.4 0.2 0 I I I 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 "its - University of Massachusetts at Amherst Summary Statistics Run Number INQOO2-full, manual 1 Num of Queries 50 L 4otal number of documents over all queries Retrieved: 50000 Relevant: 10185 Rel~ret: 8165 mcall Level Averages Recall Precision 0.00 0.1162 0.10 0.6299 0.20 0.5604 0.30 0.4920 0.40 0.4331 0.50 0.3122 0.60 0.3133 0.10 0.2268 0.80 0.1545 0.90 0.0815 1.00 0.0122 ~ge precision over all 1 ~nt docs nterpolated 0.3565 ] Level Averages t~ Recall 0.4860 0.6161 2 0.6885 0.1359 1 0.1639 0 0.1802 0.1859 -1 0.1859 0.1859 -2 0.1859 -3 0.1859 0.8 0.6 Precision 0.4 0.2 0 1 0.8 0.6 Document Level Averages Precision At 5 docs 0.6000 At 10 docs 0.6160 At 15 docs 0.6040 At 20 docs 0.6140 At 30 docs 0.5880 At 100 docs 0.5058 At 200 docs 0.4089 At 500 docs 0.2626 At 1000 docs 0.1633 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3954 NORMAL DEVIATE - Fallout-Recall Recall Recall- Precision Curve I I I - 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 0.98 0. 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 -3 -2 -1 0 1 2 3 suits - HNC, Inc. Summary Statistics Run Number HNCad1-full, automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10785 Rel~ret: 6944 0.6 ecall Level Averages Recall Precision 0.00 0.7750 0.10 0.5762 0.20 0.4934 0.30 0.4215 0.40 0.3536 0.50 0.2672 0.60 0.2041 0.70 0.1271 0.80 0.0575 0.90 0.0215 1.00 0.0007 ~ge precision over all ~nt docs nterpolated 0.2787 Level Averages Recall 0 0.4087 0 0.5261 0 0.5867 0.6272 0.6533 ) 0.6681 ) 0.6720 ) 0.6720 ) 0.6720 Document Level Averages Precision At 5 docs 0.5560 At 10 docs 0.5680 At 15 docs 0.5507 At 20 docs 0.5470 At 30 docs 0.5367 At 100 docs 0.4520 At 200 docs 0.3652 At 500 docs 0.2247 At 1000 docs 0.1389 R-Precision (precision after R docs retrieved (where R Is the number of relevant docu- ments)) Precision 0.4 0.2 0 I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 0.8 Exact 0.3474 0.6 NORMAL DEVIATE Fallout-Recall ~1 0.02 0.16 0.84 0.98 0 99 0.4< 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 3 Recall suits - HNC, Inc. L Summary Statistics Run Number HNCad2-full, feedback Num of Queries 50 1 1 Total number of documents over all queries 0.8< Retrieved: 50000 Relevant: 10785 Rel~ret: 6911 0.6 Recall-Precision Curve ~ecall Level Averages Recall Precision 0.00 0.7811 0.10 0.5920 0.20 0.5262 0.30 0.4429 0.40 0.3574 0.50 0.2767 0.60 0.2059 0.70 0.1250 0.80 0.0713 0.90 0.0218 1.00 0.0006 `rage precision over all vant docs -interpolated 0.2882 ~t Level Averages ut Recall 31 ~ 0.4132 ~20 0.5207 2 ~0 0.5790 )60 0.6233 1 )80 0.6485 0 .00 0.6643 L20 0.6676 -1 L40 0.6676 L60 0.6676 -2 L80 0.6676 -3 ?00 0.6676 Document Level Averages Precision At 5 docs 0.5560 At 10 docs 0.5680 At 15 docs 0.5507 At 20 docs 0.5470 At 30 docs 0.5513 At 100 docs 0.4784 At 200 docs 0.3714 At 500 docs 0.2243 At 1000 docs 0.1382 R-Precision (precision after R (where R is relevant docu- docs retrieved the number of ments)) Exact 0.3575 NORMAL DEVIATE - Fallout-Recall .001 0.02 0.16 0.5 0.84 0.98 0.C I I I I -~7-- -3 -2 -1 0 1 2 3 Precision 0.4 0.2 0 1 0.8 0.6 I~999 0.4~ 0.98 0.2 0.84 0.5 0 Recall I I 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve I I 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 esults - Belicore Summary Statistics j Run Number Isial-full, automatic Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10785 Rel~ret: 4756 0.6< Recall Level Averages Recall Precision 0.00 0.5897 0.10 0.3100 0.20 0.2392 0.30 0.1929 0.40 0.1482 0.50 0.1077 0.60 0.0616 0.70 0.0257 0.80 0.0159 0.90 0.0000 1.00 0.0000 rage precision over all 1 ~ant docs -interpolated 0.1307 1 it Level Averages Lit Recall 0~0 0.2200 20 0.3034 40 0.3601 ~0 0.4050 BO 0.4350 ~0 0.4576 ~0 0.4642 ~0 0.4642 ,0 0.4642 30 0.4642 )0 0.4642 Document Level Averages Precision At 5 docs 0.3720 At 10 docs 0.3340 At 15 docs 0.3200 At 20 docs 0.3210 At 30 docs 0.3060 At 100 docs 0.2664 At 200 docs 0.2092 At 500 docs 0.1406 At 1000 docs 0.0951 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Precision 0.4 0.2 0 1 0.8 L Exact 0.1937 0.6 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.84 0.98 0 ~~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I ~sIiIts - Beilcore Summary Statistics Run Number Isiasm-full, automatic Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 - Relevant: 10785 Rel~ret: 7869 Recall Level Averages Recall Precision 0.00 0.7127 0.10 0.5122 0.20 0.4698 0.30 0.4241 0.40 0.3753 0.50 0.3379 0.60 0.2758 0.70 0.2073 0.80 0.1331 0.90 0.0443 1.00 0.0000 rage precision over all vant docs -interpolated 0.3018 it Level Averages ut Recall 00 0.4272 20 0.5560 40 0.6362 60 0.6927 80 0.7323 DO 0.7493 20 0.1540 40 0.7540 60 0.7540 80 0.7540 DO 0.7540 Document Level Averages Precision At 5 docs 0.5240 At 10 docs 0.5020 At 15 docs 0.5027 At 20 docs 0.4950 At 30 docs 0.4873 At 100 docs 0.4306 At 200 docs 0.3716 At 500 docs 0.2469 At 1000 docs 0.1574 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3580 NORMAL DEVIATE - Fallout-Recall 0.6 Precision 0.4 0.2 I I I I 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 1 0.8 0.6 Recall 101 0.02 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 esults - Thinking Machines Corn. [ Summary Statistics [ Run Number TMC8-full, automatic Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 49998 0.8 Relevant: 10185 Rel~ret: 6497 Recall Level Averages Recall Precision 0.00 0.5482 0.10 0.4005 0.20 0.3411 0.30 0.2911 0.40 0.2525 0.50 0.1914 0.60 0.1341 0.10 0.0855 0.80 0.0509 0.90 0.0125 1.00 0.0000 rage precision over all 1 ~docs -interpolated 0.1939 it Level Averages ~t Recall DO 0.2962 20 0.4158 ~ 0.4942 50 0.5490 ~0 0.5911 ~0 0.6191 ~0 0.6260 ~ 0.6260 ,0 0.6260 30 0.6260 )0 0.6260 Document Level Averages Precision At 5 docs 0.3520 At 10 docs 0.3680 At 15 docs 0.3733 At 20 docs 0.3180 At 30 docs 0.3820 At 100 docs 0.3342 At 200 docs 0.2835 At 500 docs 0.1942 At 1000 docs 0.1299 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2612 NORMAL DEVIATE - Fallout-Recall 101 0.02 0.16 ( 5 0.84 0.98 0 0.6 Precision 0.4 0.2 0 Recall 1 0.8 0.6 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 Lilts - Thinking Machines Corp. Summary Statistics Run Number TMC9-full, automatic Num of Queries 50 Total number of documents over all queries 1 Retrieved: 49998 0.8 Relevant: 10185 Rel~ret: 6181 0.6 - Recall-Precision Curve `call Level_Averages Recall Precision 0.00 0.5141 0.10 0.3692 0.20 0.3195 0.30 0.2104 0.40 0.2203 0.50 0.1113 0.60 0.1144 0.70 0.0682 0.80 0.0357 0.90 0.0104 1.00 0.0000 ige precision over all mt docs iterpolated 0.1136 Document Level Averages Precision At 5 docs 0.2800 At 10 docs 0.3180 At 15 docs 0.3320 At 20 docs 0.3450 At 30 docs 0.3501 At 100 docs 0.3144 At 200 docs 0.2661 At 500 docs 0.1840 At 1000 docs 0.1237 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2436 Precision 0.4 0.2 0 Level Averages NORMAL DEVIATE - Fallout-Recall Recall Recall 0.2125 0.3846 0.4610 0.5126 0.5540 0.5827 0.5904 0.5904 0.5904 0.5904 0.5904 1 0.8 0.6 101 0.02 0.16 ( 5 0.84 0.98 0 9 0.4 2 0.98 0.2 1 0.84 0 0.5 0 I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I I 0.001 -3 -2 -1 0 1 2 3 "ults - New York University Summary Statistics ] Run Number nyuirl-category B, automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 49884 0.8 - Relevant: 3929 Reljet: 2983 0.6 - ~call Level Averages Recall Precision 0.00 0.7013 0.10 0.4874 0.20 0.4326 0.30 0.3531 0.40 0.3016 0.50 0.2637 0.60 0.2175 0.10 0.1617 0.80 0.1176 0.90 0.0684 1.00 0.0102 ge precision over all nt docs iterpolated 0.2649 J Level Averages Recall 0.2825 0.4257 0.4979 0.5494 0.5768 0.6112 0.6395 0.6574 0.6757 0.6905 0.7014 2 Document Level Averages Precision At 5 docs 0.4920 At 10 docs 0.4420 At 15 docs 0.4240 At 20 docs 0.4050 At 30 docs 0.3640 At 100 docs 0.2720 At 200 docs 0.1886 At 500 docs 0.1026 At 1000 docs 0.0597 R-Precision (precision after R docs retrieved (where R is relevant docu- the number of ments)) Exact 0.3003 NORMAL DEVIATE - ~1out-RecaII 101 0.02 -3 -2 -1 0 1 2 3 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 Precision 0.4 0.2 0 1 0.8 0.6 .999 0.4 0.98 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I ilts - New York University Summary Statistics Run Number nyuir2-category B, automatic 1 Num of Queries 50 3 Total number of documents over all queries 0.8 Retrieved: 49876 Relevant: 3929 Rel~ret: 3274 I 0.6 - Recall-Precision Curve `call Level Averages Recall Precision 0.00 0.7528 0.10 0.5567 0.20 0.4721 0.30 0.4060 0.40 0.3617 0.50 0.3135 0.60 0.2703 0.70 0.2231 0.80 0.1667 0.90 0.0915 1.00 0.0154 ~ge precision over all ~nt docs nterpolated 0.3111 ] Level Averages t Recall 0 0.3407 0 0.4886 ~ 0.5620 ~0 0.6193 ~0 0.6569 )0 0.7012 ~0 0.7202 10 0.7435 )0 0.7572 30 0.7732 )0 0.7842 Document Level Averages Precision Precision 0.4 At 5 docs 0.5360 At 10 docs 0.4880 0.2 At 15 docs 0.4693 At 20 docs 0.4390 At 30 docs 0.4067 0 At 100 docs 0.3094 At 200 docs 0.2139 At 500 docs 0.1137 At 1000 docs 0.0655 R-Precision (precision after R I docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.3320 0.6 NORMAL DEVIATE - Fallout-Recall Recall 0.84 098 0.4 0.2 0.98 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I 2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ~u1ts - New York University Summary Statistics Run Number nyuir3-category B, automatic Recall-Precision Curve Num of Queries 50 1 lotal number of documents over all queries Retrieved: 49811 0.8 Relevant: 3929 Reljet: 3281 ecall Level Averages Recall Precision 0.00 0.7528 0.10 0.5514 0.20 0.4724 0.30 0.4076 0.40 0.3621 0.50 0.3142 0.60 0.2711 0.10 0.2237 0.80 0.1691 0.90 0.0916 1.00 0.0160 lge precision over all Int docs nterpolated 0.3118 Level Averages Document Level Averages Precision At 5 docs 0.5360 At 10 docs 0.4880 At 15 docs 0.4101 At 20 docs 0.4410 At 30 docs 0.4080 At 100 docs 0.3094 At 200 docs 0.2140 At 500 docs 0.1140 At 1000 docs 0.0656 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 0.6 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I Exact 0.3321 0.6 NORMAL DEVIATE - Fallout-Recall 0.84 098 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 FalloutxlOO -2 0.02 0.001 -3 -2 -1 0 1 2 3 Recall Recall D 0.3412 D 0.4885 D 0.5629 0.6220 0.6586 0.6991 0.7206 3 0.7445 3 0.1581 3 0.1744 3 0.7866 suits - Carnegie Mellon University I Summary Statistics Run Number CLARTA-full, automatic 1 Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10785 Rel~ret: 8109 0.6 ~ecall Level Averages Document Level Averages Precision Recall Precision Precision 0.4 0.00 0.7581 At 5 docs 0.5800 0.10 0.5676 At 10 docs 0.5680 0.2 0.20 0.5038 At 15 docs 0.5547 0.30 0.4489 At 20 docs 0.5490 0.40 0.3938 At 30 docs 0.5253 0 0.50 0.3455 At 100 docs 0.4644 0.60 0.2806 At 200 docs 0.3874 0.70 0.2172 At 500 docs 0.2542 0.80 0.1463 At 1000 docs 0.1622 0.90 0.0827 R-Precision (precision after R 1 1.00 0.0070 docs retrieved (where R is ~age precision over all the number of relevant docu- ~ant docs ments)) 0.8 interpolated 0.3264 ] Exact 0.3645 Recall-Precision Curve I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I t Level Averages Pt Recall )~0 0.4407 )~0 0.5756 ~ 0.6613 ,0 0.7169 30 0.7520 )0 0.7732 ~ 0.7767 ~ 0.7767 ,0 0.7767 30 0.7767 )0 0.7767 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 301 0.02 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 `suits - Carnegie Mellon University L Summary Statistics ] F Run Number CLARTM-full1 manual Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8229 ~ecall Level Averages Recall Precision 0.00 0.1455 0.10 0.5811 0.20 0.5240 0.30 0.4622 0.40 0.4135 0.50 0.3593 0.60 0.2983 0.10 0.2312 0.80 0.1516 0.90 0.0814 1.00 0.0062 `age precision over all ant docs interpolated 0.3383 j t Level Averages Recall Fo 0.4521 0 0.5940 ~ 0.6130 0 0.7211 0 0.1696 0 0.1868 .0 0.7932 0 0.7932 0 0.1932 0 0.1932 `0 0.1932 Document Level Averages Precision At 5 docs 0.5840 At 10 docs 0.5140 At 15 docs 0.5640 At 20 docs 0.5590 At 30 docs 0.5433 At 100 docs 0.4846 At 200 docs 0.3915 At 500 docs 0.2601 At 1000 docs 0.1646 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3141 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.16 Recall-Precision Curve 1 0.8 0.6 - Precision 0.4 0.2 I I I 0 1 0.8 0.6 Recall 0.R4 0.98 0.4 0.2 2 0.98 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 "its - Swiss Federal Institute of Technology (ETH) Summary Statistics I F Run Number schaul-full, automatic 1 Num of Queries 50 j Total number of documents over all queries 1 0.8 ~-, Retrieved: 50000 1 Relevant: 10785 Reljet: 7081 j 0.6 - Recall-Precision Curve ~call Level Averages Recall Precision 0.00 0.1692 0.10 0.4912 0.20 0.4231 0.30 0.3649 0.40 0.3025 0.50 0.2529 0.60 0.1887 0.70 0.1213 0.80 0.0127 0.90 0.0403 1.00 0.0000 ~ge precision over all ~nt docs nterpolated 0.2517 1 Level Averages t Recall 31 0.3620 0.4763 2 0.5547 0.6102 1 0.6507 0 0.6743 0.6771 -1 0.6771 0.6771 -2 0.6171 -3 0.6771 Document Level Averages Precision At 5 docs 0.5360 At 10 docs 0.4960 At 15 docs 0.4800 At 20 docs 0.4720 At 30 docs 0.4513 At 100 docs 0.4000 At 200 docs 0.3302 At~500 docs 0.2158 At 1000 docs 0.1416 R-Precision (precision after R docs retrieved (where R is of relevant docu- the number ments)) Exact 0.3148 NORMAL DEVIATE - Fallout-Recall .001 0.02 0.16 0.5 0~84 0.98 0.~ I I I I -~7- -3 -2 -1 0 1 3 Precision 0.4 0.2 0 1 0.8 0.6 ~999 0.4 0.98 0.2 0.84 0.5 0 0.16 0.02 0.001 Recall I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 Fallout x 100 ~su1ts - The Environment Research Institute of Michi~an Summary Statistics Run Number erimal-full1 automatic Num of Queries 50 J TOtd'l number of documents over all~queries Retrieved: 48809 Relevant: 10185 Rel~ret: 5349 Recall Level_Averages Recall Precision 0.00 0.6409 0.10 0.3564 0.20 0.2893 0.30 0.2251 0.40 0.1693 0.50 0.1332 0.60 0.0911 0.70 0.0610 0.80 0.0310 0.90 0.0048 1.00 0.0000 rage precision over all ~ant docs -interpolated 0.1589 Document Level Averages Precision At 5 docs 0.4360 At 10 docs 0.3960 At 15 docs 0.3827 At 20 docs 0.3810 At 30 docs 0.3660 At 100 docs 0.3016 At 200 docs 0.2378 At 500 docs 0.1598 At 1000 docs 0.1070 R-Precision (precision after R docs retrieved (where R is relevant docu- the number of ments)) Exact 0.2179 Recall-Precision Curve 1 0.8 - 0.6 Precision 0.4 0.2 I I 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I it Level Averages Lit Recall DO 0.2423 20 0.3411 40 0.4053 60 0.4460 BO 0.4827 DO 0.5138 20 0.5221 40 0.5221 60 0.5221 BO 0.5221 ~0 0.5221 0.6 NORMAL DEVIATE - Fallout-Recall )Oi 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 Recall -3 -2 -1 0 1 2 3 ults - The Environment Research Institute of Michigan I Summary Statistics Recall-Precision Curve ~ Run Number erima2-full, automatic 1 1 Num of Queries 50 Total number of documents over all queries 0.8 Retrieved: 47943 Relevant: 10785 Rel~ret: 5975 0.6 ecall Level_Averages Recall Precision 0.00 0.7096 0.10 0.4059 0.20 0.3287 0.30 0.2723 0.40 0.2276 0.50 0.1698 0.60 0.1302 0.70 0.0707 0.80 0.0480 0.90 0.0011 1.00 0.0000 rage precision over all 1 ~ant docs -interpolated 0.1885 1 Document Level Averages Precision At 5 docs 0.4760 At 10 docs 0.4480 At 15 docs 0.4320 At 20 docs 0.4180 At 30 docs 0.3993 At 100 docs 0.3426 At 200 docs 0.2707 At 500 docs 0.1799 At 1000 docs 0.1195 ~R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Fxact 0.2494 Precision 0.4 0.2 - 0 1 0.8 0.6 it Level Averages NORMAL DEVIATE - Fallout-Recall Recall Tht Recall 0.98 0 0.4 00 0.2717 20 0.3824 40 0.4449 60 0.4979 80 0.5353 00 0.5611 20 0.5678 40 0.5678 60 0.5678 .80 0.5678 !00 0.5678 2 0.98 0.2 1 0.84 0 0.5 0 -1 0.16 -2 0.02 -3 -3 -2 -1 0 1 2 3 0.001 1 0Thy~~~~0.40.60.8 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 ~ults - CITRI, Royal Melbourne Institute of Technology Summary Statistics ] Run Number citril-full, automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 7439 ecall Level_Averages Recall Precision 0.00 0.8023 0.10 0.5298 0.20 0.4570 0.30 0.4027 0.40 0.3482 0.50 0.2951 0.60 0.2271 0.70 0.1514 0.80 0.0887 0.90 0.0379 1.00 0.0000 ~ge precision over all ~nt docs nterpolated 0.2804 Level Averages Recall 0~ 0.3934 0 0.5301 0 0.6112 0 0.6630 0 0.6972 0 0.7158 D 0.7192 0 0.7192 0 0.7192 0 0.7192 0 0.7192 Document Level Averages Precision At S docs 0.5680 At 10 docs 0.5320 At 15 docs 0.5173 At 20 docs 0.5140 At 30 docs 0.4913 At 100 docs 0.4224 At 200 docs 0.3541 At 500 docs 0.2340 At 1000 docs 0.1488 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3359 NORMAL DEVIATE - Fallout-Recall ol 084 0.98 0.4~ 0.2 Recall 2 0.98 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 I I I I 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 0.8 0.6 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ults - CITRI, Royal Melbourne Institute of Technology L Summary Statistics Run Number citri2-full, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 1495 ecall Level Averages Recall Precision 0.00 0.1856 0.10 0.5335 0.20 0.4611 0.30 0.4086 0.40 0.3526 0.50 0.2932 0.60 0.2306 0.70 0.1613 0.80 0.1053 0.90 0.0510 1.00 0.0097 rage precision over all ~ant docs interpolated 0.2814 it Level Averages Ft Recall Fo 0.4050 20 0.5313 2 40 0.6116 60 0.6684 80 0.7040 00 0.1216 0 20 0.1291 -1 40 0.1291 60 0.1291 -2 80 0.1291 00 0.1291 -3 Document Level Averages Precision At 5 docs 0.5600 At 10 docs 0.5280 At 15 docs 0.5133 At 20 docs 0.5000 At 30 docs 0.4900 At 100 docs 0.4354 At 200 docs 0.3606 At 500 docs 0.2358 At 1000 docs 0.1499 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 1 0.8< 0.6 Precision 0.4 0.2 - 0 Exact 0.3388 NORMAL DEVIATE - Fallout-Recall Recall 1 0.8 0.6 0.84 0.98 0 0.4~ 0.98 0.2 0.84 0.5 0 0.16 0.02 I I I 0.001 -3 -2 -1 0 1 2 3 Recall-Precision Curve I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 ~`11ts - GE Researeh and Development Center Summary Statistics j Run Number gecrd2-full, automatic 1 Recall-Precision Curve 1 L Num of Queries 50 J r Total number of documents over all queries 1 Retrieved: 50000 1 0.8 - Relevant: 10785 Rel~ret: 6080 0.6 ecall Level Averages Recall Precision 0.00 0.7047 0.10 0.4433 0.20 0.3753 0.30 0.3112 0.40 0.2503 0.50 0.2049 0.60 0.1496 0.70 0.0999 0.80 0.0690 0.90 0.0381 1.00 0.0088 ~ge precision over all ~nt docs nterpolated 0.2183 1 Document Level Averages Precision At 5 docs 0.4880 At 10 docs 0.4760 At 15 docs 0.4653 At 20 docs 0.4510 At 30 docs 0.4253 At 100 docs 0.3662 At 200 docs 0.2896 At 500 docs 0.1876 At 1000 docs 0.1216 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2816 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I Level Averages NORMAL DEVIATE - Fallout-Recall Recall Recall D 0.3221 0.4269 0.4957 0.5489 0.5850 ) 0.5968 ) 0.6014 ) 0.6014 ) 0.6014 ) 0.6014 ) 0.6014 0.6 101 0.02 5 0.84 0.98 0 0.4 0.2 - 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 -3 -2 -1 0 1 2 3 0.001 ults - Verity, Inc. Summary Statistics Recall-Precision Curve Run Number TOPIC2-fuII1 manual 1 Num of Queries 50 Total number of documents over all queries Retrieved: 31624 0.8 Relevant: 10185 Rel~ret: 5614 mcall Level Averages Recall Precision 0.00 0.8491 0.10 0.5786 0.20 0.4943 0.30 0.3906 0.40 0.2774 0.50 0.2062 0.60 0.1221 0.10 0.0663 0.80 0.0353 0.90 0.0137 1.00 0.0000 ige precision over all 1 mt docs nterpolated 0.2464 J Document Level Averages Precision At 5 docs 0.6280 At 10 docs 0.5880 At 15 docs 0.5827 At 20 docs 0.5680 At 30 docs 0.5407 At 100 docs 0.4624 At 200 docs 0.3347 At 500 docs 0.2002 At 1000 docs 0.1123 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3211 0.6 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve Level Averages Recall ) 0.3744 ~ 0.4689 ~ 0.5232 ~ 0.5446 ~ 0.5512 ~ 0.5512 ~ 0.5512 ) 0.5512 ~ 0.5512 ) 0.5512 ~ 0.5512 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 101 0.02 0.98 0 )~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 -3 -2 -1 0 1 2 3 0.001 tilts - Siemens Corporate Researchq Inc. [ Summary Statistics Run Number siems2-fuIl, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8068 `call Level_Averages Recall Precision 0.00 0.1964 0.10 0.5166 0.20 0.5267 0.30 0.4808 0.40 0.4244 0.50 0.3810 0.60 0.3088 0.70 0.2418 0.80 0.1416 0.90 0.0513 1.00 0.0000 ige precision over all mt docs riterpolated 0.3408 Document Level Averages Precision At 5 docs 0.6000 At 10 docs 0.5860 At 15 docs 0.5733 At 20 docs 0.5600 At 30 docs 0.5447 At 100 docs 0.4690 At 200 docs 0.4030 At 500 docs 0.2606 At 1000 docs 0.1614 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Fxact 0.3938 Level Averages NORMAL DEVIATE - Fallout-Recall Recall ) 0.7722 ) 0.7163 ) 0.7763 ) 0.7763 ) 0.7763 Ol 0.02 0.16 0.4 0.2 2 0.98 1 0.84 0 0.5 0 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 I I 0 1 0.8 0.6 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 -3 -2 -1 0 2 3 0.001 Its - Siemens Corporate Research, Inc. L Summary Statistics ] Run Number siems3-full, automatic ] Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 8065 all Level Averages ecall Precision ).00 0.8164 ).10 0.5849 ).20 0.5262 ).30 0.4774 ).40 0.4262 ).50 0.3771 ).60 0.3111 ).70 0.2338 ).80 0.1362 ).90 0.0513 1.00 0.0000 r~e precision over all it docs ~erpolated 0.3397 evel Averages Recall 0.4714 0.5999 0.6680 0.7207 0.7543 0.7720 0.7757 0.7757 0.7757 0.7757 0.7757 Document Level Averages Precision At 5 docs 0.5960 At 10 docs 0.5920 At 15 docs 0.5773 At 20 docs 0.5630 At 30 docs 0.5453 At 100 docs 0.4692 At 200 docs 0.3979 At 500 docs 0.2598 At 1000 docs 0.1613 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 1 0.8 0.6 Precision 0.4 0.2 0 Exact 0.3931 NORMAL DEVIATE - Fallout-Recall Recall 1 0.8 0.6 Ol 0.02 0.16 5 0.84 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 Recall-Precision Curve I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I I 0.001 -3 -2 -1 0 1 2 3 ilts - Queens College, CUNY Summary Statistics J Run Number pircs3-full, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10185 Rel~ret: 1456 call Level Averages ~ecall Precision 0.00 0.7538 0.10 0.5529 0.20 0.4127 0.30 0.4088 0.40 0.3531 0.50 0.2994 0.60 0.2311 0.10 0.1677 0.80 0.1094 0.90 0.0551 1.00 0.0145 ~e precision over all ~t docs terpolated 0.2949 evel Averages Recall 0.4118 0.5411 0.6092 0.6608 0.6990 0.7245 0.1288 0.1288 0.1288 0.1288 0.7288 Document Level Averages ________________ Precision At 5 docs 0.5640 At 10 docs 0.5560 At 15 docs 0.5467 At 20 docs 0.5460 At 30 docs 0.5200 At 100 docs 0.4466 At 200 docs 0.3660 At 500 docs 0.2342 At 1000 docs 0.1491 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3437 NORMAL DEVIATE - Fallout-Recall Ol 0.02 0.16 0.84 0.98 0 ~999 2 0.98 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 I I I I 0 Recall 1 0.8 0.6 0.4 0.2 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 `sults - Queens College, CUNY Summary Statistics ) ~ Run Number pircs4-fulI, automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10785 Rel~ret: 7464 Recall Level_Averages Recall Precision 0.00 0.7621 0.10 0.5569 0.20 0.4773 0.30 0.4108 0.40 0.3554 0~50 0.3017 0.60 0.2343 0.70 0.1702 0.80 0.1128 0.90 0.0602 1.00 0.0154 `rage precision over all .`vant docs i-interpolated 0.2981 ut Level Averages ~t Recall )00 0.4140 )20 0.5449 2 )40 0.6128 )60 0.6637 1 )80 0.7008 0 100 0.7244 120 0.7293 -1 [40 0.7293 [60 0.7293 -2 [80 0.7293 -3 ~00 0.7293 Document Level Averages Precision At 5 docs 0.5760 At 10 docs 0.5760 At 15 docs 0.5507 At 20 docs 0.5460 At 30 docs 0.5200 At 100 docs 0.4494 At 200 docs 0.3680 At 500 docs 0.2348 At 1000 docs 0.1493 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) _______________ 1 0.8 0.6 Precision 0.4 0.2 0 ~t03467 NORM ~L DEVIATE - Fallout-Recall Recall 101 0.02 0.98 0 -3 -2 -1 0 1 2 3 1 0.8 0.6 0.4~ 0.98 0.2 0.84 0.5 0 Recall-Precision Curve I I I I 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 results - VPI ~ SU Summary Statistics ] Run Number VTcms2-full manual Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10185 Rel~ret: 1616 Recall Level_Averages Recall Precision 0.00 0.8693 0.10 0.6115 0.20 0.5409 0.30 0.4591 0.40 0.3834 0.50 0.3172 0.60 0.2451 0.70 0.1659 0.80 0.0909 0.90 0.0364 1.00 0.000 ~rage precision over a vant docs 0.320:11 i-interpolated Document Level Averages Precision At 5 docs 0.6800 At 10 docs 0.6280 At 15 docs 0.6240 At 20 docs 0.6130 At 30 docs 0.5820 At 100 docs 0.4790 At 200 docs 0.3835 At 500 docs 0.2430 At 1000 docs 0.1535 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3708 0.6 ~t Level Averages NORMAL DEVIATE - Fallout-Recall ut Recall i00 0.4410 `20 0.5666 40 0.6420 60 0.6877 80 0.7179 00 0.7347 20 0.7414 40 0.7414 60 0.7414 80 0.7414 o0 0.7414 Recall 3:1 0.98 a )~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0.6 Precision 0.4 0.2 - I I I 0 1 0.8 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 `ults - Conquest Software, Inc. [ Summary Statistics Recall-Precision Curve Run Number CnQstl-full, manual 1 I II Num of Queries ½50 ~otal num ~ 0f ~ over all queries Retrieved: 50000 Relevant: 10185 Rel~ret:6494I Document Level Averages ~ecall Level Averages Precision Recall Precision At 5 docs 0.4480 0.00 0.6831 At 10 docs 0.4360 0.10 0.4552 At 15 docs 0.4427 0.20 0.4010 At 20 docs 0.4310 0.30 0.3538 At 30 docs 0.4393 0.40 0.2951 At 100 docs 0.3954 0.50 0.2361 At 200 docs 0.2997 0.60 0.1807 At 500 docs 0.1962 0.70 0.1147 At 1000 docs 0.1299 0.80 0.0725 0.90 0.0454 R-Precision (p~recisionafterRI 1.00 0.0054 docs retrieved (where R is the number of relevant docu- `rage precision over all ments)) -1 `vant docs ~-interpolated 0.2343 Exact 0.2869 NORMAL DEVIATE - Fallout-Recall ut Level Averages )ut Recall .0 Ol 0.02 0.16 0.5 0.84 0.98 0.9~999 )00 0.3320 ~20 0.4506 2 0.98 040 0.5406 0.84 060 0.5993 1 080 0.6400 0.5 0 100 0.6696 120 0.6763 -1 0.16 140 0.6763 0.02 160 0.6763 -2 180 0.6763 -3 0.001 p200 0.6763 3 -2 -1 0 1 2 3 0.8 - 0.6 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 1 0.8 0.6 Recall 0.4 0.2 0 0 0.05 0.1 0.15 0.2 Fallout x 100 slilts - Conquest Software, Inc. Summary Statistics ] Run Number CnQst2-full, manual 1 Num of Queries 50 J Total number of documents over all queries 1 Retrieved: 50000 1 Relevant: 10785 Rel~ret: 6788 ~ecall Level Averages Recall Precision 0.00 0.7202 0.10 0.5005 0.20 0.4411 0.30 0.3787 0.40 0.3242 0.50 0.2681 0.60 0.2010 0.70 0.1373 0.80 0.0947 0.90 0.0465 1.00 0.0045 ~age precision over all 1 ,ant docs interpolated 0.2633 t Level Averages it Recall Po 0.3737 ?0 0.4851 ~ 0.5676 )0 0.6290 30 0.6653 )0 0.6931 ~0 0.6960 ~ 0.6960 )0 0.6960 10 0.6960 ~0 0.6960 Document Level Averages Precision At 5 docs 0.5080 At 10 docs 0.4880 At 15 docs 0.4920 At 20 docs 0.4960 At 30 docs 0.4793 At 100 docs 0.4178 At 200 docs 0.3256 At 500 docs 0.2102 At 1000 docs 0.1358 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3140 NORMAL DEVIATE - Fallout-Recall g) Ol 0.02 2 0.98 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 I I I Recall 1 0.8 0.6 0.4 0.2 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 suitS - PRC, Inc. Summary Statistics Run Number prceol-full, manual 1 1 LTNum of Queries 50 otal number of documents over all queries Retrieved: 50000 0.8 Relevant: 10185 Rel~ret: 3323 0.6~ ~ecall Level Averages Document Level Averages Precision Recall Precision Precision 0.4 0.00 0.5963 At 5 docs 0.4120 0.10 0.3270 At 10 docs 0.4000 0.2 0.20 0.2306 At 15 docs 0.3920 0.30 0.1430 At 20 docs 0.3140 0.40 0.0866 At 30 docs 0.3521 0 0.50 0.0425 At 100 docs 0.2122 0.60 0.0323 At 200 docs 0.1914 0.70 0.0246 At 500 docs 0.1090 0.80 0.0209 At 1000 docs 0.0665 0.90 0.0131 R-Precision (precision after R 1 1.00 0 0041 docs retrieved (where R is rage precision over all the number of relevant docu- ~ant docs ments)) 0.8 -interpolated 0 1120 Exact 0.1809 0.6 it Level Averages LIT Recall 00 0.2103 20 0.2531 40 0.2811 60 0.3006 80 0.3166 00 0.3294 20 0.3335 40 0.3335 60 0.3335 80 0.3335 00 0.3335 NORMAL DEVIATE - ~11out-Reca11 Recall 1) 0.84 0.98 0 0.4 2 1 0 -1 -2 I I -3 0.98 -3 -2 -1 0 1 2 3 0.2~ 0.84 0.5 0 0.16 0.02 0.001 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I I 0 0.05 0.1 0.15 0.2 Fallout x 100 suits - PRC, Inc. Summary Statistics Run Number prceo2-full, manual Recall-Precision Curve Num of Queries 50 1 L lotal number of documents over all queries Retrieved: 50000 0.8 Relevant: 10785 Rel~ret: 3561 F 0.6 ~ecall Level_Averages Recall Precision 0.00 0.4768 0.10 0.2613 0.20 0.1807 0.30 0.1202 0.40 0.0875 0.50 0.0504 0.60 0.0328 0.70 0.0169 0.80 0.0142 0.90 0.0077 1.00 0.0031 age precision over all ant docs nterpolated 0.0904 Document Level Averages Precision At 5 docs 0.2120 At 10 docs 0.2480 At 15 docs 0.2707 At 20 docs 0.2760 At 30 docs 0.2753 At 100 docs 0.2418 At 200 docs 0.1756 At 500 docs 0.1086 At 1000 docs 0.0712 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.1703 Precision 0.4 0.2 0 Level Averages NORMAL DEVIATE - ~11out-Reca11 t Recall Recall 101 0.02 0.16 5 a84 0.98 0 0 0.1861 0 0.2402 0 0.2860 0 0.3141 0 0.3387 0 0.3564 0 0.3623 0 0.3623 0 0.3623 0 0.3623 0 0.3623 2 0.98 1 0.8 0.6 0.4 0.21 1 0.84 0 0.5 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ults - Mead Data Central, Inc. Summary Statistics Run Number UREKA2-full, manual 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50042 Relevant: 10785 Rel~ret: 2690 ecall Level Averages Recall Precision 0.00 0.2740 0.10 0.0977 0.20 0.0774 0.30 0.0701 0.40 0.0617 0.50 0.0467 0.60 0.0338 0.70 0.0242 0.80 0.0187 0.90 0.0000 1.00 0.0000 rage precisior~ over all `-~rtteidp0:l:ted0.05021 it Level Averages ut Recall 00 0.0733 20 0.1100 2 40 0.1403 ~0 0.1713 1 80 0.1970 0 .00 0.2173 .20 0.2271 -1 .40 0.2271 .60 0.2271 -2 .80 0.2271 -3 ?00 0.2271 Document Level Averages Precision At 5 docs 0.1360 At 10 docs 0.1140 At 15 docs 0.1067 At 20 docs 0.1100 At 30 docs 0.1033 At 100 docs 0.0912 At 200 docs 0.0831 At 500 docs 0.0664 At 1000 docs 0.0538 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.0793 NORMAL DEVIATE - Fallout-Recall Recall 5 C ). .-- . .9 I I 101 0.02 0.84 0.98 0. 0.98 0.84 0.16 -3 -2 -1 0 2 3 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 I I 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 1 I I 0.8 0.6 0.4 0.2 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 ~su1ts - Mead Data Central, Inc. Run Num be Recall-Precision Curve 50 1 [ Total numb Retrieved: 0.8 Relevant: 10785 Rel~ret:3282j t ~ecall Level Averages Recall Precision 0.00 0.4228 0.10 0.1880 0.20 0.1385 0.30 0.1045 0.40 0.0840 0.50 0.0643 0.60 0.0445 0.70 0.0365 0.80 0.0183 0.90 0.0000 1.00 0.0000 age precision over all ant docs interpolated 0.0819 Level Averages t Recall 0 0.1242 0 0.1144 0 0.2072 0 0.2337 0 0.2571 0 0.2740 0 0.2805 0 0.2805 D 0.2805 ~ 0.2805 ~ 0.2805 31 2 1 0 Document Level Averages Precision At 5 docs 0.2520 At 10 docs 0.2320 At 15 docs 0.2267 At 20 docs 0.2200 At 30 docs 0.2073 At 100 docs 0.1688 At 200 docs 0.1383 At 500 docs 0.0954 At 1000 docs 0.0656 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- m ents)) Exact 0.1231 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.16 0. 5 0.84 0.98 0 7 0.6 Precision 0.4 0.2 0 1 0.8 0.6 ~999 0.4 0.98 0.2 0.84 0.5 0 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I -1 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ults - University of Cahfornia, Los An~e1es Summary Statistics Run Number uclaal-category B, automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8' Relevant: 3929 Reljet: 3204 ecall Level Averages Recall Precision 0.00 0.8126 0.10 0.6075 0.20 0.5292 0.30 0.4385 0.40 0.3861 0.50 0.3210 0.60 0.2767 0.10 0.2284 0.80 0.1646 0.90 0.0938 1.00 0.0163 ige precision over all mt docs nterpolated 0.3345 Level Averages Recall () 0.3638 D 0.4872 D 0.5666 D 0.6114 ~ 0.6617 ~ 0.6944 D 0.7136 ~ 0.7335 ~ 0.1451 ~ 0.7604 ~ 0.1748 Document Level Averages Precision At 5 docs 0.5840 At 10 docs 0.5380 At 15 docs 0.4973 At 20 docs 0.4660 At 30 docs 0.4333 At 100 docs 0.3098 At 200 docs 0.2216 At 500 docs 0.1140 At 1000 docs 0.0641 R-Precision (precision after k docs retrieved (where R is the number of relevant docu- ments)) 0.6 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I Exact 0.3629 0.6 NORMAL DEVIATE - Fallout-Recall Ol 0.02 0.98 0 )~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 Recall suits - University of California~ Los An~e1es f Summary Statistics Run Number uclaa2-category B1 automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 3929 Reljet: 3059 0.6 - ~ecall Level Averages Recall Precision 0.00 0.7464 0.10 0.5880 0.20 0.5024 0.30 0.4058 0.40 0.3429 0.50 0.2698 0.60 0.2168 0.10 0.1803 0.80 0.1209 0.90 0.0695 1.00 0.0098 *age precision over all ant docs interpolated 0.2951 ] Document Level Averages Precision At 5 docs 0.5440 At 10 docs 0.5180 At 15 docs 0.4920 At 20 docs 0.4650 At 30 docs 0.4201 At 100 docs 0.2160 At 200 docs 0.1958 At 500 docs 0.1051 At 1000 docs 0.0612 RLprecision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3289 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I t Level Averages ~t Recall Fo 0.3245 ~0 0.4324 ~ 0.4885 ~ 0.5313 ~0 0.5838 ~ 0.6135 0 0.6339 ~ 0.6561 ~ 0.6698 0 0.6852 0 0.7066 0.6 NORMAL DEVIATE - ~Iout-RecaI1 101 0.02 0.84 0.98 0.4 0.2 Recall 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 Lilts - University of California, Los Angeles Summary Statistics Recall-Precision Curve un Number uclafi-category B, feedback, frozen evaluation 1 um of Queries 50 Total number of documents over all queries 0.8< etrieved: 50473 elevant: 3929 cljet: 3047 J 0.6 ecall Level Averages Recall Precision 0.00 0.8125 0.10 0.5951 0.20 0.5218 0.30 0.4113 0.40 0.3484 0.50 0.2813 0.60 0.2289 0.10 0.1915 0.80 0.1351 0.90 0.0764 1.00 0.0105 age precision over all ant docs interpolated 0.3091 1 L Level Averages it Recall Fo 0.3425 ?0 0.4483 2 ~ 0.5041 ,0 0.5489 1 30 0.5898 0 )0 0.6216 ~0 0.6431 -1 ~ 0.6653 50 0.6864 -2 BO 0.1030 -3 00 0.1112 Document Level Averages Precision Precision 0.4 At 5 docs 0.5880 At 10 docs 0.5220 0.2 At 15 docs 0.4893 At 20 docs 0.4640 At 30 docs 0.4360 0 At 100 docs 0.2884 At 200 docs 0.1992 At 500 docs 0.1058 At 1000 docs 0.0609 R-Precision (precision after R 1 docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.3459 NORMAL DEVIATE - Fa11out-R~ca11 Recall 0 0.2 0.4 0.6 08 Recall Fallout-Recall Curve I I 0.6 - 0.98 0 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 -3 -2 -1 0 1 2 3 suits - University of Illinois at Chicago Summary Statistics Run Number uicah-full, automatic 1 Recall-Precision Curve Num of Queries 50 J 1 Total number of documents over all queries 1 Retrieved: 45108 1 0.8 Relevant: 10785 Rel~ret: 4421 0.6 ~ecall Level Averages Recall Precision 0.00 0.5665 0.10 0.3272 0.20 0.2527 0.30 0.1937 0.40 0.1413 0.50 0.0712 0.60 0.0393 0.70 0.0069 0.80 0.0000 0.90 0.0000 1.00 0.0000 age precision over all 1 ant docs interpola~~.1200 Level Averages t ~0 0 0 0 0 0 0 0 0 0 0 Recall 0.2095 0.3057 0.3532 0.3860 0.4125 0.4359 0.4467 0.4467 0.4467 0.4467 0.4467 Document Level Averages Precision At 5 docs 0.3280 At 10 docs 0.3340 At 15 docs 0.3280 At 20 docs 0.3380 At 30 docs 0.3233 At 100 docs 0.2784 At 200 docs 0.2107 At 500 docs 0.1392 At 1000 docs 0.0884 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact I 0.1950 NORMAL DEVIATE - Fallout-Recall Precision 0.4 0.2 0 1 0.8 0.6 Recall )01 0.02 0.84 0.98 0 0.4 - 0.2 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 3 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I A results - City University, London Summary Statistic s] Run Number cityri-full, automatic 1 Num of Queries 50 J Total number of documents over all queries Retrieved: 50000 0.8< Relevant: 10489 Rel~ret: 6801 0.6 Recall-Precision Curve Recall Level Averages Recall Precision 0.00 0.1888 0.10 0.6028 0.20 0.5101 0.30 0.4380 0.40 0.3851 0.50 0.3070 0.60 0.2494 0.70 0.1826 0.80 0.1269 0.90 0.0606 1.00 0.0048 ~rage precision over all vant docs 0.3149 i-interpolated ut Level Averages ~ut Recall )~ 0.3610 )20 0.4444 )40 0.5009 )60 0.5433 )80 0.5791 L00 0.6074 [20 0.6294 [40 0.6476 [60 0.6636 [80 0.6784 ~0 0.6918 Document Level Averages Precision At 5 docs 0.6280 At 10 docs 0.5940 At 15 docs 0.5867 At 20 docs 0.5690 At 30 docs 0.5333 At 100 docs 0.4316 At 200 docs 0.3385 At 500 docs 0.2154 At 1000 docs 0.1360 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Precision 0.4 0.2 0 1 0.8 Exact 0.3607 0.6 - NORMAL DEVIATE - Fallout-Recall Recall ol 0.02 0.84 0.98 0. 999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I results - City University, London L Summary Statistics Run Number cityr2-fulI, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10489 Rel~ret: 7134 Recall Level Averages Recall Precision 0.00 0.8313 0.10 0.6341 0.20 0.5491 0.30 0.4990 0.40 0.4356 0.50 0.3618 0.60 0.2908 0.10 0.2305 0.80 0.1551 0.90 0.0900 1.00 0.0085 rage precision over all ~ant docs -interpolated 0.3562 J it Level Averages it Recall ~ 0.3949 ~0 0.4731 ~ 0.5331 50 0.5722 30 0.6050 )0 0.6278 ~0 0.6510 ~ 0.6696 ,0 0.6851 30 0.7065 )0 0.7181 j 1 I 1 0.8 0.6 Document Level Averages Precision _______________ Precision 0.4 At 5 docs 0.6920 At 10 docs 0.6500 At 15 docs 0.6213 0.2 At 20 docs 0.6060 At 30 docs 0.5613 0 At 100 docs 0.4494 At 200 docs 0.3588 At 500 docs 0.2261 At 1000 docs 0.1427 R- Precision (precision after R 1 docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.3877 0.6 NORMAL DEVIATE - Fallout-Recall Recall 101 0.a2 0.16 ( 5 0.84 0.98 0 0.4 0.2 2 0.98 1 0.84 0 0.5 0 -1 0.16 -2 0 02 -3 0.001 -3 -2 -1 0 1 2 3 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 results - Rutgers University Summary Statistics Run Number rutcombx-full, manual Num of Queries 50 Total number of documents over all queries Retrieved: 49983 Relevant: 10489 Rel~ret: 6244 Recall Level Averages Recall Precision 0.00 0.7739 0.10 0.5601 0.20 0.4537 0.30 0.4042 0.40 0.3458 0.50 0.2860 0.60 0.2074 0.70 0.1708 0.80 0.1180 0.90 0.0668 1.00 0.0132 erage precision over all 1 evant docs n-interpolated 0.2905 )ut Level Averages out Recall 000 0.3188 020 0.3966 040 0.4374 060 0.4733 080 0.4980 100 0.5403 120 0.5587 140 0.5765 160 0.5927 ~180 0.6044 `200 0.6169 Document Level Averages Precision At 5 docs 0.6000 At 10 docs 0.5640 At 15 docs 0.5427 At 20 docs 0.5300 At 30 docs 0.5000 At 100 docs 0.3880 At 200 docs 0.3023 At 500 docs 0.1930 At 1000 docs 0.1249 1 0.8< 0.6 Precision 0.4 0.2 0 R-Precision (precision a+ter R 1 (where R is docs retrieved the number of ments)) relevant docu- 0.8 0.6 Exact 0.3375 NORMAL DEVIATE - Fallout-Recall Recall Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0.98 0. 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I 0.001 -3 -2 -1 0 1 2 3 results - Cornell University Summary Statistics Recall-Precision Curve Run Number crnlRl-full, automatic 1 1 Num of Queries 50 Jotal number of documents over all queries Retrieved: 50000 0.8 Relevant: 10489 Rel~ret: 7164 Recall Level Averages Recall Precision 0.00 0.8268 0.10 0.1060 0.20 0.6282 0.30 0.5636 0.40 0.4900 0.50 0.4246 0.60 0.3416 0.10 0.2483 0.80 0.1543 0.90 0.0615 1.00 0.0062 .`rage precision over all vant docs -interpolated 0.3952 Document Level Averages Precision At 5 docs 0.6800 At 10 docs 0.6180 At 15 docs 0.6640 At 20 docs 0.6560 At 30 docs 0.6241 At 100 docs 0.4950 At 200 docs 0.3914 At 500 docs 0.2526 At 1000 docs 0.1553 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.4213 0.6 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I it Level Averages ~ut Recall ~00 0.4126 `20 0.5260 `40 0.6041 `60 0.6485 `80 0.6786 00 0.1056 20 0.1241 40 0.1433 60 0.1588 80 0.1685 O0 0.1194 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 101 0.02 0.16 5 0.84 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 suits - Cornell University Summary Statistics Recall-Precision Curve Run Number crnlCl-full1 automatic 1 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10489 Reljet: 7808 0.6 call Level Averages Recall Precision 0.00 0.8355 0.10 0.1202 0.20 0.6516 0.30 0.5816 0.40 0.5142 0.50 0.4518 0.60 0.3542 0.70 0.2519 0.80 0.1547 0.90 0.0677 1.00 0.0062 ige precision over all 1 mt docs j ~terpolated 0.4091 J Level Averages Recall ) 0.4336 ) 0.5602 ~ 0.6110 ) 0.6592 ~ 0.6886 ~ 0.7144 ~ 0.7338 ~ 0.7489 D 0.1588 D 0.7681 D 0.7763 Document Level Averages Precision Precision 0.4 At 5 docs 0.7240 At 10 docs 0.7000 0.2 At 15 docs 0.6840 At 20 docs 0.6660 0 0.2 0.4 0.6 0.81 At 30 docs 0.6240 0 At 100 docs 0.5156 At 200 docs 0.4104 Recall At 500 docs 0.2570 At 1000 docs 0.1562 Fallout-Recall Curve R-Precision (precision afjer R 1 docs retrieved (where R is relevant docu- the number of ments)) Exact 0.4367 NORMAL DEVIATE - F~1out-Reca11 0.8 0.6 - Recall Ol 0.02 0.16 0.98 0 0.4~ 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 snlts - University of California, Berkeley Summary Statistics Run Number Brkly4-full, automatic Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10489 Rel~ret: 6339 0.6 ~call Level Averages Recall Precision 0.00 0.7606 0.10 0.5512 0.20 0.4543 0.30 0.4004 0.40 0.3522 0.50 0.2797 0.60 0.2032 0.70 0.1665 0.80 0.1166 0.90 0.0631 1.00 0.0147 Ige precision over all mt docs nterpolated 0.2905 Document Level Averages Precision Precision 0.4 At 5 docs 0.6080 At 10 docs 0.5920 0.2 At 15 docs 0.5707 At 20 docs 0.5580 At 30 docs 0.5160 0 At 100 docs 0.3934 At 200 docs 0.3117 At 500 docs 0.1984 At 1000 docs 0.1268 ~-~recision (precision after R 1 docs retrieved (where R is relevant docu- the number of ments)) Exact 0.3332 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I Level Averages Recall () 0.3395 D 0.4263 0 0.4791 0 0.5212 0 0.5500 0 0.5747 0 0.5939 0 0.6096 0 0.6255 0 0.6388 0 0.6530 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 31 ol 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ~sults - University of California, Berkeley Summary Statistics Run Number N um of Queri 50 Total number Retrieved: 50000 Relevant: 10489 fRel~ret:7237I 1 0.8< 0.6 - Recall-Precision Curve KY ecall ~vel Averages Recall Precision 0.00 0.8361 0.10 0.6331 0.20 0.5384 0.30 0.4781 0.40 0.4231 0.50 0.3611 0.60 0.2812 0.70 0.2400 0.80 0.1714 0.90 0.0928 1.00 0.0190 rage precision over all ~ant docs -interpolated 0.3538 it Level Averages ut Recall 00 0.3969 20 0.4892 40 0.5489 160 0.5881 180 0.6227 .00 0.6458 L20 0.6690 140 0.6864 L60 0.7007 [80 0.7158 ~0 0.1277 Document ~vel Averages Precision At 5 docs 0.6600 At 10 docs 0.6360 At 15 docs 0.6160 At 20 docs 0.6040 At 30 docs 0.5753 At 100 docs 0.4608 At 200 docs 0.3650 At 500 docs 0.2310 At 1000 docs 0.1447 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Precision 0.4 0.2 0 1 0.8 0.6 Exact 0.3852 NORMAL DEVIATE - Fallout-Recall Recall 2 1 0.84 0 0.5 0 -1 0.16 -2 0.02 -3 0.84 0.98 0 0.4 0.2 0.98 3 -2 -1 0 1 2 3 0.001 0 `).2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 `esults - Universitaet Dortmund [ Summary Statistics Run Number dortPl-full, automatic Recall-Precision Curve Num of Queries 50 1 L Total number of documents over all queries Retrieved: 50000 0.8~ Relevant: 10489 Rel~ret: 7148 { 0.6 ecall Level Averages Document Level Averages Precision Recall Precision _______________ Precision 0.4 0.00 0.7899 At 5 docs 0.6600 0.10 0.6598 At 10 docs 0.6380 0.20 0.5844 At 15 docs 0.6387 0.2 0.30 0.5331 At 20 docs 0.6300 0.40 0.4765 At 30 docs 0.6080 0 0.50 0.4134 At 100 docs 0.4816 0.60 0.3289 At 200 docs 0.3904 0.70 0.2502 At 500 docs 0.2500 0.80 0.1665 At 1000 docs 0.1550 0.90 0.0801 R-Precision (precision after R 1.00 0.0041 docs retrieved (where R is 1 Ige precision over all 1 the number of relevant docu- Int docs ments)) 0.8 nterpolated 0.3800 Exact 0.4195 Level Averages 0.6 NORMAL DEVIATE - Fallout-Recall Recall ) 0.3933 ) 0.5213 ) 0.5807 0.7370 0.7511 0.7666 0.7743 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I Recall 3:1 ol 0.98 0 0.4. 2 0.98 0.2 1 0.84 0 0.5 0 -1 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ~su1ts - Universitaet Dortmund L Summary Statistics I Recall-Precision Curve F Run Number dortVl-full, automatic ~ 1 ~ Num of Queries 50 j Total number of documents over all queries 1 0.8 Retrieved: 50000 1 Relevant: 10489 Rel~ret: 7386 0.6 ecall Level Averages Recall Precision 0.00 0.7864 0.10 0.6082 0.20 0.5373 0.30 0.5023 0.40 0.4446 0.50 0.3788 0.60 0.2997 0.70 0.2415 0.80 0.1467 0.90 0.0590 1.00 0.0049 age precision over all ant docs interpolated 0.3516 t Level Averages it Recall )0 0.3912 )~0 0.4894 2 ~ 0.5521 ,0 0.5974 1 30 0.6316 0 )0 0.6602 ~0 0.6863 -1 ~ 0.7020 50 0.7152 -2 30 0.7279 -3 ~ 0.7389 Document Level Averages Precision Precision 0.4 At S docs 0.6280 At 10 docs 0.5980 0.2 At 15 docs 0.5960 At 20 docs 0.5790 At 30 docs 0.5653 0 At 100 docs 0.4520 At 200 docs 0.3661 At 500 docs 0.2352 At 1000 docs 0.1477 R-Precision (precision after R 1 docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.3947 NORMAL DEVIATE - Fallout-Recall Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I 0.6 - 0.84 0.98 0 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 I I I 0.001 -3 -2 -1 0 1 3 ~su1ts - University of Massachusetts at Amherst [ Summary Statistics [ Run Number INQOO3-full, automatic Num of Queries 50 Total number of documents over all queries Retrieved: 50000 Relevant: 10489 Reljet: 1161 ecall Level Averages Recall Precision 0.00 0.1852 0.10 0.6315 0.20 0.5296 0.30 0.4162 0.40 0.4246 0.50 0.3621 0.60 0.2195 0.10 0.2309 0.80 0.1135 0.90 0.1016 1.00 0.0122 `age precision~ over all `ant docs interpolated 0.3538 t Level Averages it Recall )0 0.3148 ~0 0.4880 10 0.5412 )0 0.5823 ~0 0.6106 )0 0.6341 ?0 0.6516 10 0.6180 )0 0.6926 30 0.1063 )0 0.1155 Document Level Averages Precision At S docs 0.6440 At 10 docs 0.6320 At 15 docs 0.6021 At 20 docs 0.5820 At 30 docs 0.5593 At 100 docs 0.4452 At 200 docs 0.3493 At 500 docs 0.2295 At 1000 docs 0.1432 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 1 0.8 0.6 Precision 0.4 0.2 0 1 0.8 Exact 0.3893 NORMAL DEVIATE - Fallout-Recall Recall Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I -` 0.6 - ~1 0~02 5 0.84 0.98 0 0.4 0.2 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I I 0.001 -3 -2 -1 0 1 3 ~esn1ts - University of Massachusetts at Amherst Summary Statistics ] I Run Number INQOO4-fulI, manual 1 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10489 Rel~ret: 7386 0.6 Recall-Precision Curve Recall Level Averages Document Level Averages Precision Recall Precision Precision 0.4 0.00 0.8489 At S docs 0.6680 0.10 0.6450 At 10 docs 0.6580 0.2 0.20 0.5327 At 15 docs 0.6200 0.30 0.4821 At 20 docs 0.5940 0 0.2 0.4 0.6 0.81 0.40 0.4304 At 30 docs 0.5747 0 0.50 0.3633 At 100 docs 0.4474 0.60 0.3054 At 200 docs 0.3547 Recall 0.70 0.2325 At 500 docs 0.2360 0.80 0.1762 At 1000 docs 0.1477 Fallout-Recall Curve 0.90 0.1054 R-Precision (precision after R 1 1.00 0.0151 docs retrieved (where R is ..rage precision over all the number of relevant docu- vant docs ments)) 0.8 -interpolated 0.3612 Exact 0.3951 Jt Level Averages ut Recall ~ 0.3956 20 0.4966 40 0.5621 60 0.6083 80 0.6396 .00 0.6649 .20 0.6860 .40 0.7025 .60 0.7203 .80 0.7313 ~00 0.7435 NORMAL DEVIATE Fallout-Recall 0.6 Recall )01 0.98 0 )~999 0.4~ 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 results - HNC, Inc Summary Statistics Run Number HNCrt1-full, automatic Recall-Precision Curve Num of Queries 50 J 1 Total number of documents over all queries 1 Retrieved: 49970 1 0.8~ Relevant: 10489 Rel~ret: 5833 0.6 Recall Level Averages Recall Precision 0.00 0.7867 0.10 0.5873 0.20 0.5083 0.30 0.4187 0.40 0.3311 0~50 0.2691 0.60 0.1962 0.70 0.1130 0.80 0.0451 0.90 0.0180 1.00 0.0023 erage precision over all -vant docs ri-interpolated 0.2810 ut Level Averages )ut Recall ~ 0.3211 D20 0.3893 ~40 0.4274 ~0 0.4554 D80 0.4791 ~0 0.5037 120 0.5152 ~0 0.5260 ~0 0.5373 ~0 0.5458 200 0.5504 Document Level Averages Precision At 5 docs 0.6120 At 10 docs 0.6020 At 15 docs 0.5853 At 20 docs 0.5740 At 30 docs 0.5480 At 100 docs 0.4228 At 200 docs 0.3223 At 500 docs 0.1924 At 1000 docs 0.1167 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3383 Precision 0.4 0.2 0 1 0.8 0.b 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I y NORMAL DEVIATE - Fallout-Recall Ol 0.02 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 ________________ I I 0.001 Recall -3 -2 -1 0 1 2 3 results - HNC, Inc. Summary Statistics Run Number HNCrt2-full, automatic 1 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8~ Relevant: 10489 Rel~ret: 6407 Recall Level Averages Recall Precision 0.00 0.8028 0.10 0.5734 0.20 0.5058 0.30 0.4316 0.40 0.3134 0.50 0.2933 0.60 0.2190 0.70 0.1373 0.80 0.0535 0.90 0.0137 1.00 0.0029 erage precision over all evant docs n-interpolated 0.2921 )ut Level Averages out Recall 000 0.3238 020 0.4185 040 0.4718 `060 0.5285 `080 0.5523 100 0.5731 `120 0.5884 140 0.6024 160 0.6117 180 0.6200 ~200 0.6281 0.6 Document Level Averages Precision Precision 0.4 At 5 docs 0.6200 At 10 docs 0.6100 0.2 At 15 docs 0.5827 At 20 docs 0.5560 At 30 docs 0.5300 0 At 100 docs 0.4090 At 200 docs 0.3296 At 500 docs 0.2095 At 1000 docs 0.1281 R-~recision (precision after R 1 docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.3638 NORMAL DEVIATE - Fallout-Recall Recall 0.6 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I )01 0.02 0.98 0 0.4 2 0.98 0.2 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 I I I I 0.001 -3 -2 -1 0 1 2 3 ~su1ts - Thinking Machines Corp. Summary Statistics Run Number TMC6-full, automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 46075 0.8 - Relevant: 10489 Rel~ret: 6061 ~ecall Level Averages Recall Precision 0.00 0.6276 0.10 0.4638 0.20 0.4142 0.30 0.3724 0.40 0.3194 0.50 0.2670 0.60 0.1950 0.70 0.1482 0.80 0.1123 0.90 0.0547 1.00 0.0211 age precision over all 1 ant docs interpolated 0.2553 J Level Averages t Recall ~0 0.2924 0 0.3747 0 0.4267 0 0.4887 0 0.5237 0 0.5472 0 0.5666 D 0.5883 0 0.6021 ~ 0.6172 ~ 0.6274 Document Level Averages Precision At 5 docs 0.4320 At 10 docs 0.4180 At 15 docs 0.4187 At 20 docs 0.4200 At 30 docs 0.4060 At 100 docs 0.3396 At 200 docs 0.2839 At 500 docs 0.1868 At 1000 docs 0.1213 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) L Exact 0.2991 NORMAL DEVIATE - Fallout-Recall *( -.- ). ( )01 0.02 0.16 5 0.84 0.98 0 2 1 -1 0 -2 -3 )~999 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 -3 -2 -1 0 1 3 0.6 Precision 0.4 0.2 0 1 0.8 0.6 0.4 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I re8ults - Thinking Machines Corp. Summary Statistics Recall-Precision Curve Run Number TMC1-full1 automatic 1 1 -I Num of Queries 50 Total number of documents over all queries Retrieved: 46122 0.8 Relevant: 10489 Reljet: 5112 0.6 - Recall Level Averages Recall Precision 0.00 0.5085 0.10 0.3828 0.20 0.3318 0.30 0.3064 0.40 0.2632 0.50 0.1964 0.60 0.1581 0.10 0.1240 0.80 0.0184 0.90 0.0453 1.00 0.0202 brage precision over all 1 `vant docs j i-interpolated 0.2045 j Document Level Averages Precision At5docs 0.3120 At 10 docs 0.3360 At 15 docs 0.3341 At 20 docs 0.3360 At 30 docs 0.3281 At 100 docs 0.2920 At 200 docs 0.2440 At 500 docs 0.1688 At 1000 docs 0.1142 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2564 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I 1 0.8 - ut Level Averages ut Recall ~ 0.2333 ~20 0.3224 ~0 0.3114 ~0 0.4123 ~0 0.4433 .00 0.4121 .20 0.4961 .40 0.5158 .60 0.5360 .80 0.5482 ~0 0.5611 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 101 0.02 0.16 ( 5 flR4 0.98 0 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 results - Beilcore Summary Statistics ) Run Number Isiri-full, automatic 1 Recall-Precision Curve 1 L Num of Queries 50 Total number of documents over all queries 1 Retrieved: 50000 0.8 Relevant: 10489 Rel~ret: 6522 J Recall Level Averages Recall Precision 0.00 0.7569 0.10 0.5281 0.20 0.4302 0.30 0.3779 0.40 0.3222 0.50 0.2446 0.60 0.1935 0.70 0.1474 0.80 0.0847 0.90 0.0295 1.00 0.0043 rage precision over al 1 `,ant docs 0.266: -interpolated it Level Averages Lit Recall 0~0 0.3042 20 0.4031 40 0.4660 60 0.5058 80 0.5365 o0 0.5635 20 0.5838 40 0.6034 0.6276 BO 0.6431 0.6554 Document Level Averages Precision At 5 docs 0.6120 At 10 docs 0.5480 At 15 docs 0.5387 At 20 docs 0.5110 At 30 docs 0.4880 At 100 docs 0.3798 At 200 docs 0.3037 At 500 docs 0.2011 At 1000 docs 0.1304 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3050 NORMAL DEVIATE - Fallout-Recall 0.6 Precision 0.4 0.2 0 1 0.8 0.6 Recall )01 0.02 0.84 0.98 0 0.4 0.2 - 2 0.98 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 3 ~u1ts - Beilcore Summary Statistics Run Number lsir2-full, automatic I 1 Num of Queries 50 j Total number of documents over all queries 1 0.8 Retrieved: 50000 Relevant: 10489 Rel~ret: 1155 J 0.6 call Level Averages ~ecall Precision 0.00 0.8533 0.10 0.6270 0.20 0.5435 0.30 0.4122 0.40 0.4080 0.50 0.3546 0.60 0.2913 0.70 0.2032 0.80 0.1282 0.90 0.0717 1.00 0.0044 ige precision over all tnt docs ,terpolated 0.3442 Level Averages Recall 0.3478 0.4676 0.5269 0.5817 0.6136 0.6407 0.6632 0.6192 0.6973 0.7079 0.7174 Document Level Averages Precision At 5 docs 0.6960 At 10 docs 0.6660 At 15 docs 0.6253 At 20 docs 0.5960 At 30 docs 0.5740 At 100 docs 0.4524 At 200 docs 0.3562 At 500 docs 0.2274 At 1000 docs 0.1431 R- Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3804 NORMAL DEVIATE - Fallout-Recall )01 0.02 0.16 ( 5 0.84 0.98 0 ½(½ 2 1 Precision 0.4 0.2 0 1 0.8 0.6 Recall ~999 0.4 0.98 0.2 - 0.84 0 0.5 0 -1 0.16 -2 0.02 -3 -7 0.001 I I I I -3 -2 -~ 0 1 2 3 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 ~su1ts - New York University Summary Statistics Run Number nyuirl-category B, automatic 1 Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 2064 Rel~ret: 1390 `call Level Averages Recall Precision 0.00 0.5400 0.10 0.3937 0.20 0.3423 0.30 0.2512 0.40 0.2263 0.50 0.2032 0.60 0.1614 0.10 0.1295 0.80 0.0905 0.90 0.0442 1.00 0.0284 ge precision over all nt docs iterpolated 0.2038 Level Averages Recall 1) 0.2413 0.2950 2 0.3314 0.3126 1 0.3872 0 0.4046 0.4181 -1 0.4328 0.4415 -2 0.4600 0.4698 Document Level Averages Precision At 5 docs 0.3360 At 10 docs 0.3240 At 15 docs 0.2933 At 20 docs 0.2790 At 30 docs 0.2401 At 100 docs 0.1412 At 200 docs 0.0939 At 500 docs 0.0489 At 1000 docs 0.0218 R-Precision (precision after R docs retrieved (where R is 0.6 Precision 0.4 0.2 0 1 the number of relevant docu- ments)) 0.8 - Exact 0.2267 0.6 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I NORMAL DEVIATE - Fallout-Recall 0.84 0.98 0 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 Recall -3 -2 -1 0 1 2 3 suits - New York University L Summary Statistics j Run Number nyuir2-category B, automatic 1 Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 2064 Rel~ret: 1610 0.6 call Level Averages ~ecall Precision 0.00 0.6435 0.10 0.4610 0.20 0.3105 0.30 0.3031 0.40 0.2631 0.50 0.2282 0.60 0.1934 0.10 0.1542 0.80 0.1002 0.90 0.0456 1.00 0.0186 ~e precision over all ~t docs terpolated 0.2331 Document Level Averages Precision At S docs 0.4280 At 10 docs 0.4000 At 15 docs 0.3613 At 20 docs 0.3260 At 30 docs 0.2740 At 100 docs 0.1108 At 200 docs 0.1018 At 500 docs 0.0515 At 1000 docs 0.0322 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.2513 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I evel Averages Recall 0.2600 0.3484 0.3946 0.4336 0.4551 0.4858 0.5033 0.5126 0.5246 0.5355 0.5486 NORMAL DEVIATE - Fallout-Recall 0.6 Recall 101 a.02 0.84 0.98 0 111111999 0.4 2 0.98 0.2 1 0~84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 _______________ I I 0.001 -3 -2 -1 0 1 2 3 ~sults - Carnegie Mellon University Summary Statistics [Run Number CLARTA-full, automatic Recall-Precision Curve [ Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 L Relevant: 10489 Rel~ret: 6811 0.6 ~call Level Averages Document Level Averages Precision Recall Precision Precision 0.4 0.00 0.8101 At 5 docs 0.6200 0.10 0.5781 At 10 docs 0.6060 0.20 0.4941 At 15 docs 0.5893 0.2 0.30 0.4368 At 20 docs 0.5730 0 0.2 0.4 0.6 0.81 0.40 0.3851 At 30 docs 0.5460 0 0.50 0.3358 At 100 docs 0.4346 0.60 0.2749 At 200 docs 0.3416 Recall 0.70 0.2074 At 500 docs 0.2184 0.80 0.1561 At 1000 docs 0.1362 0.90 0.0933 Fallout-Recall Curve R-Precision {precision after R 1.00 0.0141 1 docs retrieved (where R is ge precision over all the number of relevant docu- nt docs ments)) 0.8 iterpolated 0.3269 Exact 0.3646 Level Averages Recall 0.3545 0.4571 0.5408 0.5801 0.6132 0.6422 0.6603 0.6822 0.6959 0.7096 0.7172 0.6 NORMAL DEVIATE - Fallout-Recall 101 0.02 0.84 0.98 0 0.4 0.2 Recall 2 0.98 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 ~q~iltg - Garne~e Mellon University Summary Statistics Run Number CLARTM-full, manual Num of Queries 501 Total number of documents over all queries Retrieved: 50000 Relevant: 10489 Reljet: 6785 Document Level Averages ~ecall Level Averages Precision Recall Precision At 5 docs 0.6120 0.00 0.7191 At 10 docs 0.5940 0.10 0.5825 At 15 docs 0.5800 0.20 0.4987 At 20 docs 0.5700 0.30 0.4505 At 30 docs 0.5527 0.40 0.3848 At 100 docs 0.4396 0.50 0.3295 At 200 docs 0.3436 0.60 0.2739 At 500 docs 0.2176 0.70 0.2109 At 1000 docs 0.1357 0.80 0.1683 0.90 0.0951 R-Precision (precision after R 1.00 0.0126 docs retrieved (where R is the number of relevant docu- precision over all ments)) vant docs i-interpolated 0.3302 Fxact 0.3642 NORMAL DEVIATE - Fallout-Recall ut Level Averages )ut Recall 001 0.02 0.16 C 0.84 0.98 0.9~999 )00 0.3602 ~20 0.4615 2 0.98 ~0 0.5383 060 0.5783 1 0.84 080 0.6143 0 0.5 100 0.6371 120 0.6611 -1 0.16 140 0.6781 160 0.6946 -2 0.02 `180 0.7071 -3 0.001 200 0.7199 -3 -2 -1 0 1 2 3 Recall-Precision Curve 1 0.8 0.6 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 0.8 0.6 Recall 0.4 0.2 0 0 0.05 0.15 0.2 0.1 Fallout x 100 g results - Institute for Decision Systems Research Su~y Statistics Run Number idsra2-category B, partial topics, manual Num of Queries 10! Total number of documents over all queries Retrieved: 10000 fRelevant: 423 IRel~ret: 267 Recall Level Averages Recall Precision 0.00 0.5279 0.10 0.4577 0.20 0.4183 0.30 0.3665 0.40 0.3159 0.50 0.2956 0.60 0.2425 0.70 0.1672 0.80 0.0971 0.90 0.0115 1 00 0.0000 precision over all ~!~docsI i-interpolated 0.2504 Document Level Averages Precision At 5 docs 0.4000 At 10 docs 0.4200 At 15 docs 0.3933 At 20 docs 0.3600 At 30 docs 0.3333 At 100 docs 0.1870 At 200 docs 0.1040 At 500 docs 0.0482 At 1000 docs 0.0267 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3085 it Level Averages NORMAL DEVIATE - Fallout-Recall ut Recall ~.001 0.02 0.16 0.5 0.84 0.98 0.9~999 00 0.4079 20 0.5099 2 0.98 40 0.5397 1 50 0.5591 0.84 ~0 0.5642 )0 0.5717 0 0.5 ~0 0.5834 -1 0.16 10 0.5834 )0 0.5920 -2 0.02 10 0.5927 ~0 0.5979 -3 0.001 -3 -2 -1 0 1 2 3 1 0.8 0.6 Precision < 0.4 0.2 0 Recall-Precision Curve I I 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 1 0.8 0.6 Recall 0.4 0.2 0 0 0.05 0.1 0.15 0.2 Fallout x 100 `esults - Swiss Federal Institute of Technology (ETH) [ Summary Statistics I I Run Number schaul-full, automatic 1 L Num of Queries 50 Total number of documents over all queries 1 Retrieved: 50000 1 Relevant: 10489 Reljet: 5345 j Recall Level Averages Recall Precision 0.00 0.5925 0.10 0.4192 0.20 0.3618 0.30 0.3083 0.40 0.2453 0.50 0.1982 0.60 0.1319 0.10 0.0824 0.80 0.0485 0.90 0.0165 1.00 0.0023 `rage precision over all vant docs i-interpolated 0.1994 ut Level Averages ~t Recall ~ 0.2488 120 0.3126 140 0.3618 160 0.4031 )80 0.4549 LOO 0.4855 L20 0.5102 L40 0.5341 L60 0.5501 L80 0.5684 ?00 0.5811 Document Level Averages Precision At 5 docs 0.4000 At 10 docs 0.3960 At 15 docs 0.3680 At 20 docs 0.3550 At 30 docs 0.3500 At 100 docs 0.2910 At 200 docs 0.2350 At 500 docs 0.1561 At 1000 docs 0.1069 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 1 0.8 0.6 Precision 0.4 0.2 0 Fxact 0.2108 NORMAL DEVIATE - Fallout-Recall Recall 0.02 0.16 C 5 1 0.8 0.6 0.84 0.98 0 0.4 0.2 2 0.98 Recall-Precision Curve 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Falloutx 100 -2 0.02 -3 I I I I 0.001 -3 -2 -1 0 1 2 3 results - The Environment Research Institute of Michi~an Summary Statistics Run Number erimrl-fulI, automatic Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 41829 0.8 Relevant: 10489 Rel~ret: 4152 0.6~ Recall Level_Averages Recall Precision 0.00 0.5948 0.10 0.3190 0.20 0.2334 0.30 0.1817 0.40 0.1280 0.50 0.0841 0.60 0.0441 0.70 0.0210 0.80 0.0115 0.90 0.0020 1.00 0.0020 erage precision over all evant docs n-interpolated 0.1219 ~t Level Averages )ut Recall 000 0.1771 020 0.2227 040 0.2581 060 0.2834 080 0.3169 100 0.3383 120 0.3545 140 0.3794 160 0.3886 180 0.3995 ~0 0.4104 Document Level Averages Precision At 5 docs 0.3720 At 10 docs 0.3580 At 15 docs 0.3347 At 20 docs 0.3340 At 30 docs 0.3113 At 100 docs 0.2304 At 200 docs 0.1802 At 500 docs 0.1203 At 1000 docs 0.0830 R-Precision (precision after docs retrieved (where R the number of relevant ments)) R Is docu- Exact 0.1814 Precision 0.4 0.2 0 1 0.8 0.6 NORMAL DEVIATE - Fallout-Recall p01 0.02 0.98 0 )~999 0.4 2 0.98 0.2 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I `esults - The Environment Research Institute of Michigan Summary Statistics Run Number erimr2-full, automatic 1 Num of Queries 50 Total number of documents over all queries Retrieved: 45274 0.8 - Relevant: 10489 Rel~ret: 4475 0.6 Recall-Precision Curve Recall Level Averages Recall Precision 0.00 0.6448 0.10 0.3426 0.20 0.2643 0.30 0.2098 0.40 0.1551 0.50 0.1116 0.60 0.0568 0.70 0.0300 0.80 0.0168 0.90 0.0074 1.00 0.0006 `rage precision over all vant docs -interpolated 0.1415 Jt Level Averages ut Recall ~ 0.1891 20 0.2408 ~0 0.2940 60 0.3245 ~0 0.3517 .00 0.3797 .20 0.3958 .40 0.4092 .60 0.4219 .80 0.4343 ~00 0.4445 Document Level Averages Precision At 5 docs 0.4720 At 10 docs 0.4240 At 15 docs 0.3933 At 20 docs 0.3810 At 30 docs 0.3420 At 100 docs 0.2524 At 200 docs 0.2012 At 500 docs 0.1394 At 1000 docs 0.0895 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Precision 0.4 0.2 0 1 0.8 Exact 6~2031 NORMAL DEVIATE - Fallout-Recall Recall 0 0.2 0.4 0.6 08 Recall Fallout-Recall Curve 0.6 - Ol 0.02 0.16 1 5 0.84 0.98 0. 0.4 2 0.98 0.2 1 0.84 I I 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 resjilts - GE Research and Development Center [ Summary Statistics ] [ Run Number gecrdl-full1 manual 1 Recall-Precision Curve 1 Num of Queries 50 Total number of documents over all queries Retrieved: 44004 0.8 Relevant: 10489 Rel~ret: 6118 ~ecall Level_Averages Recall Precision 0.00 0.8112 0.10 0.6131 0.20 0.5320 0.30 0.4518 0.40 0.3988 0.50 0.3305 0.60 0.2198 0.10 0.2196 0.80 0.1393 0.90 0.0868 1.00 0.0130 age precision over all 1 ant docs J interpolated 0.3308 ] Level Averages t Recall 0 0.3899 0 0.4141 0 0.5113 0 0.5400 0 0.5625 0 0.5186 0 0.5932 0 0.6086 0 0.6220 0 0.6291 0 0.6381 0.6 Document Level Averages Precision Precision 0.4 At 5 docs 0.6080 At 10 docs 0.6060 At 15 docs 0.5821 0.2 At 20 docs 0.5660 At 30 docs 0.5440 0 At 100 docs 0.4528 At 200 docs 0.3364 At 500 docs 0.2045 At 1000 docs 0.1236 R-Precision (precision after R 1 docs retrieved (where R is 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve II>¼½ the number of relevant docu- ments)) 0.8 Exact 0.3812 0.6 NORMAL DEVIATE - Fallout-Recall )01 5 0.84 o.gs ~~999 0.4< 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 Recall -3 -2 -1 0 1 2 3 results - GE Research and Development Center Summary Statistics Run Number gecrd2-full, automatic 1 Num of Queries 50 Total number of documents over all queries 0.8 Retrieved: 49261 Relevant: 10489 Rel~ret: 4485 0.6 Recall-Precision Curve Recall Level_Averages Recall Precision 0.00 0.6802 0.10 0.3796 0.20 0.3111 0.30 0.2581 0.40 0.2108 0.50 0.1600 0.60 0.1015 0.10 0.0611 0.80 0.0360 0.90 0.0150 1.00 0.0009 ~rage precision over all `vant docs i-interpolated 0.1772 ut Level Averages ~ Recall ~00 0.2109 020 0.2171 2 040 0.3179 060 0.3513 1 080 0.3156 0 100 0.3987 120 0.4178 -1 140 0.4340 160 0.4458 -2 180 0.4559 -3 200 0.4671 Document Level Averages Precision Precision 0.4 At 5 docs 0.4760 At 10 docs 0.4460 0.2 At 15 docs 0.4280 At 20 docs 0.4130 0 0.2 0.4 0.6 0.81 At 30 docs 0.3907 0 At 100 docs 0.2930 At 200 docs 0.2275 Recall At 500 docs 0.1385 At 1000 docs 0.0897 Fallout-Recall Curve R-Precision (precision after R 1 docs retrieved (where R is the number of relevant docu- ments)) 0.8 Exact 0.2342 NORMAL DEVIATE - Fallout-Recall Recall 0.6 - P01 0.84 0.98 0 )~999 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 -3 -2 -1 0 1 2 3 results - TRW Systems and Development Division Summary Statistics Run Number trwl-full, manual 1 Recall-Precision Curve Num of Queries 50 1 Total number of documents over all queries Retrieved: 36485 0.8 Relevant: 10489 Rel~ret: 5681 0.6 - ~ecall Level Averages Recall Precision 0.00 0.1181 0.10 0.4823 0.20 0.4319 0.30 0.3706 0.40 0.3093 0.50 0.2586 0.60 0.1878 0.70 0.1471 0.80 0.1094 0.90 0.0528 1.00 0.0113 age precision over all ant docs interpolated 0.2525 Document Level Averages ________________ Precision At 5 docs 0.4640 At 10 docs 0.4100 At 15 docs 0.4507 At 20 docs 0.4300 At 30 docs 0.4247 At 100 docs 0.3770 At 200 docs 0.2929 At 500 docs 0.1847 At 1000 docs 0.1136 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3220 Precision 0.4 0.2 0 1 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 0.6 L Level Averages NORMAL DEVIATE - Fallout-Recall t Recall 0.3071 0 0.4006 *0 0.4389 0 0.4745 0 0.5004 0 0.5218 0 0.5389 0 0.5507 0 0.5614 0 0.5685 0 0.5743 g) 101 0.02 0.98 0 )~999 2 0.98 Recall 0.4 0.2 - 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 esults - TRW Systems and Development Division L L Summary Statistics Run Number trw2-full, manual Num of Queries 50 Total number of documents over all queries Retrieved: 48432 Relevant: 10489 Rel~ret: 7383 ~ecall Level Averages Recall Precision 0.00 0.8320 0.10 0.6329 0.20 0.5431 0.30 0.4694 0.40 0.4096 0.50 0.3558 0.60 0.2848 0.70 0.2132 0.80 0.1499 0.90 0.0899 1.00 0.0210 rage precision over all ~ant docs -interpolated 0.3459 it Level Averages ut Recall 00 0.3768 20 0.4753 2 40 0.5289 `60 0.5694 1 `80 0.6029 0 .00 0.6321 .20 0.6599 -1 .40 0.6770 .60 0.6930 -2 .80 0.7035 -3 !00 0.7147 Document Level Averages Precision At 5 docs 0.6680 At 10 docs 0.6240 At 15 docs 0.6107 At 20 docs 0.5810 At 30 docs 0.5453 At 100 docs 0.4554 At 200 docs 0.3623 At 500 docs 0.2350 At 1000 docs 0.1477 R-Precision (precision after R (where R is relevant docu- docs retrieved the number of ments)) ~t03807 NORMAL DEVIATE - Fallout-Recall Recall 0.84 0.98. - 0 -3 -2 -1 0 1 2 3 Recall-Precision Curve 1 0.8< 0.6 Precision 0.4 0.2 I I 0 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 1 0.8 0.6 - 0.4< 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 results - Verity, Inc. ~um mary Statistics Run Number TOPIC2-full, manual Num of Queries 50 ] 1 Total number of documents over all queries Retrieved: 40869 0.8 Relevant: 10489 Rel~ret: 5600 Recall-Precision Curve Recall Level Averages Recall Precision 0.00 0.7300 0.10 0.5963 0.20 0.4923 0.30 0.4160 0.40 0.3450 0.50 0.2139 0.60 0.1183 0.10 0.1350 0.80 0.0963 0.90 0.0289 1.00 0.0120 .~rage precision over all .~ant docs i-interpolated 0.2830 ~t Level Averages ut Recall 00 0.3293 `20 0.4000 `40 0.4399 ~0 0.4617 80 0.4843 00 0.4966 20 0.5094 40 0.5243 60 0.5355 80 0.5500 DO 0.5631 0.6 - Document Level Averages Precision Precision 0.4 At 5 docs 0.6080 At 10 docs 0.5800 At 15 docs 0.5513 0.2 At 20 docs 0.5390 At 30 docs 0.5221 0 At 100 docs 0.4028 At 200 docs 0.3048 At 500 docs 0.1805 At 1000 docs 0.1120 R-Precision (precision after R docs retrieved (where R is 1 the number of relevant docu- ments)) 0.8 Fxact 0.3418 0.6 NORMAL DEVIATE - Fallout-Recall 5 0.84 0.98 0 0.4 0.2 - Recall 2 0.98 0 02 0.4 0.6 0.8 Recall Fallout-Recall Curve I I 1 0.84 0 0.5 0 -1 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 esults - Siemens Corporate Research, Inc. ~ary Statistics Run Number siemsl-full, automatic I Num of Queries 50 of documents over all queries 50000 Retrieved: 10489 Relevant: 6666 Rel~ret: I Recall Level Averages Recall Precision 0.00 0.1391 0.10 0.5851 0.20 0.5002 0.30 0.4330 0.40 0.3631 0.50 0.3050 0.60 0.2333 0.10 0.1555 0.80 0.0860 0.90 0.0214 1.00 0.0042 ~erage precision over all levant docs )n-interpolated 0.2984 Document Level Averages Precision At 5 docs 0.6080 At 10 docs 0.5960 At 15 docs 0.5693 At 20 docs 0.5450 At 30 docs 0.5260 At 100 docs 0.4010 At 200 docs 0.3229 At 500 docs 0.2092 At 1000 docs 0.1333 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.3482 NORMAL DEVIATE F~Iout~RecaII lout Level Averages ilout Recall ~.o0i 0.02 0.16 0.5 0.84 0.98 0.9~999 0000 0.3305 0020 0.4414 2 0.98 0040 0.4944 1 0.84 0060 0.5411 ~0080 0.5731 0 0.5 )0100 0.6022 )0120 0.6232 -1 0.16 )0140 0.6427 -2 0.02 ~0160 0.6628 D0180 0.6759 -3 0.001 00200 0.6865 3 -2 -1 0 1 2 3 1 0.8 0.6 Precision 0.4 0.2 0 1 0.8 0.6 Recall 0.4 02 0 Recall~Precision Curve 0 0.2 0.4 0.6 0.8 1 Recall Fallout-Recall Curve 0 0.05 0.1 0.15 0.2 Fallout x 100 results - Queens College, CUNY Run Number pircsl-full, automatic Recall-Precision Curve [ Num of Queries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 Relevant: 10489 Rel~ret: 5913 Recall Level Averages Recall Precision 0.00 0.6360 0.10 0.4732 0.20 0.3947 0.30 0.3523 0.40 0.2999 0.50 0.2457 0.60 0.1911 0.70 0.1443 0.80 0.1094 0.90 0.0571 1.00 0.0087 rage precision over all ~ant docs -interpolated 0.2488 t Level Averages it Recall ~0 0.2840 ~0 0.3622 10 0.4133 ,0 0.4448 30 0.4690 )0 0.4951 ~0 0.5229 ~ 0.5603 )0 0.5746 30 0.5853 )0 0.5942 0.6 Document Level Averages Precision Precision 0.4 At 5 docs 0.4920 At 10 docs 0.4840 At 15 docs 0.4547 0.2 At 20 docs 0.4420 At 30 docs 0.4147 0 At 100 docs 0.3382 At 200 docs 0.2758 At 500 docs 0.1828 At 1000 docs 0.1183 R-Precision (precision after R 1 docs retrieved (where R is relevant docu- the number of ments)) 0.8 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve Exact 0.2950 0.6 NORMAL DEVIATE - Fallout-Recall 101 0.02 0.16 0.98 0 1~999 0.4 2 0.98 0.2 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 Recall -3 -2 -1 0 1 2 3 esults - Queens College, CUNY Summary Statistics ] [ Run Number pircs2-full, automatic 1 L Num of Queries 501 Total number of documents over all queries 0.8 - Retrieved: 50000 Relevant: 10489 Rel~ret: 6135 0.6 Recall- Precision Curve ~ecall Level Averages Recall Precision 0.00 0.6609 0.10 0.4959 0.20 0.4145 0.30 0.3735 0.40 0.3219 0.50 0.2638 0.60 0.2046 0.70 0.1586 0.80 0.1199 0.90 0.0656 1.00 0.0085 rage precision over all ~ant docs -interpolated 0.2664 1 it Level Averages ut Recall 31 ~ 0.3010 `20 0.3821 2 `40 0.4263 60 0.4648 1 80 0.4954 0 .00 0.5175 .20 0.5603 -1 .40 0.5795 .60 0.5958 -2 .80 0.6056 -3 ~0 0.6155 Document Level Averages Precision Precision 0.4 At 5 docs 0.5440 At 10 docs 0.5120 0.2 At 15 docs 0.4907 At 20 docs 0.4690 0 0.2 0.4 0.6 0.81 At 30 docs 0.4460 0 At 100 docs 0.3588 At 200 docs 0.2895 Recall At 500 docs 0.1901 At 1000 docs 0.1227 Fallout-Recall Curve R-Precision (precision after R 1 (where R is of relevant docu- docs retrieved the number ments)) Exact 0.3155 NORMAL DEVIATE - Fallout-Recall .001 0.02 0.16 0.5 0.84 0.98 0.~ 4-4~r I I I ~7-- -3 -2 -1 0 2 3 Recall I~999 0.8 0.6 - 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 results - VPI & SU Summary Statistics J Run Number VTcms2-ful I, manual 1 Recall-Precision Curve Num of Queries 50 J 1 Total number of documents over all queries 1 Retrieved: 50000 0.8 Relevant: 19489 Reljet: 5898 Recall Level_Averages Recall Precision 0.00 0.7524 0.10 0.5342 0.20 0.4510 0.30 0.3828 0.40 0.3063 0.50 0.2539 0.60 0.1817 0.70 0.1270 0.80 0.0809 0.90 0.0496 1.00 0.0060 rage precision over all ~ant docs -interpolated 0.2681 J it Level Averages Lit Recall 00 0.3015 20 0.3795 40 0.4320 60 0.4660 BO 0.4986 DO 0.5250 20 0.5421 40 0.5577 60 0.5745 BO 0.5916 DO 0.6021 Document Level Averages Precision At 5 docs 0.6080 At 10 docs 0.5760 At 15 docs 0.5387 At 20 docs 0.5160 At 30 docs 0.4913 At 100 docs 0.3806 At 200 docs 0.2953 At 500 docs 0.1882 At 1000 docs 0.1180 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) 0.6 Precision 0.4 0.2 0 1 0.8 Exact 0.3143 0.6 NORMAL DEVIATE - Fallout-Recall Ol 0~02 0.84 0.98 0 0.4 0.2 - 2 0.98 Recall 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve I I I I 1 0.84 0 0.5 0 0 0.05 0.1 0.15 0.2 -1 0.16 Fallout x 100 -2 0.02 -3 0.001 -3 -2 -1 1 2 3 r~q.iiltq. - Advanced Decision Systems Summary Statistics maPonlYautomatioc Total number of documents over all queries Retrieved 40423 Relevant: 5677 Rel~ret: 822 Document Level Averages Precision At 5 docs 0.0560 At 10 docs 0.0480 At 15 docs 0.0573 At 20 docs 0.0640 At 30 docs 0.0687 At 100 docs 0.0468 At 200 docs 0.0336 At 500 docs 0.0239 At 1000 docs 0.0164 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.0390 Run Number Num of Queries Recall Level Averages Recall Precision 0.00 0.1620 0.10 0.0542 0.20 0.0468 0.30 0.0273 0.40 0.0201 0.50 0.0056 0.60 0.0056 0.70 0.0000 0.80 0.0000 0.90 0.0000 1.00 0.0000 ~erage precision over all levant docs )n-interpolated 0.0195 Precision iout Level Averages NORMAL DEVIATE - Fallout-Recall Recall lout Recall ooi 0.02 0.16 0.5 0.84 0.98 0.9~999 0000 0.0236 0020 0.0311 2 0.98 0040 0.0346 0060 0.0389 1 0.84 0080 0.0429 0 0.5 `0100 0.0457 0120 0.0476 -1 0.16 0140 0.0497 )0160 0.0526 -2 0.02 )0180 0.0539 -3 0.001 )0200 0.0553 -3 -2 -1 0 1 2 3 Recall-Precision Curve 1 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 0.8 - 0.6 0.4 0.2 0< 0 0.05 0.15 0.1 Fallout x 100 0.2 results - Advanced Decision Systems Summary Statistics Run Number ADS2-ap only, automatic 1 of Queries 50 Total number of documents over all queries 0.8 Retrieved: 33034 Relevant: 5671 Rel~ret:l468I 0.6 Recall-Precision Curve Recall level Averages Recall Precision 0.00 0.2199 0.10 0.1572 0.20 0.1304 0.30 0.1160 0.40 0.0885 0.50 0.0884 0.60 0.0114 0.70 0.0612 0.80 0.0362 0.90 0.0201 1.00 0.0196 ucrage precision over all 1 levant docs )n-interpolated 0.0821 ] lout Level Averages lout Recall 0000 0.0506 0020 0.0817 2 0040 0.1070 0060 0.1248 1 0080 0.1403 0 0100 0.1568 0120 0.1643 -1 0140 0.1687 0160 0.1742 -2 `0180 0.1801 -3 `0200 0.1813 Document Level Averages Precision Precision 0.4 At 5 docs 0.1280 At 10 docs 0.1260 0.2 ¼- ~ At 15 docs 0.1320 At 20 docs 0.1300 0 0.2 04 0.6 081 At 30 docs 0.1300 0 At 100 docs 0.1118 At 200 docs 0.0838 Recall At 500 docs 0.0530 At 1000 docs 0.0294 Fallout-Recall Curve R-Precision ~precision after R 1 docs retrieved (where R is of relevant docu- the number ments)) Exact 0.1092 NORMAL DEVIATE - Fallout-Recall Recall 0.8 - 0.6 101 0.02 0.98 0 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0.02 0.001 3 -2 -1 0 1 2 3 ;ults - Daihousie University Summary Statistics Run Number DalTx2-category B, manual 1 1 Num of Queries 50 Total number of documents over all queries 0.8 Retrieved: 11987 Relevant: 2064 Rel~ret: 881 0.6 call Level_Averages ~ecall Precision 0.00 0.2984 0.10 0.2029 0.20 0.1661 0.30 0.1415 0.40 0.1056 0.50 0.0714 0.60 0.0691 0.10 0.0488 0.80 0.0111 0.90 0.0002 1.00 0.0002 3ge precision over all ~nt docs nterpolated 0.0893 1 Level Averages t Recall 0 0.1023 .0 0.1559 ~ 0.2034 0 0.2353 ~o 0.2516 )0 0.2135 ~0 0.2181 LO 0.2854 )0 0.2890 30 0.3151 )0 0.3305 Document Level Averages Precision At 5 docs 0.1320 At 10 docs 0.1280 At 15 docs 0.1301 At 20 docs 0.1310 At 30 docs 0.1213 At 100 docs 0.1040 At 200 docs 0.0645 At 500 docs 0.0334 At 1000 docs 0.0114 Precision 0.4 0.2 0 ~L Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Fxact 0.1199 NORMAL DEVIATE - Fallout-Recall Recall Ol a.02 0.16 ( 5 0.84 0.98 0. 9~999 0.4 :1:1 7' 0.98 2 1 0.84 0 0.5 -1 0.16 -2 0.02 -3 -7" 0.001 I I _______________ 3 -2 -1 0 1 2 3 Recall-Precision Curve I I I 1 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve 1 I I 0.8 0.6 0.2 I I I 0 0 0.05 0.15 0.1 Fallout x 100 0.2 ~su1ts - Systems Environment Corporation Summary Statistics Run Number SEC-category B, automatic Num of Queries 50 ] 1 Total number of documents over all queries Retrieved: 19602 0.8 Relevant: 2064 Rel~ret: 37 Recall-Precision Curve `call Level_Averages Recall Precision 0.00 0.0213 0.10 0.0000 0.20 Q.0000 0.30 0.0000 0.40 0.0000 0.50 0.0000 0.60 0.0000 0.70 0.0000 0.80 0.0000 0.90 0.0000 1.00 0.0000 precision over all terpolateci~005J Level Averages Recall 0.0030 0.0046 0.0056 0.0057 0.0073 0.0080 0.0069 0.0071 0.0076 0.0080 0.0082 Document Level Averages Precision At 5 docs 0.0040 At 10 docs 0.0020 At 15 docs 0.0027 At 20 docs 0.0050 At 30 docs 0.0060 At 100 docs 0.0038 At 200 docs 0.0025 At 500 docs 0.0015 At 1000 docs 0.0007 { R- Precision (pThecisionafterR docs retrieved (where R is he number of relevant docu- j Exact 0.0042 ~ NORMAL DEVIATE - Fallout-Recall )01 0.02 0.16 .5 0.84 0.98 0 2 0.98 1 0.84 0 0.5 -1 0.16 -2 0.02 -3 0.001 -3 -2 -1 0 1 2 3 0.6 Precision 0.4 0.2 0< A A 0 0.2 0.4 0.6 0.8 1 Recall Recall Fallout-Recall Curve 1 0.8 0.6 0.4 0.2 0 0 0.05 0.1 0.15 0.2 Fallout x 100 suits - University of Central Florida Summary Statistics Run Number UCFIS1-category B, automatic 1 Num of Queries 50 J Total number of documents over all queries 0.8 Retrieved: 50000 Relevant: 2064 Rel~ret: 102 0.6 Recall-Precision Curve ~call Level_Averages Recall Precision 0.00 0.0891 0.10 0.0286 0.20 0.0062 0.30 0.0015 0.40 0.0015 0.50 0.0011 0.60 0.0000 0.10 0.0000 0.80 0.0000 0.90 0.0000 1.00 0.0000 3ge precision over all ~nt docs~ -¼nterpolatedo.00601 Document Level Averages Precision At 5 docs 0.0400 At 10 docs 0.0260 At 15 docs 0.0187 At 20 docs 0.0170 At 30 docs 0.0127 At 100 docs 0.0070 At 200 docs 0.0039 At 500 docs 0.0024 At 1000 docs 0.0020 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.0128 1 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0.8 Recall Fallout-Recall Curve A 1 0.8 - 0.6 A A _ Level Averages t Recall 0 0.0307 0 0.0339 .0 0.0369 0 0.0314 0 0.0384 0 0.0427 0 0.0427 ~ 0.0427 ~0 0.0436 10 0.0436 )0 0.0439 NORMAL DEVIATE - Fallout-Recall Recall Ol 0.02 0.84 0.98 0. 0.4 0.2 2 0.98 1 0.84 0 0.5 0< -1 0.16 -2 0.02 -3 I I 0.001 -3 -2 -1 0 1 2 3 I ~ ~ Y 0 0.05 0.1 0.15 0.2 Fallout x 100 results - University of Illinois at Chicago [ L Summary Statistics Run Numbe Recall-Precision Curve NumofQueries 50 1 Total number of documents over all queries Retrieved: 50000 0.8 - Relevant: 10489 Rel~ret: 3067 0.6 ~ecall Level Averages Recall Precision 0.00 0.3965 0.10 0.2189 0.20 0.1543 0.30 0.1025 0.40 0.0807 0.50 0.0578 0.60 0.0361 0.10 0.0195 0.80 0.0088 0.90 0.0000 1.00 0.0000 ~ge precision over all mt docs nterpolated 0.0780 Level Averages Recall ) 0.1231 ) 0.1693 0.1977 0.2278 0.2414 0.2664 0.2814 0.2918 0.3115 0.3272 0.3373 Document Level Averages Precision At 5 docs 0.1960 At 10 docs 0.2020 At 15 docs 0.2133 At 20 docs 0.2090 At 30 docs 0.2053 At 100 docs 0.1652 At 200 docs 0.1311 At 500 docs 0.0876 At 1000 docs 0.0613 R-Precision (precision after R docs retrieved (where R is the number of relevant docu- ments)) Exact 0.1440 NORMAL DEVIATE - Fallout-Recall ~.001 0.02 0.16 a 2 1 0 -1 -2 -3 .5 0.84 0.98 a 7 -3 -2 -1 0 1 2 3 Precision 0.4< 0.2 0 0 0.2 0.4 0.6 08 Recall Fallout-Recall Curve I - 1 0.8 - 0.6 ~~999 0.4 0.98 0.2 0.84 0.5 0 0 0.05 0.1 0.15 0.2 0.16 Fallout x 100 0~02 0.001 Recall