APPENDIX B This appendix eontains tables of resUlts for the ~PSThR panel. The tables have the same descriptions as those in Appendix A. The resUlts in this appendix represent a second round of work on this task, as a previous evaluation of `IIPSTER resUlts had taken place in September. The TIPSThR contractors were able to use some (minimal) failure analyis to correct some of the easier problems found in that evaluation in time for these results. Whereas these results cannot be comp&ed directly with the ThEC-l results, they represent what can be accomplished in only one month of additional tuning for the task. 431 ults - University of Massachusetts at Amherst Summary Statistics Run number INQRYO-full, manual I Num~queries 50 Total number of documents over all queries Retrieved 10000 0.8 Relevant 16400 Rel~ret 4503 0.6 ~call Level Averages Precision 0 0.9448 0 0.6794 0 0.5250 0 0.3067 0 0.1668 0 0.1116 0 0.0926 0 0.0631 0 0.0337 0 0.0174 0 0.0000 ge precision for all points Avg 0.2674 J ge precision for 3 iediate points 0.50, 0.80) ~vg 0.2235 Document Level Averages Precision Recall Precision 0.4 at 5 docs 0.0184 0.7640 at 15 docs 0.0549 0.7293 0.2 at 30 docs 0.0992 0.6840 at 100 docs 0.2451 0.5546 at 200 docs 0.3572 0.4503 0 Recall- Precision Curve 0 0.2 0.4 0.6 ROC Averages Recall False Alarm Detection 0.00000 0.1250 NORMAL DEVIATE - Detection-False Alarm 0.00001 0.1654 0.001 0.02 0.16 C 5 0.84 0.98 0 0.00002 0.1957 3 0.00003 0.2203 ½<½ 0.00004 0.2409 2 0.00005 0.2576 0.00006 0.2738 1 0.00007 0.2882 0.00008 0.2994 0 0.00009 0.3113 -1 -~ -2 -3 __________________ I I -3 -2 -1 0 2 `its - HNC, Inc. [ Summary Statistics Run number HNCmn1~isk1 only, manual 1 Num~queries 50 Total number of documents over all queries Retrieved 9982 0.8 - Relevant 10477 Rel~et 2711 0.6 Recall-Precision Curve ~all Level Averages Precision 0.6569 0.4078 0.3005 0.2331 0.1540 0.0952 0.0338 0.0169 0.0101 0.0020 0.0020 r,e precision for all points ~vg 0.1739 v,e precision for 3 ediate points [).50, 0.80) Lvg 0.1352 Precision Document Level Averages Recall Precision 0.4 at 5 docs 0.0184 0.4400 at 15 docs 0.0469 0.4227 0.2 at 30 docs 0.0758 0.3847 at 100 docs 0.2098 0.3388 at 200 docs 0.3209 0.2711 0 0 0.2 0.4 0.6 0.8 ROC Averages Recall False Alarm Detection 0.00000 0.0726 0.00001 0.0967 0.00002 0.1039 0.00003 0.1287 0.00004 0.1301 0.00005 0.1518 0.00006 0.1677 0.00007 0.1837 0.00008 0.1857 0.00009 0.1995 NORMAL DEVIATE - Detection-False Alarm C I I 0 i01 3 2 1 0 -1 -2 .3 0.02 0.16 0.5 0.84 0.98 0 -3 -2 -1 0 1 2 3 results - University of Massachusetts at Amherst [ Summary Statistics Run number I NQ RYC-full, manual L Num~queries 50 1 Total number of documents over all queries 1 Retrieved 10000 1 0.8 Relevant 15459 *Rel~ret 4527 j 0.6 Recall Level Averages call Precision 00 0.8688 10 0.6770 20 0.5321 30 0.3515 40 0.1606 50 0.0703 60 0.0342 70 0.0000 80 0.0000 90 0.0000 90 0.0000 age precision for all points Avg 0.2449 ] age precision for 3 ~ediate points 0.50, 0.80) Avg 0.2008 Document Level_Averages Precision Recall Precision 0.4 at 5 docs 0.0202 0.7080 at 15 docs 0.0542 0.6893 0.2 at 30 docs 0.0947 0.6493 at 100 docs 0.2268 0.5478 at 200 docs 0.3457 0.4527 0 ROC Averages False Alarm Detection 0.00000 0.1161 0.00001 0.1544 0 0.00002 0.1833 3 0.00003 0.2082 0.00004 0.2262 2 0.00005 0.2414 0.00006 0.2537 1 0.00007 0.2670 0.00008 0.2797 0 0.00009 0.2899 -1 -2 -3 Recall-Precision Curve 0 0.2 0.4 0.6 0 Recall NORMAL DEVIATE - Detection-False Alarm .001 0.02 0.16 0.5 0.84 0.98 I I I I -3 -2 -1 0 1 2