Past Results

Results of previous Text REtrieval Conferences have been archived within a password protected area. These results include the participants' submissions in their raw form.

For document retrieval tasks, each result file is a list of the 1000 documents retrieved for each topic in order of decreasing similarity with the query. The associated output of the trec_eval (.gz) (trec_eval_latest.tar.gz) program is also included within that area. The TREC proceedings present the retrieval results in a finished form that is more appropriate for most people than the raw result files. In particular, the Overview paper in each Proceedings provides a general introduction to that years' results, and the trec_eval (.gz) (trec_eval_latest.tar.gz) output for each run is given in Appendix A in graphical form. People doing research on data fusion or similar techniques, however, may find the raw result files more useful.

Individuals may request access to the protected area containing the raw results by contacting the TREC Program Manager. Before receiving access, individuals will be asked to sign an agreement that acknowledges the limited uses for which the data can be used. The guidelines constraining the dissemination and publication of TREC evaluation results were created by the TREC program committee and apply to both TREC participants and any organizations requesting access to the raw TREC results. A synopsis of the guidelines is included below.

1. SCIENTIFIC OR TECHNICAL PUBLICATIONS: Scientific or technical publications, including newsletters from universities or research laboratories, should adhere to community standards for fairness and objectivity and should accurately and clearly state the limitations of the testing conditions and other factors which might influence scores. The experimental nature of the tasks, data and evaluation procedures should also be stated. The full conference proceedings should always be referenced.
2. ADVERTISEMENTS: No advertisements using the TREC evaluation results can be placed in magazines, journals, newspapers, or other publications.
3. PRESS RELEASES: Press releases about TREC results to organizations with national/international coverage are also prohibited.
4. MARKETING LITERATURE, LOCAL NEWSLETTERS: Although it is recognized that extensive evaluation discussions are not appropriate in this type of literature, it is expected that any claims made on the basis of evaluation results are accurate, that the evaluation measures used to substantiate these claims are stated, and that a reference is made to the full conference proceedings. Where promotional material is subject to prepublication revision by the media, the author should make every effort to see that the revision does not cause a violation of the guidelines.
5. WEB SITES: Web sites are a type of marketing literature with a global distribution. The guidelines for marketing literature in general apply to web sites as well. References to TREC should link to the TREC web site.
6. Cross-system comparisons may not be made with other named teams for individual tests, and may only be made when they are supported by accepted methods of statistical significance testing. Comparisons must be accompanied by the results of those tests and should reference the publication of those tests in the conference proceedings or related literature. Informal, qualitative comparisons with recognized baselines or benchmarks, and with general levels or trends in performance, must be clearly stated to be such and thus open to statistical reassessment.