TREC Statement on Product Testing and Advertising
The annual Text REtrieval Conference (TREC) is an event in which organizations
with an interest in information retrieval research take part in a coordinated
series of experiments using the same experimental data. The goal of the conference series
is to create the infrastructure necessary for large-scale evaluation of research retrieval
systems and thereby foster research into effective techniques for information access.
By design, TREC is explicitly not a venue for commercial product tests
(i.e., benchmark comparisons). A valid, informative vendor test requires a
level of control in task definition and system execution that is counter to
the scientific research goals of TREC. Insofar as TREC participants do
the same task, the results from different participating teams are comparable,
but interpretation of what those results actually represent may vary widely.
For example, commercial participants may submit results from research prototype
systems rather than their production system, or participants may deliberately
degrade one aspect of their system to focus on another aspect.
To preserve the desired, pre-competitive nature of the TREC conferences,
TREC requires all participants to sign and abide by
an agreement
concerning the dissemination and publication of TREC evaluation results. The guidelines embodied
in the agreement are meant to preclude the publication of incomplete or
inaccurate information that could damage the reputation of the conference or
its participants. In particular, the agreement prohibits any advertising based
on TREC results and sharply curtails the use of TREC results in marketing literature.
NIST and the TREC program committee are strongly committed to the ethos of
cooperation the guidelines are designed to engender, but cannot accept responsibility
for performance claims made by participants in violation of the agreement.
TREC reserves the right to prohibit violators from participating in future TREC conferences.
|
|
is an agency of the
U.S. Commerce Department
| |
Last updated: Tuesday, 23-Feb-2021 08:42:07 EST
Date created: Wednesday, January 25, 2012
privacy policy / security notice / accessibility statement
disclaimer / FOIA
|