Procedure for Proposing New TREC Tracks |
|---|
TREC home
|
|
TREC consists of a series of "tracks", a set of tasks each focused
on some facet of the retrieval problem. Examples of tracks include
retrieval of speech documents, cross-language retrieval, retrieval
of web documents, and question answering. The tracks invigorate
TREC by encouraging research in new areas of information retrieval.
The tracks also support the retrieval research community by
creating the infrastructure such as test collections necessary for
task-specific research. The set of tracks run in any particular year depends on the interests
of the participants and sponsors, as well as on the suitability of the
problem to the TREC environment. Selecting which tracks to include
in a particular TREC in the past was done informally. However,
TREC is receiving increasingly many proposals for new tracks. With
a limit of at most eight tracks that can be run at one time,
the requests for new tracks have surpassed the available space.
Thus, the TREC program committee has decided to institute a more formal
proposal mechanism for selecting tracks. Under this scheme, proposals for TREC tracks must be
submitted in writing.
For 2007, proposals are due by September 17. The proposals will
be distributed to the TREC program committee members shortly
after the deadline, and the PC will decide on the set of tracks
to run in year X+1 by the time of the conference in year X.
All tracks (both existing and proposed) must submit a proposal
by the deadline, and the PC will consider the entire set of tracks
to produce the best possible track selection for the X+1 TREC year.
Generally, the PC will have to decide to terminate an existing track
to accommodate a newly proposed track, so it is certainly possible
that no new tracks will be selected in some year. The criteria for judging a track proposal remain the same as before:
a strong advocate who is willing to be the track coordinator (track
coordinator is a volunteer position); a large enough core of
interested researchers to make the track viable; the availability
of sufficient resources such as appropriate corpora and assessors
with expertise in the area; and the fit with other tracks.
Proposals need to contain enough information for the PC to assess
the criteria above. Proposals should contain an explicit
statement of the goals of the track (i.e., what is expected to
be learned and/or what infrastructure would be created if the
track were run). If relevance judging (or some similar sort of
annotation) is required, the proposal needs to include where the
judging would occur (NIST or elsewhere?), any special qualifications
the assessors would need (special domain expertise required?),
as well as an estimate of the amount of time such assessing would
require. Any special constraints on the document sets needed
should also be noted. Finally, proposals must contain full
contact details of the proposer.
Send a proposal as either a postscript, PDF, or ASCII document
to trec@nist.gov. Proposals in formats other than these three
will be bounced unopened.
|
|
|