Procedure for Proposing New TREC Tracks

Return to the TREC home page TREC home          National Institute of Standards and Technology Home Page

TREC consists of a series of "tracks", a set of tasks each focused on some facet of the retrieval problem. Examples of tracks include retrieval of speech documents, cross-language retrieval, retrieval of web documents, and question answering. The tracks invigorate TREC by encouraging research in new areas of information retrieval. The tracks also support the retrieval research community by creating the infrastructure such as test collections necessary for task-specific research.

The set of tracks run in any particular year depends on the interests of the participants and sponsors, as well as on the suitability of the problem to the TREC environment. Selecting which tracks to include in a particular TREC in the past was done informally. However, TREC is receiving increasingly many proposals for new tracks. With a limit of at most eight tracks that can be run at one time, the requests for new tracks have surpassed the available space. Thus, the TREC program committee has decided to institute a more formal proposal mechanism for selecting tracks.

Under this scheme, proposals for TREC tracks must be submitted in writing. For TREC 2022 tracks, proposals are due by September 19, 2021. The proposals will be distributed to the TREC program committee members shortly after the deadline, and the PC will decide on the set of tracks to run in year X+1 by the time of the conference in year X. All tracks (both existing and proposed) must submit a proposal by the deadline, and the PC will consider the entire set of tracks to produce the best possible track selection for the X+1 TREC year. Generally, the PC will have to decide to terminate an existing track to accommodate a newly proposed track, so it is certainly possible that no new tracks will be selected in some year.

The criteria for judging a track proposal remain the same as before: a strong advocate who is willing to be the track coordinator (track coordinator is a volunteer position); a large enough core of interested researchers to make the track viable; the availability of sufficient resources such as appropriate corpora and assessors with expertise in the area; and the fit with other tracks.

Proposals need to contain enough information for the PC to assess the criteria above. Proposals should contain an explicit statement of the goals of the track (i.e., what is expected to be learned and/or what infrastructure would be created if the track were run). If relevance judging (or some similar sort of annotation) is required, the proposal needs to include where the judging would occur (NIST or elsewhere?), any special qualifications the assessors would need (special domain expertise required?), as well as an estimate of the amount of time such assessing would require. Any special constraints on the document sets needed should also be noted. Finally, proposals must contain full contact details of the proposer. On the flip side, proposals need to be concise and to-the-point: if your proposal is more than four pages, it is too long.

Send a proposal as either a PDF or ASCII document to Proposals in other formats will be bounced unopened.

Ian Soboroff
Chair, TREC Program Committee

The TREC Conference series is co-sponsored by the NIST, Information Technology Laboratory's (ITL) Retrieval Group of the Information Access Division (IAD)

National Institute of Standards and Technology Home Page
is an agency of the U.S. Commerce Department

Last updated: Thursday, 19-Aug-2021 14:47:49 EDT
Date created: Wednesday, 02-Sept-25
privacy policy / security notice / accessibility statement
disclaimer / FOIA