The Program Committee of the Text REtrieval Conference invites proposals for tracks for the next cycle of TREC, TREC 2027, which begins in January 2027 and will culminate in the physical conference in November 2027.
- Track proposals due Sep 27 AoE, notifications Oct 16
- All current tracks must propose to continue.
The Text REtrieval Conference (TREC) is a community-based evaluation workshop series focused on tasks in information retrieval and information access. The community evaluation process serves to build reusable resources that support retrieval research beyond the workshop itself.
TREC consists of a series of "tracks", a set of tasks each focused on some facet of the retrieval problem. Examples of recent tracks include retrieval-augmented generation, deep learning, conversational search, cross-language retrieval, and multimodal search. The tracks invigorate TREC by encouraging research in new areas of information retrieval. The tracks also support the retrieval research community by creating the infrastructure such as test collections necessary for task-specific research.
The TREC Program Committee seeks proposals for tracks to take place in TREC 2027 (January - November 2027). Track proposals may be no longer than four pages, and must include:
- A description of the retrieval task,
- What document collection will be used, and is it in hand? If not, will it be in hand by January 2027?
- How will relevance judgments be made? What special qualifications (if any) do the relevance assessors need? Are special assessment protocols proposed?
- Are automatic evaluation metrics/methods proposed, and how will their correctness be verified?
- What do we expect to learn from this track?
- Who will participate in the track?
- Whether the track will result in reusable retrieval collections,
- The names of the coordinators of the proposed track, and a description of each of their roles. (Bios are not necessary) Make sure each proposed coordinator has a clear role to play.
The set of tracks run in any particular year depends on the interests of the participants and sponsors, as well as on the suitability of the problem to the TREC environment. Proposals for tracks will be reviewed by the TREC Program Committee, and decisions will be shared in time for coordinators of new tracks to plan to attend this year's TREC Conference (November 16-18, 2026).
While NIST endeavors to support manual assessment for as many tracks as possible, it is not possible to predict our resources this far in advance. All track proposals must contain a concrete plan for topic development and relevance assessment that does not rely on NIST resources.
All current TREC tracks must submit a proposal to continue, and such proposals will be considered alongside new track proposals. Proposals for continuing tracks should indicate how the proposed iteration fits into an ongoing plan for the track. Generally, the PC will have to decide to end an existing track to accommodate a newly proposed track, so it is certainly possible that no new tracks will be selected for the coming year.
Automatic metrics and methods of assessment are, at present, research topics rather than accepted practice, but are necessary to support research on generative tasks. Tracks are welcome to propose and plan to use automated metrics, but must state how those measures will be validated. Correlation to past TREC system rankings is not sufficient to validate a new evaluation. Remember that participants expect to be evaluated, and evaluation results will be used in publications for years to come.
Track proposals must be submitted by September 27, 2026, AoE. Decisions will be returned by October 16, 2026.
The criteria for judging a track proposal are: a strong advocate who is willing to be the track coordinator (track coordinator is a volunteer position); a large enough core of interested researchers to make the track viable; the availability of sufficient resources such as appropriate corpora and assessors with expertise in the area; and the fit with other tracks.
Proposals need to contain enough information for the PC to assess the criteria above. Proposals should contain an explicit statement of the goals of the track (i.e., what is expected to be learned and/or what infrastructure would be created if the track were run). If relevance judging (or some similar sort of annotation) is required, the proposal needs to describe the assessment process, any special qualifications the assessors would need (such as special domain expertise), as well as an estimate of the amount of time such assessing would require. Any special constraints on the document sets needed should also be noted. If a track plans to use automatic evaluation methods including LLM-driven methods, the proposal must clearly state this and describe how the measures will be validated. Finally, proposals must contain full contact details of the proposer. On the flip side, proposals need to be concise and to-the-point: if your proposal is more than four pages, it is too long.
Send a proposal as a PDF document to [email protected]. Proposals in other formats will be bounced unopened. Aside from the length limit of four pages and the requirement of PDF, there are no formatting restrictions.
Ian Soboroff
Chair, TREC Program Committee
