G u i d e l i n e s


Motivating principles

For TREC 2001 the Interactive Track will carry out observational studies which maximize the realism of the searching by allowing the use of data and search systems/tools publicly accessible via the Internet. This suggests that, to the extent possible, searchers be allowed to choose tasks and systems/tools for accomplishing those tasks.

At the same time, the studies for TREC 2001 should be designed to maximize the likelihood that groups will find in their observations the germ of a hypothesis they can test for TREC 2002. This suggests that there be restrictions - some across all sites, some only within a given site - to make it more likely that patterns will emerge. The restrictions will be formalized in two sorts of guidelines: one set for all sites and another set that applies only within a site.

Cross-site guidelines

Each site will observe as many searchers as possible and appropriate. A target number of 24 is suggested.

Each searcher will work in one or more of the following domains provided by the track to all sites:

Each searcher will carry out four searches - two from a list of fully specified tasks provided to all sites and two for which only the format is predetermined but which are otherwise up to the site/searcher to create.

Each site will collect a minimal standard set of data defined roughly by the track and covering searcher characteristics and satisfaction, effectiveness, and efficiency. These will be reported in the notebook papers if appropriate to the results. The searcher characteristics data will not be collected by NIST as in previous years. Here are some examples of the kind of information that is of interest. The questionnaires will not be usable as-is, if only because they say TREC-9.

Each site will collect at least the urls of all pages visited during all searches.

The only real submission required will be the notebook paper, which should include among other things a testable hypothesis for TREC-2002. Notebook papers will be due at NIST by around 30. October. Ellen Voorhees will send out information about how to submit them. NIST will also ask each participating group for a 100-word description of the approach they have taken and something about the results. This description should be emailed to paul.over@nist.gov by 30. October.

Tasks

Here are the eight fully specified tasks:

Here are the eight partially specified tasks:

Within-site guidelines

Within the cross-site guidelines, each site must impose further restrictions of its own choice on ALL its searchers to define an area of interest for observation - to be reported to the track before the observations begin. Each site may define its own time limits for searches.

For example, a site could impose inclusive or exclusive restrictions on any (combinations) of the following: the choice/assignment of domain from the 4 provided, the data to be searched, the search system/tools to be used (e.g., search systems, meta-search systems, directories,...), functionality within a given search system/tool, the characteristics of searchers, the time allowed, the pre- search training provided, etc. Sites are also encouraged to coordinate their plans with other sites, form small teams sharing guidelines, etc.

Each site will evaluate their searches using any criteria defined in the cross-site guidelines plus any site specific evaluations.

As part of the data analysis for TREC 2001, each site will attempt to formulate a testable hypothesis for TREC 2002 and will report this as part of the results for TREC 2001.


National Institute of Standards and Technology Home Last updated: Tuesday, 22-Sep-2015 09:46:10 EDT
Date created: Monday, 12-Mar-00
For further information contact trec@nist.gov