G u i d e l i n e s

Motivating principles

For TREC 2001 the Interactive Track will carry out observational studies which maximize the realism of the searching by allowing the use of data and search systems/tools publicly accessible via the Internet. This suggests that, to the extent possible, searchers be allowed to choose tasks and systems/tools for accomplishing those tasks.

At the same time, the studies for TREC 2001 should be designed to maximize the likelihood that groups will find in their observations the germ of a hypothesis they can test for TREC 2002. This suggests that there be restrictions - some across all sites, some only within a given site - to make it more likely that patterns will emerge. The restrictions will be formalized in two sorts of guidelines: one set for all sites and another set that applies only within a site.

Cross-site guidelines

Each site will observe as many searchers as possible and appropriate. A target number of 24 is suggested.

Each searcher will work in one or more of the following domains provided by the track to all sites:

• finding consumer medical information on a given subject
• buying a given item
• planning travel to a given place
• collecting material for a project on a given subject

Each searcher will carry out four searches - two from a list of fully specified tasks provided to all sites and two for which only the format is predetermined but which are otherwise up to the site/searcher to create.

Each site will collect a minimal standard set of data defined roughly by the track and covering searcher characteristics and satisfaction, effectiveness, and efficiency. These will be reported in the notebook papers if appropriate to the results. The searcher characteristics data will not be collected by NIST as in previous years. Here are some examples of the kind of information that is of interest. The questionnaires will not be usable as-is, if only because they say TREC-9.

Each site will collect at least the urls of all pages visited during all searches.

The only real submission required will be the notebook paper, which should include among other things a testable hypothesis for TREC-2002. Notebook papers will be due at NIST by around 30. October. Ellen Voorhees will send out information about how to submit them. NIST will also ask each participating group for a 100-word description of the approach they have taken and something about the results. This description should be emailed to paul.over@nist.gov by 30. October.

Tasks

Here are the eight fully specified tasks:

• Medical
• Tell me three categories of people who should or should not get a flu shot and why.
• Find a website likely to contain reliable information on the effect of second-hand smoke.
• Buying
• Get two price quotes for a new digital camera (3 or more megapixels and 2x or more zoom).
• Find two websites that allow people to buy soy milk online.
• Travel
• I want to visit Antarctica. Find a website with information on organized tours/trips there.
• Identify three interesting things to do during a weekend in Kyoto, Japan.
• Project
• Find three articles that a high school student could use in writing a report on the Titannic
• Tell me the name of a website where I can find material on global warming.

Here are the eight partially specified tasks:

• Medical
• List two of the generally recommended treatments for _____.
• Identify two pros or cons of taking large doses of _____.
• Buying
• Name three features to consider in buying a(n) _____.
• Find two websites that will let me buy a(n) _____ online.
• Travel
• Identify three interesting places to visit in _____.
• I'd like to go on a sailing vacation in _____, but I don't know how to sail.
• Tell me where can I get some information about organized sailing cruises in that area.
• Project
• Find three different information sources that may be useful to a high school student in writing a biography of ________.
• Locate a site with lots of information for a high school report on the history of _____.

Within-site guidelines

Within the cross-site guidelines, each site must impose further restrictions of its own choice on ALL its searchers to define an area of interest for observation - to be reported to the track before the observations begin. Each site may define its own time limits for searches.

For example, a site could impose inclusive or exclusive restrictions on any (combinations) of the following: the choice/assignment of domain from the 4 provided, the data to be searched, the search system/tools to be used (e.g., search systems, meta-search systems, directories,...), functionality within a given search system/tool, the characteristics of searchers, the time allowed, the pre- search training provided, etc. Sites are also encouraged to coordinate their plans with other sites, form small teams sharing guidelines, etc.

Each site will evaluate their searches using any criteria defined in the cross-site guidelines plus any site specific evaluations.

As part of the data analysis for TREC 2001, each site will attempt to formulate a testable hypothesis for TREC 2002 and will report this as part of the results for TREC 2001.

---

Last updated: Tuesday, 22-Sep-2015 09:46:10 EDT
Date created: Monday, 12-Mar-00
For further information contact trec@nist.gov