G u i d e l i n e s

Motivating principles

For TREC 2002 the Interactive Track will complete the two-year cycle of observational studies in TREC 2001 followed now by more controlled laboratory experiments focussing on question answering using web data.

Data to be searched

The collection will be used by most participating groups as indexed and searched by the Panoptic search engine. The cited version does stemming and the homepage-finding feature is turned off. Results can be obtained in XML format by sending a query via CGI, e.g.,:

     trec.panopticsearch.com/gov/padre-sw_xml.cgi?collection=gov&query=bush

Tasks

Eight searcher tasks, analogous to those used in TREC-2001, have been generated and tested. They are listed below. All searchers must be given at least 10 minutes on each task once the actual searching begins and participating groups must report the results as of the end of the 10 minute period in their notebook papers. Groups may optionally decide to report additional results, e.g., after 5 minutes, 15 minutes, etc.

1. Looking for personal health information
2. Seeking guidance on US government laws, regulations, guidelines, policy
3. Making travel plans
4. Gathering material for a report on a given subject

A standard query will be run by all searchers and the time between pressing the search button and the return of the results logged. This information will be used to get some idea of the different response time conditions searchers may encounter.

Task formats

The searcher tasks will be formulated in one of the following ways:

1. Find any N short answers to a question, to which there are multiple answers of the same type.
2. Find any N websites that meet the need specified in the task statement

Considerations in creating tasks

Here are some goals and guidelines to be used in the process of creating the tasks:

1. Assume the searcher is an adult with at least a couple years of education beyond high school or equivalent life experience.
2. Include information about the motivating context of the task in the question - including scenarios in which a non-US citizen might plausibly be looking for information on a US government website.
3. Be as specific as possible about what constitutes an acceptable answer, e.g., number/kind of items, intended use, etc.
4. Make sure there is a stopping condition stated, if you want more than a default maximum elapsed time allowed per question (to be determined)
5. Test the question on the .GOV collection and make sure most of what is requested does not come up on the first page of results in response to a query consisting of the question or a few key words from it.

Task statements

1. You are traveling from the Netherlands, and want to bring some typical food products as gifts for your friends. What are three kinds of food products from the Netherlands that you are not allowed to bring into the US? [Government Regulation]
2. You are concerned with privacy issues related to electronic information and would like to know what laws have been passed by the US Congress regarding these issues. Identify three such laws. [Government Regulation]
3. A friend has a private well which is the family's only source of drinking water. Locate a US publication, which contains guidelines for the maintenance of safe water standards for private well use. [Health]
4. You are not sure about the safety of genetically engineered foods, and would like to find more information and research on this topic. Name four potential types of safety problems that have been raised. [Health or Project]
5. You are interested in learning more about what measures the US government has taken since 2001 to prevent Mad-Cow Disease. Identify three such measures. [Health or Project]
6. Name/find three research programs/projects that investigate the treatment/causes of dwarfism. [Project]
7. You are planning a cycling expedition along the Silk Road in Central Asia. Find a website that is a good source information about health precautions should you take. [Travel]
8. You are planning to travel to the northeast territories of India and wonder if there are any problems/restrictions for tourists. Find a website that is a good source of information about such problems/restrictions. [Travel]

Experimental design

The track will reuse the experimental protocol developed for the TREC-9 Interactive Track. The design will allow the comparison of two systems or system variants. A minimum of 16 searchers will be used. Each searcher will perform all eight tasks (two of each of the four types), half on one system and half on another.

Each searcher will issue the query "information retrieval" twice - once before beginning each set of the four searches on the two search engines in the design. The searcher will ignore the results of the search. Experimenters will collect elapsed time for these searches from the time tje search button is pressed until results start to appear. This calibrating information should be reported in the workshop notebook paper.

Evaluation

The searches will be evaluated for effectiveness, efficiency, and user satisfaction in a manner much like that in previous interactive tracks. Effectiveness will include at least whether the task was completed successfully. Efficiency will include at least the elapsed time used for each search. Instruments for the collection of minimal searcher background and satisfaction information are available here.

Schedule

• 24. May - 8 tasks specified
• 31. May - Guidelines complete
• 3. Sep - Each participating group must submit a paragraph to Paul.Over@nist.gov including a brief description of their study, and a statement of progress to-date (e.g., n of m searchers complete). No other submissions separate from the workshop paper will be required.

Last updated: Tuesday, 22-Sep-2015 13:57:32 UTC
Date created: Tuesday, 30-apr-02
For further information contact trec@nist.gov