G u i d e l i n e s
For TREC 2002 the Interactive Track will complete the two-year cycle
of observational studies in TREC 2001 followed now by more controlled
laboratory experiments focussing on question answering using web
Data to be searched
The track will use on open version of the .GOV web
collection created for the TREC 2002 Web Track as the data to be
searched. The collection is open in the sense that some links to pages
outside the collection are presented and can be followed. This means
the collection is intermediate in its stability between the live Web
used by the track last year and a completely fixed, closed version of
the .GOV collection, which was desired but not available in time.
The collection will be used by most participating groups as indexed and
searched by the Panoptic
search engine. The cited version does stemming and the homepage-finding
feature is turned off. Results can be obtained in XML format by sending
a query via CGI, e.g.,:
and getting back a padre_results packet. Experiments need not
be limited to interface defined by actual html pages returned by the
Panoptic engine. Help on the use of the
Panoptic search engine is available.
Eight searcher tasks, analogous to those
used in TREC-2001, have been generated and tested. They are listed
below. All searchers must be given at least 10 minutes on each task
once the actual searching begins and participating groups must report
the results as of the end of the 10 minute period in their notebook
papers. Groups may optionally decide to report additional results,
e.g., after 5 minutes, 15 minutes, etc.
Looking for personal health information
Seeking guidance on US government laws, regulations, guidelines, policy
Making travel plans
Gathering material for a report on a given subject
A standard query will be run by all searchers and the time between
pressing the search button and the return of the results logged.
This information will be used to get some idea of the different
response time conditions searchers may encounter.
The searcher tasks will be formulated in one of the following
Find any N short answers to a question, to which there are multiple answers of
the same type.
Find any N websites that meet the need specified in the task statement
Considerations in creating tasks
Here are some goals and guidelines to be used in the process of
creating the tasks:
Assume the searcher is an adult with at least a couple years of
education beyond high school or equivalent life experience.
Include information about the motivating context of the task
in the question - including scenarios in which a non-US
citizen might plausibly be looking for information on a
US government website.
Be as specific as possible about what constitutes an acceptable
answer, e.g., number/kind of items, intended use, etc.
Make sure there is a stopping condition stated, if you want
more than a default maximum elapsed time allowed per question
(to be determined)
Test the question on the .GOV collection and make sure most of what
is requested does not come up on the first page of results in response
to a query consisting of the question or a few key words from it.
You are traveling from the Netherlands, and want to bring some typical
food products as gifts for your friends. What are three kinds of food
products from the Netherlands that you are not allowed to bring into the
US? [Government Regulation]
You are concerned with privacy issues related to electronic
information and would like to know what laws have been passed by the
US Congress regarding these issues. Identify three such
laws. [Government Regulation]
A friend has a private well which is the family's only source of
drinking water. Locate a US publication, which contains guidelines for the
maintenance of safe water standards for private well use. [Health]
You are not sure about the safety of genetically engineered foods, and
would like to find more information and research on this topic. Name four
potential types of safety problems that have been raised. [Health or
You are interested in learning more about what measures the US
government has taken since 2001 to prevent Mad-Cow Disease. Identify three
such measures. [Health or Project]
Name/find three research programs/projects that investigate the
treatment/causes of dwarfism. [Project]
You are planning a cycling expedition along the Silk Road in Central
Asia. Find a website that is a good source information about health
precautions should you take. [Travel]
You are planning to travel to the northeast territories of India and
wonder if there are any problems/restrictions for tourists. Find a
website that is a good source of information about such
The track will reuse the experimental protocol developed for the TREC-9
Interactive Track. The design will allow the comparison of two
systems or system variants. A minimum of 16 searchers will be
used. Each searcher will perform all eight tasks (two of each of the
four types), half on one system and half on another.
Each searcher will issue the query "information retrieval" twice -
once before beginning each set of the four searches on the two search
engines in the design. The searcher will ignore the results of the
search. Experimenters will collect elapsed time for these searches
from the time tje search button is pressed until results start to
appear. This calibrating information should be reported in the
workshop notebook paper.
The searches will be evaluated for effectiveness, efficiency, and user
satisfaction in a manner much like that in previous interactive
tracks. Effectiveness will include at least whether the task was
completed successfully. Efficiency will include at least the elapsed
time used for each search.
Instruments for the collection of minimal searcher background and satisfaction information
are available here.
- 24. May - 8 tasks specified
- 31. May - Guidelines complete
- 3. Sep - Each participating group must submit a paragraph to Paul.Over@nist.gov including a brief description of their study, and a statement of progress to-date (e.g., n of m searchers complete). No other submissions separate from the workshop paper will be required.
Last updated: Tuesday, 22-Sep-2015 07:57:32 MDT
Date created: Tuesday, 30-apr-02
For further information contact firstname.lastname@example.org