Incompleteness
Effects of TREC pooling
- studied by Harman (TREC-4) and Zobel (SIGIR-98)
- did find additional relevant documents
- roughly uniform across systems
- highly skewed across topics
- systems that do not contribute to the pool not harmed; common systems not reinforced
- need for unbiased judgments argues against newer pooling schemes