Relevance Judgments
Consistency
- idiosyncratic nature of relevance judgments does not affect comparative results
Incompleteness
- the important issue is that relevant judgments be unbiased
- complete judgments must be unbiased
- TREC pooling has been adequate to produce unbiased judgments