Relevance Judgments
Main source of criticism of Cranfield tradition
- In test collections, judgments are usually binary, static, and assumed to be complete.
- But...
- �relevance� is highly idiosyncratic
- relevance does not entail utility
- documents have different degrees of relevance
- relevance can change over time for the same user
- for realistic collections, judgments cannot be complete