Judgment Sets (“Qrels”)

• Can define 3198 different 1-judge qrels

  • generated sample of 100,003 of them

• Can also define 4 combined-judge qrels

  • adjudicated (gold standard)
  • strict majority
  • union
  • intersection

• Compute system ranking for each qrels

Previous slide

Next slide

Back to first slide

View graphic version