Compute Correlation among Rankings
Kendall’s tau as correlation metric
- number of pairwise swaps to turn one ranking into other
- normalized by number possible swaps
Calculated mean tau over two sets
- all pairwise correlations in a 1000 qrels subsample of the 1-judge qrels: .9632 (15)
- each 1-judge qrels against adjudicated qrels: .9563 (18)