Judgements for this track are distributed in 3 separate qrels files: # geo-nist.qrels This file contains a series of context-number, document, and judgements. Each judgement is one of: -2: could not load 0: not geographically appropriate 1: marginally geographically appropriate 2: geographically appropriate All of the judgements in this file where made by NIST accessors. # geo-user.qrels This file contains another series of context-number, document, and judgements. This judgements in this file have the same meaning as the ones in geo-nist.qrels. All of these judgements were made by users. When calculating scores if there is judgement in both geo-nist.qrels and geo-users.qrels the one in geo-nist.qrels is used. # desc-doc.qrels This file contains judgements and timing data for profile-document pairs. For each pair a judgement on the intereset in the attraction based on the description and based on the document. Additionally how long it took to arrive at the judgement in seconds is available. For the timing data a time of -1 means that the time to judge is unavailable. Each description and document judgement is one of: -2: could not load 0: strongly uninterested 1: uninterested 2: neutral 3: interested 4: strongly interested # topics.csv This file contains a list of profile-context pairs that were judged by users. All of these profile-context pairs were judged up to rank 5 for all runs. # eval.py This script takes in 5 files: a submission file, topics.csv, desc-doc.qrels, geo-nist.qrels, and geo-users.qrels (in that order). It results a P@5 score, a MRR score, and a TBG score for each profile-context pair as well as a mean score for all three metrics. The result is a CSV formatted file with the header: run,profile,context,score,metric