| Runtag | Org | Is this run manual or automatic? | What form of the documents did you use? | What topic fields did you use? | What form of the queries did you use? | Please provide a short description of this run, including info about anything checked "Other" above. | Please give this run a priority for inclusion in manual assessments. |
|---|---|---|---|---|---|---|---|
| tech-coordinators-1MNCS-patapscoBM25htnoRM3desc (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization | 20 |
| tech-coordinators-1MNCS-patapscoBM25htnoRM3td (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization | 6 |
| tech-coordinators-1MNCS-patapscoBM25htnoRM3title (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization | 14 |
| tech-coordinators-1MNCS-patapscoBM25htRM3desc (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 17 |
| tech-coordinators-1MNCS-patapscoBM25htRM3td (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 1 (top) |
| tech-coordinators-1MNCS-patapscoBM25htRM3title (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 10 |
| tech-coordinators-MNES-fast_psq_t (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 11 |
| tech-coordinators-MNES-fast_psq_td (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 3 |
| tech-coordinators-MNES-patapscoBM25qtnoRM3desc (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization | 22 |
| tech-coordinators-MNES-patapscoBM25qtnoRM3td (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization | 8 |
| tech-coordinators-MNES-patapscoBM25qtnoRM3title (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization | 16 |
| tech-ISI_SEARCHER-ANE_run1 (ir_measures) | ISI | automatic | ['Original non-English text'] | ['title', 'description', 'narrative'] | ['English'] | ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model. | 1 (top) |
| tech-coordinators-MNES-patapscoBM25qtRM3desc (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 19 |
| tech-coordinators-MNES-patapscoBM25qtRM3td (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 6 |
| tech-coordinators-MNES-patapscoBM25qtRM3title (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 13 |
| tech-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming | 21 |
| tech-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming | 7 |
| tech-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming | 15 |
| tech-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 18 |
| tech-coordinators-MTES-patapscoBM25dtRM3td (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 4 |
| tech-coordinators-MTES-patapscoBM25dtRM3title (ir_measures) (paper) | coordinators | manual | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 12 |
| tech-coordinators-MNCD-plaid_distill_mono_ht (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 2 |
| tech_coordinators-topic_dev (ir_measures) (paper) | coordinators | manual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description', 'narrative'] | ['Other'] | These were the documents identified as somewhat or very valuable during topic development. Assessors use any query they wanted to identify the documents. | 23 |
| tech-hltcoe-MNED-plaid_distill_engeng_zs2engzho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 8 |
| tech-hltcoe-MNED-plaid_distill_engeng_zs2zhozho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 13 |
| tech-hltcoe-MNED-plaid_distill_engzho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 7 |
| tech-hltcoe-MNED-plaid_distill_engzho.mt5rerank (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 | 3 |
| tech-hltcoe-MNED-plaid_distill_engzho.mt5rerank.gpt4rerank (ir_measures) (paper) | coordinators | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 and top 30 reranked using gpt4. | 8 |
| tech-hltcoe-MNED-plaid_distill_zhozho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 11 |
| tech-hltcoe-MNED-plaid_syn_distill_engzho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on sythetix queries created by mixtral in English and documents from the collection using distillation of English, Chinese scores. Default ColBERT parameters with 1 bit for residual index | 6 |
| tech-hltcoe-MNTEH-kitchen_rankfuse (ir_measures) | hltcoe | manual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: tech-hltcoe-MNED-plaid_distill_engzho, tech-hltcoe-MTED-plaid_distill_engeng, tech-coordinators-MNES-fast_psq_td, tech-coordinators-MNES-patapscoBM25qtRM3td, tech-coordinators-MTES-patapscoBM25dtRM3td, and tech-ISI_SEARCHER-ANE_run1 | 9 |
| tech-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank (ir_measures) | hltcoe | manual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Reranked tech-hltcoe-MNTEH-kitchen_rankfuse run with mt5 | 4 |
| tech-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.gpt4rerank (ir_measures) | hltcoe | manual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Reranked tech-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank run with gpt4 | 2 |
| tech-hltcoe-MTED-plaid_distill_engeng (ir_measures) | hltcoe | manual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 10 |
| tech-hltcoe-MNED-plaid_eqsynms_distill_engzho (ir_measures) | hltcoe | manual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on 50% sythetix queries created by mixtral in English and documents from the collection using distillation of English, Chinese scores and 50% MSMARCO queries in English and MSMARCO documents translated in Chinese. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| fs1_xlmr_monot5_listgalore (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR
monoT5 - top-1K reranking with monoT5-3B-10k
listgalore - RRF of top-100 Listwise RankGPT-4o + RankLLaMA70B - window (20, 10) | 1 (top) |
| fs1_monot5_listgalore (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT)
monoT5 - top-1K reranking with monoT5-3B-10k
listgalore - RRF of top-100 Listwise RankGPT-4o + RankLLaMA70B - window (20, 10) | 2 |
| fs1_xlmr_monot5_rgpt-4o (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR
monoT5 - top-1K reranking with monoT5-3B-10k
rgpt-4o - top-100 Listwise RankGPT-4o - window (20, 10) | 3 |
| fs1_monot5_rgpt-4o (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT).
monoT5 - top-1K reranking with monoT5-3B-10k
rgpt-4o - top-100 Listwise RankGPT-4o - window (20, 10) | 4 |
| fs1_xlmr_monot5_rl3.1_70b (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR
monoT5 - top-1K reranking with monoT5-3B-10k
rl3.1 - top-100 Listwise RankLLaMA70B - window (20, 10) | 6 |
| fs1_monot5_rl3.1_70b (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT).
monoT5 - top-1K reranking with monoT5-3B-10k
rl3.1 - top-100 Listwise RankLLaMA70B - window (20, 10) | 7 |
| fs1_xlmr_monot5_rz (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR
monoT5 - top-1K reranking with monoT5-3B-10k
rz - top-100 Listwise RankZephyr - window (20, 10) | 8 |
| fs1_monot5_rz (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT).
monoT5 - top-1K reranking with monoT5-3B-10k
rz - top-100 Listwise RankZephyr - window (20, 10) | 9 |
| fs1_xlmr_monot5 (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR
monoT5 - top-1K reranking with monoT5-3B-10k | 10 |
| fs1_monot5 (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT).
monoT5 - top-1K reranking with monoT5-3B-10k | 11 |
| fs1_xlmr (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1_xlmr - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT), xlmr-based ColBERT-X trained with translate-distill for English-Chinese CLIR | 12 |
| fs1 (ir_measures) | h2oloo | automatic | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | fs1 - RRF of BM25+Rocchio-QT, BM25+Rocchio-DT, GTE QWEN2 (QT) | 13 |
| gte-qwen-desc+title (ir_measures) | h2oloo | automatic | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | GTE QWEN2 | 14 |
| bm25-rocchio-qt-desc+title (ir_measures) | h2oloo | automatic | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Pyserini QT BM25 + Rocchio - Desc + Title | 15 |
| bm25-rocchio-dt-desc+title (ir_measures) | h2oloo | automatic | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Pyserini DT BM25 + Rocchio - Desc + Title | 16 |