| Runtag | Org | Is this run manual or automatic? | What collection is this submission using? | What form of the documents did you use? | What topic fields did you use? | What form of the queries did you use? | Please provide a short description of this run, including info about anything checked "Other" above. | Please give this run a priority for inclusion in manual assessments. |
|---|---|---|---|---|---|---|---|---|
| fas-ISI_SEARCHER-ANE_run1 (ir_measures.fas) | ISI | automatic | Farsi | ['Original non-English text'] | ['title', 'description', 'narrative'] | ['English'] | ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model. | 1 (top) |
| rus-ISI_SEARCHER-ANE_run1 (ir_measures.rus) | ISI | automatic | Russian | ['Original non-English text'] | ['title', 'description', 'narrative'] | ['English'] | ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model. | 1 (top) |
| zho-ISI_SEARCHER-ANE_run1 (ir_measures.zho) | ISI | automatic | Chinese | ['Original non-English text'] | ['title', 'description', 'narrative'] | ['English'] | ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model. | 1 (top) |
| fas-coordinators-MNED1-plaid_distill_mono_ht (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| fas-coordinators-MNES-fast_psqtd (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 3 |
| fas-coordinators-MNES-fast_psqtitle (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 10 |
| fas-coordinators-MNES-patapscoBM25qtnoRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 16 |
| fas-coordinators-MNES-patapscoBM25qtnoRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 22 |
| fas-coordinators-MNES-patapscoBM25qtnoRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 9 |
| fas-coordinators-MNES-patapscoBM25qtRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 19 |
| fas-coordinators-MNES-patapscoBM25qtRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 6 |
| fas-coordinators-MNES-patapscoBM25qtRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 13 |
| fas-coordinators-MNPS1-patapscoBM25htnoRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 20 |
| fas-coordinators-MNPS1-patapscoBM25htnoRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 7 |
| fas-coordinators-MNPS1-patapscoBM25htnoRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and parsivar stemming. | 14 |
| fas-coordinators-MNPS1-patapscoBM25htRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 17 |
| fas-coordinators-MNPS1-patapscoBM25htRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 2 |
| fas-coordinators-MNPS1-patapscoBM25htRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization, parsivar stemming, and RM3 with 10 terms. Original query weight is 50%. | 10 |
| fas-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 21 |
| fas-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 8 |
| fas-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 15 |
| fas-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 18 |
| fas-coordinators-MTES-patapscoBM25dtRM3td (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 4 |
| fas-coordinators-MTES-patapscoBM25dtRM3title (ir_measures.fas) (paper) | coordinators | manual | Farsi | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 12 |
| rus-coordinators-MNED1-plaid_distill_mono_ht (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| rus-coordinators-MNES-fast_psqtd (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 3 |
| rus-coordinators-MNES-fast_psqtitle (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 11 |
| rus-coordinators-MNES-patapscoBM25qtnoRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 22 |
| rus-coordinators-MNES-patapscoBM25qtnoRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 9 |
| rus-coordinators-MNES-patapscoBM25qtnoRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 16 |
| rus-coordinators-MNES-patapscoBM25qtRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, spacy stemming, and RM3 with 10 terms. Original query weight is 50%. | 19 |
| rus-coordinators-MNES-patapscoBM25qtRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, spacy stemming, and RM3 with 10 terms. Original query weight is 50%. | 6 |
| rus-coordinators-MNES-patapscoBM25qtRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization, spacy stemming, and RM3 with 10 terms. Original query weight is 50%. | 13 |
| rus-coordinators-MNRS1-patapscoBM25htnoRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 20 |
| rus-coordinators-MNRS1-patapscoBM25htnoRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 7 |
| rus-coordinators-MNRS1-patapscoBM25htnoRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and stemming. | 14 |
| rus-coordinators-MNRS1-patapscoBM25htRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization, spacy stemming, and RM3 with 10 terms. Original query weight is 50%. | 17 |
| rus-coordinators-MNRS1-patapscoBM25htRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization, spacy stemming, and RM3 with 10 terms. Original query weight is 50%. | 2 |
| rus-coordinators-MNRS1-patapscoBM25htRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 10 |
| rus-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 21 |
| rus-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 8 |
| rus-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 15 |
| rus-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 18 |
| rus-coordinators-MTES-patapscoBM25dtRM3td (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 4 |
| rus-coordinators-MTES-patapscoBM25dtRM3title (ir_measures.rus) (paper) | coordinators | manual | Russian | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 12 |
| zho-coordinators-MNED1-plaid_distill_mono_ht (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| zho-coordinators-MNES-fast_psqtd (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 3 |
| zho-coordinators-MNES-fast_psqtitle (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index | 12 |
| zho-coordinators-MNCS1-patapscoBM25htnoRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization. | 20 |
| zho-coordinators-MNCS1-patapscoBM25htnoRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization. | 7 |
| zho-coordinators-MNCS1-patapscoBM25htRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 17 |
| zho-coordinators-MNCS1-patapscoBM25htRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 2 |
| zho-coordinators-MNES-patapscoBM25qtnoRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization. | 22 |
| zho-coordinators-MNES-patapscoBM25qtnoRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization. | 9 |
| zho-coordinators-MNES-patapscoBM25qtRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 19 |
| zho-coordinators-MNES-patapscoBM25qtRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 6 |
| zho-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 21 |
| zho-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 8 |
| zho-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 15 |
| zho-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 18 |
| zho-coordinators-MTES-patapscoBM25dtRM3td (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 4 |
| zho-coordinators-MTES-patapscoBM25dtRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 12 |
| zho-coordinators-MNES-patapscoBM25qtRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 13 |
| zho-coordinators-MNES-patapscoBM25qtnoRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title'] | ['Track-provided Google translation'] | Patapsco BM25 run using spacy tokenization. | 16 |
| zho-coordinators-MNCS1-patapscoBM25htnoRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization. | 14 |
| zho-coordinators-MNCS1-patapscoBM25htRM3title (ir_measures.zho) (paper) | coordinators | manual | Chinese | ['Original non-English text'] | ['title'] | ['Track-provided human translation'] | Patapsco BM25 run using spacy tokenization and RM3 with 10 terms. Original query weight is 50%. | 10 |
| fas-hltcoe-MNED-plaid_distill_engeng_zs2engfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 8 |
| fas-hltcoe-MNED-plaid_distill_engeng_zs2fasfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 13 |
| fas-hltcoe-MNED-plaid_distill_engfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Persian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 7 |
| fas-hltcoe-MNED-plaid_distill_engfas.mt5rerank.gpt4rerank (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Persian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 and top 30 reranked using gpt4. | 9 |
| fas-hltcoe-MNED-plaid_distill_engfas_450p (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Persian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Non-overlapping passages containing 450 tokens. | 14 |
| fas-hltcoe-MNED-plaid_distill_engfas_termpool2 (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Persian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Term pooling to remove 1/2 of tokens per passage | 16 |
| fas-hltcoe-MNED-plaid_distill_engmlir (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index | 15 |
| fas-hltcoe-MNED-plaid_distill_engfas.mt5rerank (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Persian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 | 3 |
| fas-hltcoe-MNED-plaid_distill_mono_fasfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Persian using distillation. Default ColBERT parameters with 1 bit for residual index | 12 |
| fas-hltcoe-MNED-plaid_eqsynms_distill_engfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on 50% sythetix queries created by mixtral in English and documents from the collection using distillation of English, Persian scores and 50% MSMARCO queries in English and MSMARCO documents translated in Persian. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| fas-hltcoe-MNED-plaid_syn_distill_engfas (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on sythetix queries created by mixtral in English and documents from the collection using distillation of English, Persian scores. Default ColBERT parameters with 1 bit for residual index | 6 |
| fas-hltcoe-MNTEH-kitchen_rankfuse (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: fas-hltcoe-MNED-plaid_distill_engfas, fas-hltcoe-MTED-plaid_distill_engeng, fas-coordinators-MNES-fast_psqtd, fas-coordinators-MNES-patapscoBM25qtRM3td, fas-coordinators-MTES-patapscoBM25dtRM3td, and fas-ISI_SEARCHER-ANE_run1 | 10 |
| fas-hltcoe-MNTEH-kitchen_rankfuse,mt5rerank (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: fas-hltcoe-MNED-plaid_distill_engfas, fas-hltcoe-MTED-plaid_distill_engeng, fas-coordinators-MNES-fast_psqtd, fas-coordinators-MNES-patapscoBM25qtRM3td, fas-coordinators-MTES-patapscoBM25dtRM3td, and fas-ISI_SEARCHER-ANE_run1 and reranked with mt5 | 4 |
| fas-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.gpt4rerank (ir_measures.fas) | hltcoe | manual | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: fas-hltcoe-MNED-plaid_distill_engfas, fas-hltcoe-MTED-plaid_distill_engeng, fas-coordinators-MNES-fast_psqtd, fas-coordinators-MNES-patapscoBM25qtRM3td, fas-coordinators-MTES-patapscoBM25dtRM3td, and fas-ISI_SEARCHER-ANE_run1 and reranked with mt5 followed by gpt4 on top 30 | 2 |
| fas-hltcoe-MTED-plaid_distill_engeng (ir_measures.fas) | hltcoe | manual | Farsi | ['Track-provided translations'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 11 |
| rus-hltcoe-MNED-plaid_distill_engeng_zs2engrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 8 |
| rus-hltcoe-MNED-plaid_distill_engeng_zs2rusrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 13 |
| rus-hltcoe-MNED-plaid_distill_engmlir (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index | 15 |
| rus-hltcoe-MNED-plaid_distill_engrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 7 |
| rus-hltcoe-MNED-plaid_distill_engrus.mt5rerank (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 | 3 |
| rus-hltcoe-MNED-plaid_distill_engrus.mt5rerank.gpt4rerank (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 and top 30 reranked using gpt4. | 9 |
| rus-hltcoe-MNED-plaid_distill_engrus_450p (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Non-overlapping passages containing 450 tokens. | 14 |
| rus-hltcoe-MNED-plaid_distill_engrus_termpool2 (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Term pooling to remove 1/2 of tokens per passage | 16 |
| rus-hltcoe-MNED-plaid_distill_mono_rusrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Russian using distillation. Default ColBERT parameters with 1 bit for residual index | 12 |
| rus-hltcoe-MNED-plaid_eqsynms_distill_engrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on 50% sythetix queries created by mixtral in English and documents from the collection using distillation of English, Russian scores and 50% MSMARCO queries in English and MSMARCO documents translated in Russian. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| rus-hltcoe-MNED-plaid_syn_distill_engrus (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on sythetix queries created by mixtral in English and documents from the collection using distillation of English, Russian scores. Default ColBERT parameters with 1 bit for residual index | 6 |
| rus-hltcoe-MNTEH-kitchen_rankfuse (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: rus-hltcoe-MNED-plaid_distill_engrus, rus-hltcoe-MTED-plaid_distill_engeng, rus-coordinators-MNES-fast_psqtd, rus-coordinators-MNES-patapscoBM25qtRM3td, rus-coordinators-MTES-patapscoBM25dtRM3td, and rus-ISI_SEARCHER-ANE_run1 | 10 |
| rus-hltcoe-MNTEH-kitchen_rankfuse,mt5rerank (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: rus-hltcoe-MNED-plaid_distill_engrus, rus-hltcoe-MTED-plaid_distill_engeng, rus-coordinators-MNES-fast_psqtd, rus-coordinators-MNES-patapscoBM25qtRM3td, rus-coordinators-MTES-patapscoBM25dtRM3td, and rus-ISI_SEARCHER-ANE_run1 and reranked with mt5 | 4 |
| rus-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.gpt4rerank (ir_measures.rus) | hltcoe | manual | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: rus-hltcoe-MNED-plaid_distill_engrus, rus-hltcoe-MTED-plaid_distill_engeng, rus-coordinators-MNES-fast_psqtd, rus-coordinators-MNES-patapscoBM25qtRM3td, rus-coordinators-MTES-patapscoBM25dtRM3td, and rus-ISI_SEARCHER-ANE_run1 and reranked with mt5 followed by gpt4 on top 30 | 2 |
| rus-hltcoe-MTED-plaid_distill_engeng (ir_measures.rus) | hltcoe | manual | Russian | ['Track-provided translations'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 11 |
| mlir-ATNESLH-h2oloo-rfusedbinterweave-rlp2_rg4of_clir | h2oloo | automatic | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
Interweaved across languages
Fourth Stage: RankLlama3.1-70b (Top 300) (2 passes)
Fifth Stage: RRF(RankGPT4o (top 100), Fourth Stage) | 1 (top) |
| mlir-ATNESLH-h2oloo-rfusedbinterweave-rlp2_rg4o_clir | h2oloo | automatic | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
Interweaved across languages
Fourth Stage: RankLlama3.1-70b (Top 300) (2 passes)
Fifth Stage: RankGPT4o (top 100) | 2 |
| mlir-ATNESLH-h2oloo-rfusedbinterweave-rlp2_clir | h2oloo | automatic | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
Interweaved across languages
Fourth Stage: RankLlama3.1-70b (Top 300) (2 passes) | 3 |
| mlir-ATNESLH-h2oloo-rfusedbinterweave-rlp1_clir | h2oloo | automatic | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
Interweaved across languages
Fourth Stage: RankLlama3.1-70b (Top 300) | 4 |
| mlir-ATNESLH-h2oloo-rfusedbinterweave-clir | h2oloo | automatic | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
Interweaved across languages | 6 |
| zho-hltcoe-MNED-plaid_distill_engeng_zs2engzho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 8 |
| zho-hltcoe-MNED-plaid_distill_engeng_zs2zhozho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 13 |
| zho-hltcoe-MNED-plaid_distill_engmlir (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index | 15 |
| zho-hltcoe-MNED-plaid_distill_engzho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 7 |
| zho-hltcoe-MNED-plaid_distill_engzho.mt5rerank (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 | 3 |
| zho-hltcoe-MNED-plaid_distill_engzho.mt5rerank.gpt4rerank (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Chinese using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Reranked using mt5 and top 30 reranked using gpt4. | 9 |
| zho-hltcoe-MNED-plaid_distill_engzho_450p (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Non-overlapping passages containing 450 tokens. | 14 |
| zho-hltcoe-MNED-plaid_distill_engzho_termpool2 (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries in English and documents translated into Russian using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index. Term pooling to remove 1/2 of tokens per passage | 16 |
| zho-hltcoe-MNED-plaid_distill_mono_zhozho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['Track-provided Google translation'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents translated into Chinese using distillation. Default ColBERT parameters with 1 bit for residual index | 12 |
| zho-hltcoe-MNED-plaid_eqsynms_distill_engzho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on 50% sythetix queries created by mixtral in English and documents from the collection using distillation of English, Chinese scores and 50% MSMARCO queries in English and MSMARCO documents translated in Chinese. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
| zho-hltcoe-MNED-plaid_syn_distill_engzho (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on sythetix queries created by mixtral in English and documents from the collection using distillation of English, Chinese scores. Default ColBERT parameters with 1 bit for residual index | 6 |
| zho-hltcoe-MNTEH-kitchen_rankfuse (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: zho-hltcoe-MNED-plaid_distill_engzho, zho-hltcoe-MTED-plaid_distill_engeng, zho-coordinators-MNES-fast_psqtd, zho-coordinators-MNES-patapscoBM25qtRM3td, zho-coordinators-MTES-patapscoBM25dtRM3td, and zho-ISI_SEARCHER-ANE_run1 | 10 |
| zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used rank-based fusion over the following runs: zho-hltcoe-MNED-plaid_distill_engzho, zho-hltcoe-MTED-plaid_distill_engeng, zho-coordinators-MNES-fast_psqtd, zho-coordinators-MNES-patapscoBM25qtRM3td, zho-coordinators-MTES-patapscoBM25dtRM3td, and zho-ISI_SEARCHER-ANE_run1 and reranked with mt5 | 6 |
| zho-hltcoe-MTED-plaid_distill_engeng (ir_measures.zho) | hltcoe | manual | Chinese | ['Track-provided translations'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 11 |
| zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.gpt4rerank (ir_measures.zho) | hltcoe | manual | Chinese | ['Original non-English text'] | ['title', 'description'] | ['English'] | Used rank-based fusion over the following runs: zho-hltcoe-MNED-plaid_distill_engzho, zho-hltcoe-MTED-plaid_distill_engeng, zho-coordinators-MNES-fast_psqtd, zho-coordinators-MNES-patapscoBM25qtRM3td, zho-coordinators-MTES-patapscoBM25dtRM3td, and zho-ISI_SEARCHER-ANE_run1 and reranked with mt5 followed by gpt4 on top 30 | 2 |