Runtag | Org | Is this run manual or automatic? | What collection is this submission using? | What form of the documents did you use? | What topic fields did you use? | What form of the queries did you use? | Please provide a short description of this run, including info about anything checked "Other" above. | Please give this run a priority for inclusion in manual assessments. |
---|---|---|---|---|---|---|---|---|
mlir-ISI_SEARCHER-ANE_run1 (ir_measures) | ISI | automatic | Multilingual | ['Original non-English text'] | ['title', 'description', 'narrative'] | ['English'] | ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model. | 1 (top) |
mlir-coordinators-MNES-fast_psqtd (ir_measures) (paper) | coordinators | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index. Indexed by language with score fusion | 2 |
mlir-coordinators-MNES-fast_psqtitle (ir_measures) (paper) | coordinators | manual | Multilingual | ['Original non-English text'] | ['title'] | ['English'] | Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index. Indexed by language with score fusion | 6 |
mlir-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 9 |
mlir-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 3 |
mlir-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming. | 7 |
mlir-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 8 |
mlir-coordinators-MTES-patapscoBM25dtRM3td (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 1 (top) |
mlir-coordinators-MTES-patapscoBM25dtRM3title (ir_measures) (paper) | coordinators | manual | Multilingual | ['Track-provided translations'] | ['title'] | ['English'] | Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%. | 4 |
mlir-ATNESLH-h2oloo-rfused-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o | 1 (top) |
mlir-ATNESLH-h2oloo-rfused-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o | 1 (top) |
mlir-ATNESLH-h2oloo-rfused-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o | 1 (top) |
mlir-ATNESLH-h2oloo-rfusedb-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o | 2 |
mlir-ATNESLH-h2oloo-rfusedb-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o | 2 |
mlir-ATNESLH-h2oloo-rfusedb-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o | 2 |
mlir-ATNESLH-h2oloo-rg4o-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankGPT4o | 3 |
mlir-ATNESLH-h2oloo-rg4o-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankGPT4o | 3 |
mlir-ATNESLH-h2oloo-rg4o-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankGPT4o | 3 |
mlir-ATNESLH-h2oloo-rl70b-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankLlama3.1-70b | 4 |
mlir-ATNESLH-h2oloo-rl70b-rus.trec (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankLlama3.1-70b | 4 |
mlir-ATNESLH-h2oloo-rl70b-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankLlama3.1-70b | 4 |
mlir-ATNESLH-h2oloo-rzephyr-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankZephyr | 6 |
mlir-ATNESLH-h2oloo-rzephyr-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankZephyr | 6 |
mlir-ATNESLH-h2oloo-rzephyr-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
Third Stage (on top 100): RankZephyr | 6 |
mlir-ATNESLH-h2oloo-monot5+lit5-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large | 7 |
mlir-ATNESLH-h2oloo-monot5+lit5-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large | 7 |
mlir-ATNESLH-h2oloo-monot5+lit5-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large | 7 |
mlir-ATNESLH-h2oloo-monot5-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): First Stage, monot5-3b | 8 |
mlir-ATNESLH-h2oloo-monot5-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): First Stage, monot5-3b | 8 |
mlir-ATNESLH-h2oloo-monot5-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): First Stage, monot5-3b | 8 |
mlir-ATNESLH-h2oloo-lit5-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): lit5-v2large | 9 |
mlir-ATNESLH-h2oloo-lit5-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): lit5-v2large | 9 |
mlir-ATNESLH-h2oloo-lit5-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
Second Stage (on top1K): lit5-v2large | 9 |
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) | 10 |
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) | 10 |
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) | 10 |
mlir-ATNESLH-h2oloo-spladedt+rocchio-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: SPLADE DT + Rocchio (Top 1K) | 11 |
mlir-ATNESLH-h2oloo-spladedt+rocchio-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title'] | ['English', 'Track-provided Google translation'] | First Stage: SPLADE DT + Rocchio (Top 1K) | 11 |
mlir-ATNESLH-h2oloo-spladedt+rocchio-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: SPLADE DT + Rocchio (Top 1K) | 11 |
mlir-ATESH-h2oloo-bm25dt+rocchio-fas (ir_measures) | h2oloo | automatic | Farsi | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: BM25 DT + Rocchio (Top 1K) | 12 |
mlir-ATESH-h2oloo-bm25dt+rocchio-rus (ir_measures) | h2oloo | automatic | Russian | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: BM25 DT + Rocchio (Top 1K) | 12 |
mlir-ATESH-h2oloo-bm25dt+rocchio-zho (ir_measures) | h2oloo | automatic | Chinese | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | First Stage: BM25 DT + Rocchio (Top 1K) | 12 |
mlir-hltcoe-MNED-plaid_distill_clir_scorefuse (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index | 12 |
mlir-hltcoe-MNED-plaid_distill_clir.mt5rerank.scorefuse (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index. CLIR run reranked with mt5 and then score fused. | 11 |
mlir-hltcoe-MNED-plaid_distill_clir.mt5rerank.scorefuse.gpt4rerank (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index. CLIR run reranked with mt5 and then score fused. Top 30 reranked with gpt4 | 2 |
mlir-hltcoe-MNED-plaid_distill_engeng_zs (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 6 |
mlir-hltcoe-MNED-plaid_distill_mlir_bycoll_scorefuse (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. 3 indexes by language and then score fused. Default ColBERT parameters with 1 bit for residual index | 1 (top) |
mlir-hltcoe-MNED-plaid_distill_mlir_mixedentry (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index | 7 |
mlir-hltcoe-MNED-plaid_distill_mlir_mixedentry_termpool2 (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Document terms pooled to reduce terms by half. Default ColBERT parameters with 1 bit for residual index | 10 |
mlir-hltcoe-MNED-plaid_distill_mlir_mixedpass (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed passages. Default ColBERT parameters with 1 bit for residual index | 8 |
mlir-hltcoe-MNED-plaid_distill_mlir_rr (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with round robin. Default ColBERT parameters with 1 bit for residual index | 9 |
mlir-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.scorefuse (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used score fusion over the following runs: zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, rus-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, and fas-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, which itself was fused over several systems | 13 |
mlir-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.scorefuse.gpt4rerank (ir_measures) | hltcoe | manual | Multilingual | ['Original non-English text', 'Track-provided translations'] | ['title', 'description'] | ['English', 'Track-provided Google translation'] | Used score fusion over the following runs: zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, rus-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, and fas-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, which itself was fused over several systems and then reranked with gpt4 | 3 |
mlir-hltcoe-MTED-plaid_distill_engeng (ir_measures) | hltcoe | manual | Multilingual | ['Track-provided translations'] | ['title', 'description'] | ['English'] | PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index | 4 |
mlir-IRLabAmsterdam-ANEL-titledesc (ir_measures) | IRLab-Amsterdam | automatic | Multilingual | ['Original non-English text'] | ['title', 'description'] | ['English'] | LSR trained on translated MS MARCO dataset | 1 (top) |
mlir-IRLabAmsterdam-ANEL-desc (ir_measures) | IRLab-Amsterdam | automatic | Multilingual | ['Original non-English text'] | ['description'] | ['English'] | LSR trained on translated MS MARCO dataset | 2 |
mlir-IRLabAmsterdam-ANEL-title (ir_measures) | IRLab-Amsterdam | automatic | Multilingual | ['Original non-English text'] | ['title'] | ['English'] | LSR trained on translated MS MARCO dataset | 3 |