The Thirty-Third Text REtrieval Conference
(TREC 2024)

NeuCLIR (Cross language and multilingual search) Multi-lingual retrieval task Appendix

RuntagOrgIs this run manual or automatic?What collection is this submission using?What form of the documents did you use?What topic fields did you use?What form of the queries did you use?Please provide a short description of this run, including info about anything checked "Other" above.Please give this run a priority for inclusion in manual assessments.
mlir-ISI_SEARCHER-ANE_run1 (ir_measures) ISI
automatic
Multilingual
['Original non-English text']
['title', 'description', 'narrative']
['English']
ISI's TREC 2024 SEARCHER submission is a two-stage model consisting of an initial retrieval stage and a reranking stage. The first stage is based on a neural cross-lingual model utilizing a sparse index (see https://aclanthology.org/2020.clssts-1.4.pdf); the second stage re-ranks the top 1000 documents using a cross encoder based on a fine-tuned Mistral model.
1 (top)
mlir-coordinators-MNES-fast_psqtd (ir_measures) (paper)coordinators
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index. Indexed by language with score fusion
2
mlir-coordinators-MNES-fast_psqtitle (ir_measures) (paper)coordinators
manual
Multilingual
['Original non-English text']
['title']
['English']
Probabilistic structured queries using fast-psq implementation with recommended parameters for creating statistical translation table and index. Indexed by language with score fusion
6
mlir-coordinators-MTES-patapscoBM25dtnoRM3desc (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['description']
['English']
Patapsco BM25 run using spacy tokenization and stemming.
9
mlir-coordinators-MTES-patapscoBM25dtnoRM3td (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['title', 'description']
['English']
Patapsco BM25 run using spacy tokenization and stemming.
3
mlir-coordinators-MTES-patapscoBM25dtnoRM3title (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['title']
['English']
Patapsco BM25 run using spacy tokenization and stemming.
7
mlir-coordinators-MTES-patapscoBM25dtRM3desc (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['description']
['English']
Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%.
8
mlir-coordinators-MTES-patapscoBM25dtRM3td (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['title', 'description']
['English']
Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%.
1 (top)
mlir-coordinators-MTES-patapscoBM25dtRM3title (ir_measures) (paper)coordinators
manual
Multilingual
['Track-provided translations']
['title']
['English']
Patapsco BM25 run using spacy tokenization and stemming and RM3 with 10 terms. Original query weight is 50%.
4
mlir-ATNESLH-h2oloo-rfused-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o
1 (top)
mlir-ATNESLH-h2oloo-rfused-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o
1 (top)
mlir-ATNESLH-h2oloo-rfused-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankZephyr, RankLlama3.1-70b, RankGPT4o
1 (top)
mlir-ATNESLH-h2oloo-rfusedb-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
2
mlir-ATNESLH-h2oloo-rfusedb-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
2
mlir-ATNESLH-h2oloo-rfusedb-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RRF: RankLlama3.1-70b, RankGPT4o
2
mlir-ATNESLH-h2oloo-rg4o-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankGPT4o
3
mlir-ATNESLH-h2oloo-rg4o-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankGPT4o
3
mlir-ATNESLH-h2oloo-rg4o-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankGPT4o
3
mlir-ATNESLH-h2oloo-rl70b-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankLlama3.1-70b
4
mlir-ATNESLH-h2oloo-rl70b-rus.trec (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankLlama3.1-70b
4
mlir-ATNESLH-h2oloo-rl70b-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankLlama3.1-70b
4
mlir-ATNESLH-h2oloo-rzephyr-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankZephyr
6
mlir-ATNESLH-h2oloo-rzephyr-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankZephyr
6
mlir-ATNESLH-h2oloo-rzephyr-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large Third Stage (on top 100): RankZephyr
6
mlir-ATNESLH-h2oloo-monot5+lit5-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
7
mlir-ATNESLH-h2oloo-monot5+lit5-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
7
mlir-ATNESLH-h2oloo-monot5+lit5-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): RRF: First Stage, monot5-3b, lit5-v2large
7
mlir-ATNESLH-h2oloo-monot5-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): First Stage, monot5-3b
8
mlir-ATNESLH-h2oloo-monot5-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): First Stage, monot5-3b
8
mlir-ATNESLH-h2oloo-monot5-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): First Stage, monot5-3b
8
mlir-ATNESLH-h2oloo-lit5-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): lit5-v2large
9
mlir-ATNESLH-h2oloo-lit5-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): lit5-v2large
9
mlir-ATNESLH-h2oloo-lit5-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k) Second Stage (on top1K): lit5-v2large
9
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
10
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
10
mlir-ATNESLH-h2oloo-bm25dt+spladedt+plaid-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: RRF: PLAID server (top 100), SPLADE DT + Rocchio (Top 1K), BM25 DT + Rocchio (Top 1k)
10
mlir-ATNESLH-h2oloo-spladedt+rocchio-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: SPLADE DT + Rocchio (Top 1K)
11
mlir-ATNESLH-h2oloo-spladedt+rocchio-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title']
['English', 'Track-provided Google translation']
First Stage: SPLADE DT + Rocchio (Top 1K)
11
mlir-ATNESLH-h2oloo-spladedt+rocchio-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: SPLADE DT + Rocchio (Top 1K)
11
mlir-ATESH-h2oloo-bm25dt+rocchio-fas (ir_measures) h2oloo
automatic
Farsi
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: BM25 DT + Rocchio (Top 1K)
12
mlir-ATESH-h2oloo-bm25dt+rocchio-rus (ir_measures) h2oloo
automatic
Russian
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: BM25 DT + Rocchio (Top 1K)
12
mlir-ATESH-h2oloo-bm25dt+rocchio-zho (ir_measures) h2oloo
automatic
Chinese
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
First Stage: BM25 DT + Rocchio (Top 1K)
12
mlir-hltcoe-MNED-plaid_distill_clir_scorefuse (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index
12
mlir-hltcoe-MNED-plaid_distill_clir.mt5rerank.scorefuse (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index. CLIR run reranked with mt5 and then score fused.
11
mlir-hltcoe-MNED-plaid_distill_clir.mt5rerank.scorefuse.gpt4rerank (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned for CLIR task on MSMARCO queries and documents in Chinese, Persian, or Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index. CLIR run reranked with mt5 and then score fused. Top 30 reranked with gpt4
2
mlir-hltcoe-MNED-plaid_distill_engeng_zs (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index
6
mlir-hltcoe-MNED-plaid_distill_mlir_bycoll_scorefuse (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. 3 indexes by language and then score fused. Default ColBERT parameters with 1 bit for residual index
1 (top)
mlir-hltcoe-MNED-plaid_distill_mlir_mixedentry (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Default ColBERT parameters with 1 bit for residual index
7
mlir-hltcoe-MNED-plaid_distill_mlir_mixedentry_termpool2 (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed entries. Document terms pooled to reduce terms by half. Default ColBERT parameters with 1 bit for residual index
10
mlir-hltcoe-MNED-plaid_distill_mlir_mixedpass (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with mixed passages. Default ColBERT parameters with 1 bit for residual index
8
mlir-hltcoe-MNED-plaid_distill_mlir_rr (ir_measures) hltcoe
manual
Multilingual
['Original non-English text']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in Chinese, Persian, and Russian using distillation of English,English scores with round robin. Default ColBERT parameters with 1 bit for residual index
9
mlir-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.scorefuse (ir_measures) hltcoe
manual
Multilingual
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
Used score fusion over the following runs: zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, rus-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, and fas-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, which itself was fused over several systems
13
mlir-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank.scorefuse.gpt4rerank (ir_measures) hltcoe
manual
Multilingual
['Original non-English text', 'Track-provided translations']
['title', 'description']
['English', 'Track-provided Google translation']
Used score fusion over the following runs: zho-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, rus-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, and fas-hltcoe-MNTEH-kitchen_rankfuse.mt5rerank, which itself was fused over several systems and then reranked with gpt4
3
mlir-hltcoe-MTED-plaid_distill_engeng (ir_measures) hltcoe
manual
Multilingual
['Track-provided translations']
['title', 'description']
['English']
PLAID-X implementation of ColBERT with model fine-tuned on MSMARCO queries and documents in English using distillation of English,English scores. Default ColBERT parameters with 1 bit for residual index
4
mlir-IRLabAmsterdam-ANEL-titledesc (ir_measures) IRLab-Amsterdam
automatic
Multilingual
['Original non-English text']
['title', 'description']
['English']
LSR trained on translated MS MARCO dataset
1 (top)
mlir-IRLabAmsterdam-ANEL-desc (ir_measures) IRLab-Amsterdam
automatic
Multilingual
['Original non-English text']
['description']
['English']
LSR trained on translated MS MARCO dataset
2
mlir-IRLabAmsterdam-ANEL-title (ir_measures) IRLab-Amsterdam
automatic
Multilingual
['Original non-English text']
['title']
['English']
LSR trained on translated MS MARCO dataset
3