TREC 2025 Proceedings
UCSC-SIMRAG-keyword-trained-MiniLM
Submission Details
- Organization
- ucsc
- Track
- Interactive Knowledge Acquisition Track
- Task
- Passage Ranking and Response Generation
- Date
- 2025-07-28
Run Description
- What type of manually annotated information does the system use?
- automatic: system does not use any manually annotated data and relies only on the user utterance and system responses (canonical responses of previous turns)
- How is conversation understanding (NLP/rewriting) performed in this run (check all that apply)?
- ['method uses large language models like LLaMA and GPT-x.']
- What data is used for conversational query understanding in this run (check all that apply)?
- ['method uses iKAT 23 data']
- How is ranking performed in this run (check all that apply)?
- ['method uses learned sparse retrieval (e.g.¸ SPLADE¸ etc.)']
- What data is used to develop the ranking method in this run (check all that apply)?
- ['method uses iKAT 23 data', 'method is trained with TREC Deep Learning Track and/or MS MARCO dataset']
- Please specify all the methods used to handle feedback or clarification responses from the user (check all that apply).
- ['method does not treat them specially']
- Please describe the method used to generate the final conversational responses from one or more retrieved passages (check all that apply).
- ['method uses multiple sources (multiple passages)', 'method uses large language models to generate the summary.']
- Please describe how you integrate the PTKBs in your run (check all that apply)
- [' method uses a PTKB relevance model to detect the relevant ones', " method integrates PTKBs in the response generation method (e.g. include in the LLM's prompt)"]
- Which LLM did you use to generate the final response?
- ['method uses closed-source commercial LLMs (e.g. GPT-*)']
- Please describe the external resources used by this run, if applicable.
- none
- Please provide a short description of this run.
- This run first identifies useful PTKB statements and passes the useful statements in to the LLM, which then generates a keyword-query that are good for SPLADE retrieval. The query is then evaluated through a trained SIM-RAG model (based on T5, fine-tuned on generated data from the TREC dataset), and re-generated if deemed insufficient. Then, reranking utilizes a MiniLM (initially trained on the MS-MARCO dataset) that is fine-tuned on data generated from the TREC dataset, and uses a complete (non-keyword) query generated by the base prompt. The final response generation utilizes the top 20 ranked passages.
- Please provide a priority for assessing this run. (If resources do not allow all runs to be assessed, NIST will work in priority order, resolving ties arbitrarily).
- 1 (top)
Evaluation Files