TREC 2025 Proceedings

CUET-qwen4B-v2

Submission Details

Organization
CUET
Track
Detection, Retr., and Gen for Understanding News
Task
Question Generation Task
Date
2025-08-08

Run Description

Is this run manual or automatic?
automatic
Is this run based on the provided starter kit?
no
Briefly describe this run
This run performs automated question generation for the TREC 2025 dataset using the unsloth/Qwen3-4B model, enhanced with few-shot prompting. It begins by loading the dataset of news articles and sets up a detailed prompt template containing two examples of ideal outputs to guide the LLM toward generating high-quality questions. The LangChain pipeline is used with HuggingFace's pipeline integration for efficient inference. Each topic (title and body) is passed through the LLMChain up to 3 times if needed, attempting to generate at least 10 valid, critical, investigative questions. A regex is used to extract and validate the questions. If fewer than 10 questions are generated after retries, the list is padded with "N/A" placeholders. Finally, all questions are cleaned and saved in TSV format for submission as CUET_run3.tsv.
What other datasets or services (e.g. Google/Bing web search, ChatGPT, Perplexity, etc.) were used in producing the run?
No external APIs, datasets or services (like Google Search or ChatGPT) are used during the execution of this code.
Briefly describe LLMs used for this run (optional)
The model used is unsloth/Qwen3-4B, a 4-billion parameter open-source model optimized by Unsloth for efficient inference. It is loaded with 4-bit quantization (load_in_4bit=True) to reduce GPU memory usage, and can dynamically support RoPE scaling to handle long inputs up to 2048 tokens. The model is accessed using HuggingFace Transformers and wrapped in a LangChain LLMChain with a custom prompt for few-shot learning. This allows the model to more reliably generate task-specific outputs.
Please give this run a priority for inclusion in manual assessments.
3

Evaluation Files

Paper