| Runtag | Org | What type of manually annotated information does the system use? | How is conversation understanding (NLP/rewriting) performed in this run (check all that apply)? | What data is used for conversational query understanding in this run (check all that apply)? | How is ranking performed in this run (check all that apply)? | What data is used to develop the ranking method in this run (check all that apply)? | Please specify all the methods used to handle feedback or clarification responses from the user (check all that apply). | Please describe the method used to generate the final conversational responses from one or more retrieved passages (check all that apply). | Please describe the external resources used by this run, if applicable. | Please provide a short description of this run. | Please provide a priority for assessing this run. (If resources do not allow all runs to be assessed, NIST will work in priority order, resolving ties arbitrarily). |
|---|---|---|---|---|---|---|---|---|---|---|---|
| baseline-gen-only-llama3.1-top5 (trec_eval) (ptkb.trec_eval) (paper) | coordinators | generation-only: system uses the given ranked list and only generates a response based upon that. | ['method uses large language models like LLaMA and GPT-x.'] | ['method uses other external data (please specify in the external resources field below)'] | ['method uses other ranking method (please describe below)'] | ['method is trained on other datasets (please describe below)'] | ['method does not treat them specially'] | ['method uses multiple sources (multiple passages)'] | llama3.1 | baseline-gen-only-llama3.1-top5 | 1 (top) |
| baseline-gen-only-gpt4o-top5 (trec_eval) (ptkb.trec_eval) (paper) | coordinators | generation-only: system uses the given ranked list and only generates a response based upon that. | ['method uses large language models like LLaMA and GPT-x.'] | ['method uses other external data (please specify in the external resources field below)'] | ['method uses other ranking method (please describe below)'] | ['method is trained on other datasets (please describe below)'] | ['method does not treat them specially'] | ['method uses multiple sources (multiple passages)'] | baseline-gen-only-gpt4o-top5 | baseline-gen-only-gpt4o-top5 | 1 (top) |