Draft_SOW_IRP_RAG_Approaches_Annotations_1.pdf

Statement of Work

Title: RAG Approaches Annotations: Datasets for Developing and Testing BioGen RAG Approaches

Objective/Purpose:

This project seeks to evaluate and verify the references attributed and the quality of LLM-generated answers.

Background:

Large language models (LLMs) have shown the ability to achieve state-of-the-art performance across a wide range of natural language processing (NLP) and information retrieval tasks. The National Library of Medicine (NLM) has been researching LLMs within the CHiQA-RAG project and is seeking support to help determine the effectiveness of LLMs. We anticipate sharing our findings from this work in the TREC 2024 BioGen task. Reporting will focus on (a) reference attribution and

(b) the quality and factuality of the text generated by LLMs to answer clinical questions asked by clinicians to (a) satisfy their own information needs and (b) answer health-related questions asked by their patients. For patients, we envision that the answers will be reviewed by clinicians and subsequently explained in plain language.

Scope and Description of Work:

In each task, Annotators will judge outputs of LLMs. Annotators must have Medical Doctors or Nurse Practitioner Degrees.

Task 1: Evaluation of the LLM answers for CHiQA-RAG

Annotators will receive Documents that have been generated by LLMs. A Document consists of a Question, a Generated Answer, and Cited Documents (see the Attachment: Document Example for more details). There will be 50,000 Documents in total to evaluate. Forty questions will be put through the best performing 40 LLMs to generate 25 Documents for each question.

The Annotators will evaluate if the statements and sources within the Cited Documents supports the Generated Answer as follows:

1A. Each document provided as reference will be judged on its relevance to the answer:

The Document Supports the answer — there is at least one sentence in the document that supports/agrees with the statement.

The Document Contradicts the answer – there is at least one sentence in the document that disagrees/states the opposite.

Neutral – the document is topically relevant, but lacks any information to validate or invalidate the assertion.

Not relevant – the document is not relevant to the sentence.

• 1B. Judge if the information provided in each answer statement linked to the Cited Documents o if the statement must be included/is critical to support the answer;

o if the statement is OK to have, but not critical to support the answer;

o if the statement provides too much information that clutters the answer, and o If the statement provides potentially harmful or misleading information that should not be included in the answer.

o If the generated answer statement is not support by any of the documents included with the answer Use other resources (e.g., search PubMed) to identify and capture the statement and source if support exists elsewhere.

NLM can provide Annotators with the platform that allows highlighting factual statements in the answers and labeling them as supported by the documents or not. Alternatively, companies may use their own systems that offer comparable functionality. For more information on Evaluations, please review the Attachment: Guidelines for Evaluating Answers.

Task 2: Evaluation of the LLM answers for MediQA-RAG

Annotators will receive Documents that have been generated by LLMs (see the Attachment:

Document Example for more details). There will be 50,000 Documents in total to evaluate. Twenty five questions will be put through the best performing 40 LLMs to generate 50 Documents for each question.

The Annotators will evaluate if the statements and sources within the Cited Documents supports the Generated Answer as follows:

2A. Each document provided as reference will be judged on its relevance to the answer:

• The Document Supports the answer — there is at least one sentence in the document that supports/agrees with the statement.

• The Document Contradicts the answer – there is at least one sentence in the document that disagrees/states the opposite.

• Neutral – the document is topically relevant, but lacks any information to validate or invalidate the assertion.

• Not relevant – the document is not relevant to the sentence.

2B. Judge if the information provided in each answer statement linked to the Cited Documents

• if the statement must be included/is critical to support the answer;

• if the statement is OK to have, but not critical to support the answer;

• if the statement provides too much information that clutters the answer, and

• If the statement provides potentially harmful or misleading information that should not be included in the answer.

• If the generated answer statement is not support by any of the documents included with the answer o Use other resources (e.g., search PubMed) to identify and capture the statement and source if support exists elsewhere.

NLM can provide Annotators with the platform that allows highlighting factual statements in the answers and labeling them as supported by the documents or not. Alternatively, companies may use their own systems that offer comparable functionality. For more information on Evaluations, please review the Attachment: Guidelines for Evaluating Answers.

Task 3: Evaluation of the plain language adaptations of PubMed abstracts generated by LLMs for CHiQA-RAG

Annotators will be provided with the 40 patients’ questions (same questions from Task 1) and 10 PubMed abstracts (400 total) that answer the question.

3A. For each abstract, the LLMs will be asked to identify medical terminology that is difficult to understand and provide an explanation, lay language substitution or generalization of the term. The annotators will judge 12-15 plain language terms/explanations generated by the LLM to determine the quality of the adaptation of the term to plain language in terms of:

• Technical accuracy o Does the adaptation correctly translate the term to plain language.

• Completeness o Does the adaptation convey the meaning of the term sufficiently to understand its meaning in the context of answering the patient’s question.

• Integrity o Does the adaptation use relevant plain language and present in a manner that is logical to readers

3B. For each sentence in the abstract, the LLM will be asked to translate the sentence to plain language, considering the previously presented information. For example, if an explanation of a difficult term was provided earlier, the term shouldn’t be explained again.

The annotators will judge each sentence adaptation generated by the LLM to determine the quality of the adaptation in terms of:

• Technical accuracy o Does the adaptation correctly translate to plain language.

• Completeness o Does the adaptation include all key details of abstract o Does the adaptation include enough translated details to maintain intent of abstract

• Integrity o Does the adaptation use relevant plain language and present in a manner that is logical to readers

Contract type:

Firm-fixed price contract

Period of Performance:

Tentative Period of Performance is September 2, 2024- October 4, 2024

Place of Performance:

Remote

Deliverables:

32,000 Evaluation of the LLM answers for CHiQA-RAG 40,000 Evaluation of the LLM answers for MediQA-RAG 400 Evaluations of the LLM Plain Language Translation for Medical Terminology 400 Evaluations of the LLM Plain Language Translation for Abstracts by Sentence

Inspection/Acceptance Criteria:

Datasets will be inspected on the below factors:

• Completeness in terms of quantity; 1000 datasets containing 1 Generated Answer and [x] Cited Documents.

• Relevancy; deliverables will be evaluated against standard judgements which have been developed by NLM.