SOW FOR SOURCES SOUGHT - DRAFT.pdf

CCQDER Cognitive Interviewing Studies

Statement of Work

1. BACKGROUND AND NEED

The Centers for Disease Control and Prevention (CDC) is recognized as the lead federal agency for protecting the health and safety of people – at home and abroad, providing credible information to enhance health decisions, and promoting health through strong partnerships. CDC serves as the national focus for developing and applying disease prevention and control, environmental health, health promotion and education activities. As part of CDC, the National Center for Health Statistics (NCHS) located in Hyattsville, Maryland provides statistical information that will guide actions and policies to improve the health of the American People. As the Nation’s principal health statistics agency, NCHS leads the way with accurate, relevant and timely data. Within NCHS, The Division of Research and Methodology (DRM) plans, coordinates, and stimulates the NCHS' applied and basic research program in the fields of mathematical statistics, survey design and methodology, cognitive and interpretive aspects of survey measurement, and automated statistical, graphical, and geographical technologies, and conducts research in each of these fields. The Division serves as the Center's consultants in the fields of survey design and methodology, data collection, data analysis, and question design, and formulates and implements statistical standards in these areas.

Within DRM, The Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) was established as a resource for CDC, NCHS, and other Federal statistical agencies to develop and test survey instruments. Continuing in this capacity, the fundamental mission of the CCQDER is to improve the quality and usefulness of Federal statistics by improving the reliability and validity of CDC, NCHS survey instruments and providing an understanding of the substantive meaning represented in survey statistics. As an expanding focus, the CCQDER seeks to understand how response processes and response error might differ across socio-cultural or diverse language populations—work that is necessary for developing comparable health measures internationally as well as nationally, among varied cultural and economic populations.

Various qualitative methods are used by CCQDER researchers to evaluate survey questions including:

cognitive interviewing, focus groups, and usability testing. Purposive, small samples of respondents are recruited relevant to the survey questions being evaluated to participate in an hour or 1 ½ hour cognitive interview/focus group and are given remuneration. Cognitive interviews and focus groups are audio or video recorded.

To rigorously and systematically examine the cognitive interviewing, focus group, and usability testing data, CCQDER has developed an online qualitative research software program called Q-Notes. Q-Notes was developed specifically for the management and analysis of cognitive interviewing studies. Q-Notes was designed with qualitative data principles in mind such as the simultaneous pursuit of data collection and analysis and constructing analytic codes and themes from data. CCQDER views analysis of cognitive interviews as a five level pyramid, where each level provides a specific type of understanding. Performing a complete and detailed analysis is time consuming, especially when using pen and paper or even excel spreadsheets. Q-Notes provides interviewers and analysts with real-time access to interview data. The Q- Notes application helps to overcome the challenges of analyzing large samples and supports multiple geographical regions so that comparability can be examined across multi-national and multi-lingual respondents. Q-Notes helps project managers to easily monitor the status of a study as well as the quality of interview data. Finally, Q-Notes promotes the standardization of cognitive interviewing studies, ensuring consistency across interviews and a systematic analysis. Access to CCQDER studies is limited to CCQDER staff, contracted staff, and potential collaborators of each specific study.

Findings are written into a report along with a description of the methodology used to reach conclusions.

All findings are supported by empirical evidence from interviews, adding transparency and accountability to the research.

All final reports are submitted to Q-Bank. Q-Bank is an online database that houses cognitive interviewing and other question evaluation reports, making CCQDER question evaluation studies accessible to other federal statistical agencies, the larger survey research community, and the public.

CCQDER’s core mission is to conduct cognitive interviewing question evaluation studies. Question evaluation is a vital part of survey development. Evaluation reduces non-response error, which ensures comparable data and supports data users in their analyses of survey data. In addition, the Office of Management and Budget stipulates in Section 1.4 of The Standards and Guidelines for Statistical Surveys: “Agencies must ensure that all components of a survey function as intended when implemented in the full-scale survey and that measurement error is controlled by conducting a pretest of the survey components or by having successfully fielded the survey components on a previous occasion.”

1.2 PURPOSE

The purpose of this service contract will be to conduct cognitive interviewing evaluation studies on a variety of demographic, social, and health-related topics in various languages.

2. SCOPE OF WORK

The contractor will be responsible for all activities necessary to conduct, complete, remunerate, document, and deliver the results of a specific cognitive interviewing study using test questions provided by NCHS. The range of cognitive interviews to be conducted for any one specific study is 10-200. The contractor will be required to enter into a Designated Agency Agreement (DAA) with CDC/NCHS. The contractor may be responsible for completing a CIPSEA Information Protection Plan as determined by the NCHS Confidentiality Officer and the NCHS Information Systems Security Officer. The contractor may also be required to provide inspection of contractor facility that collects, uses, or stores cognitive interviewing and cognitive interview support data under a Designated Agent Agreement (DAA).

If the execution of work to be performed by the contractor requires the hiring of sub-contractors the prime contractor must clearly state this in the proposal. Sub-contractors must be identified and the work they will perform must be defined.

3. REQUIREMENTS

The contractor must have experience conducting detailed analysis of conducted cognitive interviews, to include constructing analytic codes and themes from cognitive interviews using the five level pyramid method (Level 1 - Conducting interviews; Level 2 – Summarizing interview notes; Level 3 – Comparing across respondents; Level 4 – Comparing across groups; Level 5 – Drawing conclusions about question performance) described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July 2014.

Contracted staff conducting cognitive interviews must have a training certificate from Collaborative Institutional Training Initiative (CITI) for Social & Behavioral Research and RCR Combined training courses. Contracted staff must provide CITI expiration date to Contracting Officer Representative (COR).

4. TASKS TO BE PERFORMED

Task 4.1: Cognitive Interviewing Study This task may be funded at a later date through a contract modification if and when the CCQDER is required to conduct a cognitive interviewing study.

The contractor will be responsible for all activities necessary to conduct, complete, remunerate, document, and deliver the results of a specific cognitive interviewing study using test questions provided by NCHS. Provision of token remuneration payment to survey respondents is usual, customary, reasonable, and is in direct support of CCQDER’s mission.

Cognitive interviews will be completed in accordance with a schedule to be specified. All work under this contract shall be performed at the contractor’s cognitive testing facility, offsite in the field, or at CCQDER’s question laboratory. NCHS will provide contracted personnel whose specific task is to conduct cognitive interviews with government-furnished encrypted laptops and government-furnished encrypted flash drives. For cognitive interviews conducted at the contractor’s facility, the contractor shall furnish all the necessary personnel, services, facilities, and property (locking workstations, chairs, locking filing cabinets, office supplies, telephones (landline/mobile phone), fax, printers, and do all other things necessary for or incidental to the performance of the work as described in the following tasks. For cognitive interviews conducted at NCHS in CCQDER’s question laboratory, NCHS will provide contracted personnel access to CCQDER’s physical cognitive laboratory. NCHS will not provide any additional government-furnished property including workstations, chairs, filing cabinets, office supplies, telephones (landline/non-mobile phone), fax, printers, etc.

In summary, the contractor will perform the following activities, with close and continuing oversight from the CCQDER’s Principal Investigator (PI) assigned to the specific cognitive interviewing study:

Subtask 4.1.1 - Communicate regularly with NCHS

• To facilitate oversight, the contractor shall establish e-mail contact with the COR and the CCQDER PI assigned to the specific study within 6 business days of the specific cognitive testing contract award and maintain that contact throughout the study. The CCQDER PI assigned to the specific study will expect to be in frequent e-mail and/or telephone contact (once a week, at a minimum) with contractor’s specific study cognitive interview team to review progress.

Subtask 4.1.2 – Review background materials

• Immediately following award of a specific cognitive interviewing study, the contractor shall review CCQDER’s Office of Management (OMB) and NCHS Ethics Review Board (ERB) approved study specific clearance packages to enhance familiarity with the background and scientific basis of the project. The PI will provide the relevant materials.

Subtask 4.1.3 – Specific Study kick-off meeting

• Within 10 days of award, contractor personnel assigned to the specific cognitive interviewing study shall meet (remotely/in-person) with CCQDER PI to review the tasks and to discuss issues related to the conduct of the work.

• Within 5 business days after the initial study specific kick-off meeting, the contractor shall write a formal letter to the COR, summarizing the contractor’s understanding of any significant issues needing clarification or further elaboration, including the schedule, staffing, and other matters.

Subtask 4.1.4 - Comply with confidentiality requirements

• A Designated Agent Agreement (DAA) between NCHS and the contractor will be executed. All data collection, handling, transmittal, and storage procedures must comply with CDC/NCHS regulations for the protection of subjects. Cognitive interviewing data will be protected under NCHS confidentiality legislation including the Confidential Information Protection and Statistical Efficiency Act or CIPSEA (PL 107-347) and Section 308(d) of the Public Health Service Act (42 U.S.C. 242m). No CIPSEA information shall be removed from the contractor office or accessed remotely. Clarification will be provided in the DAA. All procedures will be reviewed by the NCHS Confidentiality Officer for compliance.

• As a part of the training for all contracted staff having access to the data, the contractor shall include awareness of staff responsibilities under the Confidential Information Protection and Statistical Efficiency Act (CIPSEA) and Section 308(d) of the Public Health Service Act (42 U.S.C. 242m). The contractor shall also ensure that all such contracted staff complete NCHS Confidentiality training at https://www.cdc.gov/nchs/training/confidentiality/training/. Submit a certificate of completion for the training, and sign a Nondisclosure Affidavit (provided by NCHS/Attachment 1). The contractor shall prepare and submit a memo to the COR that lists all persons who have completed NCHS Confidentiality Training and have also signed a Nondisclosure Affidavit. The contractor shall also send hardcopies of the NCHS Confidentiality Training certificates and original signed hardcopies of the Nondisclosure Affidavits to the COR.

After the first memo is submitted, the contractor shall submit a memo on the last day of each month listing all new persons who have taken NCHS Confidentiality Training and signed an Affidavit along with the training certificate and original signed hardcopy of the Affidavit.

• No video or audio recording data may be sent to or accessed from the contractors (and/or any subcontractors) home or telecommuting work site or removed from contractor’s office.

Exceptions: Contractors (and/or any subcontractors) may access respondent’s confidential information (e.g., respondent demographics and geographic identifiers) while working offsite.

• At the conclusion of the study, the contractor will return to NCHS any data obtained or generated for a specific cognitive interviewing study. The contractor shall destroy all study specific respondent Personal Identifiable Information (PII), after receiving confirmation from CCQDER that the recordings and all documentation including telephone screeners, informed consent documents, respondent data collection sheets, special consent documents, were received, to eliminate the ability to identify respondents participating in cognitive interviewing studies.

• The contractor will provide a memorandum to this effect detailing the disposition of data.

• The contractor may be responsible for completing a CIPSEA Information Protection Plan as determined by the NCHS Confidentiality Officer and the NCHS Information Systems Security Officer. If required, the Plan will document how the contractor safeguards and controls information in their systems, dataflow diagram(s), and associated data services before submitting the deliverables to NCHS for comments/acceptance. Access to CCQDER cognitive interviewing data must be restricted to only those approved to access the data/or collect data (i.e. NCHS designated agents).

Subtask 4.1.5 - Preparatory Activities

• Provide translations of non-English materials as necessary including questionnaire, advertisements/flyers/advance letter, telephone screener, informed consent form, respondent data collection sheet, thank you letter, special consent form, etc.

Subtask 4.1.6 - OMB and NCHS Ethics Review Board (ERB) Clearances

• All CCQDER cognitive testing studies must receive OMB clearance and NCHS ERB clearance prior to start of a specific cognitive interviewing study;

• CCQDER will garner OMB and NCHS ERB clearances on behalf of the contractor.

https://www.cdc.gov/nchs/training/confidentiality/training/

Subtask 4.1.7 - Recruitment

• Contact, screen, and schedule prospective respondents for cognitive testing according to

CCQDER’s recruitment design plan, using CCQDER OMB & NCHS ERB approved forms, following established procedures and controls;

• Ensure prospective respondents meet all the specifications of a particular study.

Subtask 4.1.8 - Data Collection

• Administer informed consent documents, acquire informed consent, and other paperwork (Respondent Data Collection Sheet) following established procedures and controls and maintain documentation until completion when documentation is to be sent to CCQDER’s COR via Federal Express;

• Conduct cognitive interviews in using the method described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July 2014;

• Operate recording equipment (government issued encrypted laptop) in order to record interviews conducted at contractor’s facility or conducted in the field following established procedures and controls;

• At the completion of the study, copy recordings onto government issued encrypted flash drive and send recordings to COR via Federal Express.

Subtask 4.1.9 - Remunerate Respondents

• During the planning of the cognitive interview, focus group, or usability test research design, CCQDER staff will determine the appropriate remuneration rate level in consultation with the NCHS ERB and the OMB;

• Provide cash remuneration to respondents participating in cognitive interviewing study following CCQDER established procedures and controls;

• Manage and maintain remuneration accounting.

Subtask 4.1.10 - Data Entry

• Enter cognitive interviewing notes into Q-Notes using the method described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July 2014;

• Cognitive interviewing notes of non-English interviews must be entered in English;

• No direct or indirect personally identifiable information (PII) will be entered into Q-Notes.

OPTIONAL Subtask 4.1.11 - Data Analysis & Report (Project Specific)

• Perform a complete and detailed analysis (in collaboration with or under the guidance of the CCQDER study PI as necessary) of conducted cognitive interviews constructing analytic codes and themes from the cognitive interviews using Q-Notes and the five-level pyramid method (Level 1 - Conducting interviews; Level 2 – Summarizing interview notes; Level 3 – Comparing across respondents; Level 4 – Comparing across groups; Level 5 – Drawing conclusions about question performance) described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July 2014.

• Prepare study report (in collaboration with or under the guidance of the CCQDER study PI as necessary) describing methodology, sample selection, recruitment, and analysis of test questions as described in Cognitive Interviewing Methodology (2014) by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla;

• Findings should be supported by empirical evidence from the conducted cognitive interviews;

adding transparency and accountability to the research;

• Submit for the PI’s review a first draft, and a final draft of the report;

• Draft and final report will be submitted as electronic files (in Microsoft WORD), and may include appendices;

• Respond to first draft and final draft comments by the PI.

OPTIONAL Subtask 4.1.12 – Travel (Project Specific)

• Travel may be required in order to conduct cognitive interviews in various geographic locations for specific projects.

OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study This optional task may be funded at a later date through a contract modification if and when the CCQDER is required by NCHS to conduct a facility inspection.

• In accordance with the CIPSEA, NCHS conducts inspections of any off-site facility that collects, uses, or stores NCHS restricted use data under a DAA. These inspections ensure the physical and information systems security of the NCHS restricted use data. The inspections assess and document compliance with protection procedures outlined in the DAA.

• Upon NCHS request, the contractor shall facilitate and assist with an on-site inspection where data and NCHS equipment are kept or used for the purposes of inspecting data security arrangements as outlined in the DAA.

• The contractor shall submit a report detailing the results of the inspection to the NCHS Confidentiality Officer within five business days of the inspection.

5. GOVERNMENT FURNISHED PROPERTY/MATERIALS/INFORMATION

NCHS will provide contracted personnel whose specific task is to conduct cognitive interviews with government-furnished encrypted laptops and government-furnished encrypted flash drives to be used for official business only.

The control and accountable record keeping for such property shall be retained by the Government. The Contractor shall remain accountable for loss or damage but will not be required to submit an annual inventory or place its own bar code on the items. The Government will provide and install CDC/NCHS bar code labels and other identification for the Government-furnished property.

For cognitive interviews conducted at NCHS in CCQDER’s question laboratory, NCHS will provide contracted personnel access to CCQDER’s physical cognitive laboratory.

NCHS will provide electronic copies of all English-language study specific NCHS ERB and OMB approved cognitive testing forms including: NCHS ERB and OMB approval notices to conduct the study specific cognitive testing study, advertisements/flyers, advance letter (if used), telephone screener, informed consent, respondent data collection sheet, thank you letter, and special consent.

NCHS will provide instructions for administering CCQDER’s informed consent and special consent for expanded use of video and audio recordings documents.

6. PERIOD OF PERFORMANCE

Total period of performance for this procurement is 09/01/2020-08/31/2025. POP assumes exercised base year + 4 option years of Task 4.1 Cognitive Interviewing Study and OPTIONAL Task 4.2 Facility Inspection for Cognitive Interviewing Study. See table below.

Description Estimated Period of Performance

Initial Award or Option

Task 4.1: Cognitive Interviewing Study 09/01/2020-08/31/2021 Initial

OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study

09/01/2020-08/31/2021 Initial

Task 4.1: Cognitive Interviewing Study 09/01/2021-08/31/2022 Option Year 1 OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study

09/01/2021-08/31/2022 Option Year 1

Task 4.1: Cognitive Interviewing Study 09/01/2022-08/31/2023 Option Year 2 OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study

09/01/2022-08/31/2023 Option Year 2

Task 4.1: Cognitive Interviewing Study 09/01/2023-08/31/2024 Option Year 3 OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study

09/01/2023-08/31/2024 Option Year 3

Task 4.1: Cognitive Interviewing Study 09/01/2024-08/31/2025 Option Year 4 OPTIONAL Task 4.2: Facility Inspection for Cognitive Interviewing Study

09/01/2024-08/31/2025 Option Year 4

7. PLACE OF PERFORMANCE

TASK 4.1/Cognitive interviewing studies including cognitive interview support are to be performed at the contractor’s facility, in the field, or at CCQDER’s question laboratory, as appropriate and necessary. Optional Task 4.2/Facility inspection for cognitive interviewing study including cognitive interview support is to be performed at the contractor’s facility.

No video or audio recording data may be sent to or accessed from the contractors (and/or any subcontractors) home or telecommuting work site or removed from contractor’s office.

Exceptions: Contractors (and/or any subcontractors) may access respondent’s confidential information (e.g., respondent demographics and geographic identifiers) while working offsite.

8. DELIVERABLES

Deliverables are to be made to the National Center for Health Statistics to the designated COR located at 3311 Toledo Road, Hyattsville, MD 20782.

Note about CCQDER deliverables: Deliverable dates cannot be attached to cognitive interviewing and cognitive interview support tasks to be performed. Sponsor’s deadlines for developing, testing, and conducting analysis of the survey questions prior to programming or fielding of the survey questions drives the majority of the tasks to be performed.

Deliverable dates will be mutually agreed to once a contract modification award has been made for a specific cognitive interviewing study.

Task Deliverable Due Date Task 4.1:

Cognitive Interviewing Study

Submit a CIPSEA Information Protection Plan if required by the DAA.

If required, deliverable date to be determined.

Complete fingerprinting and background check required to access CDC equipment using a HSPD12/PIV badge.

Immediately following award.

Complete NCHS Confidentiality Training Within 10 days of award of specific cognitive interviewing study.

Review CCQDER’s Office of Management (OMB) and NCHS Ethics Review Board (ERB) approved clearance packages to enhance familiarity with the background and scientific basis of the project.

Within 10 days of award of specific cognitive interviewing study.

Read and sign Nondisclosure Affidavit. Within 10 days of award of specific cognitive interviewing study.

Kick-off meeting with contracted personnel assigned to specific cognitive interviewing study and CCQDER PI to review tasks and discuss issues related to the conduct of the work.

Within 10 days of specific cognitive interviewing contract award.

Complete Collaborative Institutional Training Initiative (CITI) for Social & Behavioral Research and RCR combined training courses.

Provide CITI training certificate and expiration date to COR prior to start of specific cognitive testing study.

Provide translations of non-English materials as necessary including questionnaire, advertisements/flyers/advance letter, telephone screener, informed consent form, respondent data collection sheet, thank you letter, special consent form, etc.

Completed within agreed-upon timeframe.

Call, screen, and schedule prospective respondents for cognitive testing based on evolving sponsor-driven cognitive testing study schedule.

Completed within agreed-upon timeframe.

Administer informed consent documents, acquire informed consent, and other agreements/paperwork and maintain documentation until the documentation can be sent to the COR via Federal Express.

Completed within agreed-upon timeframe.

Provide cash remuneration to respondents participating in cognitive interviews.

Completed within agreed-upon timeframe.

Task Deliverable Due Date Conduct cognitive interviews in using the method described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July, 2014.

Completed within agreed-upon timeframe.

Operate recording equipment (government issued encrypted laptop) in order to record interviews conducted in the field. Copy recordings onto government encrypted flash drive and send recordings to COR via Federal Express when study has been completed.

All recordings must be sent to the COR via Federal Express 15 days after study has been completed.

Make remuneration payments to cognitive testing respondents.

Completed within agreed-upon timeframe.

Enter cognitive interviewing notes into Q-Notes using the method described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July, 2014.

All notes must be entered into Q-Notes no later than 15 days after the last interview.

Ensure that no direct or indirect personally identifiable information (PII) is included in interview notes in Q- Notes.

Within 24 hours, a CCQDER staff member will review interview notes and will delete any direct or indirect personal identifiable information (PII) if found.

As necessary, analyze conducted cognitive interviews using Q-Notes and the method described in “Cognitive Interviewing Methodology (Wiley Series in Survey Methodology)” by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla published July, 2014.

All analyses must be conducted no later than 3 weeks after the last interview.

As necessary, submit first draft, final draft, and final report to PI in Microsoft WORD as described in Cognitive Interviewing Methodology (2014) by Kristen Miller, Valerie Chepp, Stephanie Willson, Jose Luis Padilla.

First draft report must be submitted to the PI in Microsoft WORD no later than 3 weeks after the analysis is conducted.

At-least-weekly status reports on data collection. Deliverable date to be determined.

Return to CCQDER any data obtained or generated for a specific cognitive interviewing study.

Conclusion of specific contracted cognitive testing study.

Destroy all study specific respondent Personal Identifiable Information (PII), after receiving confirmation from CCQDER that the recordings and all documentation including telephone screeners, informed consent documents, respondent data collection sheet, special consent, were received.

Send memo to COR detailing disposition of data.

Optional Task 4.2:

Facility Inspection for Cognitive Interviewing Study

Upon request, contractor shall facilitate and assist with an on-site inspection where data and CDC equipment are kept or used.

Completed within agreed-upon timeframe.

Submit a report detailing the results of the inspection to the NCHS Confidentiality Officer.

Report must be received within five business days of the inspection.