Attachment 1 SOW_OPR_CHINA.docx

Statement of Work USAGM China Audience Estimation Project

1. INTRODUCTION

The United States Agency for Global Media (USAGM) is required to show evidence of performance and effectiveness. Common measures such as audience reach and reliability of programming have long served the USAGM in assessing the impact of our media on populations. As of 2023, the Agency needs new audience measures in China, a key market for Voice of America (VOA) and Radio Free Asia (RFA) and of primary strategic importance for the Agency.

Typically, the Agency uses nationally representative surveys in our markets to measure reach and performance. However, due to the sensitive and controversial nature of RFA and VOA in mainland China, USAGM does not anticipate being able to administer a nationally representative survey that directly asks respondents about their use of USAGM networks.

As such, the Agency seeks a task order to conduct research that estimates an audience reach figure for VOA and RFA offerings using non-probability samples, digital analytics data, and/or available third-party sources. The Agency prefers the estimate include TV, radio, and digital users, however, proposals that only estimate an audience reach figure for VOA and RFA digital offerings will be accepted. This estimate should be representative of either the national population or the population that uses the internet (going forward in this SOW, “representative” will be used to describe either of these two groups).

An USAGM Contracting Officer Representative (COR) and a USAGM Project Manager (PM) will be assigned after the contract award.

2. BACKGROUND AND RESEARCH NEEDS

USAGM is required to update the estimate of its audience in mainland China. The Contractor must propose a research method to accurately estimate the number of individuals in China who use VOA and/or RFA. This research must include an online panel survey of individuals in China which asks about their use of RFA and VOA. The representative estimate of users may be produced using any available combination of data (panel survey, online survey, digital analytics, etc.) and any statistical methods (regression weights, propensity weights, imputation, etc.). These methods, and their justification, should be outlined in the proposal in sufficient detail such that USAGM can understand the pros and cons of available methods.

USAGM understands that, based on the availability of different survey methods, the number of users will either be representative of internet users in China or representative of the total population and may include estimates for TV, radio, and/or digital users. The type of estimate and of which group the estimate is representative should be clearly outlined in the proposal.

The proposal must include a panel survey of individuals in China which asks about VOA and RFA use, and measures the media market and individuals’ media habits.

Results of the survey will inform decisions by senior, division and service managers regarding allocation of resources for producing and delivering content on a variety of media platforms (e.g., satellite, social media services). The results will identify opportunities offered by changes in the market (knowledge of and use of internet circumvention tools, partnerships with independent domestic and international media, etc.).

3. METHODOLOGY

There are a variety of research methods that USAGM finds acceptable to estimate USAGM usership in China, but the proposal must include a panel survey of individuals in China that directly asks about VOA and RFA use.

The panel survey must collect as much of the following information as possible considering the challenging research conditions in China:

1. the size of the audience (TV, radio, and/or digital) for VOA, RFA, or other international brands

2. the size of the regular and occasional audience for other international and domestic television, radio, and digital news agencies (where possible)

3. demographic characteristics and geographical distribution of these audiences; and

4. general patterns of media use.

In addition, a small number of questions concerning respondents’ views on various social and political questions may be included in the survey.

In addition to the data collected by a panel survey, Contractors may use any available data to help estimate users in China. USAGM will make digital analytics data and nationally representative data collected in 2017, 2014, and 2009 available to Contractors upon request.

Please detail in proposals which data and statistical methods will be used to estimate the number of USAGM users in mainland China and how much of the above information can be collected using the proposed research methods.

The estimated average interview duration for the panel surveys will be approximately 30 minutes. If a shorter or longer survey time is needed for the proposed research methods, please indicate that in the proposal and include separate costs for a 30-minute survey and the proposed survey length.

a. Timeline USAGM would like to have finalized audience estimates by the end of September 2023. However, USAGM will accept proposals that cannot meet this deadline, and prioritizes technical quality over speed. Please note, audience estimate figures may be submitted prior to the full analytical report. Contractors should include a detailed timeline estimate in their proposal.

b. Questionnaire USAGM anticipates that the questionnaire(s) will be developed alongside the Vendor, as many questions will need to be adapted for the research goals and the environment. Contractors shall indicate in the proposal in what languages the survey(s) will be administered.

The Contractor will translate the English questionnaire(s) into applicable languages. The Contractor will provide full versions of all translations in an MS Excel file. The data collection platforms must have an integrated module to allow for MS Excel translation file import and export functionality. Please note, the Contractor will send translations for approval to the USAGM-identified Program Manager AFTER the English questionnaire has been scripted into the data collection platform.

c. Sample Design and Analysis Plan

Sample size will be dictated by research methods. Contractors should indicate their preferred sample size. Contractors should include a detailed proposed sampling plan that indicates how individuals will be selected to participate in all proposed surveys. Please indicate, to the extent possible, how panels are constructed. The Contractor shall also clearly outline the group(s) that the sample is representative of (if any) and the methods that ensure that representation. In particular, the sampling plan should include details on standard demographic (age, gender, urbanicity, education) coverage, geographic coverage, coverage of ethnic groups, and additional characteristics that affect representativeness of the sample or research methods.

After selection, the Contractor will be required to submit an analysis plan to USAGM for review and approval. This plan should clearly outline how the nationally-representative estimates of users will be produced, including methods and all data sources. Please note, the information on the analytical and/or statistical methods required to produce the audience estimates is required in the proposal as well.

d. Pre-Field Data Quality Assurance

The Contractor will be responsible for creating the data collection script(s) in English and languages of the instrument.

USAGM will provide a codebook that details the required data structure that USAGM expect to be adhered to when the Contractor delivers the final data in an SPSS file.

The Contractor must include the name of the planned survey software in their proposal.

The Contractor will provide test links to the data collection script in English for USAGM approval. Once the script review using the test links has been approved, the Contractor will provide dummy data in an SPSS file. USAGM will check the dummy data for any errors that might not have been found during the questionnaire review and script review stages.

If dummy data provision is not feasible due to the survey method and/or timeline, please indicate this in the proposal and suggest alternative to ensure that the script functions as expected.

The Contractor will be responsible for the development and implementation of a coding scheme for all open-ended questions, as well as data cleaning.

e. Pre-Testing Survey

Depending on proposed methods, the Contractor will test the survey with respondents prior to the commencement of any fieldwork. The pre-test, if included, must include at least 5 USAGM users in China. The Contractor shall indicate if they plan to conduct a pre-test, and if so, details of that pre-test including methods and sample size. If a pre-test is conducted, USAGM will require a short pre-test report.

f. Fieldwork Quality Controls

Rigorous quality control measures will need to be implemented to ensure that the data used to estimate USAGM users is of the highest quality. The Contractor shall detail the pre-field, in-field, and post-field quality control measures they will use. If the Contractor proposes any interviewer administered research, including, but not limited to the pre-test), the agency requires weekly field progress reports.

g. Data Review and Processing

The Contractor must ensure the data is collected in accordance with applicable requirements on representativeness, as communicated by the USAGM’s Project Manager. The Contractor is also responsible for carrying out data quality checks as outlined in the proposal.

h. Dataset File Delivery (SPSS)

The Contractor will deliver an SPSS data file to USAGM PM and COR containing the full quota of completed interviews.

The Contractor is responsible for all analytical methods that will be used to estimate the audience size of VOA and RFA in China.

The SPSS file must include the following information for each “completed” interview;

· a unique respondent ID number,

· an interviewer ID number (if applicable),

· the date of the interview,

· the start and end times of the interview and

· the language of the interview.

The format of the start and end times must be HH:MM and these must be the physical timestamps at the point the interviewer asks the first survey question and completes the last survey question.

i. Technical Report

When delivering data, the Contractor will provide a detailed Technical Report that will enable the agency and network analysts to fully understand data collection conditions, data limitations, and processing protocols that are relevant to further analysis and interpretation.

The technical report must include a detailed, technical description of the analytical methods and processes used to estimate USAGM audience size. All scripts/syntax used to analyze the data and produce estimates will be provided to USAGM.

j. Analytical Report

Before beginning analysis, the Contractor will consult with USAGM Project Manager on report requirements. The Contractor will provide an analytical report written by a fluent English speaker, in colloquial English. The Contractor will participate in up to three rounds of revisions, which address substance, style, and copyediting errors.

The report will include but not be limited to:

1. Reach of VOA, RFA, and other international broadcasters

2. Use of and user characteristics of VOA and RFA

3. A description, written for a lay audience, of the analytical methods used to estimate USAGM’s audience

4. A detailed analysis of the media environment in China that incorporates data collected under this project and external sources, where applicable

5. Obstacles to and/or recommendations for increased reach and impact; and

6. A stand-alone executive summary of 1-2 pages featuring both text and key charts.

The full analytical report is not required at the time that data is submitted. If preferred, the Contractor can provide the dataset, technical report, and estimates of usership as early as possible (preferred end of September) and submit the full analytical report a full calendar month later.

k. Human Safety and Data Security The Contractor will submit an assessment of the risk to participants and research teams based on geo-political, public health, or weather conditions while carrying out required activities, as well as plans to limit risk, in the proposal. In particular, the Contractor shall detail the anticipated likelihood of government interference in the research, how they plan to overcome this, and how they plan to ensure participant safety. The Contractor shall also indicate if they have done sensitive research in this way previously, what degree of self-censorship they anticipate, and how they will minimize self-censorship.

Under applicable federal standards, personally identifiable information (PII) is any information that is linked or reasonably linkable to a specific individual. Examples include: name, street and email addresses, precise GPS coordinates of interview locations (3 decimal degrees and above), telephone and mobile numbers, date of birth, IP address, unique respondent ID number, interviewer ID number, PSU ID number, and codes for precise location, date of interview, start and end times of interview.

The Contractor must ensure that security and privacy controls (physical, technical and/or policy controls) are in place that mitigate the risk of unauthorized disclosure of PII collected in any phase of this project.

The Contractor must submit Privacy and Security Plans which, at a minimum, must include:

(a) a requirement that all PII be encrypted in transit and at rest;

(b) protection of PII with security and privacy controls that are commensurate with the harm to individuals that would be caused by disclosure of the information; and

(c) training of all employees and sub-contractors who will come into contact with PII gathered or handled on behalf of the USAGM in the course of work under this contract.

If recording GPS coordinates will pose a security or privacy threat to any interviewer, respondent, or members of their household, the Contractor will notify and seek approval from USAGM.

The Contractor must destroy all records of coordinates (from devices, data sets, or any other source and all copies of such data) at the end of the Period of Performance.

The Contractor must remove all survey and interview data from interview devices as used in field at the end of fieldwork.

The Contractor must securely transmit all survey and interview data from interview devices to the central data collection server.

The Contractor must securely deliver all data files to USAGM via Contractor FTP portals or Dropbox “Share” only. The data files delivered to USAGM should not include any PII data. E-mail should only be used as a communication dialogue platform. No files should be attached to any e-mail transmissions.

l. Communication

The Contractor will engage in routine communication with USAGM-identified Project Manager. The frequency of communication between the Contractor and the USAGM Project Manager will be “as required” by the phase of the project or as required by the USAGM Project Manager but must be no less frequent than twice a month during the period of performance. USAGM anticipates taking a large role in the design and data analysis of this project.

Examples of such communications may include:

· Bi-weekly phone call check-ins between the Contractor and the USAGM Project Manager.

· Debrief phone call after first interview and then additional debrief at least 2 days after all interviews are completed.

m. Contractor Personnel Requirements

Project managers employed by the Contractor on this project must have 8-10 years' experience managing survey research projects for international clients.

Statisticians employed by the Contractor on this project must have sufficient relevant education and experience in the proposed methods. This should be detailed in the proposal.

Persons translating documents from English to the languages of the survey must have experience translating technical documents.

All Contractor’s proposals must include the average turnover rate (in number of months) for field staff of any proposed subcontractor, if field staff will be used. The USAGM’s preference is that the Contractor’s average retention rates for field staff will be 24 months or longer.

4. CONTRACTOR RESPONSIBILITIES

The Contractor shall be responsible for data acquisition and basic quality control for the survey, including the following specific tasks:

· Developing, in close consultation with USAGM and stakeholder networks, the survey instrument that will align with the proposed research and estimation methods

· Translating, formatting, pre-testing, adapting, and programming of questionnaire

· Developing a detailed analysis plan which outlines the statistical methods that will be used to estimate audience size.

· Providing scripting, survey test links, and “dummy” data (if applicable) prior to launch of fieldwork, to ensure that scripting and coding of survey questionnaire functions accurately

· Arranging and supervising all aspects of data collection

· Training staff, where applicable

· Ensuring that data, including but not limited to personally identifiable information (PII), are safeguarded with security and privacy controls (i.e., physical, technical and policy controls) that are commensurate with the harm to interviewers, survey respondents and others that could result from unauthorized disclosure of the information

· Entering and processing the survey data, including carrying out data quality checks, encoding survey responses, and data cleaning if necessary

· Statistical analysis of findings and writing an analytical report

· Producing VOA and RFA audience estimates

· Compiling a detailed technical report

· Providing a final SPSS data file to the USAGM that fully adheres to the codebook mapping instructions

· Providing the syntax or scripting used to produce estimates of audience size

· Participating in routine check-in meetings

· Taking notes and preparing action steps from each meeting between the USAGM and the Contractor

5. CONTRACTOR DELIVERABLES

The Contractor will be responsible for the following deliverables:

· Timeline

· Analysis Plan

· Adapted questionnaire in English containing local market inputs

· Translated questionnaire(s)

· Survey programming (scripting)

· Survey test links

· Survey Dummy Data in SPSS, if not excluded during proposals

· Final SPSS data file(s)

· Syntax files of analysis

· Technical Report

· Analytical Report

6. DELIVERABLE ACCEPTANCE CRITERIA

The Contractor will provide work products and deliverables that comport with the acceptance criteria set forth below. The quality measures included in the acceptance criteria below will be applied to each work product and deliverable.

a. Adherence to Requirements— All work products and deliverables will comply with the Government’s requirements as defined in this Statement of Work (SOW)

b. Accuracy: All work products and deliverables will be free from errors and mistakes (all errors will be corrected) and be developed in accordance with applicable laws, regulations, policies, and procedures

c. Completeness: All work products and deliverables will have all parts or elements as required by this SOW

d. Clarity: All work products and deliverables will be easy to understand and written in colloquial English

e. Timeliness: All work products and deliverables will be available at the time required and generated on or before specified and mutually agreed to due dates or in accordance with a later mutually agree to scheduled date

f. Legend: Either the header or footer of all work products and data delivered to USAGM or USAGM’s Project Manager must be prominently labelled: “CUI-PRIV – INTERNAL USAGM USE ONLY.”

g. Format: All work products and deliverables will be submitted electronically.

h. No Significant Deficiencies - Deliverables will be free of significant deficiencies. Significant deficiencies are findings by the USAGM reviewing officials of serious omissions, inaccuracies, noncompliance with contract requirements and standards.

i. All deliverables become the property of the USAGM. The USAGM will have 10 business days from the date of submission to provide any comments or formally accept/reject each deliverable. The Contractor will retain no data rights in any Deliverable.

7. PLACE OF ACCEPTANCE

The USAGM Contracts Office (CO) or the duly authorized representative, such as the COR, as well as a Project Manager, will accept supplies and services to be provided under this contract. The USAGM COR and the USAGM Project Manager will be assigned after contract award.

8. NOTICE REGARDING LATE DELIVERY

In the event the Contractor anticipates that it will not be able to meet any delivery and/or performance requirements identified in this contract, it will immediately notify the Contracting Officer, and the Contracting Officer’s Representative (COR) orally and in writing giving pertinent rational and proposed corrective action(s). If the reason for the delay is not covered under the Excusable Delays Clause (FAR 52.249-14) of this contract, the Contractor’s notice should include the reason for its delay for the USAGM’s consideration.

The aforementioned “Notice Regarding Late Delivery” clause will not be construed as a waiver by the USAGM of any of its rights or remedies available under this contract including termination.

9. PROJECT MILESTONES AND PAYMENT SCHEDULE

The Period of Performance is six (6) months for this contract. Please note, USAGM is asking for finalized estimates of audience size by September 30, 2023.

The Contractor will submit the required deliverables to the COR in accordance with the following Project Milestone and Payment Schedule. Payment will be made when milestone deliverable is approved by the COR.

Project Milestones Billable upon approval by the COR Payment Schedule

KICK OFF:

Government provides drafts of research instruments, including

· Draft questionnaire in English

· Codebook Provided by USAGM’s COR or Project Manager

MILESTONE 1:

· Estimated timeline

· Analytical plan

· Adapted questionnaire containing local media market inputs

· Translated questionnaires

· Scripting and/or dummy data 33%

MILESTONE 2:

· Routine updates on survey implementation progress, including number of completes and demographics. (For any interviewer administered data collection, the agency will require weekly progress reports)

· Cleaned and complete dataset in SPSS

· Technical Report 33%

MILESTONE 3:

· Analytical Report

· Syntax files of analysis 33%