Attachment D - Amendment 00001.pdf

Attachment D - Amendment 00001 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

Question # Question Title RFQ Section Number/Page Number Vendor Questions Government Comment

1 Incumbent info I was wondering if this is a new requirement or if there is an incumbent that fulfilled the same/similar services? If there is an incumbent, could you please provide the contractor's name and contract number?

These services have been performed in the past by SwissPeaks under contract # BBG50-A-17-0105

2 Incumbent info

We would like to know if this is a brand new contract OR if there is (was) an incumbent performing these services.

If not brand new, could you please provide the current / previous contract number?

See response to Question #1

Can you provide insight on the volume of participants in each survey?

and/or Will the agency need assistance in determining the appropriate audience size for each type of survey instrument? (Rationale: this will help me determine how many data developers I will need for this project)

The standard sample size for National Surveys is n=2,000. In countries with large populations, sample sizes are sometimes larger (Ex. DRC, n=3,000, Russia, n=6,000). Omnibus surveys are typically 1,000, although in large countries can be higher (ex. Brazil, n=2000, India, n=3,000).

4 QA/QC and definition of defective services

RFQ Part 1, General Information, Quality Control, p.2

Re this: "The contractor shall develop and implement procedures to identify, prevent, and ensure non-recurrence of defective services," we would like to confirm that we will be apprised of QA and QC requirements placed on local vendors by USAGM, as well as the definition of "defective services" by those vendors.

Yes. USAGM will share all requirements for Vendors.

5 Survey database

RFQ Part 1, General Information, Scope, p.2

(Also referenced in Attachment A, SOW,

4. SPECIFIC TASKS, 1.7 Produce final datasets)

Re this: "The agency requires data management services to ensure the quality of survey data, standardized data collection and unification, in support of a survey database." Is there a particular database application currently in use (and what is it if so), or are we to provide a database solution?

Currently, USAGM data uses SQL to store data, and dashboards are built using PowerBI. That is the system that we anticipate using for the next year or more. This contract does not cover the provision of a database solution, just the services to ensure that the data can be easily and accurately pulled into the existing system.

6 Survey duration RFQ 1.1, p.4 RFQ reports an average "core media questionnaire" duration of 18 minutes.

Checking that this is a different instrument than the one we reviewed for Somalia in June 2024, which had a reported average length of 33:13 minutes.

Please refer to the revised SOW incorporated as part of Amendment 0001

Pre-Submission Inquiries Spreadsheet - RFQ # Solicitation # 951700-25-Q-0009

Copy of Data QC QA 01212025 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

7 Work schedule Attachment A, SOW

3. SCOPE

For us to assess our resource allocation, can you provide information on the approximate schedule on which survey components (questionnaires, derived variable plans, script-testing links, interim data, field data, final data) will be delivered to us for review, and whether we should expect to receive these materials for individual countries sequentially, or for multiple countries (or all countries) simultaneously? Additionally, during which months of the year are the bulk of tasks expected to occur?

For this year, all Full Media surveys will be fielding at approximately the same. All surveys are currently slated to begin in April, and they should all last approximately 6 week. As such, all deliverables will be sent to the vendor to review at approximately the same time (estimated within 1 week of each other), although it's unlikely that all would be sent on the exact same day.

For this year, Deliverables 1.1, 1.2, 1.3, and 1.4 will be in February and March, 1.5 in April, 1.6 in May/June, and 1.7 in June/July. These are approximate and subject to change.

The government expects the schedule to remain consistent in future years, but they are subject to change based on field team availability, dates of Ramadan, and changes to the Questionnaire.

8 Specific tasks Attachment A, SOW

4. SPECIFIC TASKS

We would like to confirm that review of sample designs, population parameters and weighting protocols are not among the tasks included in this

SOW.

Correct, those services are not requested at this time.

9 Full survey vs. omnibus review Attachment A, SOW

4. SPECIFIC TASKS

Do deliverables for omnibus surveys differ from deliverables for full surveys, and how so? Yes, omnibus surveys typically only require deliverables 1.3 and 1.7.

10 Definitions of interim data and field data

Attachment A, SOW

4. SPECIFIC TASKS,

1.5 Review Interim Data,

1.6 Field Data Review

We'd like to clarify the difference (if any) between interim data and field data.

Do interim data consist of both a substantive dataset and a contact dataset for each survey country? Are field data simply the final versions of the interim data, or do they differ as to content?

They are the same set of variables, however interim data is raw data, while final data is codebook compliant.

In the past, USAGM has not asked that contact data is checked in the interim data. This stage of review has mostly just to confirm that no filtering issues were missed during scripting. However, OPR may ask to review contact data in the future.

11 Topline tables Attachment A, SOW

4. SPECIFIC TASKS,

1.6 Produce final datasets

Does USAGM have a standard or preferred format for topline tables? Do tables contain cross-national and/or trended results?

We has previously used topline tables in excel and are generally happy with this format. However, USAGM is open to other formats as well as long as they are easy to use and intuitive. Topline tables are only produced on the single dataset (so not cross-national or trends). They display all fielded variables and derived variables by gender, age groups, education, urbanicity, and the potential of a few additional variables upon request.

12 Deliverables deadlines Attachment A, SOW

5. DELIVERABLES

Turnaround time for several of the deliverables is quite tight, given the level of scrutiny and reporting we like to deliver - a potential concern particularly if we are required to deliver materials for multiple countries simultaneously. Is flexibility in this regard possible?

Some deliverables have flexibility, but that is largely dependent on the timeline. We do typically allow for additional days for certain tasks, such as DV plan review, if the field schedule allows for that, as it all depends on field schedule. Interim data is always firm, as we do not want to lose time in field.

Copy of Data QC QA 01212025 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

13 Weekly/monthly reports

Attachment A, SOW

7. INSPECTION AND ACCEPTANCE

CRITERIA and

8. OTHER DIRECT COSTS

Beyond the deliverables listed in Section 5, Section 7 calls for "weekly reports describing the work performed" and Section 8 refers to "the Monthly Progress Reports." Can you elaborate on these - are weekly and/or monthly reports required in addition to Section 5 deliverables, and if so what is the required content of monthly reports?

During periods where there is a high volume of activities, the government would like weekly reports. During periods where there is less work, monthly reports are acceptable. Reports would contain brief description of the activities completed in the time frame.

14 Place of performance Attachment A, SOW

13. PLACE OF PERFORMANCE

Re this: "Services are expected to be executed within USAGM network/infrastructure/platforms utilizing virtual work capabilities" - we would like to confirm that we will be able to work remotely using our in-house IT.

Please refer to the revised SOW incorporated as part of Amendment 0001.

USAGM, the networks, and fieldwork vendors utilize Dropbox to tranfer files. The vendor providing data QC services will be expected to obtain their own accounts in order to work in Dropbox to receive and upload files. The vendor may work with files on their own IT infrastructure after that, and then upload completed work to Dropbox for the agency to download and archive.

15 QC Process for Full surveys

vs. Omnibus Page 2 - Scope

Is the same QC process used for both sets of surveys (10 full surveys and 20 omnibus surveys) each year? Or is the QC process different in some respects for the omnibus surveys?

For this year, Omnibus surveys will require deliverables 1.3 and 1.7, while full surveys require all deliverables. We could order additional deliverables on Omnibus surveys in the future, but do not plan to do so for Fiscal Year 2025 surveys.

16 Survey Length and Variable Types Page 2 - Scope

Approximately how many variables are on each survey (full and omnibus) and what proportion of those are open-ended? Are all 90 countries surveyed with the execution of every full and omnibus survey?

The number of variables changes, depending on the number of brands and programs contained in the survey; however a rough estimate is that full surveys range from 600-900 variables, and omnibus surveys range from 300-600.

17 Questionnaire Review Page 2 - Review Questionnaire

For Task 1.1 (Media Questionnaire Review), does the contractor review individual distinct questionnaires for each country, or would the contractor review one base questionnaire with imbedded variants on filters and coding for those countries impacted?

The Vendor will review distinct questionnaires for each market for full media surveys (QQ review is not needed for omnibus surveys). The individual QQs are all based on the same questionnaire, but significant variation exists between market. Part of the QQ review is ensuring that coding on each individual QQ matches the coding in the core QQ where applicable.

18 Accessibility of USAGM Staff to Contractor Page 2 - Quality Control

Do references to the "contracting officer" refer to the procurement officer or the point person within the USAGM research office? Will someone within the research office serve as the COR?

Contracting Officer is Procurement Officer. Contracting Officer's Representation (COR) is someone within Research Office.

19 Status of Quality Assurance Surveillance Plan Page 2 - Quality Assurance

Does USAGM Research Office already employ a Quality Assurance Surveillance Plan or is this to be developed? If a QASP is already employed, can it be shared as part of the submission process?

It is not currently employed. Vendor may submit QASP.

20 Error Rate Pages 2, 4, 5, 6

Given the significant effort assigned to reviewing for errors in various stages of the process, what is the average rate of errors in each: the script review, interim data and field data reviews? Will the body of information already encountered from prior assessments be made available to the successful contractor as new systems are being developed?

The average rate of errors varies substantially on projects, largely depending on who our fieldwork vendor works with, and the quality assurance protocols they employ.

Copy of Data QC QA 01212025 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

21 DV Plan Page 5 - Review DV Plan Does a standard Derived Variable (DV) Plan exist or is this something that must be developed uniquely for each survey?

DV plans are developed uniquely for each survey, and is a collaborative effort between USAGM and the networks. All media survey DV plans have similar elements.

22 Scripting Language Page 5 - Script Testing Is there a standard CAPI/CATI system and scripting language employed for all USAGM surveys? Is this different for the the 20 Omnibus surveys?

OPR allows fieldwork vendors to script in different applications. OPR cannot, at this time, dictate that all vendors use the same software. We reserve the right to deny a fieldwork vendor the ability to use a particular program, but only if that program cannot meet our requirements. Most fieldwork vendors typically use off-the-shelf survey software platforms such as SurveyToGo. A few vendors use proprietary, in-house applications for CAPI and CATI surveys. For script review activities, the data QC vendor would be responsible for attaining access to off-the-shelf survey software, and USAGM would work with field vendors to assist the data QC vendor with gaining access to proprietary survey software.

The omnibus surveys are all scripted in SurveyToGo, OPR does not require script review of omnibus survey scripts.

23 Time Management Page 6 - Deliverable Schedule What is the typical overall period of time from Task 1.1 (Media Questionnaire Review) through Task 1.7 (Produce final Datasets)?

Typically, this process takes 6-9 months. An example based on FY2025 is that questionnaire review would be in January, script review in February, fieldwork in April, and data QC in the end of June and July. These dates can always shift based on fieldwork and issues on the ground, and any major issues found with data QC at any level can delay the preferred timeline.

24 Security Clearance SOW Page 6 - Public Trust Public Trust clearance for the contractor may take weeks to attain, might this clearance be achieved post-award and be investigated concurrently as the project progresses?

Please refer to the revised SOW incorporated as part of Amendment 0001

25 Quantity of Surveys Page 12 - Volume II Price Quotation Attachment C shows a quantity of 12 for each Task and 32 for Task 1.7.

Should this be based on 10 surveys mentioned earlier in the RFQ or is it really 10-12 Media surveys and 20 Omnibus?

The number of media surveys each year varies based on the government's budget and the agency's research priorities. We estimate that over the lifetime of this BPA, we will have yearly 10-12 Full Surveys and 20 Omnibus surveys, but that is an estimate and could vary.

For FY2025, we have 11 full media surveys, will request partial servies on one full media survey, and will have approximately 10 omnibus surveys.

Omnibus services are more fluid than Full Surveys.

Copy of Data QC QA 01212025 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

26 Approximate number of questions RFQ Part 1, Page 2, Scope Approximately how many questions are in a full survey vs omnibus survey?

Please refer to the revised SOW incorporated as part of Amendment 0001.

The number of questions administered in each survey varies based on market.

Omnibus surveys vary in administered length from 5 - 6 minutes.

Full surveys typically take approximately 25 to 30 minutes. The number of questions administered in each survey varies based on market.

27 Open ends RFQ Part 1, Page 2, Scope Approximately how many questions are closed vs. open?

OPR contracts provide for up to 4 truly open ended questions in our surveys. However, there are many "other, specify" questions throughout the survey. The fieldwork vendor is responsible for translating and coding those response options, while the QC Vendor is only responsible for ensuring those answers are properly cleaned, translated, and coded. If the QC vendor finds that open-ends are not properly formatted, the data is sent back to the fieldwork vendor to fix.

28 Sample size RFQ Part 1, Page 2, Scope What is the average sample size for a full and omnibus survey?

The standard sample size for full media national surveys is n=2,000. In countries with large populations, sample sizes are sometimes larger (Ex.

DRC, n=3,000, Russia, n=6,000). Omnibus surveys are typically 1,000, although in large countries can be higher (ex. Brazil, n=2000, India, n=3,000).

29 Number of Scripts RFQ 1.4, page 5 How many scripts does USAGM anticipate per full and omnibus survey? Is this counted on a per-country or per-survey basis?

There will be one script (with multiple reviews) for each full media survey. For this year, OPR will not request script review on Omnibus surveys.

30 Contact attempts RFQ 1.5.2, page 5 Does USAGM mean the number of contact attempts for each respondent before substituting, or the contact information (email, phone, etc) of each respondent?

This refers mostly to ensuring that household and respondent level selection procedures are followed. This may involve looking at contact attempts by household and interviewer, confirming that contact attempts took place at appropriate times (not within 3 hours of the previous), ensuring disposition coding is accurate/reasonable, etc.

OPR does not receive data with PII from Vendors, so we would typically not ask for a review of the actual contact information of respondents.

Copy of Data QC QA 01212025 United States Agency for Global Media (USAGM) Survey Data Quality Control Services

31 Survey Platform RFQ 1.6.2 Deliverables, page 5 Does USAGM require contracted survey firms to use a specific survey platform (such as ODK, SurveyToGo, etc.) that ensures that metadata recorded from the surveys meets these requirements?

OPR allows fieldwork vendors to script in different applications. OPR cannot, at this time, dictate that all vendors use the same software. We reserve the right to deny a fieldwork vendor the ability to use a particular program, but only if that program cannot meet our requirements. Most fieldwork vendors typically use off-the-shelf survey software platforms such as SurveyToGo. A few vendors use proprietary, in-house applications for CAPI and CATI surveys.

32 Enumeration team RFQ 1.6.2 Deliverables, page 5 Will USAGM provide information such as the survey contractor’s enumeration team size and any administrative challenges during fielding (such as political unrest) to inform data review?

Yes.

33 Final dataset format RFQ 1.7, page 5 In what format does USAGM prefer final datasets?

OPR requires fieldwork vendors to submit datasets in .sav (SPSS) formatting. We sometimes also receive .dta (STATA) files, but our requirement is SPSS.

34 Number of Relevant Experiences

RFQ Evaluation Factors, Factor 3 - Relevant Experience, page 10

Does USAGM require a certain number of relevant experience projects (such as three or five)?

Please refer to the revised technical experience factor incorporated as part of Amendment 00001.

35 Price Schedule Quantity Attachment C Price Schedule, Tab "Price

Schedule by Task", Service Categories 1.1-1.6

Question about the units expressed in the "Quantity" columns (C, F, etc.). The Price Schedule has quantity (12) for service categories 1.1-1.6. Is this months? Would an appropriate response be the average estimated monthly cost based on 30 total surveys across the 12-month contract period? (SOW scope - (10) full surveys and (20) Omnibus per year = 30 surveys). Thank you.

The quantity indicates number of surveys.

36 Price Schedule Quantity Attachment C Price Schedule, Tab "Price Schedule by Task", Service Category 1.7 -

Produce final datasets

Question about the units expressed in the "Quantity" columns (C, F, etc.). The Price Schedule has quantity (32) for service category 1.7. Can you provide additional detail about what this number refers to? Is this for 32 surveys?

Would an appropriate response be (32) x the per-survey estimate for this work? Thank you.

The quantity indicates number of surveys.