21R0005 Online Labs 5_20_2021 Conference Presentation.pdf

Bureau of Consumer Financial Protection Office of Research

Virtual (Online) Research Support

RFP# 9531CB21R0005

Pre-Proposal Conference May 20, 2021 │2:00 p.m.

Agenda

1. Introductions

2. Request for Proposals (RFP) Background information Scope Objectives Requirements

Online Panels

Recruitment & Screening

Participant Payment

Matched Participant Data

Personnel

Other Information Requested

Constraints Deliverables

3. Contract Clauses/Provisions

4. Evaluation Criteria

5. Pricing Templates & Sample

Table Price Table Recommended Payment Table Participant Overhead Table

6. Questions/Answers

Introductions

Office of Procurement

Office of Research

RFP – Background Information

Dodd-Frank Act established CFPB and the Office of Research

Controlled experiments and surveys

Online studies

Acronyms:

RFP: Request for Proposals

PWS: Performance Work Statement

RFP Section B.1 (page 5) 4

RFP - Scope

Contractor will help CFPB to execute research, experiments, and studies via the internet.

Contractor will carry out support tasks only, including supporting submission to the Office of Management and Budget (OMB) for Paperwork Reduction Act (PRA) obtaining Institutional Review Board (IRB) approval, if necessary;

recruiting and screening for high-quality participants;

paying participants;

and, possibly, matching study data with administrative data.

RFP Section B.3 (pages 6-8) 5

RFP - Scope

CFPB researchers will determine recruitment requirements for each study.

Study Workflow:

CFPB sends the Contractor details of the study in a Study Request.

CFPB and the Contractor will discuss and finalize the Study Request.

Contractor provides support tasks to carry out the study.

CFPB and the Contractor meet to debrief after the study.

RFP - Objectives

1. Implement studies quickly

2. Forecast prices accurately for current and upcoming work

3. Institutional Review Board (IRB) approval, if necessary

4. Technical support for CFPB (study materials, PRA paperwork)

5. Technical support for participants

6. Recruitment of high-quality participants

7. Online panels meet the needs of a variety of studies

8. Payment amounts are flexible

9. Payments are made quickly

10. Matched data

11. Feedback

RFP Section B.5 (pages 8-10) 7

RFP - Requirements

6.1 Required Capabilities

1. IRB Approval

2. Online Panels

3. Recruitment

4. Screening

5. Participant Payment

6. Matched Participant Data

7. Personnel

6.2 Added Value

1. Online Panels

2. Recruitment and Screening

3. Matched Participant Data

4. Personnel

RFP Section B.6 (pages 10-15) 8

RFP – Online Panels

Online panel: A collection of individuals who volunteered to be contacted for the purpose of participating in research via the internet.

Offerors may propose more than one online panel for use.

For each online panel, the Offeror shall provide the information requested in the PWS. This includes, but is not limited to:

A. Conditions under which the Offeror will use the panel.

B. The panel’s approximate number of active members.

C & D. For each of the demographic groups listed in the

Requirements and, if applicable, Added Value sections:

The incidence of each group.

An estimate of the maximum sample size of each group that the Offeror believes it can recruit for a 30-minute study.

(See Example Incidence Table on pages 60-61).

E. A description of how the Offeror determines incidences.

RFP Sections 6.1.2 (pg.10-11), 6.2.1 (pg.14), 3.4 (pg.57-61) 9

Snapshot of Example PWS Incidence Table (RFP pg.60-61)

Group

Population Incidence (%) Maximum Sample Size for a 30-Minute Study (N)

Sex -- -- Male

Female Age -- --

18-29 years old 30-39 40-49 50-59

60 or older Race -- --

White Black/African American

Asian American American Indian or Alaska Native

Hispanic/Latinx -- -- Hispanic / Latinx

Non-Hispanic / Non-Latinx Education -- --

Less than a college degree Associate’s degree or technical or

Vocational degree Bachelor’s degree

Graduate or professional degree (e.g., MA, MSW, MD, JD, PhD)

RFP – Recruitment and Screening

6.1 Required Capabilities:

Participant Criteria: Age 18+ residing in the US. Participation is voluntary. Participants are accustomed to participating in online studies, including studies that take up to 30 minutes to complete.

Screening Criteria: (A) Demographics and respondents’ characteristics, (B) Participant Quality

6.2 Added Value: The Contractor can provide a sample of participants that matches the demographics of certain populations (e.g., American adults according to the U.S. census). This may increase the external validity of CFPB studies run under this Contract.

Information requested on the PWS includes:

G. The method the Offeror will use to broadcast or advertise studies to panel members. This should include example recruitment materials.

H. The method(s) the Offeror will use to screen for participants and the methods by which the Offeror will measure and determine participant quality (as defined in the SOO). If methods include data collection, such as surveys, then the Offeror shall provide example screening materials.

I. The method the Offeror will use to push links to the Bureau’s studies to eligible respondents.

RFP Section 6.1.3-4 (pg.11-12), 3.4 (pg.57-61) 11

RFP – Participant Payment

6.1 Required Capabilities:

Study Requests will describe how participants payments will be determined.

Some studies may specify fixed payments for all participants who complete a study, other studies will determine payments as a function of participants’ choices in the study.

The Contractor must be able to take payment instructions from the CFPB after a participant completes a study and pay participants individualized amounts in accordance with those instructions.

The Contractor shall confirm with CFPB that participants were paid as instructed.

Information requested on the PWS includes:

L. The method by which the Offeror will pay participants.

M. The method the Offeror proposes that CFPB will inform the Offeror which participants should be paid and how much they should be paid.

RFP Section 6.1.5 (pg12), 3.4 (pg.57-61) 12

RFP – Matched Participant Data

The Contractor shall provide to the CFPB all data collected to determine participants’ eligibility for a study.

If the Contractor screens volunteers using information collected prior to the study

(e.g., demographic information collected from all members of an online panel), then the Contractor shall provide, for each screened attribute:

The nature of the data used, a list of all possible values, and, if applicable, individual survey items.

Summary statistics for the entire sampling frame Individual values for all individuals who responded to the study’s recruitment materials.

6.2 Added Value: The Contractor provides additional demographics data about members of its online panels. This allows CFPB researchers to create more succinct study instruments and therefore increases the value of this Contract. In its response to this RFQ, the Contactor would provide a list of available demographics.

Information requested on the PWS includes:

N. The method by with the Offeror can provide matched participant data for the panel.

RFP Section 6.1.6, 6.2.3 (pg.12-14), 3.4 (pg.57-61) 13

RFP – Personnel

Key Personnel – Research Scientist:

Responsible for reviewing Study Requests to ensure each study is feasible. Shall have significant experience running online studies, including studies that include tasks for individual participants, such as surveys; as well as studies that include tasks for multiple participants, such as games (e.g., a prisoners’ dilemma game) or economic simulations (e.g., an auction).

Non-Key Personnel – Research Associate

6.2 Added Value

Has experience running online studies using the Qualtrics platform.

Has experience running online studies using oTree software

Submission includes resume and personnel narrative

RFP Section 6.1.7, 6.2.4 (pg.13-15), 3.4 (pg.61-62) 14

RFP – Other Information Requested

Information requested on the PWS also includes:

The method by which the Offeror can obtain IRB approval

F. How study durations will be determined for each study

J. The method used to provide technical assistance, as needed, to participants who need help successfully navigating issues they may encounter

K. How the Offeror defines completes and how it will determine if an individual has completed a study

RFP Section 3.4 (pages 57-59) 15

RFP – Constraints

Budget Constraint: $315,400.00 Estimate based on completion of twenty (20 studies) over the base and all options.

Actual number of Study Requests and effort may vary depending on the needs.

Option period funding is not currently secured and may impact the continuation or completion of this contract beyond the base period.

Adhere to all Security and Non-Disclosure information within the clauses of this RFP

All deliverables must be 508 compliant

RFP Section B.7 and B.8 (pages 15-18) 16

RFP – Deliverables

Contract Kickoff meeting

Final Quality Control Plan (QCP)

Intermittent Project Status Reviews

Study Study Feasibility

Finalized Study Request

Support with OMB Package

IRB Approval

High-Quality Participants

Confirmation of Participant Payment

De-Identified Demographic and/or Other Matched Participant Data

Study Wrap-Up Meeting

RFP Section B.7 and B.8 (pages 15-18) 17

Contract Clauses and Provisions

Local Clauses/Provisions

Period of Performance

FAR Clauses and Provisions

Representations and Certifications – Sam.gov

Clauses/Provisions incorporated by Reference

Base Period: Number of Option Periods:

One (1) year base period Four (4) one-year option periods

Option IAW FAR 52.217-8 Up to 6 months (may be exercised monthly)

Evaluation Criteria

Two Non-Price Factors

Technical Approach

Key Personnel

Price

The non-price factors, when combined, are approximately equal to price.

Pricing Templates and Sample Study Price Table

For most tables requested in Attachment 1, the Offeror shall provide six (6) of each of the requested tables, one for the base period and each option period (including the 6-month option), as a separate attachment.

For the purpose of evaluating offers, the CFPB will consider the total price implied by the Offeror’s Price Model (Attachment 1) for a hypothetical experiment in the base period. See Section 3.4.4 Price Comparison.

RFP Attachment 1 (pages 84-86) 20

1.0 Price Table

RFP Attachment 1 (pages 84-86) 21

Deliverable Fixed Rate

Study Feasibility $

Finalized Study Request $

Support with OMB Package $

IRB Approval $

De-identified demographic and/or other matched participant data

Study Wrap-Up Meeting $

2.0 Recommended Payment Table

Study Duration: One-minute increments from 5 to 30 minutes

Rate: Contractor’s recommended hourly rate for a study of that length.

Total Payment: Total payment implied by the rate per minute and the study duration. For example, for an hourly rate of $6 per hour ($0.10 per minute) and a study duration of 5 minutes, the third column would read $0.50 ($0.10*5).

RFP Attachment 1 (pages 84-86) 22

Study Duration (minutes) Rate ($ per minute)

Total Payment ($)

5 $ $

3.0 Participant Overhead Table

Table cells contain the price of a single complete, depending on the corresponding study duration and incidence rate.

Prices are “partially-burdened”: Rates reflect costs of all recruitment activities, including costs related to (1) advertising the study & screening participants, (2) paying participants*, and (3) technical support regarding the Contractor’s duties.

*Prices should not include any other costs, including participant payments or the cost of any deliverables listed in the Price Table.

If incidences depend on the advertised payment for the participant: Indicate that this dependence exists. Complete the tables assuming the CFPB will pay participants in accordance with the Contractor’s Recommended Payments Tables.

If Offeror’s pricing structure does not take incidence into account, may collapse the table down to a single column.

RFP Attachment 1 (pages 84-86) 23

Population Incidence (%) Study Duration (minutes) 1 2 3 4 5 6 … 100 5 $

Questions?