953121R0005 A3.pdf

(x)

9531CB21R0005

x x

1 copies of the amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted ; or (c) By separate letter or electronic communication which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR ACKNOWLEDGEMENT TO BE

RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OF OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN REJECTION OF YOUR

OFFER. If by virtue of this amendment you desire to change an offer already submitted , such change may be made by letter or electronic communication, provided each letter or electronic communication makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

x

CFP

WASHINGTON DC 20552

1700 G STREET, NW

OFFICE OF PROCUREMENT

Consumer Financial Protection Burea

06/15/20210003

13. THIS ITEM ONLY APPLIES TO MODIFICATION OF CONTRACTS/ORDERS. IT MODIFIES THE CONTRACT/ORDER NO. AS DESCRIBED IN ITEM 14.

12. ACCOUNTING AND APPROPRIATION DATA (If required) is not extended.is extended, Items 8 and 15, and returning

Offers must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended , by one of the following methods: (a) By completing

The above numbered solicitation is amended as set forth in Item 14. The hour and date specified for receipt of Offers

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

FACILITY CODE CODE

10B. DATED (SEE ITEM 13)

10A. MODIFICATION OF CONTRACT/ORDER NO.

9B. DATED (SEE ITEM 11)

9A. AMENDMENT OF SOLICITATION NO.

CODE

8. NAME AND ADDRESS OF CONTRACTOR (No., street, county, State and ZIP Code)

7. ADMINISTERED BY (If other than Item 6)CODE 6. ISSUED BY

PAGE OF PAGES

4. REQUISITION/PURCHASE REQ. NO.3. EFFECTIVE DATE2. AMENDMENT/MODIFICATION NO. 5. PROJECT NO. (If applicable)

1. CONTRACT ID CODE

AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT

05/13/2021

CHECK ONE A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE CONTRACT

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES (such as changes in paying office, C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER (Specify type of modification and authority) appropriation data, etc.) SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(b).

E. IMPORTANT: Contractor is not is required to sign this document and return __________________ copies to the issuing office.

ORDER NO. IN ITEM 10A.

14. DESCRIPTION OF AMENDMENT/MODIFICATION (Organized by UCF section headings, including solicitation/contract subject matter where feasible.)

The purpose of this amendment is to:

1. Provide answers to vendor questions, and

2. Extend the submission deadline to 3:00 pm on July 6, 2021.

See the Attached Q&A and updated solicitation document.

16A. NAME AND TITLE OF CONTRACTING OFFICER (Type or print)15A. NAME AND TITLE OF SIGNER (Type or print)

15C. DATE SIGNED 16B. UNITED STATES OF AMERICA 15B. CONTRACTOR/OFFEROR 16C. DATE SIGNED

(Signature of person authorized to sign) (Signature of Contracting Officer)

PEGGY J. WRIGHT

STANDARD FORM 30 (REV. 11/2016)

Prescribed by GSA FAR (48 CFR) 53.243

Previous edition unusable

Except as provided herein, all terms and conditions of the document referenced in Item 9 A or 10A, as heretofore changed, remains unchanged and in full force and effect .

9531CB21R0005, Amendment 3

VIRTUAL (ONLINE) RESEARCH SUPPORT

VENDOR QUESTIONS/BUREAU ANSWERS

# Question Answer

1. Should the vendors infer that we will not be asking screening questions?

See section B.5 parts F and G. Two of the objectives of this contract are that the Contractor be able to screen for high-quality participants that match the demographic requirements of the study. This is further elaborated on in section B.6, part 6.1.4.

The CFPB does not anticipate asking screening questions in addition to those asked by the Contractor. In the event that a study would require this, this requirement would be detailed in the Study Request and discussed with the Contractor.

2. Is this a small business set aside? This will be a full and open competition.

3. Does the budget of $315K include the amounts that will be paid to the participants? or will you provide additional funds for that. In particular, if you are offering variable payments depending on their choices in the experiment.

The budget estimate includes the amounts that will be paid to participants, including payments that depend on participant choices.

The budget ceiling should cover both the vendor’s proposed pricing model and the amounts participants will be paid for the duration of the contract.

4. Should we assume 100 completes for each of the 20 studies?

No. The number of completes needed will vary by study. The 100 completes listed in this hypothetical example for Price Comparison is only relevant for evaluating price proposals, and therefore is not representative of all studies under this contract, nor should be used as an estimate of all studies.

5. Will the meeting be recorded? CFPB Procurement does not record meetings.

6. Do you have a vision for the proportion of the contract that goes to participant fees vs. contractor time?

Contractor time should be considered in the fixed Price Tables and in the Participant Overhead Tables. In addition, Section 2.0 of the Pricing Tables (Attachment 1) asks each Proposer to recommend the rate that CFPB should pay participants as a function of the study duration. As discussed in that section, if an Offeror’s Participant Overhead depends on the amount that participants would be paid, then the Offeror may assume that CFPB would pay participants the rate recommended by the Offeror.

# Question Answer

7. Will there be only one award? Asking this to estimate if the potential panel demographics is very limited (e.g. 18- 26-year-olds), if it has a chance to be competitive.

The CFPB anticipates a single-award contract but reserve the right make multiple awards, or no award.

8. Participant payments, what about that is important to you?

Can happen in various ways. Direct payment at an hourly rate ($1) or some prefer sweepstakes to win larger prize.

With regards to payment, there are two types of studies (see section B.6.1.5):

(1) Studies where a participant’s payment is based on their choices/answers in the study. For example, a study might offer different payment incentives based on different options the participant can choose. In this example, one participant might make choices that lead to a $5 payment, while another makes choices that lead to a $6 payment. In these types of studies, the vendor needs to be able to compensate the participant in a specific dollar amount based on the participant’s choices/answers in the study.

(2) Studies where a participant’s payment is not impacted by their choices/answers in a study. Depending on the nature of the recruitment process, the panel may mix for recruitment sources and compensation types, particularly with hard to recruit samples. The vendor has flexibility to recommend a compensation amount and type, and these can vary across participants for the same study. For example, a vendor may recruit some participants using dollar-amounts and others using sweepstakes as compensation.

Each Offeror may propose multiple panels. Although the awarded Contractor must be able to perform studies of type (1) with at least one of its proposed panels, not all of its panels need meet this requirement.

In their proposals, Offerors should demonstrate why their proposed panels and payment methods help the CFPB to achieve the objectives of the contract.

9. Can you talk about how you think the contractor will know if someone has completed a task if all responses are completed on your servers? Or should we describe in our proposal how we recommend that this be accomplished.

In most cases, the CFPB will directly observe when participants complete a study. Because Contractors may not have access to study data, it is acceptable for an Offeror to propose that CFPB make this determination or to propose that CFPB provide the Contractor with limited summary statistics to assist the Offeror in making these determinations. We are open to hearing if the Contractor has another way to determine completes aside from CFPB notifying the Contractor when a participant has completed the study. As requested in Section E.3.4.3, part K, Offerors should still make sure to provide a description of how the Offeror defines completes.

10. Could you please tell me if the Combined Synopsis/Solicitation posted as solicitation number 9531CB21R0005 will be a follow-on requirement to any work performed currently or in the past, or will this be a new requirement?

If possible, can you please provide the most recent contract or task order number?

This is a new requirement, not a follow-on or recompete of any previous contract

11. Pricing Table, Attachment 1 - What assumptions should we make for completing the price table for each period?

i.e. For Study Feasibility, are we expected to assume a certain number of hours will be required for each study request for us to determine feasibility?

Or would you like a cumulative cost over each period/year?

For (1) Price Table in Attachment 1, the price of each deliverable should be based on how much it will cost the Contractor to provide that deliverable for each study. See the Deliverables table in section B.8.2 for more information about each deliverable.

We included “Study Feasibility” as a deliverable in case we need to ask the Contractor to provide advanced information on incidences and costs of target groups not listed in the Incidence Table. Given we cannot know how much work it takes a Contractor to determine incidence rate and other information to confirm the cost for different target groups, we defer to the Proposer to determine how much CFPB should be charged every time we make a Study Feasibility request.

12. Pricing Table, Attachment 1 - Can you give us a sense of anticipated time/presentation/deliverables for Study Feasibility meeting and how that may be different from the Finalized Study Request portion?

Study Feasibility does not require a meeting or presentation if the information can be provided easily over email. See Deliverables table in section B.8.2 and answer to question #11 for more information.

Study Feasibility gives CFPB advanced information, if necessary, to help CFPB submit a Study Request to the Contractor (see section 3.2 Study Request). Finalized Study Request refers to the feedback, meeting, and final document(s) required to finalize the Study Request we submit to the Contractor before a study can begin.

13. Pricing Table, Attachment 1 - Are there any unique factors to consider for the OMB Package?

No. In Section B.8.2 (Deliverables – Study), we provide an example of support work, which include: CFPB staff may request information about administrative aspects of the study that are needed for CFPB staff to complete PRA paperwork. The Contractor responds by providing a brief written description of the requested information, such as the Contractor’s recruitment method, procedures for information collection from participants, collection and use of any PII by the Contractor, justification for the use and amount of participant incentives, and other administrative details as requested.

The Contractor will only be responsible for providing information or materials requested by CFPB; the Contractor will not be responsible for delivery or coordination of the PRA process with OMB.

14. Pricing Table, Attachment 1 - What are the expected deliverables and resource commitments for the Study Wrap-Up Meetings aside from final data set and methodological summaries?

The Wrap-Up Meeting will be held to confirm completion of the study and for both the Contractor and the CFPB to provide feedback to improve future studies (see the Deliverables table in section B.8.2).

15. Could you provide some clarity on the difference between the Quality Control deliverable at contract submission and at the 30 day mark?

The Quality Control Plan (QCP) required at submission is a draft.

From this draft, the final QCP (Section B, 8.4) shall be completed after any input from the Bureau on the submitted Draft QCP.

16. Is there a maximum length of the proposal? Page limits have been added to the RFP. Volume I - Factor 1 shall be no longer than fifteen (15) pages. Required Contractor Input under Section C, and Solicitation Provisions FAR 52.212-3, FAR 52.204-24 and FAR 52.204-26 are limited to those clauses and provisions only, and do not require additional pages. No other page limits will apply for Volume I. No page limits will apply for Volume II.

17. You stated that there is an estimated 20 projects. What is the Maximum N a single project might require to reach your desired statistical significance? Even an educated guess would be helpful.

The contract does not specify requirements for the size of an online panel, and it does not specify requirements for the number of participants for any single study.

Sections B.6.1.2-4 describe requirements for the online panels, recruitment, and screening.

Section E.3.4.3 parts B-D request information regarding the size of the proposed panels and the expected sample sizes and incidence rates of several demographics groups.

In their proposals, Offerors should demonstrate why their proposed panels help the CFPB to achieve the objectives of the contract, as stated in B.5.

18. Would you like me to include ballpark pricing for any of the following services?

Qualtrics Offers the following:

- Survey Design

- Survey Programming

- Survey Review

- Data collection (We have access to 100 million participants internationally)

- Survey Translation (Great option if you don't want to exclude non-English speakers in the US)

- Analysis

- Reporting

- Executive summary -- 1-5 slides. With a detailed summary of core findings and action items

- Topline report -- 5-10 slides. With in-depth question group analysis, core findings, and action items

- Full report -- 10-20 slides. With in-depth question-by-question analysis, core findings, and action items

Contractors should provide the information for services and deliverables requested in the RFP. The costs of any services required to provide a deliverable should be incorporated into the deliverable estimates provided in the Pricing Tables (Attachment 1).

19. For the studies that have been built out, if any, could you provide me with the following specifications? We would love to know what a typical study might look like.

Number of completes (N):

Avg. time to complete the survey in minutes:

Target population:

Desired quotas:

Ex:

- Gender

- Age

- Race

The contract is designed to accommodate a variety of study-types. No studies have been “built out” with the details requested.

The hypothetical study described in Section E.3.4.4 was designed to give CFPB the ability to compare prices across different vendors’ pricing models. Thus, while the hypothetical study is not “typical,” it may give you a general idea of what a study under this contract could look like.

20. Can you clarify if the maximum budget for this contract includes the variable incentives that will be paid to participants based on the studies that you will run? Do you envision paying participants a flat fee for participating in addition to those variable incentives?

See answer to question #8 above.

The contract does not restrict proposals from including flat payments for participation in addition to variable incentives. These flat payments should be incorporated into the Rate and Total Payments of the Recommended Payment Table in Attachment 1. In their proposals, Offerors should demonstrate why their proposed payment methods and amounts help the CFPB to achieve the objectives of the contract.

21. Because the studies have not been designed yet, it is possible that a future study will be more complex than has been described here and as a result, the costs outlined in the price table (as shown on page 87) could be insufficient. For example, this could occur if the OMB package or the IRB approval is particularly complex due to a complex design. If this is the case, will it be possible to negotiate costs on a per study basis, for more complex studies?

(For more information on the deliverables mentioned below, see Deliverables table section B.8.2)

It is unlikely that studies will be significantly more complex than what is described in the RFP. CFPB does not anticipate that prices for OMB or IRB support will be negotiable over the life of this contract.

If a study will require participant groups for which the Contractor has not already specified incidence rates, then the CFPB will request that the Contractor provide incidence rates and maximum sample sizes for these groups. See “Study Feasibility” under Deliverables in section B.8.2 for more information.

Although this is not anticipated at this time, if, for some reason, CFPB wishes to conduct a study under this contract that is within scope of the contract but requires the Contractor to conduct a task or service that is not already specified in this contract, then CFPB would request that the Contractor provide a price for that task or service. The Contractor would be offered the opportunity to provide pricing or indicate that the firm is unable or unwilling to provide this additional task or service.

22. There are several fixed costs that are likely to be associated with this contract, that would-be one-time costs or costs irrespective of any specific study. For example, holding a kickoff meeting, a quality control plan, and intermittent project status reviews, and other contract level costs. How/where should we provide the budget information for these aspects of our budget?

Fixed costs should be fully-burdened, meaning that all costs associated with the general contract shall be incorporated into the cost of the deliverables and prices provided in Pricing Tables (Attachment 1).

23. Do you anticipate awarding a fixed priced or cost reimbursement plus fixed fee type of award?

The Bureau will be awarding this as a fixed price, level of effort contract.

24. On page 9, it says that the contractor should “obtain large samples”. What is “large”? Can you provide an indication of the minimum total sample size that would fit your needs?

Given the variety of studies and participant needs, we do not have a clearly-defined minimum total sample size per target group. The Contractor is asked to supply an Incidence Table which includes an estimate of the maximum sample size of each group that the Contractor believes it can recruit for a 30-minute study (see example in section E.3.4 Submission Instructions).

25. What deliverables need to be 508 compliant, as there are no public facing written deliverables, such as reports or documentation?

Technically all documents and deliverables within the Federal Government are required to be 508 compliant. We do not anticipate extra work will be required to make any of the deliverables listed in the RFP 508 compliant.

26. Can you define incidence for the table on page 60? As requested in Section E.3.4.3, part E, Offerors should provide a description of how the Offeror defines incidence. Our market research indicated that vendors have different ways to define and determine incidence. Knowing how the Offeror determines incidence, along with the information provided, will help the CFPB better understand the ease of obtaining certain samples and the anticipated costs.

27. Is there a page limit for the technical proposal’? Or any formatting requirements?

See Q 16 for page limitations. Formatting requirements have been added to the RFP.

28. Is it possible to extend the deadline to allow for adjustments following responses to questions?

The deadline has been extended to July 6, 2021.

3 | P a g e

May 26, 2021

SUBJECT: Request for Proposals: 9531CB21R0005 – Virtual (Online) Research Support –

Amendment 23

The purpose of this Request for Proposals (RFP) is to obtain online research support in accordance with the attached Statement of Objectives and per Federal Acquisition Regulations (FAR) Part 12 and Part 13. Per the guidelines discussed herein, CFPB requests that you provide a response to this RFP.

Based upon the evaluation of proposals received in response to this RFP, the CFPB intends to award a single contract, but reserves the right to make multiple awards or no award, to be in effect for a term of up to five (5) years.

A pre-proposal conference will be held on May 20, 2021 at 2:00 p.m. ET. Questions related to this RFP shall be received by both the Contracting Officer (CO) Peggy.Wright@cfpb.gov and the Contract Specialist (CS) Crystal.McKay@cfpb.gov in writing no-later-than (NLT) 3:00 p.m. ET on June 2, 2021.

Responses to this RFP should be received by both the CO and CS listed above NLT 3:00 p.m. ET on June 24, 2021 July 6, 2021. Responses received after that date and time may not be opened, evaluated, or considered for award.

We appreciate you taking the time to provide us with a response for this critical requirement.

Sincerely, Peggy J. Wright Contracting Officer

Enclosures

9531CB21R0005 – Amendment 23 Virtual Research Support

4 | P a g e

REQUEST FOR PROPOSALS

9531CB21R0005

VIRTUAL RESEARCH SUPPORT

TABLE OF CONTENTS

SECTION A – SF 1449

SECTION B – SCHEDULE OF SERVICES

SECTION C – CONTRACT CLAUSES

C.1 FAR 52.212-4, Contract Terms and Conditions-Commercial Items C.2 Addendum to FAR 52.212-4 C.3 FAR 52.212-5, Contract Terms and Conditions Required to Implement Statutes and

Executive orders

SECTION D – CONTRACT DOCUMENTS, EXHIBITS OR ATTACHMENTS

Reserved

SECTION E – SOLICITATION PROVISIONS AND INSTRUCTIONS TO OFFERORS

E.1 52.212-2, Evaluation-Commercial Items E.2 CFPB Solicitation Provisions E.3 Addendum to FAR 52.212-1 E.4 FAR Solicitation Provisions

SOLICITATION ATTACHMENTS AND EXHIBITS

Attachment 1 – Price Tables Attachment 2 – Non-Disclosure Agreement

5 | P a g e

SECTION B – SCHEDULE OF SERVICES

B.1 BACKGROUND INFORMATION

On July 21, 2010, the Dodd-Frank Wall Street Reform and Consumer Protection Act (the "Act") created the Consumer Financial Protection Bureau (CFPB). Section 1013(b)(1) of the Consumer Financial Protection Act, 12 U.S.C. § 5493(b)(1), establishes the Office of Research (“OR”) and assigns to it the responsibility of researching, analyzing, and reporting on topics relating to the Bureau’s mission, including developments in markets for consumer financial products and services, consumer awareness, and consumer behavior.

Controlled experimentation provides researchers with command over data collection processes, which provides several benefits. For example, researchers can maintain the assumptions of a given social science model, and the ability to alter these assumptions, which enables research that would either be less precise or impossible using administrative data. In addition, the results of experiments can be replicated and tested over different specifications of the environment or in the underlying model, implying that this type of research can, relatively easily, be expanded upon by other researchers or tested for robustness. Finally, experimentation allows researchers to conduct research that is timely and relevant to the Bureau’s greatest needs. This is in contrast to research resulting from natural experiments, the timing and topics of which cannot be influenced by Bureau researchers. While it is important to recognize that controlled experimentation itself has limitations, it is also clear that it plays an important role in the advancement of economic understanding.

Over the last decade, much social science research has transitioned from in-person laboratories and computer labs to virtual laboratories in which information is exchanged via the World Wide Web (the internet). There are at least four benefits of this approach. First, typical online participant pools include individuals with a greater diversity of demographics than do typical in-person participant pools, especially those maintained by typical colleges and universities. In addition, online participant pools are typically larger than university pools, online studies are less expensive than typical university studies, and online studies can be run more quickly than typical university studies.

B.2 DEFINITIONS

A. Online panel. A collection of individuals who volunteered to be contacted for the purpose of participating in research via the internet.

B. Participant. A participant is any individual who participates in a CFPB study under this Contract, regardless of whether they complete the study.

Section B – Schedule of Services

6 | P a g e

C. Study. Work conducted under this contract will be organized as “studies.” Studies will include tasks for individual participants such as surveys, as well as tasks for multiple participants such as games (e.g., a prisoners' dilemma game) or economic simulations (e.g., an auction).

D. Study Request. A Study Request will describe the requirements of the study, including required characteristics of participants and tasks for the Contractor.

E. Task. Work the Contractor conducts under each study is delineated as tasks. Recruitment of participants for a study is an example of a task.

F. Institutional Review Board (IRB): An IRB is a type of committee that applies research ethics by reviewing the methods proposed for research to ensure that they are ethical.

Such boards are formally designated to approve (or reject), monitor, and review behavioral research involving humans.1

G. Paperwork Reduction Act (PRA): PRA is a Federal law that seeks to limit the burden Federal agencies impose on the public for their information requirements. The CFPB is required to submit a PRA package with details about the information collection for Office of Management and Budget (OMB) review and approval. PRA approval is required prior to implementing a study.

B.3 SCOPE

Under this contract, a Contractor will help CFPB to execute research, experiments, and studies via the internet. The Contractor will carry out support tasks only, including providing support with CFPB’s submission to the Office of Management and Budget (OMB) for Paperwork Reduction Act (PRA) package; obtaining Institutional Review Board (IRB) approval, if necessary; recruiting and screening for high-quality participants; paying participants; and, possibly, matching study data with administrative data. Depending on the requirements of a study, participants may be paid a fixed amount, or they may be paid an amount that depends on their choices and/or the choices of other participants in the study.

CFPB researchers will determine recruitment requirements for each study. Some studies will require convenience samples, while other studies will require samples that match certain demographics profiles. CFPB researchers will create study materials and make them accessible to participants through the internet.

Work conducted under this contract will be organized as “studies.” Work the Contractor

1 Definition derived from Wikipedia: https://en.wikipedia.org/wiki/Institutional_review_board

7 | P a g e conducts under each study will be delineated as “tasks.” The workflow for work conducted under each contract is as follows:

3.1 Studies

At the beginning of each study, CFPB will send the Contractor a “Study Request” which will describe the requirements of the study, including required characteristics of participants and tasks for the Contractor. Each study will follow the same basic workflow:

A. CFPB sends the Contractor a Study Request.

B. CFPB and the Contractor discuss and finalize the Study Request.

C. The Contractor provides support for CFPB’s submissions to the OMB, if necessary.

D. The Contractor obtains IRB approvals, if necessary.

E. For each wave of data collection (including pre-tests/pilots):

a. The Contractor recruits and screens participants.

b. The Contractor delivers eligible screened participants to the study web address.

c. CFPB informs the Contractor which individuals completed the study and specifies the amount each participant should be paid.

d. The Contractor pays participants.

F. The Contractor provides the CFPB with matched administrative data, including responses to screener questions.

G. CFPB and the Contractor meet to debrief after the study.

3.2 Study Requests

To initiate a study under this contract, CFPB will send the Contractor a “Study Request.” A Study Request will describe the requirements of the study, including required characteristics of participants and tasks for the Contractor. The Contractor will offer proposed changes to the Study Request. CFPB and the Contractor will then discuss and finalize the Study Request. Each Study Request will include, at least:

A. The required demographics of participants B. The required number of participants C. Suggested recruitment materials and suggested screener questions/techniques, if applicable D. An estimate of study length E. Needs regarding IRB approvals and/or submissions to the OMB.

F. A description of the web addresses for the study, including whether the addresses will be individualized, and examples of the web address(es) for the study

8 | P a g e

G. The structure and timing of data collection, including (1) whether there will be a pilot/pre-test and the number of participants required, (2) the number of waves of data collection and the number of participants required for each, and (3) the approximate timing of data collection waves and the pilot/pre-test, if relevant (e.g., the number of days between the completion of a pilot and wave one of data collection).

H. A description of how participant payments will be determined, including whether participants will be paid a flat payment or whether payment will depend on participants’ choices. The description will also include minimum, maximum, and expected payment amounts.

I. Required data matching (if applicable), including, data from the Contractor’s screening process; and including (optional) supplemental data matching.

B.4 PERIOD OF PERFORMANCE

The base period of performance for this contract shall be one (1) year, plus four (4) one-year option periods. In accordance with FAR 52.217-8, this contract may be extended for up to six

(6) months beyond the final option period.

B.5 STATEMENT OF OBJECTIVES

A. Implementation - The Contractor enables CFPB to implement online studies quickly.

Each stage of each study is implemented quickly, within the timeframes listed in the deliverables table (see section B.8 Deliverables). In addition, after each round of data collection, when CFPB is ready, the next round is begun quickly, if applicable. The Contractor’s solution shall clearly provide CFPB the ability to implement online studies within this time frame.

B. Forecasting Prices - The Contractor provides enough information for the CFPB to accurately forecast the prices of studies before submitting a Study Request. The Contractor’s solution, Price Model, and estimates of incidences of demographic groups in the Contractors’ online panel(s) shall provide the means for CFPB to estimate studies’ costs prior to submitting a Study Request.

C. Institutional Review Board (IRB) Approval - The Contractor is able to obtain IRB approval. The Contractor’s solution shall demonstrate the ability to lead and obtain IRB approval, if necessary.

D. Technical Support for CFPB - The Contractor is able to provide information to assist CFPB in creating Study Request materials and to fill out PRA paperwork.

9 | P a g e

E. Technical Support for Participants – Participants are able to receive technical assistance, as needed, to help them successfully navigate issues they may encounter when accessing a study, accessing or completing screening materials, or with receiving payment. The Contractor’s solution shall demonstrate the ability to provide technical support to participants.

F. Participation - The Contractor recruits high-quality participants. The Contractor’s solution shall clearly demonstrate the ability to recruit and screen high-quality individuals who are human beings (i.e., not robots or “bots”) with demographics accurately reported, and who answer questions attentively and honestly.

G. Online Panels - Online panels meet the needs of a variety of studies. The Contractor’s solution shall clearly demonstrate the ability to access to one or more online panels from which it can recruit and screen for participants. These panels allow CFPB to meet several goals, depending on the needs of the study:

a. Recruit quickly. The Contractor is able to quickly recruit high-quality participants from a convenience sample.

b. Recruit based on demographics. The Contractor is able recruit high-quality participants based on sex, race, or age; descriptors of their situations such as status as a veteran, status as a student, employment status, or household income;

or descriptors of their experiences such as current or previous uses of financial products like mortgage loans, credit cards, or savings accounts. The Contractor can use this capability to create diverse samples or samples that match the demographics of certain populations (e.g., American adults).

c. Obtain large sample sizes. The Contractor is able to recruit large samples of high-quality participants from both convenience samples and samples selected based on demographics.

H. Payment amounts are flexible. The Contractor’s solution shall clearly demonstrate the ability to pay participants amounts according to the needs of each study, including for studies in which participants’ payments are determined by their choices.

I. Payment Time - The Contractor’s solution shall demonstrate the ability to deliver participant payments quickly, within the timeframe specified in the deliverables table (see section B.8 Deliverables).

J. Matched Data - Matched data helps CFPB understand how participants compare to the full sample and/or population. The Contractor’s solution shall demonstrate ability to quickly provide accurate matched data, including, at least, whatever data is used to recruit and screen for participants, after each round of data collection.

K. Feedback - Feedback from the Contractor improves the contract. After each study, the Contactor and the CFPB discuss how to alter communications or logistics so that the contract may better meet the above objectives.

10 | P a g e

B.6 REQUIREMENTS

6.1 Required Capabilities

The Contractor shall possess the following capabilities.

6.1.1 Institutional Review Board (IRB) Approval

The Contractor shall obtain all approvals required to conduct CFPB studies involving human subjects, including obtaining approval from its IRB, if applicable.2

6.1.2 Online Panels

The Contractor shall recruit participants from one or more online panels. The Contractor shall be familiar with the demographics of the members of each online panel it uses, and shall be able to provide estimates of incidences and sample sizes for each of the following demographic groups in each of the panels it intends to use in service of this Contract:

A. Sex. Male and female B. Age. 18-29 years old, 30-39, 40-49, 50-59, and 60 years old or older C. Race. White, Black/African-American, Asian-American, and American Indian or Alaska

Native D. Hispanic/Latinx and non-Hispanic/non-Latinx E. Education. Less than a college degree, associate’s degree or technical/vocational degree, bachelor’s degree, graduate or professional degree (for example, MA, MSW, MD, JD, PhD)

F. Annual House Income. Less than $20k, between $20k and $29,999, $30k-$39,999, $40k- $59,999, $60k-$79,999, $80k-$99,999, $100k-$150k, greater than $150k

In addition, in order to confirm the cost of each study before the study is implemented, the Contractor shall be able to provide incidences of demographic groups specified in each Study Request.

6.1.3 Recruitment

All participants recruited by the Contractor shall meet the following criteria:

A. They joined the panel(s) and participate in studies voluntarily.

2 At the Contractor’s request, CPFB will provide the Contractor with the artifacts to facilitate approvals. These may include, for example, a description of the experiment.

11 | P a g e

B. They are accustomed to participating in studies via the internet. This means, at least, that each member has access to a reliable internet connection and access to a modern browser.

C. They are accustomed to participating in studies that take up to 30 minutes to complete.

D. They are over the age of 18 and reside within the United States.

6.1.4 Screening

The Contractor shall recruit and screen individuals from online panel(s) for participation in CFPB studies. If the Contractor has information about its panel(s)’ members’ demographics and quality (as defined below), then the Contractor may recruit the appropriate individuals using this information rather than screening using additional data collections such as surveys. Screening criteria will be of two types:

A. Demographics. The Contractor must be able to screen respondents based on respondents’ characteristics, including descriptors of themselves such as sex, race, or age; descriptors of their situations such as status as a veteran, status as a student, employment status, or household income; or descriptors of their experiences such as current or previous uses of financial products like mortgage loans, credit cards, or savings accounts.

B. Participant quality. The Contractor must be able to screen out low-quality respondents to its recruiting materials. For example, the Contractor shall be reasonably able to screen out “bots,” respondents from “server farms,” respondents attempting to “retake” the study (either after completing the study once or after already being screened), as well as respondents demonstrating a lack of care or attention in their responses to the screener.3

6.1.5 Participant Payment

Study Requests will describe how participants payments will be determined. Although some studies may specify fixed payments for all participants who complete a study, other studies will determine payments as a function of participants’ choices in the study. The Contractor must

3 “Bots” are computer programs that automatically complete studies. “Server farms” are used to bypass online panel location restrictions or otherwise disguise their identity. Both are described in Chmielewski and Kucker (2019).

Available at https://journals.sagepub.com/doi/abs/10.1177/1948550619875149

Individuals attempting to retake a study after already completing the study or after being excluded by screening are described in Wessling, Huber, and Netzer (2017). Available at https://academic.oup.com/jcr/article/44/1/211/3737805

Measures showing attention, effort, and sincere intention to complete the task are described by Kennedy et al.

(2020). Available at https://www.cambridge.org/core/journals/political-science-research-and-methods/article/shape-of-and-solutions-to-the-mturk-quality-crisis/521AEEB9A9753D5C6038440BD123826C

12 | P a g e therefore be able to take payment instructions from the CFPB after a participant completes a study and pay participants individualized amounts in accordance with those instructions. In addition, before the study is complete, the Contractor shall confirm with CFPB that participants were paid as instructed.

6.1.6 Matched Participant Data

6.1.6.1 The Contractor shall provide to the CFPB all data collected to determine participants’ eligibility for a study. This includes data collected for individuals who were determined to be eligible as well as those who were not. These data shall include unique identifiers to allow the CFPB to match these data to participants’ responses in the study, but shall be otherwise de-identified.

6.1.6.2 If the Contractor screens volunteers using information collected prior to the study (e.g., demographic information collected from all members of an online panel), then the Contractor shall provide:

A. For each screened attribute, the nature of the data used (e.g., survey data or administrative data), a list of all possible values, and, if applicable, individual survey items.

B. For each screened attribute, summary statistics (i.e., mean, minimum, maximum, quartiles, variance, population/sample size) for the entire sampling frame (i.e., those who received, or had access to, the recruitment materials). If a study uses more than one screened attribute, then, upon request, the Contractor shall provide conditional summary statistics.

C. For each screened attribute, individual values (e.g., survey responses) for all individuals who responded to the study’s recruitment materials. As above, these data shall include a unique identifier to allow for matching, but shall otherwise be de-identified.

6.1.7 Personnel

A single person may fill multiple roles if that person meets those roles’ requirements. However, contractor may not invoice for multiple roles filled by a single person.

Key Personnel – Research Scientist The Research Scientist shall be responsible for reviewing Study Requests to ensure each study is feasible. The Research Scientist shall have significant experience running online studies, including studies that include tasks for individual participants, such as surveys; as well as studies that include tasks for multiple participants, such as games (e.g., a prisoners’ dilemma game) or economic simulations (e.g., an auction).

13 | P a g e

Non-Key Personnel – Research Associate The Research Associate shall be responsible for carrying-out most tasks required of the Contractor, including providing documentation in support of CFPB’s submissions to OMB and providing matched data. The Research Associate shall have experience running online studies, recruiting and screening participants, and paying participants.

6.2 Added-value Capabilities

The following items would add value to the Bureau but are not required. They will help CFPB identify the highest value Offers.

6.2.1 Online Panels

The Contractor provides estimates of the incidence of the following groups in its online panels:

A. Those who are, or those who identify as, current students of an institution of higher education

B. Those who are, or those who identify as, borrowers with outstanding higher education loans

C. Those who are, or those who identify as, active-duty members of the United States Armed Forces or as members of the National Guard or Reserves

D. Those who are, or those who identify as, veterans of the United States Armed Forces or as members of the National Guard or Reserves

E. Those who have access to a computer/laptop or smart phone to complete studies.

6.2.2 Recruitment and Screening

The Contractor can provide a sample of participants that matches the demographics of certain populations. For example, the Contractor can provide a sample that matches the demographics of American adults, as measured by the U.S. census. This capability may increase the external validity of CFPB studies run under this Contract.

6.2.3 Matched Participant Data

The Contractor provides additional demographics data about members of its online panels. This allows CFPB researchers to create more succinct study instruments and therefore increases the value of this Contract. In its response to this RFQ, the Contactor would provide a list of available demographics. Each Study Request would then specify a subset of this list to be provided for the study. After data collection for a study is complete, the Contractor would then provide data as

14 | P a g e specified above in section 3.1 (e.g., summary statistics for the full sampling frame, and individual values for all recruited individuals).

6.2.4 Personnel

Research Scientist

A. The Research Scientist has experience running online studies using the Qualtrics platform.

B. The Research Scientist has experience running online studies using oTree software.

Research Associate

A. The research associate has experience running online studies using the Qualtrics platform.

B. The Research Associate has experience running online studies using oTree software.

B.7 CONSTRAINTS AND RISKS

7.1 Budget Constraint

The budget for this contract, base and all options, is not expected to exceed $315,400.00. This budget estimate is based on completion of twenty (20) study requests over the life of the contract. The actual number of Study Requests and effort may vary depending on the needs of the CFPB. Option period funding is not currently secured and may impact the continuation or completion of this contract beyond the base period.

7.2 Contractor shall adhere to all Security and Non-Disclosure information within the clauses of this RFP.

7.3 All deliverables must be 508 compliant

B.8 DELIVERABLES

The following deliverables are required under this contract and shall be submitted to the Contracting Officer’s Representative (COR) and Program Manager for the Study Request (PM).

The deliverable should be submitted electronically, unless specified otherwise.

15 | P a g e

8.1 Contract

The following is a list of deliverables related to this contract.

Deliverable Due Date and Format

Kickoff meeting Within 10 business days after award, the Contractor shall meet in-person or remotely with CFPB staff.

Final Quality Control Plan

(QCP)

Within five (5) business days of receiving CFPB feedback, the Contractor shall make adjustments to the QCP. The final QCP shall be delivered within one (1) month after the award.

Intermittent Project Status Reviews

As requested, or as necessary, to successfully carry out a study.

The format will depend on the request and can include meetings (in-person or remote), phone calls, and by email.

8.2 Study

The following is a list of deliverables related to each study conducted under this contract. To provide a sense of the type of work conducted for the deliverable, examples of possible work requested are included in italics.

Study Feasibility, if applicable

If applicable, within eight (8) business days of receiving a request, provide incidences for proposed groups, maximum sample sizes for proposed groups, and other information that will allow CFPB to confirm the cost of running a specific study.

This will not be necessary if the incidences and maximum sample sizes are already known from the initial proposal or other studies executed under this contract.

Finalized Study Request

1. Initial Study Feedback: Within three (3) business days of receiving a Study Request, the Contractor shall respond with proposed changes, if any. Examples of work include: Contractor provides recommendation on proposed changes, which panel(s) to use, recruitment method, and confirms details of payment method and readily available data.

2. Study Meeting: Within five (5) business days of receiving a Study Request, the Contractor shall meet in-person or remotely

16 | P a g e with CFPB staff to discuss the contractor’s feedback and finalize the Study Request.

3. Finalized Study Request: If changes to materials such as the recruitment or screening materials are required after the Study Meeting, then the Contractor shall make these changes and send the finalized Study Request within five (5) business days of receiving the requested changes.

Support with OMB Package, as requested

Within five (5) business days of receiving a request for support, the Contractor shall provide requested information or materials in support of a submission to OMB.

The Contractor will only be responsible for providing information or materials requested by CFPB; the Contractor will not be responsible for delivery or coordination of the PRA process with

OMB.

Examples of work include: CFPB staff may request information about administrative aspects of the study that are needed for CFPB staff to complete PRA paperwork. The Contractor responds by providing a brief written description of the requested information, such as the Contractor’s recruitment method, procedures for information collection from participants, collection and use of any PII by the Contractor, justification for the use and amount of participant incentives, and other administrative details as requested.

IRB Approval, if applicable

Prior to PRA approval from OMB (OMB approval typically takes a minimum of 8 weeks), the Contractor shall provide confirmation of IRB approval to carry out the study.

Examples of work include: Contractor coordinates and completes submission for IRB review. The CFPB provides the Contractor with information or materials, as needed, for the Contractor to submit the IRB request for approval.

High-Quality Participants

Unless otherwise specified in the finalized Study Request, the Contactor shall begin recruiting for the first wave of data collection (or the pilot/pre-test, if applicable) no later than seven

(7) days after the Study Request is finalized. The Contractor shall email a confirmation when recruitment has started.

17 | P a g e

In most cases, the CFPB will directly observe when participants complete a study. In cases where a Contractor is needed to confirm completes, the Contractor shall send a confirmation within the timeframe(s) specified in the Study Request.

Confirmation of participant payment

Within one (1) business day of confirmed participant completion of the study, the Contractor shall pay the participant the amount specified by the CFPB and send confirmation of participant payment. The confirmation shall allow CFPB to confirm the date and amount of payments made for each participant who completed the study but should not include any direct-identifying

PII.

De-identified demographic and/or other matched participant data

Provided within timeframe as applicable to the Study Request.

The format of data will be determined during Study Request Meeting. The method of secure transfer will be determined after the contract is awarded.

Study Wrap-Up Meeting

Within ten (10) business days of the completion of the Bureau’s Study, the Contractor shall meet in-person or remotely with CFPB staff to confirm completion of study and provide feedback to improve future studies.

8.3 Inspection - All deliverables will be inspected for content, completeness, accuracy and conformance to order, contract or agreement requirements by the COR, or other authorized representative designated in the order, contract or…