ROA OTA-23-002 Frequently Asked Questions Set 2 dated 1 March 2023.pdf

1 March 2023 All of Us Participant and Partner Services Center (OTA-23-002) – Frequently Asked Questions (Set 2)

Question #1

The ROA requirements include the selected vendor’s platform supporting integration for incorporating clinical/EHR and claims data, as well as other data sources to augment the All of Us research base. Integration capabilities to incorporate external data are mentioned many times. This procurement seems to be focused on the overall research platform for All of Us and doesn’t call for bidders to propose specific external data sources. Will one or more ROAs/RFPs be released in the near future to establish these data feeds? If so, can you share any information about the timing and scope of those procurements?

Response #1

The Participant and Partner Services Center (PPSC) will support the All of Us Research Program within a network of systems requiring integration and either bidirectional or unidirectional data exchange. Integrations will support functions ranging from research data collection (from the program or a third-party system), data transfer to the Data and Research Center for storage and researcher access, and information return such as genetic information and results. This information will be provided in detail in a later phase of the competition process.

The PPSC platform's role in Electronic Health Records (EHRs) acquisition for the All of Us Research Program will be focused in two thematic areas. First, the collection and management of participants’ ‘Authorization to Share My EHRs for Research’. Within All of Us, this process is structured and implemented to be inclusive of Health Insurance Portability and Accountability Act (HIPAA) authorization requirements as well as legal requirements across the U.S. States and Territories. For a small subset of states, participants receive state-specific authorization forms to meet state requirements. Participants who join through our VA partner sites receive a VA-specific authorization form that meets the VA's requirements to share EHRs. Second, the PPSC platforms or partner integrations need to be capable of supporting EHR and healthcare claims data collection broadly using patient-mediated (i.e., right-of-access based) directives and data transfer methods leveraging Fast Healthcare Interoperability Resources (FHIR) standards.

The program does not anticipate the PPSC will support the implementation of passive data linkages that leverage participant consent and HIPAA authorization to acquire EHR or healthcare claims data from sources such as Health Information Exchanges (HIE) or Health Information Network (HIN) partners and services. The program has recently published NOT-PM-23-002 to notify of our intention to establish the All of Us Center for Linkage and Acquisition of Data (CLAD) under a future Research Opportunity Announcement (ROA). CLAD will expand All of Us Research Program research and analysis opportunities by acquiring new passive data streams and providing support, as needed, for existing data streams such as EHRs.

Question #2

Is being an existing awardee or participant partner in the All of Us consortium considered a conflict of interest for this award?

Response #2

Being an All of Us Program Consortium member does not itself cause an organizational conflict of interest (OCI).

If you believe there may be an actual, potential, or perceived conflict of interest as described in the PPSC ROA Section 9.5, please submit the following information to AOU-ROA@nih.gov with subject line “OTA-23-002 - Participant and Partner Services Center OCI Concern”:

-Name

-Company name

-Description of work being performed related to work being requested in the

PPSC ROA

Identify all relevant facts concerning any past, present, or anticipated interests (family member, financial, contractual, organizational, or otherwise, as contractor, subcontractor, or consultant) relating to the work requested in the PPSC ROA.

Such interests may have a bearing on whether you or your company has an actual or apparent conflict of interest with respect to being able to render impartial, technically sound, and objective assistance or advice, and without being given an unfair competitive advantage.

The NIH will then review the information and confirm whether an actual, potential, or perceived OCI exists and if a mitigation plan is needed or if future participation is not possible due to the OCI found.

Question #3

Does the All of Us Research Program stipulate or require reuse or replacement of specific technology systems for the Participant and Partner Services Center?

Response #3

No. The program is competing the Participant and Partner Services Center ROA as an open competition in order to identify and select the technology platforms and services that best align with and support the future requirements of the program.

Offerors should propose the technology solutions they feel best meet the capabilities requested for the Research Technology Systems (RTS) and Program Management and Delivery (PMD) Cores.

Question #4

If a bid addresses both cores and passes through Phase 1, will NIH specify which core(s) passed through Phase I?

Response #4

Yes. NIH will specify. Cores will be reviewed and scored individually. Offerors who propose to support both the RTS and PMD cores, and are invited to Phase II, will be informed if both or a single core was invited to proceed to Phase II.

Question #5

Does NIH plan to release those who passed through Phase I? And, if so, does NIH encourage bidders to pursue any partnerships based on the initial pass in Phase I for Phase II full technical submissions?

Response #5

No, NIH does not plan to publicly release partners that proceed to Phase II.

Question #6

How does the NIH envision winners of the bid interacting and functioning among the NIH Outreach and Engagement Team & Communications team?

Response #6

All of Us awardees function as a consortium. Functional meetings and standard operation procedures will be established between NIH and program partners to support the operational and coordination needs between partners.

Question #7

Does the NIH expect that the participant portal for DVs (direct volunteers) and the participant portal for HPOs (healthcare provider organizations) both be phased out and replaced by a one new system built and implemented by bid winners? Or will winners inherit aspects of the original DV and HPO portal systems to build upon?

Response #7

The Program is competing the Participant and Partner Services Center ROA as an open competition to identify and select the technology platforms and services that best align with and support the future requirements of the program. Offerors should propose the technology solutions they feel best meet the capabilities requested for the Research Technology Systems (RTS) and Program Management and Delivery (PMD) Cores. The All of Us Research Program does not stipulate or require reuse or replacement of specific technology systems for the Participant and Partner Services Center.

Question #8

How many researchers are currently conducting program-approved ancillary studies?

Response #8

Initial ancillary studies for the All of Us Research Program are being conducted in collaboration with NIH Institutes and Centers. The program envisions that many ancillary studies will be supported in the future and is actively working to establish processes for proposal, evaluation, and approval of future ancillary studies.

Question #9

Does the NIH expect service providers to offer a Right to Audit clause to support All of Us? Our platforms undergo annual audits for compliance with frameworks such as Statement on Standards for Attestation Engagement (SSAE) 16 System and Organization Controls (SOC) 1, SOC 2, SOC 3; International Organization for Standardization (ISO) 27001; and Payment Card Industry Data Security Standard (PCI- DSS) Level 1. The results of these audits can be provided to the NIH as desired under non-disclosure agreement. Is this acceptable to meet the NIH’s requirements?

Response #9

All partners are required to comply with the All of Us Research Program Data Security Framework documented in the Terms & Conditions (Attachment A) in Article XI, Section S. Offerors should review the security framework and requirements to determine if their proposed solution can meet program requirements.

Question #10

If a platform’s incident response reports are audited by a third-party assessment organization (3PAO), the results of these audits may be shared with the NIH under the non-disclosure agreement. Will this meet the NIH's requirements?

Response #10

All partners are required to comply with the All of Us Research Program Data Security Framework documented in the Terms & Conditions (Attachment A) in Article XI, Section S. Offerors should review the security framework and requirements to determine if their proposed solution can meet program requirements.

Question #11

How often do you intend to communicate with participants? How many messages would be deployed at a time? In a year? What kinds of automations do you anticipate and what might trigger them?

Response #11

In a recent 12-month period, the All of Us Research Program platforms were supporting greater than 200 ongoing email campaigns and tens of thousands of additional ad hoc campaigns with over 13M emails sent. Program platforms were supporting greater than 100 ongoing SMS campaigns and more than one thousand additional ad hoc campaigns with over 1.7M SMS messages sent.

Additional communications via phone are commonly used to facilitate participant engagement and retention, and letter mail is used to selectively communicate key program milestones or opportunities.

Question #12

What are the legacy All of Us systems? What technology stack do they have for each?

What versions of the technology stack are in use today? How large are the data sets in the legacy systems?

Response #12

The All of Us Research Program has multiple platforms in service currently supported by the Participant Technology Systems Center (PTSC) and the Participant Center (TPC) awards. Legacy systems will be determined based on the outcome of the Participant and Partner Services Center competition. Further details and data volumes will be provided at a later stage of the competition.

Offerors should propose a solution that they feel best meets the capabilities requested in the ROA.

Question #13

January 30, 2023 is listed as the date for questions to be answered during the Pre- Submission Seminar. Will NIH accept questions after January 30th for further clarifications? If so, what is the cut-off date for other questions?

Response #13

Yes. We encourage offerors to review the ROA and submit questions as soon as feasible. Questions must be submitted to AOU-ROA@nih.gov at least 14 calendar days prior to the white paper submission due date to ensure response;

if submitted less than 14 calendar days from the March 20th submission deadline, they may not be answered.

Question #14

What is the timeframe for notifications to offerors on Phase II?

Response #14

The tentative offeror invitation schedule for invitation to Phase II: Full Proposals is May 10, 2023.

Question #15

What is the timeframe for notifications to offerors on Phase III?

Response #15

The tentative offeror invitation schedule for invitation to Phase III: Capabilities Demonstration is August 1, 2023.

Question #16

What is the timeframe for notifications to offerors on Phase IV?

Response #16

The tentative offeror invitation schedule for invitation to Phase IV: Full Budget / Cost Proposal is September 26, 2023.

Question #17

What is the timeframe for notifications to offerors on Phase V?

Response #17

The tentative invitation schedule for Phase V: Other Transaction award is December 22, 2023.

Question #18

“Experimentation, randomization, statistical sampling, and A/B Testing tools to support protocol and user experience testing, delivery, and evaluation of existing and new program opportunities under consideration”

Please elaborate on the kind of experimentation, randomization, statistical sampling, and A/B testing tools.

Response #18

The All of Us Research Program is using experimentation in the form of embedded randomized controlled trials or A/B testing. This is done to test versions of protocols or test participant communications strategies to optimize participation and engagement while minimizing both participant burden and cost associated with outcomes achieved. Randomization is also used frequently to create sub-study audience segments for testing or when protocols cannot be implemented among all eligible program participants.

Question #19

Related to: ‘Enterprise analytics data systems and tools to be used for marketing purposes, participant engagement, experience sampling, and program reporting.’

Who are the consumers of these data?

Will the data scientists be working with or possibly integrated with the Office of Data and Analytics (ODA)?

Response #19

Collectively, the RTS Core Tools and PMD Core Services supported by PPSC must support a wide range of program partners and use-cases. Audiences will include 1) study staff and managers, 2) program administrators, and 3) ancillary studies teams. NIH All of Us Program Officers, as well as staff and leadership from program offices and divisions, work closely and collaboratively with partners to coordinate the oversight, goals, and successful delivery of program objectives.

Analytics data systems and tools will be leveraged to support a mix of workflow-based processes, audience characterization, segmentation and experience tailoring, program monitoring and evaluation, etc.

Question #20

For the Program Management and Delivery Core - Indirect Research Support Team, would the government please describe what NIH envisions for “The team will manage and support …. All of Us partner collaborations?”

Response #20

In addition to deployment, configuration, and use of PPSC systems, the PPSC team must provide training and support for study staff on all relevant PPSC software, systems, and tools.

Study staff and managers from program partners are a key audience of support for the PPSC and deployed technology must support their engagement, enrollment, and retention efforts.

Overall, All of Us awardees function as a consortium. Program management meetings and standard operating procedures will be established between functional teams to support the operational and coordination needs between partners.

Question #21

Related to: ‘Enterprise analytics data and tools to be used for marketing, participant engagement, experience sampling, and reporting’.

What is the government’s vision for these tools and their implementation?

Could the government please further break down the current categories of work that fall within this area?

Response #21

Analytics data systems and tools will be leveraged to support a mix of workflow-based processes, audience characterization, segmentation and experience tailoring, program monitoring and evaluation, etc.

Question #22

Related to: ‘Developing and implementing data analytics, including regular reporting and dashboards for internal program use, protocol monitoring, and user experience testing and evaluation.’

Can the offeror assume PMDC-based staff (Program Management and Delivery Core) will collaborate with ODA to support the projects that fall within the scope of PMDC management? If yes, What shall be the duties of the Office of Data and Analytics in these instances?

Response #22

Collectively, the RTS Core Tools and PMD Core Services supported by PPSC must support a wide range of program partners and use-cases. Audiences will include 1) study staff and managers, 2) program administrators, and 3) ancillary studies teams. NIH All of Us Program Officers, as well as staff and leadership from Program offices and divisions, work closely and collaboratively with partners to coordinate the oversight, goals, and successful delivery of program objectives.

Analytics data systems and tools will be leveraged to support a mix of workflow-based processes, audience characterization, segmentation and experience tailoring, program monitoring and evaluation, etc.

Question #23

Related to: ‘Supporting program and platform requirements for data quality monitoring, assurance, and control.’

Could the government please share existing data quality standards that have been set by the program?

Response #23

A broad data quality framework is used within the All of Us Research Program with more specific methods and standards for data quality defined within specific data types. Further details will be provided at a later stage of the competition.

Question #24

Related to: ‘Technology, methods, and services to enable digital access and sustainable participation.’

Is the goal here to address digital access or all inequities involved in sustaining participation?

Response #24

The priority for technology, methods, and services to enable digital access and sustainable participation is anticipated to focus on 1) access to technology and/or broadband service; and 2) training and support to enable technology and service utilization to support program participation. The program acknowledges that no initiatives supported by the PPSC will be capable of addressing all inequities involved in sustaining participation.

Question #25

Can you provide the list of reviewers in case of possible conflicts of interest?

Response #25

Reviewer lists are confidential and will not be released publicly. Reviewer conflict of interest is assessed and managed prior to reviewer assignment.

Question #26

Can an organization apply for one core, or should the application encompass both?

Response #26

Offerors may apply to support one or both cores, depending on their organizations’ specific capabilities. An organization is limited to a single submission but may be included as a collaborating partner on more than one application.

Question #27

What are the budget expectations for each of the years following the first 15 months?

Response #27

Cost estimates for optional periods are not being requested in Phase I proposals but may be requested in Phase IV for planning and budgeting purposes. The Phase IV offeror may be requested to provide an estimate for the optional periods based on additional details provided in future phases of the competition process.

Question #28

Are third tier subcontracts allowable through this mechanism?

Response #28

Yes, third tier subcontracts are allowable, but any lower tier agreements with subcontractors must include the flow down terms required in the OTA Terms and Conditions for the PPSC award.

Question #29

Is there a limit to the number of listed principal investigators?

Response #29

No, offerors may determine and define their leadership and management structure to best meet the capabilities and milestones outlined in the Participant and Partner Services ROA.

Question #30

What should offerors name their one white paper file if they wish to submit a proposal covering both cores?

Response #30

White paper files for white papers covering both cores should be labeled “ROA OTA-23-002 White Paper by [insert name of organization] - RTS and PMD Cores”

Question #31

How do we obtain a copy of the February 2, 2023 PPSC pre-submission webinar?

Response #31

NIH will be posting the 2/2/2023 PPSC webinar on the PPSC SAM.gov announcement once our team has completed making it 508 compliant.

Question #32

For the two Cores identified in the ROA (RTS & PMD), how do you foresee handling the division of responsibilities? Would you expect this to be mostly set at the start, or would you expect these partners to change this over time e.g., as systems mature?

Response #32

Offerors are encouraged to propose partnerships within their proposal to provide research services and technology expertise across cores within a single award.

NIH may also select separate offerors to support individual PPSC cores. All selected partners accepting an award must work collaboratively to achieve the program’s goals and the PPSC milestones and objectives. If multiple awards are issued under this announcement, the Program Management and Delivery (PMD) core will serve as the administrative coordinating center for PPSC, managing cross-core coordination and project delivery. The All of Us Research Program functions as a consortium so program partners should expect a collaborative working relationship to be developed during the onboarding phase to support cross partner and system interactions and management. Division of responsibility is anticipated to align with the capabilities and functions requested from each core in the ROA.

Question #33

Are undocumented individuals included among the participant cohorts for the PPSC?

Relatedly, are there any restrictions/qualifications for individuals who express interest in participating in PPSC services?

Response #33

Enrollment in the All of Us Research Program is open to adult individuals living in the United States with the ability to provide informed consent for themselves.

Enrollment and participation are not currently open to individuals who are incarcerated.

Question #34

Is an ATO (Authorization to Operate) and FISMA (Federal Information Security Management Act) Moderate FEDRamp (Federal Risk and Authorization Management) security stance required for this?

Response #34

The PPSC platforms will require an Authorization to Operate (ATO) consistent with the NIH security requirements outlined in the ROA and the attached Terms and Conditions (Appendix A).

Question #35

Is AOU interested in recruitment efforts as well? What is the main goal of the program?

To get to how many millions of people?

Response #35

The All of Us Research Program aims to enroll 1M or more participants in the program that reflect the rich diversity of the United States. The PPSC will interface with and support recruitment and enrollment partners by providing tools, services, and support for program operations.

Question #36

Is there a requirement that IT support personnel must be US citizens? Is there a requirement to use AWS Gov or Azure Gov for cloud infrastructure resources?

Response #36

The Terms and Conditions (Appendix A) outline restrictions on foreign affiliation for staff based outside of the United States. U.S. citizenship is not a requirement for U.S. based staff with proper authorizations for employment. The NIH does not have a preferred cloud services vendor. Any platform selected must be capable of supporting the security requirements for program platforms and infrastructure.

Question #37

Are there current awardees for the services included in PPSC?

Response #37

Please see FAQ #32 above - the response is consistent with this question and communicate that: "The milestones in year 1 of the PPSC award will facilitate a transition from the program’s current participant and study staff services – supported by multiple awards, including the Participant Technology Systems Center (PTSC) and the Participant Center (TPC) – to the PPSC award team and their resultant services. Proposed technology systems and research services must be capable of integrating with All of Us platform partners such as the Data and Research Center (DRC), configuring to the protocol, providing cross platform testing and staff training, and coordinating production launch within 8-12 months from the time of the award. Following successful production launch, the PPSC will transition existing participant user accounts from any legacy systems."

Question #38

May we get more information on the ATO – Authorization to Operate. Is that provided by NIH after an audit (and that is renewed and subject to follow on audits)?

Response #38

Additional details will be provided at a later phase of the competition. The PPSC milestones and terms and conditions overview the requirements expectation for system security and privacy.

Question #39

Is there a list of IT security controls NIH expects? We understand NIST (National Institute of Standards and Technology) 800-53 Moderate and have experience with some nuance of controls – where some may be waived.

Response #39

Additional details will be provided at a later phase of the competition. The PPSC milestones and terms and conditions overview the requirements expectation for system security and privacy.

Question #40

Will the capabilities scores for each core be shared with the offeror?

Response #40

Per the PPSC ROA, the NIH does not intend to provide technical or programmatic feedback on white paper submissions. However, offerors will receive notification should the All of Us Research Program decide not to provide an invitation for the next Phase.

Question #41

Is there any possibility of taking this global?

Response #41

Enrollment in the All of Us Research Program is open to adult individuals living in the United States with the ability to provide informed consent for themselves.

Enrollment and participation are not currently open to individuals who are incarcerated.

Question #42

Of the existing technology partners (PTSC and TPC), how many participants have been enrolled on each platform?

Response #42

Up to date information on enrollment is provided in our data snapshots on the All of Us Research Hub: https://www.researchallofus.org/data-tools/data-snapshots/ https://www.researchallofus.org/data-tools/data-snapshots/