AoU_CLAD_ROA-OTA-23-003 FAQ Set 2_7 April 2023.pdf

National Institutes of Health (NIH) Office of the Director (OD) All of Us Research Program

Frequently Asked Questions Set 2 All of Us Center for Linkage and Acquisition of Data (CLAD) No. OTA-23-003

Issue Date: April 7, 2023

Question 1. a) If the Principal Investigator, Program Manager, and De-Identification/Disclosure Expert keys must have a biosketch, and the biosketch form asks for “eRA COMMONS USER NAME (credential, e.g., agency login),” then do all three key personnel roles require accounts in eRA COMMONS? b) If the answer to the above is yes, does NIH require that the Program Manager and De-Identification/Disclosure Expert also be associated with the application in some way, or do those roles just require a user name/account?

Answer: Per CLAD ROA Amendment 2 posted on 3/30/2023, the NIH biosketch is only required for proposed Principal Investigator and/or Program Manager, which will be required to register in eRA Commons. A proposed De-Identification/Disclosure Expert is now considered other Key Personnel and only a resume is required.

Question 2. Regarding the Cost proposal template provided in Appendix F. In each milestone tab, in column I which is titled “Total Salary to Complete Milestone Over Milestone Period of Performance”: In this cell should we put a salary number only, or salary and fringe?

Answer: The “Total Salary to Complete Milestone Over Milestone Period of Performance” is the total salary (salary + fringe) for the time period to complete the milestone for that personnel based on their percent of FTE proposed for the effort. A revised Appendix F has been posted in the attachments of the CLAD ROA posting on SAM.gov to add clarity and rename column I to “Total Labor Cost to Complete Milestone Over Milestone Period of Performance”.

Question 3. Can you please clarify if references can/should be cited? And if yes, does this count towards the 32-page limit?

Answer: References may be included. References count towards the page limit.

Question 4.Would NIH agree to allow an Open Source Software (OSS) contributor to work on OSS libraries to achieve NIH goals, while allowing those contributions to be open source?

Anything specifically built for this CLAD ROA would be kept separate and remain proprietary to NIH. For tools that we build for NIH, they would be proprietary to NIH. If any changes need to be made in the underlying OSS libraries to create the tools, would NIH allow the newly developed capabilities to be shared back with the open source community?

Answer: The All of Us Research Program supports the use of open source tools, and will accept the use and development of open source libraries and tools for CLAD. However, the All of Us Research Program OTPO reserves the right to determine if new or improved OSS tools will be made available to the open source community, pending considerations of program security and programmatic considerations. The All of Us Research Program will not fund the distribution or support of the open source libraries, code, documentation, and related artifacts created for CLAD to the open source community by the CLAD awardee beyond the specified requirements of the CLAD award.

Question 5. ROA Main Document/Cost Proposal. Regarding the statement "All subawards shall provide a separate cost break and narrative per milestone at the same level of detail required of the prime," in the ROA's Cost Proposal question, an answer from the NIH to a previous question (Question 1) was "The offeror can have subawards with fixed priced labor rate arrangements for commercial services and fixed priced commercial products. Any fixed price cost shall still breakdown labor categories, hours, and itemized list of materials/equipment and travel in proposal." Can the NIH please confirm that the prime offeror would present subcontract labor costs with commercial hourly rates and labor hours? If so, would subcontractors be required to submit cost breakdown by salary and fringe in a sealed bid from subcontractors to NIH to justify hourly labor rates?

Answer: Subcontractor proposed Fixed Price Labor rates for commercial services do not need to be broken down by salary and fringe but should reflect a breakdown of the labor category, loaded labor rate, and hours proposed per labor rate.

Question 6. ROA 8.1.2 Proposal Format. If Offerors are permitted to use smaller text for charts, tables, figures, and illustrations as long as it is legible when the page is viewed at 100%, may offerors also use a different font type other than Times New Roman for the smaller text, such as Arial?

Answer: Yes, as long as it is legible when the page is viewed at 100%.

Question 7.Would the government allow the Arial Family (or other) font within tables and/or graphics?

Answer: Yes, as long as the tables and/or graphics are legible when the page is viewed at 100%".

Question 8. Is there a single data file within the DRC that the CLAD will need to link to, or are there multiple files that would each need to be separately linked to the CLAD? If so, how many?

Answer: The DRC will provide data to CLAD that will allow for linkage. This may consist of privacy preserving record linkage tokens or PII data.

Question 9. In section 8.1.3, the government states: “The offeror can have sub awards with fixed priced labor rate arrangements for commercial services and fixed price commercial products.”

Can the government please provide additional context and definition of “commercial services”?

How will the government evaluate if a sub-award is commercial or non-commercial?

Answer: Commercial services are services that are offered and sold competitively in substantial quantities in the commercial marketplace based on established catalog or market prices for specific tasks performed or specific outcomes to be achieved. The NIH is not going to make a commercial determination but the offeror should explain how the sub award effort being performed is appropriate for a fixed priced arrangement in the cost narrative.

Question 10. In section 8.1.3, the government states: “All subawards shall provide a separate cost break and narrative…” Are commercial partners required to provide a detailed cost breakdown? If no, will the government please update the proposal instructions?

Answer: Any fixed price cost shall breakdown labor categories, hours, and itemized list of materials/equipment and travel in proposal.

Question 11. In section 8.1.3, the government states: “All subawards shall provide a separate cost break and narrative…” Will the government accept sealed submissions from sub-award partners?

Answer: Subcontractor (subaward) cost breakdown and narratives should be submitted with the prime offeror’s proposal per the ROA unless subcontractor’s business practices prevent disclosure of cost breakdown to the prime offeror. Subcontractors' request to submit cost breakdowns directly to the NIH can be submitted to allofusdtpdawardrequirements@nih.gov. Requests must be received before the proposal due date of April 17, 2023 at 3:00pm ET. Subcontractor cost should also be received by the proposal due date of April 17, 2023 at 3:00pm ET. If the subcontractor is going to submit cost details directly to NIH, this must be stated in the Prime’s proposal.

Additionally, each subcontractors' overall cost must be included in the Prime's cost proposal.

Question 12. The government states “the awarded funds may not be used for…… purchase of laptops, ….. maintenance and repair,…. Proposal costs…,” etc. Costs such as software/hardware, Bid & Proposal costs, and leasehold maintenance are allowable under FAR regulations and may be included in offeror’s established provisional indirect rates. Can the government please confirm these costs are unallowable as direct costs OR are they unallowable as both direct and indirect? If they are unallowable as both direct and indirect, but included in the offeror’s negotiated indirect rate statement, how would the government request offerors to propose modifications to the established provisional indirect rates?

Answer: Per the ROA, the Terms and Conditions attached in Appendix A serve as a baseline for Other Transactions agreements awarded by the All of Us Research Program in Phase III. The NIH may modify these terms and conditions throughout the selection process. Requests by offeror(s) to modify the terms and conditions language will be considered by the NIH on a case-by-case basis and negotiated as deemed appropriate.

Discussion/negotiation of agreement terms and conditions will take place with the invitees of Phase II and Phase III. The principles included in FAR Part 31 may be applied for the purposes of determining allowable and unallowable costs.

mailto:allofusdtpdawardrequirements@nih.gov

Question 13.Would the Government allow submission of the Cost Response as a separate Proposal file?

Answer: Unless required to be a separate attachment per eRA ASSIST instructions, the individual sections in ROA Section 8.1.3 are not required to be separate attachments but each section must be clearly labeled and contain the information and adhere to the page limitations stated in the ROA.

Question 14. Are there any GFE that NIH will provide the awardee?

Answer: The Government will not be providing any Government Furnished Equipment

(GFE).

Question 15. Are there enterprise tools for Agile management (e.g., Jira), document storage tools (e.g., Confluence or SharePoint), and DevSecOps and CI/CD tools (e.g., GitHub, Jenkins, etc.) that will be available for the contractor to use?

Answer: All of Us does not provide any software infrastructure that the awardee will be required to use, with the exception of the Cyber Security Assessment and Management Application. This is used to submit and maintain security documentation. Our team is migrating to this tool:

https://www.cisa.gov/resources-tools/resources/cyber-security-assessment-and-mana gement-csam-application

Question 16. Does the data delivered to the All of Us system need to be formatted as an OMOP formatted file, or does NIH want us to describe our recommended delivery format? ROA PG 12:

“Ontology and Terminology services: Ensure incoming CLAD data is compliant with agreed upon healthcare data/interoperability standards. Map data to the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) or another format that eases the ingestion of the data by the DRC. Provide terminology services as needed to fulfill relevant milestone objectives.” ROA Pg. 13: “During the base period of this award the DLC will produce several data linkages as described in section 5.3. Linked datasets will be delivered to the All of UsDRC in the OMOP CDM standard format or another format that eases the ingestion of the data by the DRC.” ROA Pg 22. “Objective 6.4.3 - Prepare mortality data consistent with OMOP and All of Us Research Program RW tiers in preparation for transfer to the DRC”

Answer: The government would prefer mapping to OMOP but will accept other formats as long as it will ease ingestion by the DRC.

Question 17. Does NIH want to see letters of interest from partners as part of the proposal submission?

Answer: Letters of interest from partners are not required or needed.

Question 18. On p. 12 and elsewhere in the solicitation, the ROA requires that the awardee “Map data to the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) or another format that eases the ingestion of the data by the DRC”.

However, in the milestones/objectives, there are multiple objectives specifically requiring data preparation consistent with OMOP. Is OMOP the required or preferred format?

Answer: The government would prefer mapping to OMOP but will accept other formats as long as it will ease ingestion by the DRC.

Question 19. Can the government provide more detail on the EHR data elements to be acquired during the pilot? For example, does the government seek unstructured data that could be mined through natural language processing (NLP)?

Answer: This is a pilot to assess the number of All of Us Research Program participants that are successfully identified through HIN/HIE, the years of data available for these participants, the percentage of data that does not duplicate what is already in the All of Us Research Program database, as well as the breadth and depth of data acquired. We do not have a specific set of data elements that are to be acquired during the pilot. Milestone 11 Objective 11.1 is a requirements gathering step that will help frame out these details.

Currently, All of Us is not releasing clinical notes to the Researcher Workbench.

However, we are experimenting with NLP to enable this. So, generally, we’d like to capture as much clinical health data as possible.

Question 20. If the proposer is not encouraged to propose with a HIN/HIE as a teaming partner, can the government advise on placing pricing placeholders in the proposal budget? Or, is this subcontract funding to HIN/HIE partners outside of this OTA?

Answer: The government is not encouraging or discouraging you from having a teaming partner or partners. However, all of the work related to this pilot will be within the CLAD award. There will not be a separate award to connect to an HIN/HIE.

Question 21. Can the proposer include a HIN/HIE as part of the proposal either as a confirmed partner or as a letter of support?

Answer: The government is not encouraging or discouraging you from having a teaming partner or partners. However, all of the work related to this pilot will be within the CLAD award. There will not be a separate award to connect to an HIN/HIE.

Question 22. Does the government have a preference for QHIN HIN candidates to participate in the pilot?

Answer: No

Question 23. Please confirm the tool development described in the Milestones sections is meant to be executed against the All of Us Google hosted platform, and not against the CLAD system, and that access to the All of Us system will be provided.

Answer: Tools developed in the milestones section should be executed in the CLAD environment to demonstrate they operate properly. They will then be shared with the DRC for deployment to the Researcher Workbench.

Question 24. Can we assume that if we receive an invitation to participate in Phase II we will be informed which All of Us single IRB (sIRB) we can use on our FWA application? If not, how can we obtain that sIRB for our FWA application?

Answer: For the purposes of an organization completing a new FWA, the All of Us IRB information, below, should be referenced as the IRB on which they will rely for review of human subjects research:

FWA: FWA00025248

IRB: IRB000010472

For more information on filing an FWA please refer to the Office of Human Research Protections website at:

https://www.hhs.gov/ohrp/register-irbs-and-obtain-fwas/fwas/index.html.

Question 25. Section - 5.2.3 Data Linkage Core (DLC) and Appendix D ROA Paragraph Reference Section 5.2.3: Of note, the RW has three tiers:

1) a Public Tier with data that anyone can browse;

2) a Registered Tier for registered data for authorized users; and

3) a Controlled Tier that may be accessed only by researchers who have completed all data access requirements and trainings.

Appendix D: All of Us currently makes participant data available to researchers on the RW through the following three tiers:

a) Public Tier: aggregate data and summary statistics;

b) Registered Tier: participant-level data subject to a high degree of privacy-preserving obfuscation and transformation; and

c) Controlled Tier: participant-level data with additional data types and fewer such transformations.

Given the public tier only releases aggregate data and summary statistics and Appendix D does not include any variable listing for the public tier - Would the government confirm that linked individual data is not expected to be shared via the public tier?

Answer: CLAD linked individual row level data will not be shared via the public tier (https://databrowser.researchallofus.org/).

Question 26. ROA Main Document/Data Platform Core. Is the intent of the Master Study Participant Index (MSPI) for the CLAD awardee to maintain a crosswalk between All of Us Participant identifiers, and tokens used to identify All of Us participants in linked datasets? For example, if there are multiple tokenization methods used across different data vendors, is it expected that a table of identifiers and matched tokens across data vendors be maintained centrally in the Data Platform by the CLAD awardee and shared with the DRC? Or is it expected https://www.hhs.gov/ohrp/register-irbs-and-obtain-fwas/fwas/index.html that only tokens and linked data be shared with the DRC to avoid re-identification of tokenized data?

Answer: Yes, the intent is to maintain a crosswalk and the provenance of All of Us participant identifiers, tokens, and linked datasets, regardless of the source or methods.

This will be maintained within CLAD. Initially, we do not anticipate this will be shared with the DRC, however that may change if needed.

Question 27. ROA Main Document/Data Linkage Core. Please elaborate on the extent to which analytics tools developed by CLAD to be deployed for the Researcher Workbench should “jumpstart” analyses on linked datasets acquired by CLAD? For example, we assume that analytical tools at a minimum are expected to query the schema of linked datasets, present descriptive statistics of variables, and can serve as a starting point for statistical and/or machine learning models. To what extent should the analytical tools build such statistical and machine learning models? Are there specific models of interest to be developed?

Answer: For each of the data linkage milestones there is an objective for requirements gathering and tool development. During the execution of these objectives the awardee will propose the type of analytic tools that would be most useful for researchers using these data to understand the structure of the dataset. Basic descriptive statistics / univariate analyses by demographic characteristics should be included. Beyond this machine learning and other models may be explored if there is a specific type of analysis / general problem that would serve as a starting point for researchers.

Question 28. Section - 5.2.2. Data Platform Core (DPC), 5.2.3. Data Linkage Core (DLC) ROA Paragraph Reference 5.2.2. “The program expects that All of Us data is not commingled with non-All of Us data (e.g., data from other studies). 5.2.3. The DLC will be responsible for linking externally acquired datasets to All of Us Research Program participant records. These data will be used by investigators on the RW. It will also help fill in gaps in program participants’ longitudinal record and their lived experience. The DLC will use the DLP to execute the work required to produce linked datasets. All linked datasets will be stored within the DLP. These datasets will be transferred to the DRC.” IAW Section 5.2.2, "AoU data is not (to be) commingled with non-AOU data"; however, IAW Section 5.2.3, the "DLC will be responsible for linking externally acquired datasets to All of Us Research Program participant records."

Would the government please clarify whether any actual AoU EHR data (beyond PII or tokens from the AoU EHR participants) will be sent to CLAD by the DRC for linking AoU EHR data with external data.

Answer:We do not anticipate at this time sending EHR data from the DRC to CLAD.

This can be explored if needed. One place where this may need to be considered is for de-duplication efforts. However, not all DRC EHR data may be needed under such circumstances. This may be determined during requirements gathering for specific linkages (e.g, the HIN/HIE pilot).

Question 29.What is the typical number of hours needed for a vendor to complete your ATO process?

Answer:We cannot provide a direct answer to this question as it is dependent upon a number of factors and varies by project. We do estimate that achieving ATO will take between 9 and 12 months to complete.

Question 30. In Section 8.1.3 of the ROA it states: "All subawards shall provide a separate cost break and narrative per milestone at the same level of detail required of the prime," and "The offeror can have subawards with fixed priced labor rate arrangements for commercial services and fixed priced commercial products. Any fixed price cost shall still breakdown labor categories, hours, and itemized list of materials/equipment and travel in proposal." Can the NIH please confirm that the prime offeror would present subcontract labor costs with commercial hourly rates and labor hours? If so, would subcontractors be required to submit cost breakdown by salary and fringe in a sealed bid from subcontractors to NIH to justify hourly labor rates?

Answer: Subcontractor proposed Fixed Price Labor rates for commercial services do not need to be broken down by salary and fringe but should reflect a breakdown of the labor category, loaded labor rate, and hours proposed per labor rate.

Question 31. Are all Key Personnel submitted within the Technical Proposal expected to have an eRA Commons Login included in their NIH Biosketches?

Answer: Per CLAD ROA Amendment 2 posted on 3/30/2023, the NIH biosketch is only required for proposed Principal Investigator and/or Program Manager which will be required to register in eRA Commons. Only a resume is required for all other proposed Key Personnel.

Question 32. ROA 5.2.2 Pg11 Identity Management, Participant Identity: Utilization of a Master Study Participant Index (MSPI) for positive participant identification, authentication, data provenance, tracking of sourced data, and related functionality." Could you please explain what All of Us means by "utilization of the MSPI for authentication?" as required in the ROA?

Answer: It is anticipated that the MSPI will be used to authenticate individuals suggested by data sources to match All of Us study participants as indeed valid matches, or not.

Question 33. Are the use of URLs/hyperlinks allowed throughout the proposal, apart from biosketches?

Answer: Only the contents of the proposals can be reviewed and evaluated therefore proposals should not contain URLs/hyperlinks.

Question 34. Can you please verify: Only the Principal Investigator and PM will include 4-page biosketches? In other words, if additional individuals are named Key Roles per 8.1.3 item 10, they will NOT require 4-page biosketches, only 2-page resumes?

Answer: Yes, only the Principal Investigator and PM require 4-page max biosketches and only up to 2 page resumes are required for all additional Key Personnel.

Question 35.We are part of a larger team for our proposal. Can our team, which is a subcontractor to a larger prime, also subcontract to a different entity. Can we do a sub on a sub?

Answer: Yes, 3rd Tier sub-subcontractors are allowed but lower tier agreements will require all the flow down terms required in the OTA Terms and Conditions for the CLAD award.

Question 36. Can the Government please clarify what type of contract will be awarded:

Milestone Payment, Firm-Fixed-Price, Time & Materials, etc.?

Answer: Due to uncertainties and flexibility needed in the project, a fixed price agreement would not be appropriate, therefore a cost-reimbursable or cost-reimbursable plus fee payment agreement would be considered. The offeror can have sub awards with fixed priced labor rate arrangements for commercial services and fixed priced commercial products. Any fixed price cost shall breakdown labor categories, hours, and itemized list of materials/equipment and travel in proposal

Question 37. Section - 8.1.3 Proposal Content, Item 10 - Key Personnel ROA Paragraph Reference Information Systems Security Officer (ISSO): The ISSO will be responsible for ensuring that all CLAD systems, tools, platforms, and software adhere to NIH and All of Us security requirements. The ISSO will work with the All of Us Information Systems Security Officer and OTPO. Would the government clarify whether the ISSO listed as a key personnel belongs to the Program Compliance Core as shown in Section 8.1.3 under Key Personnel or if the ISSO is a key personnel in the Data Platform Core?

Answer: As described in the ROA Section 8.1.3, the ISSO is in the Program Compliance Core. Also note that the offeror may identify an alternative list or additional key personnel and their associated LoE to successfully execute this work. Offerors must have at least a Principal Investigator or Program Director as key personnel.

Question 38. Section - FAQ Amendment 1 ROA Paragraph Reference Question 35 IAW Question 35, Letters of Support are not required to be submitted with Prime's Letter of Intent submission to the government. Would the government please confirm neither Letters of Support nor Letters of Commitment are required for proposal submission.

Answer: Letters of Support or Commitment are not needed or required for Phase I proposals.

Question 39. ROA 8 Pg32 Milestone Plan. Provide a table that includes for each milestone:

(bullet 4). Estimate total cost: the estimated total cost for the milestone. Since the milestone table requires "Estimated Total Cost in Dollars", should the table be included within the Cost Volume narrative and not included with the Technical Proposal?

Answer: The costs in the milestone table are high level cost estimates for each milestone and should match directly with the full cost proposal.

Question 40. ROA Main Document/Key Personnel. Are all Key Personnel submitted within the Technical Proposal expected to have an eRA Commons Login included in their NIH Biosketches?

Answer: Per CLAD ROA Amendment 2 posted on 3/30/2023, the NIH biosketch is only required for proposed Principal Investigator and/or Program Manager which will be required to register in eRA Commons. Only a resume is required for all other proposed Key Personnel.

Question 41.Will the NIH please confirm that the costs referenced in Section O of Appendix A are prohibited from being billed as direct costs against the contract, but are permitted to be included in the Recipient’s government-approved indirect cost structure to the extent that they are allowable per FAR?

Answer: Per the ROA, the Terms and Conditions attached in Appendix A serve as a baseline for Other Transactions agreements awarded by the All of Us Research Program in Phase III. The NIH may modify these terms and conditions throughout the selection process. Requests by offeror(s) to modify the terms and conditions language will be considered by the NIH on a case-by-case basis and negotiated as deemed appropriate.

Discussion/negotiation of agreement terms and conditions will take place with the invitees of Phase II and Phase III. The principles included in FAR Part 31 may be applied for the purposes of determining allowable and unallowable costs.

Question 42. Reference 8.1.3 11.c. and Ts and Cs Appendix A Article II Term B Termination Provisions. Please clarify the intended contract type? The ROA states in part "Due to uncertainties and flexibility needed in the project, an adjustable cost model (cost-reimbursable or cost-reimbursable plus fee) will be considered." and “The Government has no obligation to pay Recipient beyond the last completed and paid milestone if Recipient decides to terminate.”.

Please clarify how contract payments will be made both in the baseline period and in the optional periods.

Answer: Per the ROA, the Terms and Conditions attached in Appendix A serve as a baseline for Other Transactions agreements awarded by the All of Us Research Program in Phase III. The NIH may modify these terms and conditions throughout the selection process. Requests by offeror(s) to modify the terms and conditions language will be considered by the NIH on a case-by-case basis and negotiated as deemed appropriate.

Discussion/negotiation of agreement terms and conditions will take place with the invitees of Phase II and Phase III.

Question 43. Please provide a link or a format for the valid and updated NIH biosketch template.

Answer: A sample of a NIH Biosketch can be found at https://grants.nih.gov/grants/forms/biosketch.htm

Question 44. Clarification for Upload In ASSIST (Section 5) there is a Single Upload Field for ‘Leadership Plan’. Is this intended to be a Multi-PI Plan (though this is not mentioned in the Proposal Content section) or is this the ‘Leadership and Management Plan’ noted on Page 32 | https://grants.nih.gov/grants/forms/biosketch.htm

8.1.3 Proposal Content | Section 6? Should there be two documents? Or just the ‘Leadership and Management Plan’ and it should be uploaded in ASSIST in Section 5?

Answer: There should be only one Leadership and Management Plan submitted. In Section 8.1.3 for Proposal Content, the Leadership and Management Plan is addressed. If multiple PIs are proposed it would be addressed in the Leadership and Management Plan as part of the description of leadership and management processes to execute the award.

Question 45. Is it expected that we prepare the Human Subjects and Clinical Trials forms in

ASSIST?

Answer: The ROA does not require Human Subjects and Clinical Trials forms to be completed.