Responses_to_Questions_AHRQ_19-10003_final.pdf

Response to Questions AHRQ-19-10003 CAPSTONE Page 1 of 8

CAPSTONE

AHRQ-19-10003

Proposal Questions

Thank you for your interest in the Request for Proposal (RFP). Below is a key to the responses to the questions found on the following pages:

“No solicitation amendment necessary” – This response is given when 1) the requirements are clearly articulated in the solicitation to the maximum extent possible, 2) the information is not available or 3) dependent on offerors approach.

“Please see solicitation amendment” – This response is given is the response to the question affect a revision to the RFP. Please see the amended RFP posted to Federal Business Opportunities (FBO).

A response to the question is provided – A response/answer is given; a clarification is necessary, but the response does not require an amendment to the RFP.

CAPSTONE Page 2 of 8

1. Is this a follow-on requirement? If so, would it be possible for you to provide the incumbent information (we need the contract number and contractor)?

This is a new requirement.

2. I see that it shows “Set Aside: Total Small Business”. We are a small business (as defined by the NAICS Code) but we are not Woman Owned, Minority Owned or in located in a 8(a) HUBZone. Can we still participate in the bidding process?

Any small business under NAICS code 541690 is eligible to participate.

3. Can the Project Director be employed by a subcontractor?

No solicitation amendment necessary.

4. If the Project Director can be employed by the subcontractor, to what extent would the offeror's proposal be less competitive?

See response to question 3.

5. Will AHRQ consider electronic delivery of the proposals?

No Solicitation amendment necessary.

6. SOW pg. 17 Task 3: Recruitment and Configuration - Please clarify what is meant by at least 10 sites from each of the following: Primary Care Registry, Specialty Registry and Integrated Health System. Does a site mean a practice TIN or a practice site location? Is it 10 sites per each registry type (primary care and specialty) and do they also need to be from the same integrated health system or could they be across different systems.

Please see solicitation amendment.

7. SOW pg. 18 Task 2: Convene a Stakeholder Panel - Please clarify how many stakeholder panel meetings will be conducted over the Phase II period of performance (20 months)? Is it 6 or 7?

Six.

8. SOW pg. 16, 18 Phase I: Task 2: Convene a Stakeholder Panel and Phase II: Task 2:

Convene Stakeholder Panel - Is the Stakeholder Panel in Phase I the in Phase II? Or is a new Stakeholder Panel convened in Phase II?

9. SOW pg. 19 Task 4: Develop products to facilitate the adoption of the AHRQ core set of depression outcome measures in EHRs and Task 7: Final Report and Artifacts - If the contractor proposes a previously developed and copyrighted software solution as the platform, will the software solution and any enhancements/configurations as a result of this project be considered a product and/or tool under this contract and thus reusable as an artifact under public domain/open source to other clinical sites and registries after the expiration of the contract’s period of performance?

10. SOW pg. 19 Task 5.2: Patient Reported Outcome Measures - Is it required that we create a point of entry for patient reported data outside of a visit to their provider?

CAPSTONE Page 3 of 8

11. Section L pg. 82 - Will AHRQ consider removing the Cover Page and Table of Contents from the page limit?

12. Section L pg. 82 - Can you please clarify what is required for the “Mandatory Qualification” and the instructions for that item?

13. Section L pg. 82 - Can you please clarify what is required for the “Bibliography” and the instructions for that item?

14. Section L.7 includes a list of the sections of the Technical Proposal that are restricted by the 35-page limit. This list includes items "Mandatory Qualification" and "Bibliography" though no other references to these were made in the body of the RFP. Could the Government clarify whether we need to include these items in our proposal?

Please see solicitation amendment

15. Section L pg. 84 - What are the data elements that you don’t want a contractor to use when de-identifying data?

16. Section L pg. 84 - If we are required to get letters of commitment from the registries and a health system, is the expectation that the offeror will enter into a subcontract with the registries and a health system to compensate them for their work? That being said, if a subcontract is required does that mean that the registries and health system have to be a small business? If that’s the case that would limit the pool of registries and health systems that would be able to participate on this project.

No solicitation amendment necessary. Also refer to the Limitation on Subcontracting.

17. Section L.6 calls for a 12-point Times New Roman font. Would this restriction apply to technical diagrams?

18. Section C, Phase I, Task 3, Recruitment and Configuration. Please describe and provide examples of an "integrated healthcare delivery system."

See AHRQ Compendium of Health Systems.

(https://www.ahrq.gov/chsp/compendium/index.html) and ACTION network (https://www.ahrq.gov/research/findings/factsheets/translating/action3/actionIIIpar tners.html) for examples.

19. Section C, Phase I, Task 3, Recruitment and Configuration. Would the Government provide a list of the sites and contacts for each of the categories available to the Contractor?

20. Section C. What would be the target registry/registries for the scoped data?

https://www.ahrq.gov/chsp/compendium/index.html https://www.ahrq.gov/research/findings/factsheets/translating/action3/actionIIIpartners.html https://www.ahrq.gov/research/findings/factsheets/translating/action3/actionIIIpartners.html

CAPSTONE Page 4 of 8

21. Section C, Phase II. What sort of environments, technologies, and tools would be made available for Phase II of the project?

None.

22. Section C, Phase II. Are there specific and existing technologies (e.g., integration engine, replication engine, portals, etc.) that are intended for Phase II and/or would the Government consider other technology solutions?

23. Section L.7. Are any harmonized outcome measure "values" captured by PsychPro or PRIME today?

24. Can you share a context diagram covering data sources, integration, and target systems?

25. Is it the Contractor's responsibility to identify the source registries, or would the Government list these registries?

See response to question 19.

26. Section H.11 refers to the NIH Grants guide. Would the Government be awarding a grant or a contract to the winning bidder?

This is a contract award.

27. Section B.5 indicates that information is To Be Determined. Will this information be provided before the proposal due date?

This information will be provided upon contract award.

28. Likewise, Section G.1 indicates information about Key Personnel is To Be Determined. Will this information be provided in advance of the proposal due date?

See response to question 27.

29. Which elements of the solution would require 508 compliance?

Please refer to Section F.2.

30. Section L.9, A. If a Contractor chooses Option 2, would the Government consider alternatives to the requirement for personal data, such as payroll?

31. Given the U.S. holiday calendar, would the Government consider extending the proposal due date?

32. In Attachments F and G itemization of Consultant Labor Costs (Hours and $) are requested to substantiate compliance to RFP reqts. B.4.a.9 indicates Consulting fees are unallowable.

Should this be read to mean Consulting Costs itemized and supported in the Business Proposal including Attachments F and G will be authorized/allowable?

Consultant labor costs may be allowable as direct costs under this contract once reviewed and authorized by the Contracting Officer.

CAPSTONE Page 5 of 8

33. Section L.9 In several places the RFP mentions the place of performance (Government site). This implies there is a Govt. site where the assigned staff will work. Please clarify. It seems the nature of the work will involve some travel and interaction with Healthcare delivery sites and organizations-is there a USG site designated?

34. Section L-9 Both Option 1 and 2 in L.9.A include a requirement to provide certified cost or pricing data. This is statutory (TINA). Per the applicable regulations, certified C&P data should not be required in a competitive solicitation (reference FAR 15.403-1 (b) (1). Will it be acceptable to state that an Attachment D certification can be provided at time of award, if adequate price competition is not present?

The Government intends to award without negotiations therefore it is requiring Offerors to provide certified cost and pricing data with original proposal submission.

35. Section L.6(b) Please clarify that each graphic must have 1.5 line spacing between images within a graphic.

No solicitation amendment necessary.

36. L.6(b) L.7 Please clarify that the 508 Template and Resumes need to confirm to the font, margin and line spacing requirements.

The appendices do not have to follow the technical proposal font, margin and line spacing requirements.

37. L.6.I to L.6.III Please confirm that there no additional requirements for the hardcopy submission, i.e. Tabs, additional CD markings, Spine labeling, binding requirements, etc.

No solicitation amendment necessary.

38. Pg 15. The proposal describes a requirement to develop data governance and data use agreement templates for secondary use. Is the expectation that we develop new data use agreement templates even if there may be agreements already in place with some of the registries? Can we reuse existing agreements or create riders where appropriate? Finally, is there a requirement that data users or registries or secondary users will sign agreements within the period of performance?

39. Phase II: Task 3 Pg 18 "The contractor shall obtain final IRB approval for data collection and exchange and submit documentation of IRB approval at all sites to the COR". It is unclear what exactly IRB approval is needed here for-- this for the "pilot analysis" developed by the panel? Is the contract awardee expected to act as a PI and steward a research project that would go through approval? Is there a requirement to complete this study and if so, is it within the option year? What if a participating site does not have a formal IRB? Finally, is there a requirement that 10 sites are selected that will be able to complete IRB approval within the short time window of the option year? This requirement may lead to the need to recruit far more than 10 sites to get those which can complete approval within the option year window and limit the kind of participating sites.

The contractor must follow all procedures at the study sites and registries for IRB review and approval for participation of patients according to the protocol for the project.

CAPSTONE Page 6 of 8

40. Section L.9 Please clarify the files and formats to be included on the CD, i.e. MS Word.

PDF, etc.

No solicitation amendment necessary.

41. Section L.7.B On page 84 (Technical Proposal/Technical Approach) the RFP states that “ANY DATA collected, stored, processed, or transmitted on any information system associated with this contract ARE DEIDENTIFIED.” How should this requirement be interpreted in light of the need to collect PRO data directly from patients (implying knowledge of their email, etc.) and the need to link the PRO data to the clinical data? – Clarify that they mean identified data should not be shared down stream.

See response to question 15.

42. Phase II: Subtask. 5.2 pg 19 - The current SOW describes a process obtaining data primarily from EHR systems. Many institutions collect PRO data in ancillary systems that are not formal EHRs. We presume that AHRQ means HIT systems broadly that interact with the clinical HIT system when they describe "EHR data". Is this assumption correct and for this project is any data available relevant to the measures within scope regardless of the system of origin?

Yes.

43. Phase II: Task 5 pg 19 - Page 19 states that the “contractor shall maintain engagement with at least 10 clinical sites from each of the two registries.” Can AHRQ clarify whether the sites must already be submitting data to those registries or whether these can be new sites that we onboard to registry reporting? Would a de novo registry qualify as one of the two registries if it fits into the PCP or specialty registry categories? Could AHRQ make available a list of preferred or high-quality candidate registries? Should registries selected for this project also be a QCDR? If there are any limitations on what kind of facility can participate as a qualifying site, such as the amount or duration of data submitted, the kind of clinical setting, etc, please provide them.

See response to questions 19 and 6.

44. H.9.1.D. On page 40 the RFP states that the system is expected to contain PII. Can AHRQ clarify why Overall Risk is rated as Low if PII is involved and data is about psychiatric conditions? Also, please check for consistency with page 84 which seems to state PII can’t be included.

45. B.4.A. On page 6, “Information Technology hardware or software” are listed as disallowed under direct costs. This seems to exclude work on IT tools and systems that are required under the scope of the contract. Would labor to develop software solutions under the tool-building phase be allowed? Also, please confirm that other IT costs, such as hosting or licensing, that support the tools used to achieve the objective under this project, are allowed.

IT hardware and software purchases do not include labor or licensing or hosting fees (if required).

46. Phase II: Subtask 5.2 pg. 19 - Phase 2, Subtask 5.2: Is AHRQ open to existing commercial products for use as part of a site's data capture and extraction strategy? Would the proposal be more attractive if part of the deliverables are (1) open source; (2) open core w transferability; (3) SMART on FHIR app made available to AHRQ and the community for no cost?

CAPSTONE Page 7 of 8

See response to questions 9 and 38.

47. Phase II: Task 3 pg. 17 - Recruiting sites for implementation projects is often challenging.

Does AHRQ allow project funding to support services or payments for costs associated with acting as a participating site? Is there funding specifically for this purpose or is the awardee expected to find 10 sites within less than 2 years who can self-fund their participation?

See response to question 16.

48. The RFP states on page 84, "The methods described in the protocol should ensure that ANY DATA collected, stored, processed or transmitted on any information system associated with this contract ARE DEIDENTIFIED." Could you please define 'associated with'? The need to return information to the clinician within the EMR workflow implies the need for Personally Identifiable Information (PII) to populate the EMR with the specific information, but is such an 'information system' considered 'associated' with this contract? The RFP also indicates on page 40 that the contract will involve PII.

49. Does the answer to Question 48 impact any security requirements?

50. The RFP requires interaction and integration among many different systems (registries, EHRs, PRO systems, etc.). Do the security requirements included in this contract apply to these existing systems? Note, it is not feasible for existing registries and EHRs to become compliant with the security requirements listed here.

51. The RFP refers to 'government information' in several places. Could you please define 'government information' in the context of this RFP? Are data collected for the registries participating in this project considered 'government information'? This relates to the security questions above.

52. Under subtask 5.3 (page 20), the RFP states, 'If such natural language processing and machine learning models are used for processing textual information, the contractor shall validate the models for performance by sampling patient records and having physician expert review compared to the output of the algorithms.' Is nurse expert review acceptable?

53. There are several references in the RFP to open source software and commercial off the shelf software. Would the use of an open source product, such as the FDA's MyStudies app, not be suitable for this project?

No solicitation amendment needed.

54. With respect to page 43, what components of this project, if any, would require a valid Authority to Operate (ATO)?

CAPSTONE Page 8 of 8

55. Are all of the reporting requirements for security incidents included in the RFP applicable for the level of data and type of system envisioned for this project?

56. Some security documents are not available via the links included in the RFP (e.g., the HHS Information Technology General Rules of Behavior on page 43, the HHS Security Data Warehouse Escalation Memorandum, July 2013, on page 44). Could these be provided?

57. What is the 'mandatory qualification' item included in the Page Limit Summary table on page 82?

See response to question 12.

58. Page 87 refers to 'major subcontractors.' What constitutes a 'major' subcontractor?

No solicitation amendment necessary. There is no set definition for major a subcontract.

59. Which registries are you collecting?

60. What type of data are you interested in collecting?

61. Will actual onsite agendas need to be addressed? (Objective 1, page 15)

62. Is the end goal to have data (structured and unstructured) to automatically/directly transfer between EHR and registries?

63. Any requirements, constraints, restraints with the development of the tool?

a. Web-based?

b. Programming language?

c. Etc.

No.

64. From our experience over the past 26 years and more recently, structured data still has a few issues with accurate transfer between systems. More so, unstructured data has yet to be accurately and efficiently transferred between systems. Is or will there be a part in Phase 2 that allow for manual validation of data across systems?

See response to question 62.

65. Will Service Disabled Veteran Owned Small Business be allowed to bid on this?

See response to question 2.

66. Can a registry team with multiple Prime contractors?

The solicitation does not preclude a registry from teaming with multiple offerors for the purpose of proposal submission.