AHRQ-19-10003_Amendment_1.pdf

Amendment #1 - AHRQ-19-10003 CAPSTONE

Attachment #1, Amended RFP Modification Summary

Changes are listed below and are highlighted in the attached amended RFP.

Item

No.

Part or Section Description Page No.

RFP

1. Section A – Solicitation Form Changed Proposal due date to

January 9, 2019

2. Section C – Statement of Work Corrected typo 15

3.

Section C – Statement of Work – Overall

Objectives

Added additional information to overall objectives and goals

4.

Section C – Statement of Work – Overall

Objectives

Added information pertaining to use of publically available or open source tools and confirmation of intent to use in proposal submission

5. Section C - Overall Objectives Context diagram added 16

6.

Section C – Phase I: Task 3 Recruitment and Configuration

Clarified the requirement of sites for the Recruitment and Configuration

Task

7.

Section C – Phase I: Task 3 Recruitment and Configuration Clarified site locations 18

8.

Section C – Phase II: Task 2: Convene

Stakeholder Panel

Additional information was added to clarify the stakeholder panel requirement

9.

Section C – Phase II: Subtask 5.3 Use of

Natural Language Processing for

Unstructured Text Data

Requirement was updated to remove “physician” and add

“clinician”

10.

Section H.9 – Information Security and

Privacy Requirements – Rules of

Behavior

Section 3 was revised to remove link and add text that the HHS

Information Technology General

Rules of Behavior will be provided after contract award.

11.

Section H.9 – Information Security and

Privacy Requirements – Rules of

Behavior

Section 4 was revised to remove the link and add text that the HHS

Security Data Warehouse

Escalation Memorandum will be provided upon award

Section H.9 – Information Security and

Privacy Requirements – Rules of

Behavior

Section 10 was revised to remove the link add text that the HHS/AHRQ

Contractor Employee Separation

Form will be provided after contract award

12.

Section H.9 – Information Security and

Privacy Requirements – Rules of

Behavior

Section 11 was revised to add text that the HHS Contract Closeout

Guide (2012) will be provided upon award.

13 Section L.7 Page Limit Summary Table Removed Mandatory Qualifications and Bibliography from the table

14. Section L.7 Technical Approach Updated to clarify requirement on de-identifying data

15.

Section L.9 Business Proposal

Instructions

Removed “Government Site” form the Option 1 and Option 2

16. Attachment C

Removed language pertaining to discounts offered in consideration of onsite (Government Site) place of performance from template

Reminder: Please pay particular attention to Section L.9 Business Proposal Instructions. It is important for offerors to strictly follow these instructions in their business proposal volume.

Request for Proposal v. 9/5/18; FAC 100 AHRQ-19-10003 Amd#1 CAPSTONE Page 1 of 104

Request for Proposal No. AHRQ-19-10003 Amendment #1

PART I - THE SCHEDULE

SECTION A - SOLICITATION FORM

Date Issued: November 15, 2018 Amendment #1 Issued: December 14, 2018 Date Questions Due: November 29, 2018 02:00 PM, EST Please Note: All questions and inquires will be processed on a non-attribution basis.

Date Notice of Intent Due: December 17, 2018 Date Proposals Due: January 9, 2019 11:00 AM, EST

You are invited to submit a proposal to the Agency for Healthcare Research and Quality (AHRQ) for Request for Proposal (RFP) No. AHRQ-19-10003 entitled “CAPSTONE - Outcome Measure Harmonization and Data Infrastructure for Patient Centered Outcomes Research in Depression”. Your proposal must be developed and submitted in accordance with the requirements and instructions of this RFP.

Please Note: This RFP is a total small business set aside. See FAR 52.219-6 and 52.219-14, herein.

Solicitation Information: Please review the table of contents and refer to the appropriate section of this RFP for requirements, proposal instructions and evaluation information. The Government intends to award this contract without negotiations, so it is imperative that all questions are addressed during the solicitation phase, and that all proposals submitted include the required information and accept the terms and conditions as stated. Failure to comply with proposal instructions or submit required information may result in the removal of a proposal from consideration for award. Please review Section L carefully.

Proposal Intent: If you intend to submit a proposal in response to this solicitation, please inform the Contracting Officer of your intent by completing the Proposal Intent Response Form (See section L.11 and Attachment A) and send it to the Contracting Officer no later than December 17, 2018. You may mail it to the address below; fax it to 301-427-1740, Attention: Tara Bertolini, Contract Specialist/Contracting Officer or email it to tara.bertolini@ahrq.hhs.gov.

Questions: All Questions regarding this solicitation shall be received in this office no later than the date shown above. All questions shall be submitted electronically by e-mail to Tara Bertolini, Contract Specialist/Contracting Officer, at the following email address:

tara.bertolini@ahrq.hhs.gov. Subject line shall read: Proposal Questions RFP No. AHRQ-19- 10003. Answers to questions and all solicitation amendments will be posted to the Federal

Business Opportunities web page: www.fbo.gov. It is your responsibility to monitor the web site.

CAPSTONE Page 2 of 104

Additional Information:

a. A time and materials contract is contemplated for a four (4) month base period and one (1), twenty (20) month option period.

b. The Government anticipates awarding one contract from this solicitation on or about May 15, 2019.

c. Proposals submitted in response to this solicitation must state that they are valid for a period of at least 120 days (reference FAR clause 52.215-1 in Section L of this document.)

d. The NAICS code for this requirement is 541690 with a size standard of $15M.

Proposal Submission: The proposal shall be signed by an authorized official to bind your organization and must be received by the due date specified above. The proposal must be submitted to the following address:

Agency for Healthcare Research and Quality (AHRQ)* Division of Contracts Management* 5600 Fishers Lane* Mail Stop 06N34B* Rockville, Maryland 20857* Attn: Tara Bertolini, (301) 427-1705*

*All boxes must be clearly labeled with this information.

Please read this section carefully. Some mail and package delivery procedures have changed due to security controls. All mail and packages are screened at a local offsite facility before being delivered to 5600 Fishers Lane.

The U.S. Postal Service’s “Express Mail” does not deliver to our Rockville, Maryland address.

Please do not use this means of delivery for proposal submission.

Proposals delivered by commercial package delivery company (example: Fedex, UPS, DHL, etc.):

Submit and label as instructed above. These companies are familiar with the delivery procedures at 5600 Fishers Lane.

Hand carried or hand carry courier service: The 5600 Fishers Lane building security does not allow packages to be delivered directly to the building. Hand carried or courier delivered proposals should be labeled as instructed above, however they must be delivered directly to the offsite screening facility at 12225 Wilkins Avenue Rockville MD 20852. The operating hours at the Wilkins Avenue facility are 7:00 AM to 3:30 PM Monday through Friday. Please note that anyone who arrives at the Fishers Lane location with a delivery will be directed to the screening facility, however this will cause delay. Packages will be accepted and date/time stamped at the screening facility. Please allow ample time as proposals cannot be date/time stamped until they have gone through security.

CAPSTONE Page 3 of 104

System for Award Management: In accordance with Federal Acquisition Regulation, all contractors must be registered in the System for Award Management (SAM) database in order to conduct business with the government [See Section I - FAR clause 52.204-7 System for Award Management]. Access the System for Award Management (SAM) at http://www.sam.gov.

Reminder: The Procurement Integrity Act prohibits the unauthorized disclosure of, inquiry about, or acceptance of procurement information. Communications with any other individual outside the AHRQ contracting office may result in rejection of an offeror’s proposal. All requests for any information concerning this RFP should be referred to Tara Bertolini, (301) 427-1705 or e-mail:

tara.bertolini@ahrq.hhs.gov. Please note e-mail requests should state subject as RFP AHRQ-19- 10003.

CAPSTONE Page 4 of 104

Table of Contents

PART I ‐ THE SCHEDULE

SECTION A ‐ SOLICITATION FORM

SECTION B ‐ SUPPLIES OR SERVICES AND PRICES/COSTS

SECTION C ‐ DESCRIPTION/SPECIFICATION/WORK STATEMENT

SECTION D ‐ PACKAGING AND MARKING

SECTION E ‐ INSPECTION AND ACCEPTANCE

SECTION F ‐ DELIVERIES OR PERFORMANCE

SECTION G ‐ CONTRACT ADMINISTRATION DATA

SECTION H ‐ SPECIAL CONTRACT REQUIREMENTS

PART II – CONTRACT CLAUSES

SECTION I CONTRACT CLAUSES

PART III‐ LIST OF DOCUMENTS, EXHIBITS AND ATTACHMENTS

SECTION J – SOLICITATION LIST OF ATTACHMENTS

PART IV– REPRESENTATIONS AND INSTRUCTIONS

SECTION K ‐ REPRESENTATIONS, CERTIFICATIONS, AND OTHER STATEMENTS OF OFFERORS

SECTION L ‐ INSTRUCTIONS, CONDITIONS AND NOTICES TO OFFERORS

SECTION M‐ EVALUATION FACTORS FOR AWARD

CAPSTONE Page 5 of 104

SECTION B - SUPPLIES OR SERVICES AND PRICES/COSTS

B.1 BRIEF DESCRIPTION OF SUPPLIES OR SERVICES

“CAPSTONE - Outcome Measure Harmonization and Data Infrastructure for Patient Centered Outcomes Research in Depression”. See Section C for a complete description.

B.2 TIME & MATERIALS

All work shall be performed in accordance with the Statement of Work. No hours shall be billed for labor categories or labor rates not included as part of this contract as detailed below:

Labor Category Hourly Rate

Type of Material Not to Exceed Amount

TOTAL CEILING PRICE BASE PERIOD/Phase I (Labor and Materials) $

B.3 OPTION PRICES

If the Government exercises its option(s) pursuant to the option article in Section H of this contract, the following applies:

Option Period 1/Phase II

Labor Category Hourly Rate

CAPSTONE Page 6 of 104

Type of Material Not to Exceed Amount

TOTAL CEILING PRICE OPTION PERIOD 1/Phase II (Labor and Materials) $

B.4 PROVISIONS APPLICABLE TO DIRECT COSTS

a. Items Unallowable Unless Otherwise Provided

Notwithstanding the clauses, ALLOWABLE COST AND PAYMENT incorporated into this contract, unless authorized in writing by the Contracting Officer, the costs of the following items or activities shall be unallowable as direct costs:

(1) Acquisition, by purchase or lease, of any interest in real property;

(2) Rearrangement or alteration of facilities;

(3) Purchase or lease of any item of general purpose-office furniture or office equipment regardless of dollar value. (General purpose equipment is defined as any items of personal property which are usable for purposes other than research, such as office equipment and furnishings, pocket calculators, etc.);

(4) Accountable Government property;

(5) Travel to attend general scientific meetings;

(6) Foreign Travel;

(7) Any costs incurred prior to the contract's effective date;

(8) Rental of meeting rooms not otherwise expressly paid for by the contract;

(9) Any formal subcontract arrangements not otherwise expressly provided for in the contract;

(10) Consultant fees;

(11) Information Technology hardware or software;

(12) Food and Beverages; and

CAPSTONE Page 7 of 104

Note: Food and Beverages are unallowable costs. Contracting Officer Authorization for Food and Beverages will be only made as specifically authorized by statute, which is extremely rare. The contractor shall not provide food or beverages of any type. The contractor shall not accept food or beverages, even if included in, or provided ‘free’ as part of a meeting/conference ‘package’.

b. Travel Costs

This contract is subject to the provisions of Public Law (P.L.) 99-234 which amends the

Office of Federal Procurement Policy Act to provide that contractor costs for travel, including lodging, other subsistence, and incidental expenses, shall be allowable only to the extent that they do not exceed the amount allowed for Federal employees. The Contractor, therefore, shall invoice and be reimbursed for all travel costs in accordance with Federal Acquisition Regulations (FAR) 31.205-46.

The following applies to Educational Institutions: The Contractor shall invoice and be reimbursed for all travel costs in accordance with OMB Circular A-21 - "Cost Principles for Educational Institutions."

No funding or expenditures will be authorized under this contract for travel or meeting expenses of any person that is not an employee of the contractor or approved subcontractor in accordance with the terms of the contract. See 31 USC § 1345 for appropriations restrictions.

B.5 CONTRACTING OFFICER AUTHORIZATION

Other provisions of this contract notwithstanding, approval of the following items within the limits set forth is hereby granted without further authorization from the Contracting Officer.

To Be Determined

CAPSTONE Page 8 of 104

SECTION C - DESCRIPTION/SPECIFICATION/WORK STATEMENT

Independently and not as an agent of the Government, the Contractor shall be required to furnish all the necessary services, qualified personnel, material, equipment, and facilities, not otherwise provided by the Government, as needed to perform the Statement of Work as follows:

Statement of Work

Background As of April 2018, the AHRQ Registry of Patient Registries (RoPR) (see https://patientregistry.ahrq.gov/) contains records for almost 4000 registries for many different reasons including research, quality improvement, public reporting, and post market surveillance.

These registries represent a huge investment in data infrastructure and a potential wealth of data that could be reused for patient centered outcomes research. However, the lack of standardization of definitions in the registries makes analysis across registries impossible.

To address this issue, AHRQ has developed a core set of harmonized definitions of patient centered outcome measures in five different clinical areas (atrial fibrillation, asthma, depression, lung cancer and lumbar spondylolisthesis) in an existing Patient Centered Outcomes Research Trust Fund (PCORTF) funded project (see https://effectivehealthcare.ahrq.gov/topics/registry-of-patient-registries/outcome-measures-framework) . Despite the recognized value of these harmonized core outcome measures, stakeholders are reluctant to put resources into implementing them because of three main perceived barriers:

1. Burden on patients and clinicians to collect patient reported outcomes from patients;

2. Disruptions to clinical care when collecting structured data from clinicians or challenges in extracting high quality information from unstructured data already in patient records;

and

3. Difficulty working with electronic health records.

This project will address the barriers cited by registries and demonstrate feasibility and value by incorporating data collection in a way that fits into the workflow of busy practices and facilitates communication between specialists and primary care. This project will demonstrate how incorporation of harmonized outcome measures to clinical systems could improve clinical care, and improve compliance and sustainability for registry and clinical system developers. For example, this project will allow streamlined reporting of practices to CMS Qualified Clinical Data Registries (QCDRs) which will lead to less burden and higher payments for the practices under the Medicare Incentive Payment System (MIPS) and thus be more sustainable. Further, by collecting higher quality data into registries, the registries may be able to become platforms for pre and post market studies of medical products (for example, by enabling nesting of clinical trials in registry infrastructure) leading to revenue for the clinical sites while simultaneously lowering the cost of clinical research (see https://www.ctti-clinicaltrials.org/projects/registry-trials).

This project will specifically address data infrastructure for treatment of depression. Depression is a common condition managed by both primary care and specialty care providers.

Antidepressants are one of the most frequently prescribed drug classes during physician office visits (https://www.cdc.gov/nchs/fastats/drug-use-therapeutic.htm) yet depression remains

CAPSTONE Page 9 of 104 poorly recognized and inadequately treated in clinical care in the United States (https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2546155). Second line therapies include antipsychotics and implantable medical devices, but there is little evidence on the effectiveness of these treatments and which patients would benefit from these therapies.

Clinical care has important uncertainties that could be addressed by data in registries and patient care encounters.

The final list of harmonized outcome measures for depression are:

OMF

CATEGORY

OUTCOME

MEASURE

DEFINITION

MEASUREMENT

NOTES

Survival All-cause mortality

Death from any cause, reported in 12-month intervals.

Survival Death from suicide

Patient with a diagnosis of major depression or dysthymia who died from suicide, reported in 12-month intervals.

This should be captured where feasible; however, it should be noted that this information may not be recorded accurately or available to all providers.

CAPSTONE Page 10 of 104

OUTCOME

MEASURE

DEFINITION

MEASUREMENT

NOTES

Clinical Response

Improvement in Depressive Symptoms – Remission

Patient age 18 or older with a diagnosis of major depression or dysthymia and an initial PHQ-9* score > 9 who demonstrates remission defined as a PHQ-9 score less than 5.

*The PHQ-9 or another brief, publicly available, validated patient-reported instrument with empirically derived cutpoints equivalent to the PHQ-9 cutpoints for remission and response and for which an evidence-based crosswalk to the PHQ-9 exists should be used to measure clinical response. Other measures may be used in addition for research or other purposes.

Timeframe for measurement:

6 months (+/- 60 days)

12 months (+/- 60 days)

In some implementations, it would beneficial to capture earlier responses and remissions and to obtain higher degrees of follow-up.

Additional measurements outside of the windows listed above are recommended as supplemental measures.

CAPSTONE Page 11 of 104

OUTCOME

MEASURE

DEFINITION

MEASUREMENT

NOTES

Clinical Response

Improvement in Depressive

Response dysthymia and an initial PHQ-9* score > 9 who demonstrates a response to treatment defined as a PHQ-9 score that is reduced by 50% or greater from the initial PHQ-9 score.

reported instrument with empirically derived cutpoints equivalent to the PHQ-9 cutpoints for remission and response and for which an evidence-based crosswalk to the PHQ-9 exists should be used to measure clinical response. Other responses and remissions and to obtain higher degrees of follow-up.

Additional measurements outside of the windows listed above are

CAPSTONE Page 12 of 104

Clinical Response

Worsening in Depressive

Recurrence dysthymia and an initial PHQ-9* > 9 who demonstrates remission (defined as a PHQ-9 score < 5) of at least two months’ duration and subsequently experiences a recurrence of a depressive episode, defined as a 50% increase in PHQ-9 score or defined as a PHQ-9 score > 9 OR hospitalization for depression or suicidality.** reported instrument with empirically derived cutpoints equivalent to the PHQ-9 cutpoints for remission and response and for which an evidence-based crosswalk to the PHQ-9 exists should be used to measure clinical response. Other

**This definition was proposed by the workgroup. Data accruing from ongoing registries are needed to assess the feasibility of using this definition to capture recurrence.

responses and remissions and to obtain higher degrees of follow-up.

Additional measurements outside of the windows listed above are

Events of Interest

Adverse Events Depression treatment-related adverse events. Use of a brief, CAPSTONE Page 13 of 104

OUTCOME

MEASURE

DEFINITION

MEASUREMENT

NOTES

publicly available, validated measurement tool to capture adverse events is recommended.

Reported in 12-month intervals.

Events of Interest

Suicide Ideation & Behavior

Selection of ‘several days’, ‘more than half the days’ or ‘nearly every day’ option on PHQ-9 item 9 (“Thoughts that you would be better off dead or of hurting yourself in some way”).

Supplemental assessments of suicide ideation and behavior should be completed for patients who screen positive for suicide ideation on the PHQ-9 or when a clinician has concerns about suicidality. Supplemental assessments should be completed using an appropriate, brief, validated instrument, such as the Concise Health Risk Tracking (CHRT) scale. Includes nonfatal suicide attempts/suicide attempt behaviors, planning/preparatory acts, active suicidal ideation.

Reported in 12-month intervals (in conjunction with the PHQ-9 suicide item).

Patient Reported

Depression-specific Quality of Life

Depression-specific quality of life should be measured using a brief, validated, publicly available instrument that is appropriate for the population of interest, such as the Quality of Life Enjoyment and Satisfaction Questionnaire (Q-LES- Q). Reported in 12-month intervals.

CAPSTONE Page 14 of 104

OUTCOME

MEASURE

DEFINITION

MEASUREMENT

NOTES

Resource Utilization

Depression-related Resource Utilization

All resource utilization (as measured by cost) related to treatment or management of depression, including medications, psychotherapy, office visits, urgent care center visits, ED visits, and hospitalizations. Reported in 12-month intervals.

Resource Utilization

Work Productivity

Work productivity loss (overall work impairment/absenteeism plus presenteeism), as measured by the Work Productivity and Activity Impairment Questionnaire (WPAI), reported in 12-month intervals.

The AHRQ Confidentiality Statute, 42 U.S.C. 299c-3(c) and (d), applies to the activities conducted under the contract. The statute protects both identifiable individuals and identifiable establishments and provides, in part, that data may not be used for any purpose other than the purpose for which it was supplied absent the consent of the individual or establishment supplying the data. Neither compliance with HIPAA nor IRB approval necessarily ensures that the requirements of the AHRQ Confidentiality Statute are met.

References:

AHRQ RoPR system: https://patientregistry.ahrq.gov/

Information about the RoPR system and Outcome Measure Repository including the library of harmonized outcome measures, design documents:

https://effectivehealthcare.ahrq.gov/topics/registry-of-patient-registries/abstract/

AHRQ Applications and Systems Development Requirements. Appendix 2A in AHRQ Publishing and Communications Guidelines. AHRQ Publication No. 08(-09)-M049, April 2009. Agency for Healthcare Research and Quality, Rockville, MD.

https://www.ahrq.gov/research/publications/pubcomguide/index.html

AHRQ Web Site Guidelines in AHRQ Publishing and Communications Guidelines. AHRQ Publication No. 08(-09)-M049, April 2009. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/research/publications/pubcomguide/pcguide2.htmlAHRQ Information Security and Privacy Policy. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/policy/electronic/privacy/infosecurity.html

CAPSTONE Page 15 of 104

Overall Objectives/Goals of the Contract

The overall objectives of this contract are to facilitate and advance the use of registries and registry research.

Specifically, the objectives of this contract are:

(1) Collect electronic health data from existing structured and unstructured information in Electronic Health Records (EHRs) and directly from patients at multiple clinical sites at fixed intervals for patient outcomes associated with treatment for depression

(2) Transfer the collected data to electronic patient registries

(3) Develop governance and data use agreement templates with the registries so that data may be made available to external researchers.

(4) Develop infrastructure and tools to allow replication for other clinical registries and clinical systems. This project will examine whether this approach to data collection and transfer enhances the ability to use registries for research on patient outcomes.

At the end of the contract, a prototype should demonstrate that data can be collected from patients and clinicians without disruptions in clinical workflow, transferred to registries, and provided back to clinicians for patient care and quality improvement. The tools developed in this contract should allow maturation of the prototype into an infrastructure that can support research with expansion to other clinical sites and sustainability with no additional infrastructure funding from the government. Researchers should be able to use the infrastructure to apply for research grants and carry out research projects.

The purpose of the contract is also to create and use reusable publicly available and/or open source tools, and governance and data use agreement templates that will remain publically available after the contract ends. If any proprietary or existing tools or templates are used or modified, the contractor must follow the instructions prescribed at FAR § 52.227-17, incorporated into the contract. In accordance with § 52.227-17(c)(2), the contractor must obtain the contracting officer’s prior written permission before incorporating data not first produced under this contract into data delivered under this contract. This requirement applies to the incorporation of any previously developed or proprietary software platform or data elements proposed to be used in fulfillment of this contract. Please note that in accordance with this provision the contractor is responsible to obtain a license of the specified scope for the benefit of the Government in order to support the free, public dissemination of these products.

Offerors must specify in their proposal if their technical approach includes incorporation of proprietary or copyrighted data (data not first produced under this contract) and confirm their intention to convey the appropriate licenses as required.

The anticipated outcomes of this project are:

Phase I:

Risk analysis of number and ability of EHRs to send/exchange data to the registries

CAPSTONE Page 16 of 104

Phase II:

1. Tools for clinicians and patients to facilitate integration of the harmonized depression outcome measures into electronic health records (EHRs) and registries so that these data will be available for clinical research, patient-centered outcomes research, quality improvement and implementation research.

2. Proof of concept for a standards-based approach (an approach that uses technical standards for data extraction/collection/exchange) for collecting and reporting patient outcomes information to clinicians within their workflow and simultaneously transmitting the data to registries to make it available for research.

3. Tools to make it easier for researchers and registry developers to integrate registries with clinical systems.

4. Governance and data use agreement templates that can be used by registries to make data available to external researchers.

AHRQ is only funding the feasibility of creating a connection between the registries and the EHRs. AHRQ will neither own the connection nor will it fund the ongoing operations and maintenance cost for the connection.

This figure illustrates how data might flow between all the systems:

CAPSTONE Page 17 of 104

Phase I: Risk Analysis (May 15, 2019 – Sept 23, 2019)

Task 1: Project management and Reports

Subtask 1.1: Kick-Off Call

At the beginning of the project, the contractor shall participate in an extended initial conference call with the COR and appropriate AHRQ staff to discuss the overall objectives and specific elements of work to be performed. During this call, any changes that are needed to the protocol and timeline that was submitted with the proposal shall be discussed.

Contact information for participants and signed conflict of interest statement for any new project team members who have been added since the original proposal was submitted to AHRQ will be provided from the contractor to the COR. The contractor shall schedule the call within 3 days of the Effective Date of the Contract (EDOC). Within 3 days of the call, the contractor shall send by email to the COR a summary of the conference call.

Deliverable: Call summary and a signed conflict of interest statement for any new members of the project team members who have been added since the original proposal was submitted to AHRQ.

Subtask 1.2: Conference Calls / Status Reports

Beginning one month after the Kick-Off Call, the contractor shall participate in monthly conference calls with the COR and other appropriate AHRQ staff to discuss the report and progress to completing the work.

Within 3 days of the call, the contractor shall send by email to the COR a summary of the conference call along with a concise progress report. The progress report shall document progress in completing tasks and any problems encountered and how they were addressed, and any foreseeable problems.

Deliverables: Call summaries and progress reports

Task 2: Convene a Stakeholder Panel

The contractor shall in consultation with AHRQ and ASPE, organize a stakeholder panel of no more than 20 members representing consumers of outcomes information, such as patient organizations, provider organizations, payers, quality measurement groups, life sciences organizations, research funders, and federal agencies. The contractor shall prepare the stakeholder panel roster and meetings agendas and other meeting materials at least one week before the meetings and submit to the COR. The panel will provide initial feedback on the protocol through a web-based meeting. This initial feedback will include how the project can contribute to overcoming barriers for clinical sites (especially primary care) to participate in data collection infrastructure that can be used for multiple purposes, including quality improvement and research. The feedback will also include how to incorporate elements into the project that will enhance sustainability and adoption of lessons learned at additional sites after the end of

CAPSTONE Page 18 of 104 this project to build data capacity. The contractor shall incorporate suggestions from project partners, AHRQ and the stakeholder panel review into the modified protocol (task 3). The contractor shall prepare meeting notes and submit to the COR no later than one week after the stakeholder meetings.

Deliverables:

Stakeholder panel roster, meeting agendas and other meeting materials, meeting notes.

Task 3: Recruitment and Configuration

The contractor shall recruit

1) 10 sites associated with an integrated healthcare delivery system

2) 10 sites already participating with a primary care focused registry (e.g., PRIME Registry https://primeregistry.org/ or other registry) which are independent practices or affiliated with different delivery systems.

2) 10 sites already participating with a specialty-focused registry (e.g., APA registry https://www.psychiatry.org/psychiatrists/registry or other registry) which are independent practices or affiliated with different delivery systems.

The sites should be separate practice locations.

The contractor shall prepare all documents necessary for Institutional Review Boards (IRB) approval and submit to the COR and the IRBs as required for each site. The contractor shall also begin to prepare information needed for documents for OMB Paperwork Reduction Act clearance.

The contractor shall evaluate the registries against the requirements for registry-readiness to receive/exchange data that was submitted with the proposal and modify the protocol to meet these requirements. The contractor shall also evaluate the specific EHRs at each clinical site for readiness to send/exchange data to the registries. The contractor shall submit the modified protocol to the COR together with a risk analysis of the whether the objectives of the contract can be met in Phase II.

Deliverables: IRB documentation, modified protocol and risk analysis.

Note: Dependent on findings of the risk analysis, AHRQ and ASPE will make determination on whether to proceed to Phase II. AHRQ and ASPE may consult experts at the Office of the National Coordinator of Health Information Technology (ONCHIT) to review the risk analysis to determine whether technical feasibility is appropriately assessed.

Phase II: Tool development. (Sept 24, 2019 to May 15, 2021)

Task 1: Project Management and Dissemination

Subtask 1.1: Project Management and Reports

The contractor shall participate in monthly conference calls with the COR and other appropriate AHRQ staff to discuss the report and progress to completing the work.

CAPSTONE Page 19 of 104

Within 3 days of the call, the contractor shall send by email to the COR, a summary of the conference call along with a concise progress report. The progress report shall document progress in completing tasks and any problems encountered and how they were addressed, any foreseeable problems.

Deliverables: Call summaries and progress reports

Subtask 1.2: Dissemination Plan

The contractor shall finalize the preliminary dissemination plan that was submitted with the contractor proposal and submit to the COR. The dissemination plan shall address the objectives, benefits, and activities of this project, and how the tools may be used by other registries and clinical sites. The plan should also include a list of professional meetings (for example, ISPOR, DIA, Health Datapalooza) where information about this project may be presented.

The contractor shall submit abstracts and present the project at at least five professional meetings over the period of performance.

Deliverables: Final dissemination plan

Subtask 1.3:

The contractor shall work with AHRQ to obtain OMB approval for data collection under the Paperwork Reduction Act (PRA). The contractor shall submit all required documents and responses to OMB questions to the COR and other AHRQ personnel as needed.

Deliverables: OMB PRA documents.

Task 2: Convene Stakeholder Panel

The contractor shall update the stakeholder panel which was formed in Phase I on project progress four times per year through web based meetings. The contractor shall prepare the stakeholder panel roster and meetings agendas and other meeting materials at least one week before the meetings and submit to the COR. The contractor shall prepare meeting notes and submit to the COR no later than one week after the stakeholder meetings. In the final quarter of the contract, the stakeholder panel will review the final reports and outputs of the project, prioritize research questions on depression care that can be answered using data from harmonized depression outcome measures, and provide feedback on the deliverables.

Deliverables:

Stakeholder panel roster, meeting agendas and other meeting materials, meeting notes.

Task 3: Configuring registries and Final IRB approval

The contractor shall work with the two registries to assure their clinical registries are configured to support the receipt, exchange and delivery of clinical and patient reported outcomes specific

CAPSTONE Page 20 of 104 to AHRQ’s harmonized depression measures (receipt from EHRs and a patient mobile application (task 5.2); exchange between the registries and delivery back to the clinical environment). The contractor shall test whether the two registries are ready for the next steps to using test data. The contractor shall develop and submit to the COR a report on registry configuration and challenge using test data.

The contractor shall obtain final IRB approval for data collection and exchange and submit documentation of IRB approval at all sites to the COR.

Deliverables: Report on registry configuration and challenge using test data Documentation of final IRB approval

Task 4: Develop products to facilitate the adoption of the AHRQ core set of depression outcome measures in EHRs

In order to facilitate the adoption of the AHRQ core set of depression outcome measures in EHRs, the contractor shall develop standardized implementation models, Fast Healthcare Interoperability Resources (FHIR, a standard for exchanging healthcare information electronically) profiles as appropriate, and an implementation guide following all of the procedures specified for the HL7 collaboration (https://www.hl7.org/fhir/implementationguide.html). The contractor shall deliver these documents to COR and work with Health Level 7 (HL7) to make the documents freely available on the HL7 website and provide information needed to implement the outcome measure definitions in EHRs.

Deliverables: Standardized implementation models, FHIR and/or similar profiles and implementation guide.

Task 5: Data extraction and exchange

The contractor shall maintain engagement with at least 10 clinical sites from each of two registries and an integrated health system and work with the registries on ongoing issues that may arise with registry configuration. The contractor shall pilot collection of the harmonized outcome measures as described in the modified protocol (task 3).

Subtask 5.1: Structured data collection

The contractor shall pilot collection of outcome measures linked to existing codes and value sets and extract the information from EHRs.

Subtask 5.2: Patient Reported Outcome Measures

The collection of the Patient Reported Outcomes (PROs) at fixed intervals will be particularly important to enable comparisons of outcome measures across provider types and care settings; currently, patients are seen and PROs collected at a wide variety of time points depending on their provider type, access to care/insurance status, and depression severity. Some of the sites have some capacity to collect PRO data at their sites when the patient is seen in an office visit. The contractor shall augment this capacity

CAPSTONE Page 21 of 104 through mobile/web technologies to achieve the collection at fixed intervals independent of visits (i.e. mobile, web). If possible, the contractor shall leverage the FY17 OS- PCORTF PRO project ‘Advancing the Collection and Use of Patient-Reported Outcomes (PROs) through Health Information Technology,’ outputs (in particular the specifications and standards for integration into the EHR and allow collection at fixed intervals independent of patient visit to the provider). In addition, the contractor shall develop a function for automated reminders to patients to fill out the forms at fixed time intervals.

The contractor shall conduct electronic data collection of patient-reported outcomes (including the Patient Health Questionnaire (PHQ)-9) at baseline and every six months for one year, and generate outcome measurements that are presented back to the clinicians through standards-based integration with EHRs. The contractor shall combine the electronic data collection from patients with the clinical data extracted from the EHR, and provide comparison reporting based on the OMH measures.

Subtask 5.3: Use of Natural Language Processing (NLP) for Unstructured Text Data

Much of the information in EHRs is collected as unstructured text data in “notes” sections.

Natural language processing is increasingly used to extract data from EHRs and several commercial and open source options for implementing NLP exist. Where direct collection of structured data from EHRs, clinicians and patients is not feasible, the contractor shall investigate and, where feasible, collect relevant unstructured data from the EHRs. The contractor shall process the data with existing tools using natural language processing and machine learning to identify specific data elements from the text information for specific characteristics or outcomes, such as adverse events and death by suicide, that could not be collected using the OMH standards directly. If such natural language processing and machine learning models are used for processing textual information, the contractor shall validate the models for performance by sampling patient records and having clinician expert review compared to the output of the algorithms.

Subtask 5.4: Data Quality

A premise of this work is that high quality data can be captured in the EHR and sent to registries for use in research. To show that the predecessor steps yield data of high quality, the contractor shall validate data quality using an AHRQ-approved method.

Examples of techniques include source system loop back verification, ongoing source-to-source verification, data-issue tracking, and sampling data from EHRs and data sent to registries comparing results of outcome measure computation.

Subtask 5.5: Reporting to registries and back to clinicians

The contractor shall collect and report outcomes information from and to clinicians within their workflow through their EHR. These outcomes data will be submitted to registries that use the same data elements. Providing outcomes information within the clinician workflow through the EHR will enable clinicians to use that information in clinical decision-making. The selection of the standards-based approach will be determined during the

CAPSTONE Page 22 of 104 contract period, with consideration of the EHRs used at the participating sites and whether those EHRs have fully implemented FHIR resources.

Task 6: Pilot Data Analysis and Prioritizing Topics for Patient-Centered Outcome Research

There are many patient-centered outcomes research questions that could be answered using real-world, robust data (i.e., EHR data sent to registries). The contractor together with the stakeholder panel shall design a pilot where the baseline registry data from the 2 selected registries, including those extracted from clinical records, can be obtained and used in analysis to answer a sample research question. This pilot analysis will serve as proof of concept that clinical data harmonized on depression outcome measures sent to registries can be robust enough to answer research questions.

The contractor shall work with the registries to produce governance and data use agreement templates. The contractor together with the stakeholder panel will create and prioritize a list of research topics related to depression care that can be answered through the use of depression outcome measures harmonized across data sources. The contractor shall prepare and submit to the COR a report detailing pilot data analysis and high priority topics for patient-centered outcome research on depression care using real world, robust data and the approach used to determine prioritization.

Deliverable: report detailing pilot data analysis and high priority topics for patient-centered outcome research on depression care using real world, robust data and the approach used to determine prioritization

Task 7: Final report and artifacts

The contractor shall develop a draft and final report including methods used for data extraction, submitting data to the registries and back to clinicians including barriers encountered (both technical and institutional) and solutions to those barriers and assessment of lessons learned applicable to implementation of harmonized outcome measures in other clinical areas. Specifically, the final report and artifacts shall include:

Tools for clinicians and patients to facilitate integration of the harmonized depression outcome measures into electronic health records (EHRs) and registries so that these data will be available for clinical research, patient-centered outcomes research, quality improvement and implementation research.

Proof of concept for a standards-based approach (an approach that uses technical standards for data extraction/collection/exchange) for collecting and reporting patient outcomes information to clinicians within their workflow and simultaneously transmitting the data to registries to make it available for research.

Tools to make it easier for researchers and registry developers to integrate registries with clinical systems.

Governance and data use agreement templates that can be used by registries to make data available to external researchers.

CAPSTONE Page 23 of 104

The draft report will be posted on the AHRQ website by AHRQ for public review and also reviewed by invited peer reviewers, project partners AHRQ and the stakeholder panel. The contractor shall prepare a final report and written disposition of comments received and submit to the COR.

The contractor shall also prepare all artifacts from the project that may be reused in other clinical sites and registries to AHRQ in a format that can be made accessible to all interested stakeholders.

All reports and artifacts shall be consistent with AHRQ guidelines (see references in background section) as applicable, including 508 compliance.

In addition, the contractor shall develop drafts of at least two manuscripts suitable for publishing in peer reviewed journals based on the project final report and deliver to the

COR.

Deliverables: Draft report, Final report and disposition of all comments, Drafts of two manuscripts, Project artifacts

CAPSTONE Page 24 of 104

SECTION D - PACKAGING AND MARKING

All deliverables required under this contract shall be packaged, marked and shipped in accordance with Government specifications. At a minimum, all deliverables shall be marked with the Contract Number, Contractor Name and Item Number. The Contractor shall guarantee that all required materials shall be delivered in immediate usable and acceptable condition. Note as appropriate on the face page of each deliverable, or on the binding, (1) one volume only, or (2) volume 1 of 2, etc.

CAPSTONE Page 25 of 104

SECTION E - INSPECTION AND ACCEPTANCE

E.1 INSPECTION AND ACCEPTANCE

a. The Contracting Officer or the duly authorized representative will perform inspection and acceptance of materials and services to be provided.

b. For the purpose of this SECTION the Contracting Officer’s Representative is the authorized technical representative of the Contracting Officer.

c. Inspection and acceptance will be performed at:

Agency for Healthcare Research and Quality 5600 Fishers Lane Rockville, Maryland 20857

E.2 CLAUSES INCORPORATED BY REFERENCE (FEB 1998)

This contract incorporates the following clause by reference, with the same force and effect as if it were given in full text.

52.246-6 Inspection Time-and Material and

Labor Hour (May 2001)

CAPSTONE Page 26 of 104

SECTION F - DELIVERIES or PERFORMANCE

F.1 PERIOD OF PERFORMANCE

a. The base period of performance (Phase I) of this contract shall be from May 15, 2019 through September 23, 2019.

b. If the Government exercises any Options pursuant to the OPTION PROVISION Article in Section H of this contract, the period of performance will be revised as follows:

Option Period Option Period Option Period 1/Phase II September 24, 2019 – May 15, 2021

F.2 DELIVERABLE SCHEDULE

The items specified for delivery below are subject to the review and approval of the Contracting Officer’s Representative (COR) before final acceptance. The contractor shall be required to make revisions deemed necessary by the COR. The deliverable schedule may only be changed by a contract modification signed by an AHRQ Contracting Officer. Days mean calendar days.

The contractor shall submit the following items electronically to the COR:

Task Deliverables Due Date Recipient Applicable Section 508 Standards

Phase I: Risk Analysis (May 15, 2019 – Sept 23, 2019) 1.1

Call summary, COI forms

6/1/19

AHRQ COR NA

1.2

Call summaries and progress reports monthly AHRQ COR NA

2 Stakeholder panel roster, meeting agenda and other meeting materials, 1 week prior to the meeting date

AHRQ COR,

Stakeholder panel

302 Functional Performance Criteria

602 Support Documentation

603 Support Services

All WCAG A & AA Success Criteria except:

2.4.1 Bypass

Blocks

2.4.5 Multiple

Ways

CAPSTONE Page 27 of 104

Consistent Navigation

3.2.4

Identification meeting notes 1 week after the meeting date; No later than: 8/1/19

AHRQ COR,

Stakeholder panel

302 Functional Performance Criteria

602 Support Documentation

603 Support Services

All WCAG A & AA Success Criteria except:

2.4.1 Bypass

Blocks

2.4.5 Multiple

Ways

Consistent Navigation

3.2.4

Identification

IRB Documentation No later than:

8/15/19 IRB submission 302 Functional

Performance Criteria

602 Support Documentation

603 Support Services

All WCAG A & AA Success Criteria

3 Modified Protocol No later than:

8/15/19

Public 302 Functional Performance Criteria

602 Support Documentation

603 Support Services

CAPSTONE Page 28 of 104

All WCAG A & AA Success Criteria

3 Risk Analysis No later than:

8/15/19

Phase II – Tool development (Sept 24, 2019 to May 15, 2021) 1.1

Call summaries and progress reports monthly AHRQ COR NA

1.2 Dissemination Plan 12/1/2019 AHRQ COR NA

1.3 OMB documents Draft

documents by Sept 30, 2019

Modifications as needed.

Stakeholder panel roster, meeting agendas and other meeting materials, meeting notes

Every 3 months

AHRQ COR NA

3 Report on registry configuration and challenge using test data

3/15/20 AHRQ COR NA

3 Documentation of Final IRB approval

3/15/20 AHRQ COR NA

Standardized implementation models, FHIR and/or similar profiles and implementation guide

3/15/21 Public 302 Functional Performance Criteria

602 Support Documentation

603 Support Services

All WCAG A & AA Success Criteria

6 Report on pilot data analysis and high priority topics for PCOR research on depression

5/15/21 Public 302 Functional Performance Criteria

602 Support Documentation

603 Support Services

CAPSTONE Page 29 of 104

Draft report Final report, disposition of comments, all artifacts and manuscripts

3/15/21 5/15/21

Public 302 Functional Performance Criteria

602 Support Documentation

603 Support Services

Copies of any deliverables should be sent to the Contracting Officer upon request.

Agency for Healthcare Research and Quality ATTN: Contracting Officer Division of Contracts Management, OMS 5600 Fishers Lane Mail Stop 06N34B Rockville, Maryland 20857

F.3 CLAUSES INCORPORATED BY REFERENCE (FEB 1998)

This contract incorporates the following clause by reference, with the same…