RFP_-_AHRQ-19-10006_-_QI_V_-_Amendment_1_(05-09-19).pdf

Request for Proposal (RFP) – Amendment #1 AHRQ-19-10006 Changes Listing Page i

Amendment #1

AHRQ-19-10006:

Annual Updates and Technical Support of AHRQ Quality Indicators for Public Use (QIs) V

Amended RFP Modification Summary

Changes are listed below and are highlighted in the attached amended RFP.

Item no.

Section Description Page No.

1 Section C.9.3.2 In the final sentence in this section, the mention of “impact stories” was removed.

2 Section C.11.2.1 Within this section, the last sentence of the 2nd paragraph (related to “impact stories”) was removed.

3 Section C.10.1 The second paragraph of this section was removed, and replaced with the highlighted paragraph.

4 Section F.2 Deliverable Schedule

The deliverable “C.10.1 – Plan for HCUP Partner workgroup” was removed. The deliverable “C.10.1 – Summary of activities performed to support HCUP Partner workgroups” was revised to remove “workgroups.”

5 Section L.7 Technical Proposal Instructions

Removed “Statistician” from Key Personnel list, and replaced it with “Senior SAS Programmer.”

6 Section L.7 Technical Proposal Instructions

Revisions to “Senior SAS Programmer” section (formerly “Statistician” section).

7 Section M – Evaluation Factors for Award

Replaced “Statistician” in Key Personnel List with “Senior SAS Programmer.”

Amendment #1 to Request for Proposal v. 1/30/19; FAC 101 AHRQ-19-10006 Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 1 of 118

Request for Proposal No. AHRQ-19-10006

PART I - THE SCHEDULE

SECTION A - SOLICITATION FORM

Date Issued: April 30, 2019 Amendment #1 Issued: May 9, 2019 Date Questions Due: May 7, 2019 at 10:00 AM EST Please Note: All questions and inquires will be processed on a non-attribution basis.

Date Notice of Intent Due: May 31, 2019 at 10:00 AM EST Date Proposals Due: June 10, 2019 at 1:00 PM EST

You are invited to submit a proposal to the Agency for Healthcare Research and Quality (AHRQ) for Request for Proposal (RFP) No. AHRQ-19-10006 entitled “Annual Updates and Technical Support of AHRQ Quality Indicators for Public Use (QIs) V”. Your proposal must be developed and submitted in accordance with the requirements and instructions of this RFP.

Please Note: This RFP is a total small business set aside. See FAR 52.219-6 and 52.219-14, herein.

Solicitation Information: Please review the table of contents and refer to the appropriate section of this RFP for requirements, proposal instructions and evaluation information. The Government intends to award this contract without negotiations, so it is imperative that all questions are addressed during the solicitation phase, and that all proposals submitted include the required information and accept the terms and conditions as stated. Failure to comply with proposal instructions or submit required information may result in the removal of a proposal from consideration for award. Please review Section L carefully.

Proposal Intent: If you intend to submit a proposal in response to this solicitation, please inform the Contracting Officer of your intent by sending an email to Lisa Slavin, Contract Specialist at Lisa.Slavin@ahrq.hhs.gov no later than May 20, 2019 at 10:00 AM EST. Subject line shall read:

Proposal Intent RFP No. AHRQ-19-10006.

Questions: All Questions regarding this solicitation shall be received in this office no later than the date shown above. All questions shall be submitted electronically by e-mail to Lisa Slavin, Contract Specialist, at the following email address: Lisa.Slavin@ahrq.hhs.gov. Subject line shall read: Proposal Questions RFP No. AHRQ-19-10006. Answers to questions and all solicitation amendments will be posted to the Federal Business Opportunities web page: www.fbo.gov. It is your responsibility to monitor the web site.

http://www.fbo.gov/

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 2 of 118

Additional Information:

a. A cost plus fixed fee contract is contemplated for a period of one year, with four one-year options. Also included is a Labor Hour line item to support optional tasks.

b. The Government anticipates awarding one contract from this solicitation on or about September 30, 2019.

c. Proposals submitted in response to this solicitation must state that they are valid for a period of at least 120 days (reference FAR clause 52.215-1 in Section L of this document.)

d. The NAICS code for this requirement is 541511 with a size standard of $27.5 million.

Proposal Submission: The proposal shall be signed by an authorized official to bind your organization and must be received by the due date specified above. The proposal must be submitted to the following address:

Agency for Healthcare Research and Quality (AHRQ)* Division of Contracts Management* 5600 Fishers Lane* Mail Stop 06N34B* Rockville, Maryland 20857* Attn: Lisa Slavin, (301) 427-1824

*All boxes must be clearly labeled with this information.

Please read this section carefully. Some mail and package delivery procedures have changed due to security controls. All mail and packages are screened at a local offsite facility before being delivered to 5600 Fishers Lane.

The U.S. Postal Service’s “Express Mail” does not deliver to our Rockville, Maryland address.

Please do not use this means of delivery for proposal submission.

Proposals delivered by commercial package delivery company (example: Fedex, UPS, DHL, etc.):

Submit and label as instructed above. These companies are familiar with the delivery procedures at 5600 Fishers Lane.

Hand carried or hand carry courier service: The 5600 Fishers Lane building security does not allow packages to be delivered directly to the building. Hand carried or courier delivered proposals should be labeled as instructed above, however they must be delivered directly to the offsite screening facility at 12225 Wilkins Avenue Rockville MD 20852. The operating hours at the Wilkins Avenue facility are 7:00 AM to 3:30 PM Monday through Friday. Please note that anyone who arrives at the Fishers Lane location with a delivery will be directed to the screening facility, however this will cause delay. Packages will be accepted and date/time stamped at the screening facility. Please allow ample time as proposals cannot be date/time stamped until they have gone through security.

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 3 of 118

System for Award Management: In accordance with Federal Acquisition Regulation, all contractors must be registered in the System for Award Management (SAM) database in order to conduct business with the government [See Section I - FAR clause 52.204-7 System for Award Management].Access the System for Award Management (SAM) at https://www.sam.gov/SAM/.

Reminder: The Procurement Integrity Act prohibits the unauthorized disclosure of, inquiry about, or acceptance of procurement information. Communications with any other individual outside the AHRQ contracting office may result in rejection of an offeror’s proposal. All requests for any information concerning this RFP should be referred to Lisa Slavin, (301) 427-1824 or e-mail:

Lisa.Slavin@ahrq.hhs.gov. Please note e-mail requests should state subject as RFP AHRQ-19- 10006.

https://www.sam.gov/SAM/

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 4 of 118

Table of Contents

PART I - THE SCHEDULE

SECTION A - SOLICITATION FORM

SECTION B - SUPPLIES OR SERVICES AND PRICES/COSTS

SECTION C - DESCRIPTION/SPECIFICATION/WORK STATEMENT

SECTION D - PACKAGING AND MARKING

SECTION E - INSPECTION AND ACCEPTANCE

SECTION F - DELIVERIES OR PERFORMANCE

SECTION G - CONTRACT ADMINISTRATION DATA

SECTION H - SPECIAL CONTRACT REQUIREMENTS

PART II – CONTRACT CLAUSES

SECTION I - CONTRACT CLAUSES

PART III- LIST OF DOCUMENTS, EXHIBITS AND ATTACHMENTS

SECTION J – SOLICITATION LIST OF ATTACHMENTS

PART IV– REPRESENTATIONS AND INSTRUCTIONS

SECTION K - REPRESENTATIONS, CERTIFICATIONS, AND OTHER STATEMENTS OF OFFERORS

SECTION L - INSTRUCTIONS, CONDITIONS AND NOTICES TO OFFERORS

SECTION M- EVALUATION FACTORS FOR AWARD

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 5 of 118

SECTION B - SUPPLIES OR SERVICES AND PRICES/COSTS

B.1 BRIEF DESCRIPTION OF SUPPLIES OR SERVICES

“Annual Updates and Technical Support of AHRQ Quality Indicators for Public Use (QIs) V”.

See Section C for a complete description.

B.2 ESTIMATED COST AND FIXED FEE

a. The Estimated Cost (exclusive of fees) for performance of the work under the base period of this contract, including direct and indirect costs is $ To Be Determined (TBD).

b. The Fixed Fee for the base period of this contract is $TBD. The fixed fee shall be paid in installments based on the percentage of completion of work, as determined by the Contracting Officer. Payment shall be subject to the withholding provisions of the Clause ALLOWABLE COST AND PAYMENT and FIXED FEE incorporated herein.

c. The total contract value, represented by the sum of the estimated cost plus the fixed fee for the base period is $TBD.

Period of Performance

Services Estimated Cost

Fixed Fee Total

Base Period Severable $ $ $

B.3 OPTION COSTS

If the Government exercises its option(s) pursuant to the option article in Section H of this contract, the Government’s total contract amount represented by the sum of the estimated cost and fixed fee will be revised as follows:

Period of Performance

Services Estimated Cost Fixed Fee Total

Option Period 1 Severable $ $ $ Option Period 2 Severable $ $ $ Option Period 3 Severable $ $ $ Option Period 4 Severable $ $ $

The base period and each option are separate and distinct from one another for accumulation of costs and billing purposes.

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 6 of 118

OPTION TASKS

a. This Labor Hour portion establishes a separate line item to support optional tasks, in accordance with the Statement of Work, Section C.14.

b. The contractor shall bill for direct labor hours in accordance with the burdened fixed hourly rates set forth below.

NOTE: If the offeror’s proposed labor category labels are different than what is listed in the table below, the offeror shall provide a crosswalk explaining how the proposed labor categories correlate to the table below.

c. The total ceiling price for optional tasks, over the life of the contract is $2,000,000.

Labor Category Burdened Labor Rate by Contract Year

Base Year Option Year

Option Year

Option Year

Option Year

Senior Management/ Project Investigator (PI)

Associate Management/ Project Director (PD)

Intermediate/Technical Staff

Junior Analyst

Other Technical Staff (e.g., editors, etc.)

Senior Management/Project Investigator 15 or more years of experience with strategic leadership and serves as key decision maker and contract oversight. Proficient in several high-level programming languages and serves as a key resource in solving complex technical problems. Manages projects to meet budgets and schedules. Serves as the point of contact with technical and government contracting personnel on contract related matters.

Associate Management/Project Director 10 or more years of experience with supervising others working on research projects, and monitors the quality, timeliness, and cost-effectiveness of their work. Proficient in several high-level programming languages and serves as a key resource in solving complex technical problems. Assists other staff in resolving project problems, especially in technical areas. Revises project deliverables prepared by other staff for technical quality and effectiveness of communication. Manages projects to meet quality standards, deliverable schedules, and budgets.

Intermediate/Technical Staff 5 or more years of experience with reviewing and analyzing programming specifications, and converting programming specifications into computer programming code. Creates and runs computer programs. Tests, debugs, and verifies that computer programs operate according to specifications. Prepares written documentation for computer programs and

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 7 of 118 analysis files that may be used by other internal staff or by clients or by the public.

Proficient in several high-level programming languages. Works on specialized programming projects that require advanced programming or technological advances

Junior Analyst 1 or more years of experience with providing assistance to programmer and research staff, including performing data processing tasks. Has working knowledge of program languages. Under close supervision, utilize packaged programs such as SAS to create and document analysis files.

Other Technical Staff 5 years or more experience providing clinical expertise and knowledge in several clinical coding system and serves as a key clinical resource. Experience in ICD-10 diagnoses, current procedural terminology (CPT) and Healthcare Common Procedure Coding System (HCPCS) coding and clinical software (DRGs, disease staging, etc.).

d. The optional tasks shall be subject to the following procedures:

1. Once per year, the Government may issue the contractor a modification request containing a statement of work for an option task. These tasks shall be accomplished on a labor hour basis.

2. The contractor shall review the statement of work for the optional task(s) and submit a proposal for the labor hours associated with accomplishing the optional task, using the labor categories and burdened labor rates included in the table above. A technical proposal may be requested.

3. Once negotiations (if necessary) are concluded, a modification shall be issued to incorporate optional task(s) into the appropriate sections of the contract.

i. A ceiling price for each specific optional task will be established.

ii. The sum of all optional tasks exercised under this contract must stay within the total ceiling price established above.

B.4 PROVISIONS APPLICABLE TO DIRECT COSTS

a. Items Unallowable Unless Otherwise Provided

Notwithstanding the clauses, ALLOWABLE COST AND PAYMENT, and FIXED FEE, incorporated into this contract, unless authorized in writing by the Contracting Officer, the costs of the following items or activities shall be unallowable as direct costs:

(1) Acquisition, by purchase or lease, of any interest in real property;

(2) Rearrangement or alteration of facilities;

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 8 of 118

(3) Purchase or lease of any item of general purpose-office furniture or office equipment regardless of dollar value. (General purpose equipment is defined as any items of personal property which are usable for purposes other than research, such as office equipment and furnishings, pocket calculators, etc.);

(4) Accountable Government property;

(5) Travel to attend general scientific meetings;

(6) Foreign Travel;

(7) Any costs incurred prior to the contract's effective date;

(8) Rental of meeting rooms not otherwise expressly paid for by the contract;

(9) Any formal subcontract arrangements not otherwise expressly provided for in the contract;

(10) Consultant fees;

(11) Information Technology hardware or software; and

(12) Food and Beverages;

Note: Food and Beverages are unallowable costs. Contracting Officer Authorization for Food and Beverages will be only made as specifically authorized by statute, which is extremely rare. The contractor shall not provide food or beverages of any type. The contractor shall not accept food or beverages, even if included in, or provided ‘free’ as part of a meeting/conference ‘package’.

b. Travel Costs

This contract is subject to the provisions of Public Law (P.L.) 99-234 which amends the

Office of Federal Procurement Policy Act to provide that contractor costs for travel, including lodging, other subsistence, and incidental expenses, shall be allowable only to the extent that they do not exceed the amount allowed for Federal employees. The Contractor, therefore, shall invoice and be reimbursed for all travel costs in accordance with Federal Acquisition Regulations (FAR) 31.205-46.

The following applies to Educational Institutions: The Contractor shall invoice and be reimbursed for all travel costs in accordance with OMB Circular A-21 - "Cost Principles for Educational Institutions."

No funding or expenditures will be authorized under this contract for travel or meeting expenses of any person that is not an employee of the contractor or

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 9 of 118 approved subcontractor in accordance with the terms of the contract. See 31 USC § 1345 for appropriations restrictions.

B.5 CONTRACTING OFFICER AUTHORIZATION

Other provisions of this contract notwithstanding, approval of the following items within the limits set forth is hereby granted without further authorization from the Contracting Officer.

To Be Determined

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 10 of 118

SECTION C - DESCRIPTION/SPECIFICATION/WORK STATEMENT

Independently and not as an agent of the Government, the Contractor shall be required to furnish all the necessary services, qualified personnel, material, equipment, and facilities, not otherwise provided by the Government, as needed to perform the Statement of Work as follows:

Statement of Work

Unless otherwise noted under a specific task, this Statement of Work applies to the base period and all options (if exercised).

The Center for Quality Improvement and Patient Safety (CQuIPS) within the Agency for Healthcare Research and Quality (AHRQ) of the United States Department of Health and Human Services (DHHS) requires the annual updating and technical support of the AHRQ Quality IndicatorsTM (QIs) (2019-2024).

The AHRQ QI Annual Update and Technical Support and Contract shall focus solely on the operational aspects of updating the QI software and supporting the users of AHRQ QIs who are focused on quality improvements by providing updated software, technical specifications, technical assistance, user friendly interface and tools and documentation for AHRQ QIs. The objectives of this statement of work are:

Update accurate and user-friendly AHRQ Quality Indicators (QI) public-use software, technical specifications, and documentation for use in the measurement and quality improvement of the delivery of health care services

Provide technical assistance to users of the AHRQ Quality Indicators (QIs)

To meet these objectives, the contractor shall perform the following activities:

1) Prepare timely, uniformly formatted AHRQ QI documentation, including technical specifications and technical documents

2) Maintain customer-oriented, streamlined AHRQ QI software to facilitate user’s calculation of AHRQ QI rates

3) Review and provide recommendations for the AHRQ QI Website, which serves as the project’s primary vehicle for public outreach, and virtual repository for project information and documentation

4) Deliver customer-oriented technical support on AHRQ QIs technical documentation, software and products developed for the AHRQ QI project

5) Maintain and support customer-oriented communication with AHRQ QI users

C.1 BACKGROUND INFORMATION

C.1.1 AHRQ Mission and Vision AHRQ is the federal agency tasked with producing evidence to make health care safer, higher quality, more accessible and more affordable, and to work within HHS and other partners to make sure that the evidence is understood and used. One of the basic foundations for producing evidence on how to improve quality is quality measurement.

As one of twelve agencies within the DHHS, AHRQ supports research and tools that

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 11 of 118 help people make informed decisions and that improve the quality of health care services. AHRQ, along with other DHHS agencies, participates in the National Quality Strategy (NQS), a nationwide effort to align public and private interests to improve the quality of health and health care for all Americans.

One way AHRQ carries out its mission and vision is to provide researchers, data organizations, health care planning administrators, hospital executives, purchasers, insurers, local and state agencies, and federal agencies with the tools to assess the quality of care being provided in hospitals. AHRQ also uses data on hospital admissions to provide information about potentially preventable admissions, to help health departments, accountable care organizations (ACOs), Medicaid agencies, and others target primary care and public health efforts. Quality measures, such as the AHRQ QIs, are widely used for research, health care planning, and quality improvement, throughout the United States (US).

C.1.2 History of the AHRQ Quality Indicators (QIs) Originally, the AHRQ Quality Indicators were developed as part of the Healthcare Cost and Utilization Project (HCUP). These indicators were created in response to requests from State partners who contributed data to HCUP and from researchers seeking to understand health care quality using administrative data.

The Agency embarked on a process to expand, refine and severity-adjust the HCUP QIs through the Evidence-based Practice Center (EPC) at University of California-San Francisco and Stanford University. This effort resulted in the development of the AHRQ QIs that are based on administrative claims data. Over the last several years the Agency has contracted with several organizations and academic institutions to continue the work on enhancing and refining the AHRQ Quality Indicators.

C.1.3 Characteristics and Uses of the Current AHRQ QIs The nature and the uses of the QIs have evolved considerably during the past decades. The original HCUP Quality Indicators were primarily used for quality improvement efforts and were based upon a "common denominator" hospital discharge data set, (i.e. using administrative data elements collected by all or nearly all hospital data systems through the Uniform Bill) (CMS, Medicare Claims Processing Manual, 2013). The current AHRQ QIs take advantage of data enhancements, improvements in coding practices and advanced statistical methodologies where possible. New data elements for which the evidence suggests inclusion in the measure specifications improves the accuracy and the reliability of the measures, such as Present-on- Admission (POA) and laboratory values, have been incorporated. Furthermore, the QIs now include sophisticated programming knowledge, rigorous precision, and risk adjustment for comparisons across hospital performance. The current AHRQ QIs are complex measures with sophisticated risk adjustment methodology and applications for missing data.

As a result of this evolution, the uses and the users of the AHRQ QIs are much more extensive and diverse than in the early years. The AHRQ QIs now serve multiple purposes, including research, needs assessments for planning at the local, state, and national levels, hospital and community quality improvement initiatives, performance assessment through private and public reporting. Users of the AHRQ QIs vary and include researchers, State data organizations, community health organizations, hospitals and hospital systems, hospital associations, State Medicaid agencies, Centers for Medicare & Medicaid Services (CMS), large private purchasers and public-private purchaser coalitions, consumer groups, and local, State and Federal health

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 12 of 118 care planning and monitoring organizations. State hospital associations, state data associations, hospitals and hospital systems use AHRQ QIs to help them understand how their hospital(s) compares on various quality dimensions in relationship to other hospitals, the state, or national average. Many of these organizations use the AHRQ QIs in hospital-level reporting programs.

Researchers, quality improvement officers, hospital administrators and community health planners use AHRQ QIs to ask questions that provide initial feedback about clinical areas appropriate for further, more in-depth analysis, such as: How can the Prevention Quality Indicators help identify quality of care issues with ambulatory care sensitive conditions? What can they signal about primary care access or outpatient services in a given community? Or how does the hysterectomy rate in a given area compare with the national average? State hospital associations can use the AHRQ QIs to do quick screens of hospital quality and primary care access. Hospitals and hospital systems nationwide can use the AHRQ QIs in quality improvement initiatives where the QIs are used to identify a potential quality problem, interventions are put in place and the QIs are used to determine whether the intervention had an impact.

Managed care organizations, business-health coalitions, and others are poised to begin assessments using hospital discharge data to answer questions such as: Can we design community interventions to reduce hospital admission rates in areas with high rates of diabetes complications? Which AHRQ QIs can be incorporated into performance management initiatives for our member hospitals?

Federal and State policymakers use the AHRQ QIs to track health care quality in the United States over time and to assess whether health care quality is improving, for example: How does the rate of coronary artery bypass graft procedures vary over time and across regions of the United States? Another such example is the use of the AHRQ QIs to track national health care quality and disparities in the Congressionally-mandated National Healthcare Quality and Disparities Reports.

In addition to the examples above, the AHRQ QIs are included on the Center for Medicare & Medicaid Services (CMS)’ Hospital Compare website https://www.medicare.gov/hospitalcompare/Data/Data-Updated.html# as well as have been included in the methodology used for U.S. News and World Reports ranking of the top 100 hospitals in the US; are part of the Leapfrog Safety Score, and Partnership for Patients; and are used by numerous hospitals and hospital systems in their quality improvement programs.

C.1.4 Overview of the AHRQ QI Indicators

C.1.4A General Overview The AHRQ QIs consist of four modules: the Prevention Quality Indicators or PQIs, the Inpatient Quality Indicators or IQIs, the Patient Safety Indicators or PSIs and the Pediatric Quality Indicators or PDIs. The indicators in the four modules are reported at either hospital level or area level. There are currently approximately 45 hospital level indicators and about 25 area level indicators.

Prevention Quality Indictors (PQIs) The Prevention Quality Indicators (PQIs) are a set of measures that can be used with hospital inpatient discharge data to identify quality of care for "ambulatory care sensitive conditions." These are conditions for which good outpatient care can potentially prevent the need for https://www.medicare.gov/hospitalcompare/Data/Data-Updated.html

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 13 of 118 hospitalization or for which early intervention can prevent complications or more severe disease. The PQIs are population based and adjusted for covariates. There are a total of 14 PQIs, including 10 individual area-level indicators and 4 composite indicators. The PQIs are population based and adjusted for gender and age. Even though these indicators are based on hospital inpatient data, they provide insight into the community health care system or services outside the hospital setting. For example, patients with diabetes may be hospitalized for diabetic complications if their conditions are not adequately monitored or if they do not receive the patient education needed for appropriate self-management.

The PQIs can be used as a "screening tool" to help flag potential health care quality problem areas that need further investigation; provide a quick check on primary care access or outpatient services in a community by using patient data found in a typical hospital discharge abstract; and, help public health agencies, State data organizations, health care systems, and others interested in improving health care quality in their communities.

With high-quality, community-based primary care, hospitalization for these illnesses often can be avoided. Although other factors outside the direct control of the health care system, such as poor environmental conditions or lack of patient adherence to treatment recommendations, can result in hospitalization, the PQIs provide a good starting point for assessing quality of health services in the community. Because the PQIs are calculated using readily available hospital administrative data, they are an easy-to-use and inexpensive screening tool. They can be used to provide a window into the community — to identify unmet community health care needs, to monitor how well complications from a number of common conditions are being avoided in the outpatient setting, and to compare performance of local health care systems across communities.

Inpatient Quality Indicators (IQIs) reflect quality of care for adults inside hospitals and include:

Inpatient mortality for medical conditions; inpatient mortality for surgical procedures; utilization of procedures for which there are questions of overuse, underuse, or misuse; and volume of procedures for which there is evidence that a higher volume of procedures maybe associated with lower mortality. There are currently 19 IQIs, including 2 composite indicators. Risk adjustment of the current IQIs include: patient demographics, the 3M All Patient Refined-DRGs (APR-DRGs), Major Diagnostic Category (MDC), transfer status, point-of-origin, and procedure days, when available.

The IQIs can be used to help hospitals identify potential problem areas that might need further study; provide the opportunity to assess quality of care inside the hospital using administrative data found in the typical discharge record; include mortality indicators for conditions or procedures for which mortality can vary from hospital to hospital; include utilization indicators for procedures for which utilization varies across hospitals or geographic areas.

Patient Safety Indicators (PSIs) also reflect on the quality of care for adults inside hospitals, but focus on potentially avoidable complications and iatrogenic events. There are currently 18 PSIs, including 1 composite indicator. Risk adjustment of the current PSIs include: age, sex, modified-DRG, MDC, comorbidities, transfer status, point-of-origin, and procedure days, if available. In addition, one of the indicators contains specifications for risk stratification. These indicators are some of the most frequently used indicators in quality improvement and public reporting initiatives.

The PSIs can be used to help hospitals identify potential adverse events that might need further study; provide the opportunity to assess the incidence of adverse events and in hospital

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 14 of 118 complications using administrative data found in the typical discharge record; include indicators for complications occurring in hospitals that may represent patient safety events. Indicators also have area level analogs designed to detect patient safety events on a regional level.

Pediatric Quality Indicators (PDIs) reflect both quality of care for children below the age of 18 and neonates inside hospitals and identify potentially avoidable hospitalizations among children. The PDIs focus on potentially preventable complications and iatrogenic events for pediatric patients treated in hospitals, and on preventable hospitalizations among pediatric patients (i.e., a combination of IQIs, PSIs and PQIs for the pediatric population). There are currently 15 individual PDIs, including 8 risk-adjusted provider-level indicators, and 7 area-level indicators; there are 3 composite indicators, as well.

Risk adjustment models for the current PDIs may include: age, gender, birth weight, Modified DRG, MDC, Clinical Classification Software groupings of conditions, transfer status, point of origin and procedure days. In addition, some of the indicators have risk-stratified categories.

The PDIs apply to the special characteristics of the pediatric population; screen for problems that pediatric patients experience as a result of exposure to the healthcare system and that may be amenable to prevention by changes at the provider level or area level; and, help to evaluate preventive care for children in an outpatient setting, and most children are rarely hospitalized.

The AHRQ QIs rely on data elements and standard coding practices found within billing or claims data, as provided on the Uniform Bill (UB-04) (http://www.nubc.org/). The National Uniform Billing Committee (NUBC) and the Centers for Disease Control and Prevention’s National Center for Health Statistics provides guidance on the standard coding of diagnoses and procedures (e.g., http://www.cdc.gov/nchs/icd/icd9.htm and http://www.cdc.gov/nchs/icd/icd10.htm ), Present-on-Admission indication of conditions and external cause of injury, patient demographics (gender, race/ethnicity), expected payer, admission status and point-of-origin, and discharge disposition. All of the technical specifications and methodology used to develop the AHRQ QIs can be found on the AHRQ QI website: www.qualityindicators.ahrq.gov

C.1.4.B Methodology Overview The current empirical methods of the AHRQ QIs are described in a comprehensive AHRQ QI report entitled, Quality Indicator Empirical Methods, posted on the AHRQ QI website (www.qualityindicators.ahrq.gov).

An indicator may describe results at the hospital-level or at an area or population-level. Some indicators indicate the number of times a medical procedure was performed within a facility (volume indicators). Most indicators, however, are expressed as ratios with a numerator of counts of hospitalizations with a condition or outcome of interest (e.g., mortality) and a denominator of the population at risk (e.g., discharges or census population estimates).

The AHRQ QI software calculates several rates, including the observed rate in the population of interest, the expected rate based on the reference (nearly universal) population, the risk-adjusted rate that accounts for the mix of patients in the hospital, and the smoothed rate (or performance score) that accounts for some of the noise in the data.

The risk adjustment of the AHRQ QI modules varies by the type of indicator. Hospital-level indicators are risk-adjusted by deriving a hierarchical logic regression model that uses a generalized estimating equations (GEE) approach to account for clustering at the provider level.

When GEE models do not converge, a multivariable logistic regression model that does not account for correlation is employed. Area-level indicators are risk adjusted for age and gender, and optionally for poverty.

http://www.nubc.org/ http://www.cdc.gov/nchs/icd/icd9.htm http://www.cdc.gov/nchs/icd/icd10.htm http://www.qualityindicators.ahrq.gov/

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 15 of 118

Smoothed rates (or performance scores) are the weighted average of the risk-adjusted rates and the reference population rates (or a regression determined target estimate based on provider attributes such as volume or prior performance). The smoothed rate is derived using empirical Bayes shrinkage to result in a rate that minimizes noise from the estimates. The shrinkage estimator is based on signal and noise variance estimates. When there is a fixed comparative rate of interest, it is possible to parameterize the smoothed rates posterior probability distribution to assign performance categories for public reporting.

C.1.4.C Description of AHRQ QI Documentation Documentation is the cornerstone feature of the AHRQ QI program, as it serves as the primary resource for the AHRQ QI algorithms and for the methods on how the AHRQ QIs were created.

Documentation includes (a) the technical specifications of each indicator, (b) technical reports that support the AHRQ QIs, including software manuals, detailed descriptions of the AHRQ QIs including numerators, denominators, reliability and validity estimates, and interpretation of rates, reports that describe how and why measures were developed, benchmarks of AHRQ QI rates/ratios, and lists of publications that use the AHRQ QIs. Ideally, documentation is created and updated in tandem with the creation of each version of the AHRQ QIs and as necessary to correct documentation errors noted by users.

C.1.4.D Description of AHRQ QI Software The AHRQ QI program is unique in that it provides publicly available software tools that calculate the AHRQ QI rates. This software legally resides in the public domain and is available as a free download from the AHRQ QI website. It allows users to calculate QI measure results based on their own administrative data set using a standard desktop computer. The QI software tools help support the dissemination of the standardized, transparent quality measures and ensures that the measures are calculated in a consistent and accurate manner.

The AHRQ QI software tools are a key part of AHRQ’s QI program. They help to support AHRQ’s objective to disseminate standardized, transparent quality measures; to ensure that the measures are calculated in a consistent and accurate manner; and to remove barriers to their use. The QI software is currently released in two different platforms: a SAS®† application and a Windows application called WinQI. The SAS software was first released in the late 1990s. It consists of several modules of SAS code and requires a SAS license to run https://www.qualityindicators.ahrq.gov/Software/Default.aspx

The Window-based software, known as WinQI, was first released in 2005 (http://www.qualityindicators.ahrq.gov/software/WinQI.aspx ). It was created in order to provide an easy-to-use, low-cost option for calculating the QIs that was not dependent on licensed software. Developed on Microsoft® Visual Studio using C# and .NET, AHRQ WinQI runs on Windows operating systems and requires only freely available software components: AHRQ-produced software, Microsoft .NET (for runtime environment and core software libraries), and Microsoft SQL Server Express (for data storage and manipulation). Table 1 lists the differences between SAS QI v2018 ICD10 CM/PCS7 and WinQI v2018 ICD10 CM/PCS.

Both the SAS and WinQI software are available in Version 2018 ICD 10 CM/PCS as 64-bit application. The 64-bit applications are targeted for Windows 7 and above operating systems.

Both the SAS and WinQI versions of the QI software are updated on an annual basis to reflect changes in the AHRQ QI technical specifications. New software versions and updated technical specifications are released simultaneously. Routine annual updates include yearly International https://www.qualityindicators.ahrq.gov/Software/Default.aspx http://www.qualityindicators.ahrq.gov/software/winQI.aspx

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 16 of 118

Classification of Diseases, 10th revision, Clinical Modification/Procedure Coding System, Medicare severity-Diagnostic Related Groups (MS-DRG), Major Diagnostic Categories (MDC), updated version of the 3MTM APR-DRG grouper, new Census population files, and risk adjustment parameters.

As the AHRQ QIs matured and their use expanded the QI software tools also transitioned from small research prototypes to widely used, production quality software. This growth in volume, visibility, and use brought a significant need for software sophistication and support. AHRQ’s priorities regarding the QI software tools include:

• ensuring that they remain accessible, low cost, and user friendly

• updating and expanding the tools to keep pace with changes in the QIs and the field of software technology

• supporting software users with well-organized, accurate documentation and responsive technical assistance

• employing good software development and testing practices to ensure timely, accurate, and robust software products

C.1.4.E Description of AHRQ QI Website The AHRQ QI Website (http://www.qualityindicators.ahrq.gov) is the main vehicle through which information on the AHRQ QIs is disseminated. The website provides detailed information on the QIs, the specifications and methodology for the measures, as well as the SAS and windows-based software which calculates the QI rates. Specifically, the AHRQ QI website should serve as a central repository for all project information including:

- General information about the project

- Technical specifications for each indicator

- Software instruction manuals

- Downloadable SAS, WinQI and other software files

- Results of software testing

- Technical documentation to support the previous development and current use of the

AHRQ QIs

- Documents benchmarking AHRQ QI rates

- References to new findings, publications, and research notes based on AHRQ QIs

- Links to quality improvement tools and initiatives (e.g., AHRQ QI Toolkit, National

Quality Strategy)

- Contact information for QI User Support

- Documents specific to history and evolution of each of the AHRQ QIs, including clinical, statistical and methodological rationale for each indicator

Not only is the AHRQ QI website useful to users of the measures, but it is also one of the most important resources for understanding the AHRQ QIs. See Section C.1.3 for current users of the AHRQ QI website.

Note: Potential Offerors should rely heavily on the AHRQ QI Website (http://www.qualityindicators.ahrq.gov) to gain information pertinent to this request for proposal.

C.1.4.F Process for Updating the AHRQ QI Software The AHRQ QI technical specification and software have an established updating process which currently comprises the following tasks, many of which may be performed in parallel. The

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 17 of 118 following tasks comprise the core of AHRQ QI software updating process and are provided here as background information only:

1. Obtain inpatient hospital billing data from the Healthcare Cost and Utilization Project (HCUP) central distributor to serve as the reference population and testing database for the AHRQ QI software

2. Obtain other data and software to support the calculation of the AHRQ QIs (e.g., 3M Limited License APR-DRG grouper software)

3. Perform routine coding, grouping, and software updates

4. Compile list of proposed refinements based on AHRQ requirements, issues found during release of the most recent version of the AHRQ QIs, and technical assistance questions

5. Assess validity, reliability, feasibility and/or acceptability of proposed refinements to the technical specifications**

6. Update technical specifications**

7. Calculate risk adjusted models and estimate risk adjusted parameter estimates**

8. Update SAS, WinQI and any other software to reflect updates to the technical specifications

9. Test specifications and software

10. Document changes to specifications and software

11. Release AHRQ QI modules to the public

**Task requires substantial clinical and statistical analysis which shall be conducted by AHRQ under separate mechanisms and then shared with the Contractor as needed.

C.1.5 CURRENT USER DEMAND

In recent years, the utilization of the AHRQ QIs has increased significantly. Approximately 150 user inquiries are received each month. There are currently 2,400 subscribers to the listserv, approximately 15,000 visits to the website monthly, and between 65,000 to 85,000 file downloads of QI documentation and software monthly. As mentioned earlier, this rapid uptake of the QIs has been due to many factors including a demand for transparent, standardized measures using readily available data, for quality improvement.

C.3 GENERAL REQUIREMENTS

The Contractor shall maintain such internal policies and procedures as may be required to ensure a high-quality, smooth operation of a project of this scope, sophistication, and complexity, including such training materials as may be necessary to introduce new employees to the project. The Contractor shall maintain a highly effective internal quality control process that assures that the deliverables developed under the contract accurately and consistently address the stated objectives and accurately portray substantive content. The Contractor shall ensure the accuracy of the content of all materials and products generated under this contract.

The contractor shall maintain an infrastructure to provide ongoing technical support and updating of the AHRQ QI measures and tools. AHRQ shall provide the contractor with extensive documentation on the project and access to all relevant materials describing past approaches and activities.

The Contractor is responsible for coordinating the receipt and tracking license agreements of all data files and software needed for the project on behalf of AHRQ. Data obtained for the purpose of the performance of this contract may not be used for any other purpose outside of this contract.

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 18 of 118

Specific task requirements are detailed in the following sections.

C.4 PROVIDE PROJECT MANAGEMENT

The Contractor shall develop and implement methods to ensure that project tasks and subtasks are coordinated so that work progresses in an orderly, efficient manner. The Contractor shall develop and implement a method for planning future project activities so that tasks are completed within expected timeframes and budgets. The Contractor shall implement quality control procedures to monitor and evaluate the quality of the products that shall be delivered.

The Contractor shall document all proposed project management methods and software tools for review by the Contracting Officer Representative (COR).

C.4.1 Support Meetings and Conference Calls The Contractor shall meet with the COR and others designated by the COR for their technical expertise or substantive interest in the project. Multiple activities must take place simultaneously to accomplish the work of this project. AHRQ staff assumes an active role on many tasks, and clear communication is necessary to prevent errors and maintain the quality of the AHRQ QIs and other products.

C.4.1.1 Participate in Conference Calls with AHRQ The Contractor shall participate in formal project management conference calls with the COR and other staff as assigned, as frequently as once a week. The purpose of these meetings shall be to review the progress of work, obtain guidance from AHRQ for task completion, identify problems and discuss possible solutions, review specific deliverables, and review future plans under the contract. The Contractor shall prepare and send an agenda 2 days in advance of each conference call and a weekly summary 3 days after the conference call which highlights key activities and any problems encountered, organized by task.

In addition, to project management conference calls, the Contractor shall participate in formal task-specific conference calls with the COR or other AHRQ staff assigned, potentially as frequently as once a week, to review task progress, receive input from AHRQ on how to achieve tasks, identify problems and discuss possible solutions and plan for future work. The Contractor is expected to provide an agenda 2 days in advance of each conference call.

C.4.1.2 Participate in other Communications with the COR It is anticipated that less formal communications by telephone or email shall also occur on occasional basis as needed between the COR and other AHRQ staff as assigned and the Contractor. It is expected that the Contractor respond to all telephone or email queries within 24 hours, unless other arrangements have been made.

C.4.2 Prepare for and Support Annual Site Visit At the start of each contract year, the Contractor shall prepare for and present the project status at an annual site visit with the COR. Key staff from the Contractor’s team shall travel to the AHRQ offices in Rockville, MD to conduct an in-person meeting. At the meeting, the contractor shall review the previous year’s progress of work including successes, problems, solutions and plans for the upcoming year with respect to project budget, staff and other resources, timelines and deliverables.

C.4.3 Prepare an Annual Project Management Plan

Annual Updates and Technical Support of AHRQ QIs for Public Use V Page 19 of 118

Effective management of the project is critical to its success. The Contractor shall assess the needs and complexities of the project and staff to determine the most effective, efficient and streamlined approach for the project. The Contractor shall document this approach, provide a rationale for the approach, and describe and demonstrate (where possible) the efficiencies it offers. The Contractor shall prepare and submit this information as part of the annual project management plan which also provides for the budgeting, monitoring, effective communication and documentation of all contract activities and costs by major tasks. The plan shall include an annual work plan for the tasks to be completed, including an updated delivery schedule and timeline for the project tasks and identify and recommend any changes in approach or process to the creation and management of the AHRQ QI databases and other tasks. The plan shall include such items as procedures for ensuring adequate availability of staff, efficient use of computer and programmer resources on each task, the performance of tasks in a timely manner, and procedures for ensuring data security and confidentiality.

Annually, at 11 months after the start of the contract year, the Contractor shall submit a draft project management plan for the next contract year (if the option is exercised), which includes a project plan for each task.

Annually, at 6 months after the start of the contract year, the Contractor shall also provide a status update of the management plan.

C.4.4 Provide Progress Reports Monthly, the Contractor shall submit to the COR…