SECTION_C.pdf

RFP-CMS-RMADA-2014

SECTION C – DESCRIPTION/SPECIFICATIONS/WORK STATEMENT

C.1 Background

Under this Research, Measurement, Assessment, Design and Analysis (RMADA) Indefinite Delivery Indefinite Quantity (IDIQ) umbrella contract, the Centers for Medicare & Medicaid Services (CMS) will award task orders (TOs) for a wide range of analytic support and technical assistance activities that support models and demonstration programs created or derived under the auspices of the Patient Protection & Affordable Care Act (ACA), and future health reform legislation where new delivery and payment reform models are enacted.

The demands of new reforms created under ACA have redefined the way CMS approaches and conducts research activities and demonstrations affecting Medicare, Medicaid, CHIP, and uninsured populations. The role of state and private sector payers is also redefined as many of the new models include multiple payers working in collaboration with CMS to reform the care delivery system. The RMADA will provide CMS with a robust tool to meet those challenges. Some of the major activities this umbrella contract will address include the following: designing, maintaining and refining model/demonstration design and operations;

monitoring model site implementations; designing and carrying out surveys and other data collection activities; obtaining and analyzing secondary data sources including Medicare, Medicaid and Children’s Health Insurance Program (CHIP), and private payer sources that support model design and evaluations. Some other evaluation activities envisioned under the RMADA include reporting on formative and summative analyses, providing rapid cycle quarterly evaluation feedback to all model participants and CMS, and the creation of summative annual and final program findings.

CMS is the Federal agency responsible for the operation and oversight of the Medicare/Medicaid programs and the Federal participant in the State-operated Medicaid and CHIP. CMS’s mission is to assure health care security for the general public through the management of programs that include Medicare, Medicaid, State Children Health Insurance Program(s), and provides leadership in the broader health care marketplace to improve the health of all citizens. As an effective steward of public funds, CMS is committed to strengthening and modernizing the nation’s health care system to provide access to high quality care and improved health at lower costs. To help accomplish this mission, CMS was given broader authorities through ACA for the planning, analysis, implementation, and rapid cycle evaluation of innovation and demonstration activities to determine effectiveness and feasibility for broader dissemination, scale and sustainability. This includes creating and testing innovative payment and service delivery models, building collaborative learning networks to facilitate the collection and analysis of innovation, the implementation of effective practices, developing necessary technology to support this activity, conducting rapid cycle analysis of the programs and participants, and close to real time reporting of interim findings and summative findings as feasible. The use of this umbrella contract is expected to assist CMS in carrying out its mission by facilitating validation and disseminating information about new care models and payment approaches to serve Medicare, Medicaid, and Children’s Health insurance Program (CHIP) beneficiaries seeking to enhance the quality of health and health care and reducing cost through improvement.

C.2 Technical Considerations

The work awarded under the RMADA will involve the design, implementation and evaluation of a broad range of research and/or payment and service delivery models to test their potential for reducing expenditures for Medicare, Medicaid, CHIP, and uninsured beneficiaries while maintaining or improving quality of care.

New models/demonstrations and research afforded under ACA are different from past CMS demonstrations and evaluations, and have changed the way we approach and conduct these activities. The scope of activities of these new models can include all areas of health care research: costs, access, quality, service delivery models, and financing and payment approaches. They tend to be larger, more complex, and can be scaled without Congressional approval if programs can demonstrate cost savings while maintaining or improving quality. These new models/demonstrations can often be characterized as being highly visible and tightly integrated with other programs.

Today’s models/demonstrations afforded under ACA often interact with one another or with national CMS programs in terms of overlapping provider participation, regulatory structures, and use of the same information technology tools and services. Such interactions must be carefully anticipated and managed to account for complex legal, operational, and policy priorities. Today’s models/demonstrations also have more program integrity considerations and involvement from enforcement agencies. Unlike past demonstrations, there is a tremendous focus on shared learning activities among participants because lessons learned from other industries demonstrate that diffusion of knowledge is a key factor in the expansion of models. Furthermore, today’s models/demonstrations typically move from design to announcement and implementation in less than a year, allowing little room for missed deadlines or coordination lapses. Most draw intense interest and scrutiny from market stakeholders and the press as well as Congress and other policymakers. Health care costs continue to rise and there are high expectations for CMS to both rapidly assess these models/demonstrations, and scale those most promising nationally without any delay.

Expectations for support of today’s models/demonstrations have also evolved. Unlike past demonstrations, greater emphasis is being placed on supporting one or more phases of the model lifecycle. This requires close coordination with multiple CMS and HHS entities, and the ability to respond quickly to evolving situations. This work may require the expertise of multi-disciplinary staff including economists, physicians and other health professionals, pharmacists, social scientists, actuaries, statisticians, health policy analysts, and systems analysts and programmers. Contractor staff must also have a wide range of health care and health policy expertise, and the ability to adequately staff multiple large scale CMS models at the same time. Contractor staff should have the analytic and management ability to accurately assess program risks, design and, if necessary, implement multiple contingency plans for successful model implementation. Because CMS engages model participants in a “high touch” manner, support staff must be able to provide intensive technical assistance in an accessible, client-oriented manner and at scale to all model participants. -

The need for analyses based on real time claims and utilization data is a unique factor that distinguishes today’s evaluation of models as opposed to prior demonstrations. Some other considerations include the need for quarterly reports for all models rather than just those at risk, continuous feedback to providers as part of improvement efforts, regression based approaches as opposed to actuarial approaches, and significantly more focus on the use of qualitative methods such as case studies. These methods provide insight into factors associated with process analyses and help determine the success or failure of specific aspects of the relevant programs. Furthermore, because Innovation Center models often include collaboration among multiple payers and other entities, the current evaluation approaches will need to account for the need to gather, coordinate, and analyze private payer and other private data sources. In addition, evaluations involving other entities, such as payers, should plan to examine the role of CMS as a convener and how the model is received by both participating and non-participating affected parties.

C.3 Requirements

Tasks that the Contractor may be required to perform under the RMADA, include but are not limited to:

A. Model Design:

Identify problem and opportunity to address (business case)

CMS is building a diverse portfolio of innovative service delivery and payment models. Typically a model conducted by the Center for Medicare and Medicaid Innovation (CMMI), begins with the selection of a concept or opportunity it wants to explore. It is then defined at a high level in a condensed, written business plan referred to as an Innovation Center Investment Plan or ICIP Light. For CMMI, the objective of the ICIP is to define the problem being addressed, the target audience, framing the basic model concept and identifying potential evidence to support the proposed approach.

In defining the problem, opportunity, and/or concept, the Contractor shall consider CMS’ written business plan for the proposed model the capabilities of the company/firm and the capacity of the company/firm perform the following tasks:

1. Conduct Literature review, evidence assessment, environmental scan, and market research;

a. Exploratory analyses: Conducting an environmental scan and literature review for evidence and examples to support the problem/concept definition, preliminary model/demonstration design and strength of the opportunity; analysis involving the collection of primary and/or secondary data and using quantitative or qualitative analytical methods.

b. Preparation of written materials and participation in briefings: Prepare summary report(s) that integrates findings from the literature review and data analyses, identifies current gaps in knowledge; and presents refined study questions, options to further address policy issue(s); relevant parameters, and recommendations for further action; drafting and/or editing the written business case, including support in definition of problem and opportunity, target populations, available evidence, documenting risks and assumptions in the proposed model; preparing written briefing materials and conducting oral presentations to senior-level CMS, Departmental officials, and/or Department contractors. .

c. Need for waivers: Identify regulatory and policy issues and requirements, including if waivers are necessary.

2. Acquire and analyze data;

a. Obtain CMS, private health care sector and/or State data: In order to obtain all CMS data necessary for defining the problem and/or opportunity for improvement, including CMS claims and non-claims data (up to and including 100% of CMS claims files), enrollment/eligibility information, and State data, the contractor should be knowledgeable about the current CMS data sources . Evaluation contractors are currently required to work in the CCW environment, but should be knowledgeable about all CMS environments including the IDR and the mainframe. Whether or not these resources will be available is project specific and dependent on numerous factors. Contractors should have the capability and be prepared to propose an approach that obtains data directly from the CMS data sets and other appropriate sources. CMS will provide data that it already has in its possession, but the bidder should not make an assumption that CMS will be responsible for providing data that it does not currently have. For information that is not found in Medicare claims, Contractors will be required to propose appropriate, timely, and cost-effective ways to access the data. This could include obtaining Medicaid data directly from states or sites, hospital level data from the AHA survey, or using other publically available data sources. Primary data collection may also be required. CMS will work with the contractor to identify the best data source. Inasmuch as existing data collection tools can be made available to the contractor, the CMS will work with the contractor to facilitate access. As part of the process of obtaining CMS data, the Contractor shall be responsible for obtaining any necessary Data Use Agreements (DUAs) and for adhering to CMS data security policies and procedures.

3. Meet with subject matter experts;

a. Stakeholder engagement: Consulting with experts in and outside of CMS, in the public and private sector; having preliminary discussions with federal and/or State officials or other experts to refine the policy issues to be examined and identify additional data sources and reports to be examined.

a. Facilitate Stakeholder Opportunities for Input: As pertinent to a given TO, the Contractor may be instructed to develop and carry out a detailed plan for generating participant, stakeholder and/or subject matter expertise input in any relevant phases of an evaluation or research TO including the design phase, the creation of data collection tools, the analysis phase, and the report writing phases. As part of this activity, the contractor may be tasked with facilitating the collection of stakeholder input, compiling, analyzing and assessing the input received and in incorporation of the input into the evaluation and research activities as appropriate.

4. List of intervention choices: Identify specific intervention strategies, benefits, financing options, etc.

a. Actuarial analyses: Performing actuarial analyses to evaluate likely impacts and payment options and methodologies, and simulate benchmark methodologies; provide a detailed financial impact model in the form of either a waiver cost estimate or business case model that evaluates return on investment/savings.

b. Quantitative and qualitative statistical analyses: using standardized statistical methods perform high level analyses linking outcomes to improved health, healthcare and lower costs.

c. Rapid Cycle Evaluation: The evaluation approach with CMMI is rapid cycle with close to real time production of findings.

5. Assess the evaluability of the options.

a. Using statistical tools and justifiable assumptions about likely behavioral responses under the model/demonstration, determine whether the proposed options will yield sufficient sample sizes to answer key questions.

b. Assess whether it will be possible to construct adequate comparison groups and whether the necessary data will be available.

c. Assess what other competing programs in the marketplace might dilute the impact of the intervention or contaminate the comparison group.

6. Analyze Return on Investment (ROI) for different options.

The contractor will develop estimates of ROI for various program design options in order to assist CMMI in selecting the option that is most likely to meet the ROI objectives while also meeting other objectives such as high quality health service delivery.

Develop Innovation Center Concept Brief

After the Problem and Opportunity have been defined, the Contractor shall perform tasks related to supporting CMS in designing a model/demonstration to test an innovative payment and/or service delivery model. For models/demonstrations being conducted under CMS authority, CMS leadership approves further definition of the concept into a fully formed Innovation Center Investment Plan (ICIP).

1. Develop Specific Model Design & Operational Features and clearly define what is being test, by whom, where, how long, and the specific purpose of the tests.

2. Identify beneficiary population and target geographic locations and options

3. Identify provider populations

4. Identify specific intervention strategies, benefits, and/or financing changes

5. Identify payment operational requirements and strategies

6. Identify data operational requirements and collection strategies

7. Develop preliminary evaluation approach Based on a general project design, the contractor will prepare a preliminary design detailing the relevant research questions of interest and how the questions of interest will be addressed.

The design approach should, at a minimum, specify the items listed below:

• Statement of the research questions and hypotheses to be addressed by the evaluation.

• At a high level, a description of the data and analytic approach to addressing the research questions and hypotheses.

To improve understanding of the impact of different initiatives, the

Contractor shall identify which CMMI priority measures found in the Priority Measures for CMMI Monitoring and Evaluation (PMME) document should be applied across study populations in the monitoring and evaluation of the initiative in the design plan and as updates to the measurement strategy are made.

The contractor shall also propose any additional measures specific to the initiative, study populations, and research questions, as necessary. When the contractor chooses to propose measures that are not a part of the PMME document, the Contractor shall be prepared to provide documentation outlining topics including, but not limited to, identifying where measure modifications are needed, which measures are not applicable to the evaluation at hand, and rationales for proposed measurement modifications or deviations. Where the existing measures do not meet the needs of the monitoring and evaluation at hand, the Contractor shall propose new measures to address the needs and research questions relevant to the task order.

The Contractor shall consider the appropriateness of the new measure using criteria such as, but not limited to, the strength of scientific evidence, data burden on participants, and unintended consequences of the measure. For additional information see Priority Measures for CMMI Monitoring and Evaluation (See Appendix 1)

• Design and perform simulations (repeated);

B. Solicit

1. Provide support to CMS in all aspects of administering the solicitation/application/requests for proposal (used interchangeably) process, including developing business requirements and implementing a web-based application process, providing support/help.

2. Assist CMS with the review process, including developing a scoring rubric/methodology to assign scores to proposals, identifying, training, and convening reviewers/technical expert panels, collecting their scores, and providing them with technical assistance.

3. Conduct readiness review and vet potential participants for program integrity related issues

4. Assist CMS in developing briefing materials and decision memos regarding recommendations for selection for participation in models/demonstrations

5. If needed, revise waiver cost estimates or financial models to include in clearance package.

C. Build, Run & Evaluate

Model/demonstration designs are reviewed and approved by the CMS Administrator, the Department of Health and Human Services, and the Office of Management and Budget. The Contractor may be requested to assist in the development of some non-sensitive materials describing the model/demonstration, which may be used during the government’s decision-making process. Once approved, CMS may initiate implementation and operation of the model/demonstration.

The Contractor will design and carry out an appropriate monitoring approach to identify potential unintended consequences of the program, detect whether participants make progress and quality improvements, and assess whether programmatic requirements are being adequately followed. The monitoring activity will serve as a primary source of information by tracking the operation of participants within the initiative. This includes general operational progress such as improvements in services furnished by participating providers, and performance progress such as improvements in care quality, care experience, and other measureable indications.

In supporting model/demonstration implementation and operation, the Contractor shall have the capacity to perform the following tasks

Build

1. Orientation/ongoing technical assistance to model/demonstration participants

a. Orientation/ongoing technical assistance to participants: Providing support to model/demonstration participants selected to test the payment and/or service delivery intervention(s) related to a variety of operational activities including, but not limited to the use of data, interpretation of findings, payment, learning and diffusion, evaluation, data collection, quality/performance measurement, etc.; troubleshooting for IT issues such as electronic file transfers to/from participant organizations, establishing an inquiry system/process and/or help desk to respond to inquiries; proactively address participants’ information needs and use a variety of media to provide training, including webinars, conference calls, face-to-face meetings, and written materials to educate them about protocols. Requirements will be project specific. The vendor will be expected to perform all the requirements in each Task order under the IDIQ solicitation.

b. Provide Technical Assistance for Self-Administered Design Plans: Where relevant, the Contractor shall provide technical assistance for the design of Awardees self-evaluations. The goal of this task is to ensure that the design and execution of the self-evaluation plan is feasible and informative. Further, this technical assistance should, to the extent possible, correct deficiencies in self-evaluations that would significantly impact CMS’ ability to independently evaluate the proposed projects.

However, this task should not impede progress on completing other tasks. Further, we hope that the self-evaluation skills gained through this task will be sustainable beyond the life of the projects.

2. Develop and Prepare Initiative-Specific Standard Operating Procedures &

Programmatic Protocols (i.e. payment administration, marketing, grievances and appeals, stakeholder input)

3. Develop operational requirements and systems for enrollment/attribution, including developing outreach strategy/materials, claims algorithms, etc.;

maintain master enrollment list of beneficiaries and/or providers (comprehensive and point in time)

a. Providing support to CMS and model/demonstration participants in whatever method is used for enrollment. Models/demonstrations may enroll providers and/or beneficiaries, and use a variety of methods including active (recruitment) or passive (attribution) methodologies. The Contractor shall provide support to CMS in running attribution methodologies, preparing and maintaining current lists of model participants, taking into account potential overlaps with other programs and entering them, as needed, into the CMS Master Database Management system (MDM); conduct appropriate information exchanges with the CMS Center for Program Integrity (CPI) and the Division of Technical Payment Policy, the Office of Inspector General (OIG), the Department of Justice, and other enforcement bodies, as directed by CMS. In order to be shared, data must have the appropriate level of encryption. Guidance regarding data encryption and transmission is provided by CMS as part of the data use agreement process and CMS policies for handling data. For manual exchange of data between federal agencies, privacy requirements for both agencies must be met.

4. Develop operational requirements and systems for sharing data, including processes for administering data use agreements

a. Support Data Sharing: Producing appropriate data files, establishing mechanism for transfer of data files to participants, ensuring timely transfer of data files to participants and complying with all CMS data use requirements, including administering data use agreements with all model/demonstration participants, as needed. The contractor shall ensure accurate documentation of participation/data sharing status for beneficiaries/patients and providers.

b. Distribute Data Files: The Contractor may be instructed to distribute data to CMS and to participants. The rate of such distributions may be as frequent as quarterly to all participants, and therefore may require extensive resources and IT capabilities from the Contractor. For all data distributed by the Contractor, appropriate documentation and evidence of compliance with DUAs, including the return of original datasets or documentation/evidence of their destruction or archival, shall be provided

c. Analytic Files: The Contractor shall distribute limited and focused analytic files based on claims or other evaluation data to designated persons within the Awardee sites as needed as part of the process of analysis, the contractor will be responsible for the creation of analytic data files. The creation of analytic data files to be used to answer the evaluation questions will include the development and documentation of the necessary codes, outputs, quality assurance structures and controls to guarantee the analysis results. In the approach for creating the analytic files, the Contractor must accommodate the need for rapid-cycle analysis.

The Contractor shall have an approach to the pulling of data and the analysis thereof that allows for the reporting of findings on a monthly or quarterly basis. All CDs, DVDs, data tapes, and other files shall be cleaned, edited, and ready for use for use in administering the program while it is still operational.

5. Develop operations for calculating and administering various types of payments, including care management fees, incentive payments , shared savings

a. Assisting CMS in calculating the appropriate payment rate for the selected model/demonstration participants, including using risk adjustment methodologies, maintaining payment information history for use in monitoring and evaluation, responding to participants’ payment questions, and providing general technical support to CMS for ongoing updates and adjustments to initially agreed upon payment rates.

Contractor requirements could include collecting or disbursing funds in addition to designing the approach and fee structure.

6. Develop process for responding to beneficiary/provider inquiries

a. Supporting CMS in interface with 1-800-Medicare, State Medicaid agencies, or other organizations and/or conduct other approaches for monitoring beneficiary concerns related to the administration of the model/demonstration.

7. Create/maintain systems for collecting performance data:

a. Obtain appropriate claims, survey, and clinical data for measurement, conduct necessary trainings for participants, establish new or use existing IT platforms and software as appropriate. The Contractor may be required to determine performance benchmarks, calculate quality scores, perform site reviews and audits where required, and create performance reports for participants. Information regarding nuances of Medicare claims data is available through Research Data Assistance Center, Resdac:

http://www.resdac.org/ Other information that is not able to be obtained through retrospective claims-based analyses that may need to be collected through surveys or other mechanisms is project specific. The contractor shall propose appropriate supplemental data collection techniques.

8. Develop operational requirements for monitoring

a. The Contractor will determine the need for and obtain the necessary data and conduct all relevant analyses to assist CMS in providing ongoing oversight. Such data collection and analysis activities will be based on data from participants, CMS and other secondary data, routine surveys and other primary data, ad hoc data. The frequency at which the Contractor must obtain secondary data may be as frequent as monthly and may be as frequent as quarterly collection for primary data.

Quantitative and qualitative analysis of the various types of data should be rapid, methodologically sound, robust, comprehensive, and appropriate to the respective research questions and the target audiences. Analyses may need to incorporate extensive analysis of performance and evaluation data in order to ensure that learning is focused on those elements of models where the data appears to demonstrate that the intervention has been effective or ineffective. Upon identification of potential issues of concern, the Contractor will notify CMS or a CMS Contractor with auditing responsibilities for further investigation of any suspicious patterns and trends.

9. Maintain critical systems of record

10. Administrative support

a. Provide administrative support to CMS workgroups that include CMS and HHS staff. Support may include scheduling meetings, preparing materials for meetings, developing meeting minutes and action items

Run

1. Administer beneficiary and/or provider enrollment/attribution process and maintain master enrollment list of beneficiaries (comprehensive and point in time)

a. The Contractor shall maintain records on participation status, data on attributes of the participants, data on performance scores, and produce ad hoc operational reports upon request by CMS.

2. Calculate and process various types of payments (disbursed or received).

3. Create and share data, as applicable, with model participants

4. Respond to beneficiary/provider inquiries

5. Provide technical assistance

a. The contractor shall, as required, provide technical assistance to the operating sites to improve project implementation, such as in soliciting participants, developing referrals, working with community partners, and designing processes to operate the project design."

6. Refine the model/demonstration

a. Updating any aspect of model/demonstration design and operations, as needed, to improve its success in improving health and care, and reducing costs.

7. Conduct routine and ad hoc program monitoring activities

a. The contractor shall monitor participants for non-compliance with program rules and terms and conditions of the model/demonstration and report to CMS any findings

8. Create and produce monitoring reports

a. The Contractor will design and prepare reports in the frequency and format appropriate to the data type, target audience, and evolving nature of the programs. The intended audiences of the reports are CMS and model participants. For reports to CMS, the content should be appropriate for addressing the needs for program oversight, evaluation, and learning and diffusion. For reports to participants, the content should include information that would be helpful for participants to track their progress among peers and to gauge areas for improvement. Reports to participants may include varying levels of grouping (i.e., at the state, hospital system, and hospital level or other levels) as appropriate to the research questions at hand and the program or demonstration being examined.

b. The contractor shall provide technical assistance to report recipients needed to understand and interpret the reports.

c. The formats of the reports should address all of the needs of the audiences, and may include, but not be limited to, trend reports, dashboards, customizable data tables and charts, summary statistics, and summary findings. The reports must be produced at a rapid manner, and may range from immediate notification to annual reports, depending on the content and urgency of the information to be delivered. At a minimum, the Contractor should have the capacity to generate a large volume of monitoring reports on a quarterly basis.

Evaluate

Demonstration/Model evaluation and/or research activities include the steps needed to plan, develop, and carry out evaluation of CMS-sponsored models or conduct CMS sponsored research activities, quantitative and qualitative analyses to address the research questions that are the focus of the model/research, and extensive report writing. The contractor should be prepared to meet project specific objectives which may include analyses specific to the demonstration and analyses that require extrapolation. It is anticipated that these evaluation activities may on occasion include the development of Public Use Data Files from the model/research evaluation data. .

Depending on the contract, there may be a need for the contractor to host data, but they would not be required to maintain this after the contract ends. At the conclusion of the contract, the Contractor shall provide CMS with a copy of all analytic data files generated during the study. Following are examples of activities/functions that normally would be expected to be included in many evaluation task orders:

1. Preliminary Evaluation Design

Development of a framework, including specification of testable research hypotheses, and programmatic and policy process and outcome measures of interest

a. The Design Plan shall provide a detailed and comprehensive overview of the methodological and analytical steps to be taken to address all of the purposes and research questions pertinent to the specific task order. For the evaluation, the design approach must be sufficiently detailed to comprehensively describe the intervention being tested and how the effects of the intervention will be assessed. The evaluation/research design plan so developed must address all relevant research questions for a given task order and be implementable in a robust yet cost effective manner. As part of this effort the Contractor shall design and carry out process analyses (qualitative analyses to assess project processes) and impact analysis.

b. The design approach should, at a minimum, specify the items listed below:

• Statement of the research questions and hypotheses to be addressed by the evaluation.

• How each question or hypothesis will be answered or tested by the research design.

• Addressing what data will be necessary in order to test the relevant hypotheses. For data elements necessary for the carrying out relevant analyses, the contractor shall make a critical assessment of the availability, cost to obtain, if any, and content and quality of existing data sources. The evaluation design shall further propose additional data collection activities that may be desirable and specify sampling designs and survey instruments as required.

• An analysis plan that specifies which statistical treatments of the data will be utilized.

• An analysis plan that incorporates, where appropriate, standardized metrics and analytic approaches that are coordinated within participants of each initiative or model and potentially across other Innovation Center or CMS initiatives.

• Identify quality performance strategies and metrics

• Identify awardee base data reporting requirements

• Identify preliminary learning and diffusion strategy and drivers

• Identify preliminary actuarial and scaling strategies

• Design and perform simulations (repeated)

2. Development of detailed and comprehensive overview of the methodological and analytical steps, including identification of methodological and statistical analytic techniques to be used

3. Determination of data needs and approach to obtaining data

a. If the approved Design Plan calls for primary data collection in the form of interviews, focus groups, or site visits or other approaches, the Contractor shall be responsible for developing, for each separate data collection activity, the associated materials. Where relevant, the contractor is responsible for obtaining any necessary approvals and clearances for the use of the data collection instruments created. This may include but is not necessarily limited to receiving approval for use under the Paperwork Reduction Act.

b. Create and design the materials associated with data collection. The

Contractor shall:

• Incorporate existing measures and constructs, where appropriate. The existing items should be ones that have undergone validity testing, preferably in a sample representative of the population in which the instrument will be administered.

• If an existing data collection instrument is not available or is not suitable for addressing the needs of the evaluation, the Contractor shall develop its own data collection instrument with all necessary associated materials. To the extent that pilot testing is necessary or relevant to these processes, the contractor shall carry it out in accordance with any OMB-PRA restrictions. At this time, some of the projects that will be included in the IDIQ will be subject to paperwork reduction act (PRA) requirements. Projects that are funded under Section 3021 of the ACA are PRA exempt.

4. Develop methodologies and execute calculations for financial and payment simulations in support of the model design and evaluation

5. Conduct evaluations:

a. Policy analyses: Conducting policy analyses to assess emerging issues and changing policy directions in health care using a variety of research techniques and developing a high level financial impact model for the proposed concept.

The contractor shall reviewing Medicare, Medicaid, CHIP and other Federal and State laws, regulations, policies, and programs to understand how they contribute to the problem or the opportunity. The contractor analyze the limits of any activity that CMS may undertake to address a problem, and identify potential solutions; identifying and exploring the types and sources of information available on a selected issue and assess, through limited review, the potential relevance and value of each.

6. Creation of Primary Data Collection Materials

a. Conduct Primary Data Collection Activities: The Contractor is responsible for obtaining any necessary primary data in order to carry out the agreed upon approaches in Design Plan. All activities associated with collecting, processing, quality assurance activities and editing the data shall be included. Within primary data collection activities, the following are common sources

b. Participant Submitted data: The Contractor must design the approach to collect and obtain any participant reported data. The Contractor must be able to carry out the data collection process with minimal provider burden. This may entail IT capabilities that allow the Contractor to abstract data (quantitative from sources such as claims or primary data and/or qualitative from sources such as records reviews from clinical and/financial systems or focus groups) from existing provider data systems where appropriate, and to have the capacity to receive and track large volumes of data transmitted from participants to CMS and the Contractor.

c. Case Studies: As appropriate, the Contractor shall conduct case studies to collect in-depth information that can be used primarily for three purposes: 1) to inform feedback provided to the participating sites or to CMS, 2) to inform CMS oversight of the program, and 3) to derive information that can be used to address the evaluation and monitoring questions. The Contractor must have sufficient resources to conduct large numbers of case studies in a manner that is both methodologically sound and appropriate for the scope of the initiatives.

d. Surveys: As appropriate the Contractor shall conduct surveys of Beneficiary and/or Beneficiary Proxies, Organizations, Providers, Stakeholders, and other persons or groups as relevant to the specific TO.

e. Interviews: As appropriate the Contractor shall conduct interviews with specific subject matter experts, participant staff members, Beneficiaries and/or Beneficiary Proxies, Organizations, Providers, Stakeholders, and other persons or groups as relevant.

f. Learning Network Observation: As appropriate the Contractor shall observe and document Learning Network activities necessary to address the research questions.

g. Other: As appropriate, the Contractor shall collect any necessary data as agreed upon in the design report.

7. Obtain CMS and/or State data

a. Obtain CMS, private health care sector and/or State data: The Contractor shall make arrangements to obtain all data necessary for supporting the design of the model/demonstration, including CMS claims and non-claims data, enrollment/eligibility information, and State and private health care sector data as required for the task. As part of the process of obtaining CMS data, the Contractor shall be responsible for obtaining any necessary Data Use Agreements (DUAs) and for adhering to CMS data security policies and procedures. The contractor should anticipate the need to link Medicare and Medicaid data. CMS has an ID within its data sets that links Medicare and Medicaid data at the level of the individual beneficiary. Further, CMS emphasizes the need to collect Health Identification Numbers from research participants in an effort to aid in linking Medicare and Medicaid data.

b. CMS Claims and Eligibility Data: Eligibility and claims data are available through a variety of approaches that differ in ways including the timeliness of the creation of data pulls relative to the services being examined and the relative ease of access. Under current CMS data policies, there are a number of data retrieval mechanisms for Medicare, Medicaid, and CHIP administrative data and claims.

In addition, for specific evaluation projects, the contractor may need to obtain Medicaid and/or non-Medicaid data directly from States. As warranted by the need for rapid cycle results, the Contractor must develop data flow processes to ensure the Contractor receives data in a timely fashion with an awareness of detailed nuances for each program’s data systems and strengths and weaknesses. CMS is committed to facilitating the Contractor’s access to these claims data to the extent possible; however, it is ultimately the Contractor’s responsibility to ensure they are able to effectively access and use their chosen data retrieval mechanism.

c. Non-Claims Data: The Contractor shall arrange for the timely access to non-claims data sources as relevant to the analysis needs for both participants and relevant comparison group members. The Contractor shall assess the need for, and if necessary obtain data from other data sources. CMS does not assist the contractor in collecting data.

8. Conduct Primary Data Collection Activities, including collection of participant submitted data, site visits, focus groups, surveys, and interviews

a. The data collection portion of the design plan shall specify in detail what data collection activities will be undertaken with respect to the collection of both primary and secondary data. It should, at a minimum data collection plans should contain the purpose of the data collection, the source of the data, draft data collection instruments, discussion of samples sizes and sampling technique, detailed time tables and budget estimates, and cost of obtaining the data.

b. The Contractor would also be required to prepare any required Office of

Management and Budget reports clearance packages for the implementation of the data collection plan.

9. Carry out all qualitative and quantitative analyses of primary and secondary data necessary to address the research and/or evaluation questions in accordance with the approved design plan

a. The Contractor shall carry out all qualitative and quantitative analyses of primary and secondary data necessary to address the research and/or evaluation questions in accordance with the approved design plan. The qualitative analyses should be viewed as the context within which the quantitative analyses are described. That is, the qualitative and quantitative analyses should not be approached as distinct and separate activities. One should inform the other in a very explicit and deliberate manner. The Contractor shall inform the COR of any significant or irreconcilable problems encountered with the data analysis and of any deviations from the analysis plan. Traditional qualitative research methods such as grounded theory, content, analysis, and/or phenomenology are all commonly used. If the statistical method requires such methods as Multinomial Logit or Probit models then that would be more appropriate. The contractors will be requires to produce data files as often as meets the needs of the program and is cost-effective. The timing of the distribution of data files will be impacted by data completeness and processing time. The contractor shall propose an appropriate lag regarding distribution to address these issues.

b. The contractor shall be prepared to perform analysis by type of intervention or other relevant categories. To the extent possible, the Contractor will evaluate the project using data at the individual level to answer all applicable research questions. In some cases, we anticipate the need for or benefit of grouping similar participants together to improve the generalizability and statistical precision of results, as well as benefitting from economies of scale.

10. Create and produce quarterly or monthly reports for CMS

a. The evaluation approach within CMS is rapid cycle with close to real time production of findings as feasible. It is expected that the Contractor has the capabilities to carry out rigorous regression based approaches for quarterly reports.

b. The contractor shall propose a methodology to calculate ROI. We anticipate that this will align with established and published methodologies as appropriate, and where it does not align with established and accepted methodology, justification should be provided regarding the rationale. Processes for determining ROI should facilitate cross program comparison whenever possible.

c. Findings must be available to CMS and all of the participants throughout the life of the project. Specifically, CMS will expect to receive timely information on findings that can be used by policy makers to make decisions about programmatic changes and to gauge progress. CMS expects to provide model participants with feedback reports to support their continued improvement, which may lead to modifications of interventions as participants learn and become more sophisticated. In accordance with Federal Law, all reports shall be required to be 508 compliant. Contractors are expected to be sufficiently versed in and staffed for 508 activities so as to not appreciably delay the distribution of reports after receipt of the approval of the final version of the document.

d. Other Reports -- The Contractor shall be responsible for all other reports that may need to be generated at a frequency ranging from monthly to semi-annual to an ad hoc basis. One example of such reports include quarterly program compliance reports for use by CMS for the purposes of overseeing the administration of the program in terms of adherence to the rules set forth in the participant agreements.

11. Create and produce quarterly feedback reports for model participants

a. The Contractor shall prepare quarterly or monthly reports appropriate for distribution to model participants or other designated parties involved in the administration of a model or initiative. The reports should describe the results of monitoring activities and interim evaluation findings at a granular level that allows for an examination of results at and within a participant.

12. Create and produce annual reports for CMS

a. The Contractor shall provide annual reports reflecting evaluation and/or research activities occurring in each of the contract years. Each Annual Report should be reflective of the work described in the Design Plan and the research questions.

b. The annual reports are expected to address findings of interest among significant subpopulations of interest. The annual report shall contain the key take away messages and executive summaries. Each annual report should be cumulative and summative, and should address trends and changes over time, and should be able to stand as a final evaluation product. The annual report at the end of the final option period (or the last exercised option) will be of special emphasis, and will represent the final report of the contract.

13. Create and produce Final report for CMS

i. The Contractor shall submit the one (1) original and two (2) copies of the approved Final Report to CMS upon completion of the period of performance set forth in the task order. The Final Report shall present the final results of the evaluations of each participants of the initiative and the overall assessment of the model. The final results shall address all of the research questions outlined in the Evaluation Design Report and describe the methods used to obtain these findings. The Final Report should draw from the Evaluation Design Report and build on the previous rapid cycle and annual reports.

14. Conduct evaluative support for operations, L&D and scaling

Model Learning and Diffusion

The learning and diffusion activities undertaken by the contractor will support the aims of raising awareness and engagement, achieving results, and disseminating new and promising ideas.

The learning and diffusion activities function as a vehicle to design, test, implement, and, ultimately, spread new models of care and payment delivery being developed. By serving multiple projects of varying sizes and in varying stages of readiness, contractors will coordinate across projects to ensure successful learning and implementation throughout program models.

In support of this effort, the contractor shall:

1. Manage and coordinate the learning systems and improvement networks for CMS program models, including:

• Development and execution of a results-oriented learning system including development of aims, key drivers, change strategies and measures;

• Leadership, guidance, and project management for learning communities;

• Content (curriculum) development necessary to achieve desired results, using expert faculty in the topic areas of need;

• Training in basic to advanced content of quality improvement methods and tools and operations research, including the Model for Improvement and use of the PDSA cycle. (API, Langley, et al.).

• Baseline and ongoing assessment of organizations participating in program models, supporting planning and content execution of learning community

• Faculty recruitment, management, and deployment;

• In-person meeting planning, logistics, and facilitation;

• Web-based meeting planning, logistics, and facilitation;

• Change package development and revision for program models; capturing and packaging key financial, business, clinical care and patient experience elements to facilitate structured learning and spread; and

• Development and…