CalWORKs_County_CSA_Report_Template_and_ACL_19-108.pdf

November 27, 2019

CALIFORNIA DEPARTMENT OF SOCIAL SERVICES

EXECUTIVE SUMMARY

ALL COUNTY LETTER NO. 19-108

This letter prepares counties for the California Work Opportunity and Responsibility to Kids (CalWORKs) Outcomes and Accountability Review (Cal-OAR) Continuous Quality Improvement (Cal-CQI) process. This letter includes the following: Cal-CQI process and requirements; details on each Cal-CQI component; report submission requirements; technical assistance, training, and county resource updates; the Cal- CQI Instruction Manual and other resources.

November 27, 2019

ALL COUNTY LETTER (ACL) NO. 19-108

TO: ALL COUNTY WELFARE DIRECTORS

ALL CALWORKS PROGRAM SPECIALISTS

ALL CALFRESH PROGRAM SPECIALISTS

ALL COUNTY WELFARE-TO-WORK COORDINATORS

ALL CONSORTIA PROJECT MANAGERS

ALL COUNTY REFUGEE PROGRAM COORDINATORS

SUBJECT: CALIFORNIA WORK OPPORTUNITY AN D RESPONSIBILITY TO KIDS

(CalWORKs) OUTCOMES AND ACCOUNTABILITY REVIEW (Cal-OAR) CONTINUOUS QUALITY IMPROVEMENT (Cal-CQI) PROCESS

REFERENCE: SENATE BILL (SB) 89 (CHAPTER 24, STATUTES OF 2017);

WELFARE AND INSTITUTIONS CODE SECTION 11523; ACL 19-40;

ALL COUNTY INFORMATION NOTICE (ACIN) I-49-18.

This letter provides County Welfare Departments (CWDs) with initial guidance for the Cal-CQI process, including reporting instructions. Information regarding Cal-OAR development was previously distributed in ACIN I-49-18, while the Cal-OAR performance measures and data file instructions were distributed in ACL 19-40.

Cal-OAR and Cal-CQI Overview

The purpose of Cal-OAR is to establish a local, data-driven program management system that facilitates continuous improvement of county CalWORKs programs by collecting, analyzing, and disseminating outcomes and best practices. Cal-OAR is designed to promote critical reflection of the CalWORKs program in order to better engage CalWORKs clients.

Cal-OAR encourages intentionality in county CalWORKs program design, service delivery models, and activity structure. This provides opportunities for adults to build core capabilities critical to manage adversity, succeed in navigating a long-term career trajectory, and balance the challenges of life, family and work. Through collaboration and partnership, counties and the state will execute a joint vision for CalWORKs http://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB89 http://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?sectionNum=11523.&lawCode=WIC http://www.cdss.ca.gov/Portals/9/ACIN/2018/I-49_18.pdf?ver=2018-08-02-151059-167 http://www.cdss.ca.gov/Portals/9/ACL/2019/19-40_ES.pdf

All County Letter No. 19-108 Page Two outcomes of interest that propel county CalWORKs programs beyond the narrow scope of the federal Work Participation Rate.

Pursuant to SB 89, Cal-OAR consists of three main components: performance measures, a CalWORKs county self-assessment process, and a county CalWORKs system improvement plan, including a peer review component.

Continuous Quality Improvement is a process by which organizations improve business processes and outcomes through regular self-assessment, analysis and ongoing data collection. Counties have a wealth of knowledge and Cal-OAR is designed to draw upon individual CWD experiences to evaluate challenges and successes, explore opportunities for overcoming challenges, and lift up promising practices and strategic, targeted, innovations.

Cal-CQI Process and Requirements

The Cal-CQI process is structured in three-year cycles. The first three-year cycle begins July 2019 and concludes June 2022.

During each three-year cycle, CWDs are required to:

• Conduct a County Self-Assessment: Includes local stakeholder engagement and a CalWORKs County Self-Assessment (Cal-CSA) Report.

• Develop a System Improvement Plan: Includes peer review participation and a CalWORKs System Improvement Plan (Cal-SIP) Report.

• Implement & Evaluate Strategies for Improvement

• Report Progress: Includes continuation of the peer review process and a Cal-

SIP Progress Report.

Page Three

The graphic below depicts the major activities within the Cal-CQI process.

The county Cal-OAR team: Establishing the County Cal-OAR team early in the Cal-CQI cycle is critical to ensure success during the Cal-OAR process. The CDSS recommends convening the county team responsible for Cal-OAR as early in the process as possible, and prior to conducting the Cal-CSA. The team will work collaboratively throughout the Cal-CQI cycle with CDSS, local stakeholders, and other relevant partners. The Cal-CQI Instruction Manual (Attachment 1) contains details on who might be included, or whom counties may find most valuable for inclusion.

Cal-CQI Process Components

The following sections provide conceptual descriptions of each Cal-CQI process component introduced in the previous section. The attached Cal-CQI Instruction Manual (Attachment 1) contains more detailed process recommendations and reporting requirements for the Cal-CQI. The manual is also available on the Cal-OAR website.

CDSS will collaborate with, and coach CWDs through, all Cal-CQI steps. Additional information regarding technical assistance can be found in the CDSS Technical Assistance and Statewide Training Section.

https://www.cdss.ca.gov/inforesources/CalWORKs/Cal-OAR

Page Four

1. The Cal-CSA

Each CWD is to conduct a county self-assessment specific to its CalWORKs program to:

• Comprehensively evaluate service array, program implementation, and participant outcomes;

• Identify strengths and challenges of current program practices and resource deployment; and,

• Describe how local operational decisions and systemic factors affect outcomes.

The Cal-CQI Instruction Manual provides specific questions CWDs must address within the Cal-CSA and optional considerations for use in assessment and analysis.

Information gathered during the Cal-CSA process will be compiled within the Cal-CSA report.

Performance Measures

The Cal-OAR consists of performance measures that align with the CalWORKs logic model and emphasize program activities that lead to meaningful participant progress toward key outcomes. An analysis of the performance measures is one component of the Cal-CSA. The Cal-CSA organizes the performance measures into groups of related measures to support a holistic assessment of the CWD’s CalWORKs program.

Attachment 2, the Cal-OAR Conceptual Framework (i.e. logic model) is a rich tool CWDs may utilize during the self-assessment process. This framework helps focus CWD attention on process points critical to improve outcomes of interest. It can also help CWDs pinpoint strategic areas for targeted program improvement, while enhancing awareness of the factors that are most under county influence.

Only a subset of the Cal-OAR performance measures will be included in the initial Cal-CSA. However, CWDs may include relevant and available information on the performance measures that are not required for inclusion. The attached measures summary (Attachment 3), along with ACL 19-40, describes the phases of implementation and what measures must be included in the first Cal-CSA.

Local Stakeholder Engagement

The CWDs are required to engage and collaborate with local stakeholders in the Cal- OAR process. Stakeholder engagement and collaboration is critical to gain additional information and insight pertaining to measure performance, strategies for improvement, http://www.cdss.ca.gov/Portals/9/ACL/2019/19-40_3.pdf

Page Five and a better understanding of the impact of system changes on CalWORKs clients and interrelated programs.

More information regarding local stakeholder engagement, including required partners and resources to aid in the stakeholder engagement process, can be found within the Cal-CQI Instruction Manual and the Cal-OAR website.

2. Peer Review Participation

County Peer Review is a required component of the Cal-OAR process and will take place following finalization of the Cal-CSA throughout Cal-SIP implementation. The CWD will document their peer review experience within the Cal-SIP and the C al-SIP Progress Reports.

Peer Review is an opportunity for CWDs to:

• Share promising and best practices;

• Discuss and brainstorm ideas to innovate and refine their CalWORKs programs;

• Discuss successes and challenges;

• Troubleshoot road blocks; and,

• Develop solutions with peer counties.

The CDSS is currently developing the peer review process. More information will be available on the Cal-OAR website. CDSS is also developing a series of tools and resources to support CWDs in the Peer Review process.

3. The Cal-SIP

Based on information gathered and reported in the Cal-CSA, each CWD will develop a plan for improving their CalWORKs program. CWDs will:

• Select a measure or set of measures for focused improvement; and,

• Develop strategies to improve performance on the chosen measures.

Information gathered during this process will be compiled within a Cal-SIP report.

Selecting Measures for Improvement

The CWDs, in partnership with CDSS, have discretion regarding which measure(s) are included in the Cal-SIP Report. Selection of measures should be strategic and targeted in order to maximize the impact of the Cal-SIP.

In selecting measure(s) for focused improvement, CWDs should consider:

• Measures that show the least growth or improvement;

http://www.cdss.ca.gov/inforesources/CalWORKs/Cal-OAR

Page Six

• Measures that a CWD can implement, and evaluate the results of, within the current Cal-CQI cycle;

• Measures that allow for the most impactful program improvement; and,

• Other measures assigned a higher priority by the CWD for various reasons (such as targeting improvement to a certain population).

The CWD will detail the process used to select the measures for improvement within the Cal-SIP, including any supporting information and data.

Developing Strategies for Improvement

The CWDs will develop strategies to improve performance on the selected measures.

These strategies may be derived or based on social service research, proven innovative practices developed by other counties, nationally recognized practices, etc. The CDSS, and a third-party partner, will provide support throughout this process, coaching counties through measure selection and implementation. All improvement efforts should fit within the Cal-OAR Conceptual Framework (i.e. logic model) and be targeted around activities that influence the outputs contributing to areas of challenge. The CWD will detail the process used to select the improvement strategies within the Cal-SIP.

Strategy implementation may be county-wide or through targeted pilots. Piloting, prior to scaling strategies county-wide, is a useful strategy to avoid change fatigue and identify implementable solutions. If a pilot or limited road test is used, the Cal-SIP must include a description on how the CWD plans to apply a successful implementation to the county as a whole, provided the pilot is successful.

When developing implementation strategies CWDs should consider the cyclical nature of Cal-OAR. Not all desired strategies can be implemented in a single cycle, and many long-term changes may require incremental steps to fully implement.

4. Implementing and Evaluating Strategies for Improvement

Once the Cal-SIP is approved by the County Board of Supervisors and CDSS, the CWD will work to implement its improvement strategies. The Cal-CQI Instruction Manual provides guidance for strategy implementation, including use of work plans to delineate activities and track task completion. If strategies will take multiple years to implement, CWDs should develop milestones that allow for incremental progress checks to allow for plan adjustments and reconfigurations, as needed.

The CWDs should also strategically consider which methods to use in order to set, track, and evaluate the impact of implemented strategies on program performance. The iterative nature of many implementation strategies should be part of this consideration.

Page Seven

There are a variety of research tools available that may be used to evaluate the effectiveness of strategy implementation. As noted above, CDSS and a third-party partner will provide support and coaching throughout this process.

5. Cal-SIP Progress Report

Each CWD will track and report measure and strategy implementation progress within the Cal-SIP Progress Report. The Cal-SIP Progress Report will include in-depth information on CWD successes and barriers in reaching performance goals and identify any necessary adjustments to improvement strategies. This summary will bridge the current and next Cal-CQI cycles.

Report Submission Requirements and Approvals

The CDSS will provide interactive templates which CWDs will use to compile the Cal- CSA, Cal-SIP and Cal-SIP Progress Report. The Cal-CSA template will be made available in the Winter of 2019, with the Cal-SIP and Cal-SIP Progress Report templates following after.

The Cal-CQI Instruction Manual includes guidance on the report approval and submission processes. All reports will be published to the CDSS website upon receiving final required approvals. For each report, the following approvals are required:

• Cal-CSA: The CWD Director is required to approve the Cal-CSA prior to submission to CDSS.

• Cal-SIP: The CWD Director is required to approve the Cal-SIP prior to submission to CDSS. Once CDSS has reviewed the Cal-SIP, CWDs are required to have the County Board of Supervisors approve the Cal-SIP. The Cal-SIP may alternatively be approved by a county’s chief elected official, where applicable.

• Cal-SIP Progress Report: The CWD Director is required to approve the Cal-SIP Progress Report prior to submission to CDSS.

Cal-CQI Due Dates

The CWDs will be organized into cohorts for reporting purposes. Report due dates are assigned by cohort and follow the same sequential timing across cohorts.

The specific schedule detailing when each CWD must submit their Cal-CSA, Cal-SIP, and Cal-SIP Progress Report is available on the Cal-OAR website.

Page Eight

CDSS Technical Assistance and Statewide Training

The CDSS will provide robust training to counties utilizing a variety of formats, focusing on the information necessary for CWDs to successfully implement Cal-OAR. The training and technical assistance will include the following components, which will be timed to provide training most relevant to the current Cal-OAR implementation step.

• Cal-OAR Webinars will cover the foundations of Cal-OAR, including the intent and development of Cal-OAR, Cal-OAR performance measures, and data flow and entry. These webinars are scheduled throughout 2019. Recorded webinars and any accompanying materials, including PowerPoint slides, hand-outs, and job aids will be made available on the Cal-OAR website after each training.

• CalWORKs Training Academy 2019 will have sessions dedicated to Cal-OAR implementation. More information is available at the CalWORKs Training Academy website.

• In-Person Regional Trainings will provide performance-based training to conduct the county self-assessment. These in-person trainings will be delivered to CWD staff in early 2020.

• Cal-OAR Forums provide county staff with an opportunity to ask questions around anything Cal-OAR related, including webinar content, official guidance, or implementation questions. These ongoing calls will occasionally have a focused topic of discussion, such as data quality assurance, peer reviews, etc. Details are available on the Cal-OAR website.

• Ongoing technical assistance and coaching will be provided throughout the Cal-OAR cycle in a structured way, as well as on an as-needed basis. CDSS staff are available to help coach counties through the Cal-CQI process itself, as well as report development. The collaborative process to determine Cal-OAR teams will begin in the Winter of 2019; county coaching will begin in January of 2020; and collaborative drafting of the Cal-CSA will begin in February of 2020.

https://www.cdss.ca.gov/inforesources/CalWORKs/Cal-OAR https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cdss.ca.gov%2Finforesources%2FCalWORKs%2FCalWORKs-Training-Academy&data=02%7C01%7CJulianna.Vignalats%40DSS.ca.gov%7C27dae863920c4b4a1ebb08d6d5612971%7C0235ba6b2cf04b75bc5dd6187ce33de3%7C0%7C0%7C636931010617987088&sdata=tipXEc7P0SM9zTjeiu0ndOaM3Hyeto310ZfHfW7tT34%3D&reserved=0

Page Nine

Cal-CQI Resources

The CDSS is developing a variety of resources for CWDs to use in the Cal-CQI process. Once available, these resources will be provided on the Cal-OAR website.

• Resource Toolkits: To assist CWDs with stakeholder engagement, peer review, resource mapping, data analysis, data quality assurance, and strategy development, implementation and evaluation.

• Data Dashboard: All Cal-OAR performance measures, by county, will be posted on the dashboard, along with other contextual data. The dashboard serves as a hub of data and information for counties to utilize in their Cal-CQI reports, as well as a tool in making program management decisions.

• Cal-CQI Cohort Structure: During the Cal-CQI process, CWDs will be organized into cohorts for reporting submission purposes, with due dates assigned to each cohort. Various characteristics were considered when developing the reporting cohorts, such as geographic region, county size, population density, and county operations.

Attached to this letter are the following resources:

• Attachment 1, Cal-CQI Instruction Manual: This document contains instructions for each step in the Cal-CQI process. While the Manual contains instructions for all measures, the measures required for the first Cal-CQI cycle will be limited, as noted in ACL 19-40.

• Attachment 2, Cal-OAR Conceptual Framework: This document provides a visual depiction of the CalWORKs program logic model and interaction with the Cal-OAR performance measures. The framework is intended to assist CWDs in assessing their CalWORKs program and to focus in on the program activities that lead to key outcomes.

• Attachment 3, Cal-CQI Measures Reference: This document provides a list of measures required for the first Cal-CSA.

These resources are also available on the CDSS Cal-OAR website.

Other resources, external to CDSS developed materials, may aid counties in the Cal- OAR process. For example, counties may have access to tools that assist in the Cal- SIP development by prompting counties to assess their current state against an aspirational state. The CDSS encourages counties to use such tools when available and appropriate in order to more efficiently and effectively engage in t he Cal-OAR process.

https://www.cdss.ca.gov/inforesources/CalWORKs/Cal-OAR

Page Ten

Contact

Additional information regarding Cal-OAR can be found at the Cal-OAR website. If you have questions concerning this letter or Cal-OAR, please contact your CalWORKs Engagement Bureau CQI Specialist at (916) 654-2137 or email the Cal-OAR inbox at Cal-OAR@dss.ca.gov.

Sincerely, Original Document Signed By:

Jennifer Hernandez Deputy Director Family Engagement and Empowerment Division

Attachments mailto:Cal-OAR@dss.ca.gov

ACL 19-108

Attachment 1

CalWORKs Outcomes & Accountability Review

Continuous Quality Improvement Instruction Manual

Family Engagement & Empowerment Division CalWORKs & Family Resilience Branch

ACL 19-108 Attachment 1

TABLE OF CONTENTS

TABLE OF CONTENTS

SECTION 1: INTRODUCTION

ABOUT THIS MANUAL

CAL-OAR BACKGROUND & DEVELOPMENT

CALWORKS CONTINUOUS QUALITY IMPROVEMENT (Cal-CQI) PROCESS

PEER REVIEW

CDSS TECHNICAL ASSISTANCE

THE COUNTY CAL-OAR TEAM

ENGAGING LOCAL STAKEHOLDERS

SECTION 2: CALWORKS COUNTY SELF-ASSESSMENT

OVERVIEW

Cal-CSA DEVELOPMENT PROCESS Cal-CSA REPORTING REQUIREMENTS

SECTION 3: CALWORKS SYSTEM IMPROVEMENT PLAN

OVERVIEW

Cal-SIP DEVELOPMENT PROCESS Cal-SIP REPORTING REQUIREMENTS Cal-SIP TARGET MEASURE SUMMARY REQUIREMENTS

SECTION 4: CAL-SIP PROGRESS REPORT

OVERVIEW

Cal-SIP PROGRESS REPORT DEVELOPMENT PROCESS Cal-SIP PROGRESS REPORT REPORTING REQUIREMENTS

APPENDIX:

QUESTIONS FOR SYSTEM ANALYSIS

INITIAL ENGAGEMENT

ONGOING ENGAGEMENT

SUPPORTIVE SERVICES

EDUCATION

EMPLOYMENT AND WAGES

EXITS/REENTRIES

SECTION 1: INTRODUCTION

ABOUT THIS MANUAL

This instruction manual outlines the California Work Opportunity & Responsibility to Kids (CalWORKs) Outcomes and Accountability Review (Cal-OAR) Continuous Quality Improvement process with guidance to County Welfare Departments (CWDs) on how to complete required reporting elements and meet other statutory requirements for Cal- OAR. It provides the basic process, tools, and resources for engaging in program assessment and system improvement.

This manual includes five core sections:

Section 1. Introduction – includes general Cal-OAR information

Section 2. County Self-Assessment – includes process and requirements

Section 3. System Improvement Plan – includes process and requirements

Section 4. Progress Reports – includes process and requirements

Section 5. Appendix – includes additional analysis questions for the Cal-CSA

Additional resources and toolkits are available on the

CDSS Cal-OAR website.

CAL-OAR BACKGROUND & DEVELOPMENT

Cal-OAR was developed by the California Department of Social Services (CDSS) in partnership with stakeholders, including; current and former CalWORKs clients, CWDs, the County Welfare Director’s Association, workforce representatives, community college and education system representatives, Tribal Temporary Assistance to Needy Families (TANF) programs, advocate and research organizations, Legislative staff, as well as a variety of other subject matter experts.

The purpose of the Cal-OAR is to establish a local, data-driven program management system that facilitates Continuous Quality Improvement of the CalWORKs program by collecting, analyzing, and disseminating outcomes and best practices. The Cal-OAR was established by Senate Bill 89 (Chapter 24, Statutes of 2017) and codified in Welfare and Institutions Code Section 11523.

Throughout the Continuous Quality Improvement process, CWDs will: analyze their CalWORKs programs, using the Cal-OAR process and outcome measures; identify areas for improvement and design a system improvement plan; implement improvement

California Department of Social Services Cal-OAR: Continuous Quality Improvement http://www.cdss.ca.gov/inforesources/CalWORKs/Cal-OAR http://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB89 http://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=WIC&sectionNum=11523.

strategies; and, provide regular progress updates. The Cal-OAR process is structured in three-year Continuous Quality Improvement cycles.

The Cal-OAR includes both process and outcome performance measures. Both the process and outcome measures shall be used to inform the CalWORKs county self-assessment and CalWORKs system improvement plan.

For the process measures:

• Data reported during the first three-year cycle will establish both county and state-wide baselines

• And, after the first three-year cycle, target thresholds will be established in consultation with counties and other stakeholders

For the outcome measures:

After the first three-year cycle, statewide standards may be established if there is enough reason

CALWORKS CONTINUOUS QUALITY IMPROVEMENT (Cal-CQI)

PROCESS

The Cal-CQI is an iterative process that includes an in-depth self-assessment of the county CalWORKs program, development of a plan to improve the county CalWORKs program, implementation of the improvement plan, and progress updates. Each component is multi-step and includes certain reporting requirements.

California Department of Social Services Cal-OAR: Continuous Quality Improvement

During each three-year Cal-CQI cycle, CWDs will produce and submit three written reports:

1. CalWORKs County Self-Assessment (Cal-CSA)

2. CalWORKs System Improvement Plan (Cal-SIP)

3. Cal-SIP Progress Report

An overview of each required written report is included below. Specific details on the requirements relating to each written report can be found in Sections 2, 3, and 4 of this manual.

1. CalWORKs County Self-Assessment

• The purpose of the Cal-CSA is for the CWD to comprehensively assess their CalWORKs program.

• CWDs are to identify strengths and weaknesses of current program practices and resource deployment and describe how local operational decisions and systemic factors affect program outcomes.

• The Cal-CSA helps CWDs identify areas of focus and best practices for the Cal- SIP. Allowable services and activities may be implemented or enhanced as strategies within the Cal-SIP.

• CWDs are to actively consult and collaborate with local stakeholders while completing the Cal-CSA.

2. CalWORKs System Improvement Plan

• The purpose of the Cal-SIP is for the CWD to describe how it will improve its CalWORKs program performance based on information learned during the Cal-

CSA.

• The CWDs will focus on either a single measure, multiple measures, or a programmatic grouping of a related measures for strategic, focused improvement throughout the remainder of the three-year cycle.

• The CWD will describe the improvement strategies to be implemented.

• The Cal-SIP will include a peer review element, in order for each CWD to gain additional insight and technical assistance (see Peer Review).

• The Cal-SIP must be approved by the CWD’s County Board of Supervisors, or, if applicable, Chief Elected Official.

3. Cal-SIP Progress Reports

• The purpose of the Cal-SIP Progress Report is for the CWD to describe and analyze the progress made since the Cal-SIP.

• The Cal-SIP Progress Reports shall describe CWD successes and/or barriers in reaching performance goals and identify any necessary changes to the Cal-SIP.

• One Cal-SIP Progress Report is required during each three-year Cal-CQI cycle.

PEER REVIEW

Peer Review is required within the Cal-SIP process. It provides the opportunity for CWDs to gain insight from one another regarding program implementation, operation, and improvement strategies. This instruction manual provides basic information regarding what is required of CWDs to meet the Cal-OAR statutory requirements for Peer Review (see Peer Review). Additional resources for CWDs consideration, can be found on the Cal-OAR website.

CDSS TECHNICAL ASSISTANCE

CDSS will provide technical assistance to CWDs throughout the Cal-OAR process, including training. CDSS technical assistance and training will include, but is not limited to:

• Calculation of performance measures.

• Coaching through and assistance with development, review, and certification of the Cal-CSA, Cal-SIP and Cal-SIP Progress Reports, including ongoing check-ins to review interim progress.

• Due dates for all three Cal-CQI reports.

• Coordination of Peer Review process, including matching CWDs for review.

• Resources for use during county peer review.

• Engagement regarding performance measure data and analyses, including ongoing check-ins to review interim change.

California Department of Social Services

• Assistance with identification of system improvement strategies and dissemination of best practices in service delivery.

• Access to resources and information only available at the state level, including connecting CWDs to resources both within and outside of CDSS.

• Access to current literature and research on operationalizing program improvement.

• Methods and strategies for engaging local stakeholders.

THE COUNTY CAL-OAR TEAM

Establishing the County Cal-OAR team early in the Cal-CQI cycle is critical to ensure success in assessing the CalWORKs program, and implementation of improvement strategies. The team will work collaboratively throughout the Cal-CQI cycle with local stakeholders and other relevant partners.

The County Cal-OAR team should include all CWD staff responsible for and contributing toward the Cal-CQI cycle, including a point of contact with CDSS. In addition, a county may want to engage a broader group of county staff for consultation, including those in leadership roles, program managers/specialists, front line managers and caseworkers, and staff who work in data collection/processing.

ENGAGING LOCAL STAKEHOLDERS

Stakeholder engagement and collaboration at the local level is a required element of the Cal-CQI cycle Engagement and collaboration is critical to gain additional information and insight related to measure performance, strategies for improvement, and to better understand the impact of system changes on CalWORKs clients and interrelated programs.

Required local stakeholders include:

• County CalWORKs administrators, supervisors, and caseworkers

• Current and former CalWORKs clients

• County human services agency partners

To the extent possible and relevant, local stakeholders should also include:

• Community College representatives

• Tribal organizations

• Local Workforce Development Board

• Adult Education providers

• Domestic abuse service providers

• Local housing and continuum of care programs

• County behavioral health departments

• County drug and alcohol programs

• Community-based service providers

• Organizations that represent CalWORKs clients (e.g. legal services organizations)

• Child care resource and referral programs

• Alternative payment programs

Additional information and resources for engaging the local stakeholder community can be found in subsequent sections of this instruction manual and on the CDSS Cal-OAR Website.

Client Satisfaction Survey

To promote stakeholder engagement within the Cal-OAR process, CDSS, Mathematica Policy Research, and current and former CalWORKs clients have collaboratively developed a CalWORKs client satisfaction survey. Data collected from this survey is a rich source of information available to CWDs for the Cal-CQI cycle. More information on how to use survey data within the Cal-CQI cycle can be found in the Cal-CSA Section, within the Stakeholder Engagement and Feedback and Performance Measure Analysis subsections.

California Department of Social Services

SECTION 2: CALWORKS COUNTY SELF-ASSESSMENT

OVERVIEW

The Cal-CSA is the first step of the Cal-CQI cycle and is completed once every three years. Through the Cal-CSA, CWDs will critically assess their CalWORKs program, analyze performance measures, identify strengths and weaknesses, identify and describe how local operational decisions and systemic factors effect outcomes, identify best practices, and identify areas for system improvement. The assessment guides the CWD in determining where improvement efforts should be focused to maximize positive outcomes for CalWORKs clients. The Cal-CSA will be utilized by the CWD as a starting point for the Cal-SIP.

This section provides a suggested self-assessment process flow, and instruction on the Cal-CSA reporting requirements. As an overview, the Cal-CSA report includes the following sections:

• Executive Summary – a high level overview of information contained in the Cal-CSA

• Introduction – a brief introduction to the Cal-CSA report, including standardized Cal-OAR background language

• Demographics – demographic data and analysis

• Agency Characteristics – overview of the county infrastructure, community partners, and financial/material resources available to the county CalWORKs program

• Stakeholder Engagement and Feedback – a discussion of the stakeholder engagement that occurred during the Cal-CSA report, including information gathered on the CalWORKs client experience and results from the client satisfaction survey

• Performance Measure Analysis – includes performance measure data, general analysis, system analysis, practice changes to consider, and stakeholder feedback

• Barrier Removal Services – discussion on the availability of and client engagement in barrier removal services

• Summary of Cal-CSA Findings – summary of the Cal-CSA

• Appendix – space for the CWD to include evidence of the CWD Director’s approval of the Cal-CSA, details on the team who contributed to developing the Cal-CSA, and tables, charts, and graphics referenced throughout the Cal-CSA report.

Reporting requirements and instructions are detailed in the Cal-CSA Reporting Requirements subsection.

Cal-CSA DEVELOPMENT PROCESS

The following is a suggested approach for developing the Cal-CSA report. Steps which are noted as ‘required’ must be completed by the CWD, pursuant to statute.

Convene the County Cal-OAR Team

• Convene the CWD staff responsible for completing the Cal-CSA - see The County Cal-OAR Team subsection. Assess the engagement need and availability of CalWORKs line staff and their supervisors; their insight and feedback are key in understanding program outcomes and spurring ideas for improvement.

• Begin consulting with CDSS on the Cal-CSA report.

• Consider initiating contact with key local stakeholders to determine availability and participation interest.

Gather and Organize Data

• CDSS will calculate the CWD’s performance measures using data reported by CWDs and other data sources (see ACL 19-40 for data sources). These primary data sources have been selected to establish a standard of information.

• When other data is available to the CWD that provides a more comprehensive representation of the measure, the CWD may (at its option) include the measure calculation utilizing the secondary data source(s) in the Performance Measure Analysis subsection. This will be provided, in addition to, the CDSS calculation, not in place of.

• Secondary data sources are county specific quantitative data sources. When a secondary data source is used, the CWD must:

o Annotate the data source alongside the data o Provide a description of the data source o Explain why the data source was selected.

California Department of Social Services

• Organize data to share with stakeholders and aid in analysis; this should include demographic data elements, performance measures, and other relevant contextual information that aid in performance measure(s) interpretation.

Engage Stakeholders

• Select methods for stakeholder consultation; this may include, but is not limited to: large stakeholder meetings, focus groups, surveys, and interviews.

• Refer to Engaging Local Stakeholders subsection for the list of required stakeholders.

• Engage and collaborate with stakeholders in a meaningful discussion regarding performance measures data, client satisfaction survey results (See Client Satisfaction Survey subsection), and other relevant information to inform the Cal-CSA.

The county will utilize the information collected from stakeholder engagement and through the client satisfaction survey to support the Cal-CSA.

Analyze Data

• Data analysis will be conducted to: assess performance on established measures; identify areas of success, weaknesses in current practice, and barriers to system improvement; explore areas for focused improvement efforts.

• CWDs may utilize information from previous Cal-CQI cycles to identify what has worked well and where additional improvements are needed (not applicable during first Cal-CQI cycle).

• Refer to Cal-CSA Reporting Requirements for details on what must be considered in the analysis.

Compile the Cal-CSA Report

• CWDs are required to organize the analysis into the Cal-CSA report and prepare the Cal-OAR Report Signature Sheet.

• Details on the required Cal-CSA report components can be found in the Cal-CSA Reporting Requirements subsection.

• Utilize the associated Cal-CSA Template and this Instruction Manual to prepare the Cal-CSA report.

Cal-CSA Report Approval

• Obtain County Director approval and submit the approved Cal-CSA and Cal-OAR Report Signature Sheet to CDSS.

• CDSS will engage the CWD in a collaborative review process (via email and/or conference call). The CWD will answer questions and adjust the Cal-CSA, as needed.

• Once finalized, the report will be posted publicly on the CDSS Cal-OAR website.

Note regarding Data Deidentification: To protect the privacy of individuals, CDSS follows the data deidentification guidelines developed by the California Health and Human Services Agency. Data deidentification procedures must be followed when data elements (and any percent values calculated from the data) contain fewer than 11 individuals. If the data used to calculate a performance measure(s) or associated sub-rate/demographic breakdowns contains fewer than 11 individuals, the rate cannot be shared publicly. Cal-CSAs will follow these data deidentification policies prior to online posting.

Cal-CSA REPORTING REQUIREMENTS

The Cal-CSA Report will include the following components:

1. Executive Summary

The Executive Summary is a high-level overview of the information in the Cal-CSA; it should not exceed two pages. The CWD should identify:

• Common themes found in Cal-CSA.

For example, strengths or barriers that impact multiple facets of the program, key demographic data, and resources.

• Current strategies that may be considered as promising practices for peer counties.

• Areas of the program that may benefit from peer review/support.

• Which measure(s) are being considered for the Cal-SIP.

2. Introduction

In the introduction section of the Cal-CSA, the CWD provides a brief introduction to the Cal-CSA report, including:

• Standardized Cal-OAR background language (this can be found in the Cal-CSA template).

• Describe how the CWD approached the Cal-CSA report.

• Methods used to gather stakeholder feedback.

3. Demographics

In this section the CWD identifies and describes the overall county population, CalWORKs population, and conducts a demographic analysis. The CWD should refer to data provided in this section to support analysis in subsequent sections of the Cal- CSA. Frequencies and rates should be included where applicable; tables or graphs may be included in the Appendix.

Please note: CDSS will provide the county profile and demographic data elements, unless otherwise noted. Please refer to the CDSS Cal-OAR website data toolkit for details and data sources of each data element/information.

County Profile The following data elements and information will be included in the County Profile.

• Poverty and Deep Poverty rates

• Child poverty rates

• Median household income

• Unemployment rate

• Primary employment industries (top five)

• Emerging employment sectors – optional information, to be included by the CWD

(please reference the CDSS Cal-OAR website data toolkit for details on where they information may be found)

• Proportion of adults with a high school degree or equivalent

• Proportion of adults with a bachelor’s degree or equivalent

• County or Continuum of Care Families Homelessness Rate

• Rural population indicator

• List of federally recognized tribes

Demographic Data The following data elements and information will be included for both the overall county population (when available), and the county CalWORKs population.

• Average Age of Adults (adults are those 18 years of age or older)

• Percentage of households/CalWORKs assistance units that have a child four years of age or younger

• Gender identity of Adults

• Sexual orientation of Adults

• Race/Ethnicity of Adults

• Spoken language of Adults

If the CWD is able to differentiate the above data points by regions within the CWD (e.g.

urban versus rural, by zip code, or county office), CDSS recommends that the CWD provide data specific to each region.

Demographic Analysis Provide an analysis of data presented in the Demographic Data subsection.

California Department of Social Services

At a minimum, the subsection should capture:

• Strengths and needs of the communities served.

• How does the county demographic profile compare with CWD and statewide demographics?

For example, is the poverty rate higher in the county, is the median household income lower?

• Analysis of how the overall CWD demographic makeup impacts the ability of the CWD to provide services to the CalWORKs population.

• Challenges or opportunities these findings present for the CWD and how this information aids in the identification of Cal-SIP strategies.

• Analysis of the variation in demographics and service delivery by region, if applicable, and how the CWD may target service delivery based on this information.

• Significant changes or trends in the demographic profile since the submission of the last Cal-CSA, and how those changes have impacted the CWD’s performance measures and service delivery (not applicable during first Cal-OAR cycle).

4. Agency Characteristics

In this section, the CWD shall provide information regarding CWD infrastructure, relevant partnerships with local groups, and details regarding available financial/material resources:

County Infrastructure Provide a description and analysis of the CWD organizational structure and workforce.

This may include the following:

Organizational Structure:

• Description of the CWD’s overall structure, identifying where the CalWORKs program falls within that structure.

• How the CWD has operationalized CalWORKs to meet the needs of the population served (See the Demographics section to support analysis).

• Description of how the organizational structure impacts CalWORKs business practices and service delivery (this information may also be explored within applicable performance measure analysis sections).

• Past changes that have impacted county practice, provision of services, and performance measures (where applicable).

County Offices:

• The number and locations of CalWORKs offices. Specify urban versus rural.

• For each office specify: operations, service array, and other services offered (i.e.

other human service or workforce programs).

• Discuss whether eligibility and Welfare-to-Work (WTW) services are co-located, use of call-centers.

• Describe office location in relation to the geographic dispersion of poverty within the county.

Staff Functions:

• Describe how cases are assigned to workers and whether specialized caseload units exist.

• Describe average caseload size by service component, including specialized staff (e.g. those serving domestic abuse survivors).

Workforce:

• Describe staff onboarding and ongoing training requirements.

• Discuss skill gap or organizational/workforce risks

For example, high vacancy/ turnover rates, worker skill level, average time in job and/or average years of experience working with families, succession planning.

Please note: CWD staff are required participants in the Cal-CSA report. CDSS recommends including staff perspective in this analysis. This may be accomplished by conducting confidential focus groups with direct service staff and their supervisors.

Partnerships Partnerships are critical for service delivery in the CalWORKs program. This section of the Cal-CSA report will detail how the CWD collaborates with community partners and include an analysis of the impact these partnerships have on service delivery.

Below are examples of potential partners:

• Governmental (i.e. behavioral health; child welfare and child support; housing and homelessness; resettlement agencies; probation and court systems;

community colleges and adult education; regional and local workforce development boards and job centers; board of supervisors; school districts; local law enforcement; public health)

• Community Based Organizations (i.e. providers of domestic abuse, substance abuse and mental health services; food banks; cultural centers; child care providers; training centers; legal and civil rights services; veterans services;

resource and referral agencies)

• Any other relevant stakeholders in the community, including but not limited to private sector and Tribal TANF Partners

Discuss how the CWD collaborates and coordinates with community partners. At a minimum, the discussion should consider:

• How do current partnerships impact business practices and service delivery?

• Are there any pending or potential partnerships that may impact service delivery?

• What service/partnership gaps currently exist? What impact does that have on service delivery, and what strategies or long-term plans does the CWD have to address these gaps?

If specific partnerships impact a performance measure, a brief description should be included in this section with further analysis in the appropriate Performance Measure Analysis or Barrier Removal sections.

Financial/Material Resources This section is intended to provide an overview of unique resources available to the CWD and the impact these resources have on the county CalWORKs program.

At a minimum, the discussion should describe:

• Additional funding streams or material resources (i.e. funding in addition to the program allocations provided through CDSS) which support the CWD and have an impact on services provided to the CalWORKs population.

• How these funding streams or material resources impact the CWD’s performance measures.

• Whether and why any specific program allocations provided through CDSS (e.g.

specific allocations for mental health or similar services) within the past three fiscal years were only partially used, or if CWD expenditures in an area exceed its allocation during those years.

5. Stakeholder Engagement and Feedback

Stakeholder engagement and collaboration is a critical element of the Cal-CSA, and the Cal-CQI. Stakeholder feedback and contributions should be included within the body of the report, both in the specific stakeholder feedback sections as well as integrated into the overall analysis.

This section provides the CWD with space to discuss the overall stakeholder engagement process, participants, contributions made by stakeholders, client experience, and results of the client satisfaction survey.

Overall Stakeholder Engagement The CWD shall summarize and provide results of the stakeholder engagement process.

CWDs will list the stakeholders engaged in the process in Appendix B: Cal-OAR Team and Stakeholders.

At a minimum the discussion must include:

• Describe the process used to identify stakeholders.

• Describe how the CWD engaged stakeholders, including the method and frequency of engagement

• Results of stakeholder engagement – a summary of feedback should be included here, with references to appropriate performance measure analysis where additional, more complete analysis is to be conducted.

Client Experience CalWORKs clients are required participants in the Cal-CSA report. The CWD shall discuss the CalWORKs client experience with the program.

At a minimum the discussion should include:

• Client experience at CalWORKs offices, including: average wait time, comfort and security of the office, whether a play area is provided for the children, etc.

• The reasons why, and the ways in which, families utilize the CalWORKs program.

• Other client outreach methods utilized, as applicable (e.g. focus groups).

Client Satisfaction Survey CalWORKs clients can provide unique information on the CalWORKs program, strengths, and challenges. The CWD shall discuss the client satisfaction survey results and how they were utilized throughout the Cal-CSA process.

When results from the survey necessitate further analysis and understanding, CDSS suggests that CWDs engage clients in focus groups. Please refer to the Stakeholder Engagement Toolkit on the CDSS Cal-OAR website for additional information on focus groups.

This discussion shall include:

• A summary of findings from the client satisfaction survey.

• Where the results of the survey align with measure performance results, and where they diverge.

• How did the survey results influence the county measure analysis?

6. Barrier Removal Services

Barrier removal services are critical to assure CalWORKs clients can succeed in WTW and on a path toward economic mobility. In this section, the CWD discusses the availability of and client engagement in the following key barrier removal services:

• Domestic Abuse Services

• Mental Health Services

• Substance Abuse Services

• Learning Disabilities Screening, Assessment and Accommodations

The CWD may also use this space to discuss any other barrier removal services it offers to ensure client success in the program (e.g. language services, legal expungement). The below subsections offer guiding questions which may be used by

California Department of Social Services

CWD’s to provide the information required for Cal-CSA barrier removal services reporting.

In the Barrier Removal Services section CWDs shall report the following: (1) Key Data points found in the section below, (2) the process by which the CWD identifies, offers, and refers clients to barrier removal services, (3) the utilization of and client access to barrier removal services, and (4) if applicable, a description of specialized trainings or service providers related to delivering barrier removal services.

Key Data For each data point, the CWD must report:

Please note: CDSS will provide the data elements related to the Learning Needs Screening only.

• Average monthly number of OCAT recommendations for:

o Domestic abuse services o Substance abuse services o Mental health services

• Average monthly number of individuals referred for:

o Domestic abuse services o Substance abuse services o Mental health services

• If available, of the individuals referred for domestic abuse, substance abuse, and/or mental health services:

o What proportion have an exemption, good cause, or program waiver?

o What proportion are receiving these services through the Family

Stabilization program?

o What proportion are also receiving Homeless Assistance or Housing

Support Program services?

• Average monthly number of individuals that completed the Learning Needs Screening during the OCAT appraisal o Average monthly number of referrals generated for the Learning Needs Evaluation from the OCAT appraisal

• Average monthly number of individuals that waived the Learning Needs Screening during the OCAT appraisal

Barrier Identification, Offer of Services, and Referral Process For each type of barrier removal service, provide a description of:

• The most common points within the program in which client barriers are identified.

• When in the CalWORKs process are barrier removal services discussed, and how the CWD packages or “markets” services to clients.

• The way barrier removal services are discussed with clients and how the CWD helps clients make informed decisions.

• The steps taken to refer a client for barrier removal services once the barrier is identified.

• How the CWD follows up to see if clients are receiving services.

• How safety plans are used in the CWD for survivors of domestic abuse.

• How and…