Safe atHome ExecutiveSummary sept28.pdf

Impact of Safe@Home on placement and permanency outcomes:

results of a quasi-experimental study

Outcomes for Unsafe Kids At-Risk of Placement

Outcomes for Unsafe Kids in Foster Care

Findings

Key Finding

Days in Out-of-Home Care Days in Out-of-Home Care o f c hi ld re n w ith o pe n ca se o f c hi ld re n w ith o pe n ca se months from case start

On average, one year from case opening, children served by Safe@Home spent 153 days in out-of-home care, compared to 228 days in the comparison group.

On average, one year from case opening, children served by Safe@Home spent 58 days in out-of-home care, compared to 221 days in the comparison group.

28% of children in foster care served by Safe@Home still had open cases after a year, compared to 57% in the comparison group.

153 days58 days

221 days 228 days

Placement Prevention

Sarah Kaye, Ph.D. & Lucia M. Reyes, Ph.D.

This summary presents results of a rigorous quasi-experimental study examining the effectiveness of Safe@ Home, implemented by Clark County Department of Family Services (CCDFS) as part of their comprehensive SAFE practice model. Children served by Safe@Home were matched to a comparison group of children served before Safe@Home was available in their community. Children were matched based on age, race/ethnicity, previous history with child welfare, and safety threats. All children in the study were determined to be unsafe by CCDFS.

kaye Implementation Evaluation&

Lower rates of out-of-home placement

Children who received Safe@Home experienced:

180 children whose families received

Safe@Home to prevent their placement in foster care were matched with 344 children in the comparison group

330 children whose families received Safe@Home after being placed out-of-home to facilitate reunification were matched with 507 children in the comparison group

87% of children served by Safe@Home achieved permanency with a parent, compared to 53% in the comparison group.

88% of children served by Safe@Home achieved reunification with a parent, compared to 53% in the comparison group.

months from case start 24 3612 24 361248 48

21% 84%

Higher rates of permanency with parent

Fewer days in foster care

Shorter time to case closure

21% of Safe@Home cases were placed out-of-home, compared to 84% in the comparison group.

Placement prevention effects were sustained for over 12 months after the end of Safe@Home.

26% of children served by Safe@Home still had open cases after a year, compared to 53% in the comparison group.

For more information about this study, please contact Sarah Kaye at sarah@kayeimplementation.com.

Safe@Home is an in-home parent skill-based intervention designed to keep children together with their families. Its goal is to minimize time in foster care or prevent out-of-home placement altogether for children that have been determined to be unsafe with their family. Safe@Home offers safety services in the categories of:

• behavior management (supervision and monitoring, stress reduction, behavior modification),

• social connection (parenting assistance, home management, social networking),

• crisis management (suicide prevention, relapse prevention, problem solving),

• separation (respite, day care, activities), and

• resource support (employment assistance, housing assistance, food, clothing, furnishings).

Safe@Home is tailored to address the unique safety threats identified by CCDFS for each family, and its service objectives, frequency, and duration can vary from family to family. Safe@Home is delivered by trained, professional, community-based Safety Managers who work with children’s caseworkers to manage safety plans and make adjustments as needed. Service intensity is customized to ensure that threats are sufficiently managed, needs are met, and children are safe while at home.

Safe@Home serves children of all ages who are determined by CCDFS to be unsafe with their families and who have cases opened for ongoing service

(1) Placement Prevention: Children and their families receive Safe@Home as part of in-home safety plans that are intended to prevent out-of-home placement.

(2) Reunification: Children and their families receive Safe@Home in order to reunify children and parents/caregivers following out-of-home placement.

Safe@Home is the community-based safety management component of a comprehensive child welfare practice model, the Safety Assessment and Family Evaluation (SAFE) developed by Action for Child Protection.

SAFE provides structured assessments and decision-making criteria that guide case activities within the public child welfare agency.1 The practice model was implemented by CCDFS during the Safe@Home study period.

Detailed information about Safe@Home is available in its intervention manual.2 Implementation supports for Safe@Home include:

• multiple family assessments,

• training for caseworkers and community-based safety managers,

• training for supervisors,

• case consultation,

• fidelity review tools, and

• implementation planning and technical assistance that is tailored to the needs and goals of the implementing agency.

For more information, visit action4cp.org/our-services/practice-model or contact Todd Holder at todd.holder@actionchildprotection.org.

Safe@Home Intervention

1Holder, T. (2021) Safety Assessment and Family Evaluation model: A systematic change based approach to public child welfare intervention. Child Welfare, 99(2).

2Holder, T., Holder, W. & Kleinedler, J. (2016). Safety Assessment Family Evaluation. Safe@Home: A community based approach to safety management [Manual]. ACTION for Child Protection.

About Safe@Home

Service Populations

The intervention study sample was drawn from children whose families received Safe@Home in Clark County, Nevada, between August 2015 and April 2019. Using population-level administrative data, children that received the intervention were matched to children from a historical comparison group who were also determined to be unsafe and served by CCDFS prior to Safe@Home implementation in their community.

Children in both service populations were matched to comparison children at the completion of the initial assessment. Children were matched based on child age, child race/ethnicity, previous in-home or out-of-home (OOH) child welfare case, and impending danger threats (i.e., specific safety threats identified by CCDFS during the initial assessment). Matching was conducted using coarsened exact matching (CEM).3-4

A focal child for each family was randomly selected after matching to account for non-independence.

3Blackwell, M., Iacus, S., King, G., & Porro, G. (2009). cem: Coarsened exact matching in Stata. The Stata Journal, 9(4), 524-546.

https://doi.org/10.1177/1536867X0900900402 4King, G., & Nielsen, R. (2019). Why propensity scores should not be used for matching. Political Analysis, 24(4). https://doi.org/10.1017/pan.2019.11 5Wilson, S. J., Price, C. S., Kerns, S. E. U., Dastrup, S. D., & Brown, S. R. (2019). Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, version 1.0, OPRE Report # 2019-56, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S.

Department of Health and Human Services. Retrieved from: https://tacfs.org/docs/psc_handbook_v1_final_508_compliant.pdf 6Cox, D. R. (1972). Regression models and life tables. Journal of the Royal Statistical Society, 34, 187–220.

7Sashegyi, A., & Ferry, D. (2017). On the interpretation of the hazard ratio and communication of survival benefit. The Oncologist, 22(4), 484–486.

https://doi.org/10.1634/theoncologist.2016-0198 8Beachy-Quick, K., Lee, C., McConnell, L., Orsi, R., Timpe, Z., & Winokur, M. (2018). SafeCare Colorado program evaluation report 2014-2017. Colorado Office of Early Childhood.

9Huhr, S., & Wulczyn, F. (2020a). Do intensive in-home services prevent placement?: A case study of Youth Villages’ Intercept® program. The Center for State Child Welfare Data. https://fcda.chapinhall.org/wp-content/uploads/2019/10/YV-Intercept-Results-1-8-2020-final.pdf 10Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8(4), 448.

11Grant, R. L. (2014). Converting an odds ratio to a range of plausible relative risks for better communication of research findings. BMJ, 348, f7450.

Research Methods Sample Selection

Analysis The evaluation team utilized several strategies to ensure a rigorous quasi experimental study, consistent with standards and procedures outlined by the Title IV-E Prevention Services Clearinghouse.5

• Analysts developed and used data quality assurance standards to ensure valid and reliable measurement.

• Statistical analyses included logistic regressions, beta regressions, and Cox proportional hazard models.

• Final statistical models controlled for child age, child race, and impending danger threats

• Analysts accounted for differences in observation periods between the intervention and historical comparison group, as well as “censored” permanency outcomes for cases that were still open at the end of the observation period.6-7

• Strategies used to analyze sustained effects were informed by analysis strategies used by other in-home parent skill-based programs rated by the Clearinghouse as supported and well-supported.8-9

• Analysts calculated effect sizes following guidance from the Clearinghouse to compare results across different statistical models.10-11

Limitations This study was limited by a retrospective design that relied entirely on administrative data, which restricted the outcomes that could be examined. Contemporaneous comparison groups are typically preferred over historical comparison groups when it is feasible. To minimize potential confounding factors due to historic comparison periods, analysts (1) carefully matched comparison cases to increase internal validity, and (2) sampled from discrete periods of time to accommodate other potential influences identified by CCDFS leadership -- most notably SAFE implementation timeframes.

Results

Both of the study populations demonstrated strong “baseline equivalence” with their matched comparison groups. Meaning, at the beginning of their cases, the intervention groups and comparison groups were similar to one another. Any minor differences between groups were statistically controlled during analysis.

Placement Prevention Reunification Safe@Home n = 180 Matched Comparison n = 344 Safe@Home n = 330 Matched Comparison n = 507 Child age, years, M(SD) 6.2 6.2 5.1 5.4

Black 23.9% 27.9% 24.6% 30.6% White 33.3% 32.9% 36.4% 34.5%

Hispanic 28.3% 28.8% 25.5% 25.3% Other/unknown 14.4% 10.5% 13.6% 9.7%

Previous In-home or OOH program 22.2% 23.0% 26.4% 24.5%

ID: Maltreatment 2.2% 2.0% 4.6% 3.9% ID: Child Functioning 3.3% 3.5% 5.5% 5.1%

ID: Parenting 71.7% 74.4% 69.7% 72.6% ID: Adult Functioning 75.6% 79.1% 88.2% 87.2%

ID= impending danger threat *None of the differences between the intervention and comparison group were statistically significant

Table 1. Characteristics of the intervention samples and their matched comparison groups

Child Characteristics

Results revealed that children who received Safe@Home experienced:

Placement Prevention Reunification Outcome d Effect Size d Effect Size

OOH placement prevention (during case episode) 1.95 Large favorable - -

OOH placement prevention (sustained for 12 months after end of services) 1.07 Large favorable - -

Length of time in OOH .81 Large favorable .22 Small favorable Case closure within a year .67 Medium favorable .75 Medium favorable Permanency with a parent .57 Medium favorable .53 Medium favorable

Study found no significant differences in maltreatment after case closure or re-entry into OOH at 6 months or 12 months.

Table 2. Summary of outcomes and effect sizes

Lower rates of out-of-home placement Fewer days in foster care

Higher rates of permanency with parent Shorter time to case closure

This rigorous quasi-experimental study offers strong initial support for average population-level treatment effects of Safe@ Home in key outcome areas that are important to child welfare systems’ goals of keeping children safe and families together.

Outcomes

The prevention of out-of-home placement effect for children in the placement prevention population was sustained for 12 months after the end of Safe@Home services.

Effect sizes describe the magnitude of the difference between the intervention and comparison groups.

Conventional cutoffs describe effects as small (d = 0.20 to .49), medium (d = .50-.79), and large (d ≥ .80).