FAM-25-039 Attachment O DSS Post Adoption Best Practices Toolkit #1.pdf

Post-Adoption

Best Practices Toolkit June 2024

Duke Storen, VDSS Commissioner

Toolkit #1: Fidelity, Quality Assurance;

Assessment; and Evaluation

Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Table of Contents

Purpose of the Toolkit

Fidelity assessment

Additional resources on measuring fidelity

Quality assurance and continuous quality improvement

QA and CQI strategies

Additional resources on family engagement

Additional resources for QA and CQI

Assessment activities

Family assessment instruments

Additional resources for family assessment

Evaluation of program outcomes

Evaluation fundamentals

Additional resources for evaluation planning

Conclusion

Appendix A – Logic Model

1 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Purpose of the Toolkit

In order for the Virginia Department of Social Services (VDSS) to understand the quality of the Regional Post-Adoption Consortiums (RPAC) work, the RPAC Lead Agencies are required to complete and submit monthly and annual reporting tools.

Specifically, the goal of the reports is to help VDSS understand the scope and quality of services being offered, how funding is being used to ensure service goals are met, problems encountered (such as gaps in service), training and support offered to staff, and outcomes for families receiving services.1 The purpose of the toolkit is to provide tools to evaluate implementation and outcomes and maintain/improve effectiveness of the consortium model. This toolkit describes the following (click on the link to navigate to the section):

1 Services fall under two main categories:

• Case management: Case management involves activities intended to result in the outcome of supported families and appropriate service referrals.

• Direct services: Direct services involve activities intended to result in the outcome of family stability and well-being.

Fidelity assessment Determining whether and the degree to which the activities that comprise case management and/or direct services are carried out as intended

Quality assurance and continuous quality improvement

Identifying ways to improve case management and/or direct service activities, including identifying and using strategies to encourage families to participate in case management and/or direct services

Assessment activities Using specific approaches for consistently and objectively measuring activities and/or outcomes

Evaluation of program outcomes

Identifying whether and the degree to which targeted outcomes are improving or meeting benchmarks, and, depending on the evaluation design, gaining insight into whether case management activities collectively and/or the collective activities of each direct service are the reason for any changes/differences in outcomes

VDSS Regional Post-Adoption Consortium Service (RPAC) Model

The Virginia Department of Social Services Division of Family Services (VDSS-DFS) provides funding to private and a limited number of local departments of social services (LDSS) agencies for post-adoption services and support (PAS).

In 2021 VDSS-DFS launched the RPAC approach to funding services as a way to ensure a consistent, comprehensive array of PAS to families across the Commonwealth. In order to understand how well the RPAC approach is working, Child Trends gathered data from adoptive parents, young adults who were adopted, and staff at both public and private child welfare agencies about their experiences with PAS, as well as staff competency in providing PAS.

2 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Fidelity assessment

Fidelity assessment involves using a set of criteria, or indicators, to identify whether each activity has been implemented (i.e., carried out) the way it was supposed to be. The indicators and overall fidelity assessments are customized to pertain to each individual level of case management and/or each type of direct services. The purpose of data collection using fidelity assessments is to identify areas in which implementation is functioning as intended, uncover challenges to address, and determine where adjustments may be needed.2

Table 2 is an example of how to conceptualize and organize key components of fidelity to a model, such as the RPAC model, and how to define indicators of fidelity. This includes how to measure the extent to which the indicator was accomplished, either partially or fully. All parties should agree on these definitions. The table should be updated at regular intervals, such as quarterly for the VDSS report. The number of indicators can vary for each outcome. This ideally would be based on your theory and/or logic model of how, why, and when you expect to see a change in the indicator. See Appendix A for the RPAC logic model, with indicators of success laid out according to outputs, and desired short, intermediate, and long-term outcomes.

Table 1: Glossary for Fidelity Table

Key component Outcome of interest related to key component

Indicators The items that together show whether you are accomplishing your key outcome measure

Unit of implementation What you intend to measure to determine if your outcome occurs

Data source(s) Where you will obtain data to measure your indicators

Data collection (who, when)

Who will collect the required data and at what intervals

Score for levels of implementation at unit level

What score indicates successful implementation at the single unit level such as the partner/attendee level, and what score indicates failure to achieve successful implementation. This score should be based on a review of relevant literature and discussion among members of the consortium using their practice experience.

Roll-up to next higher level if needed

What score indicates successful implementation at the composite unit level, such as at the meeting level as opposed to the partner/attendee level, and what score indicates failure to achieve a successful outcome.

Expected sample for fidelity measure

This is the number of units from whom data was collected divided by the total of all eligible units. Each level may have a different sample size, depending on the data collection plan/process.

2 Watkins, C. & Hornak, R. (2022). What is fidelity? State Implementation and Scaling-up of Evidence-based Practices.

https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/What%20is%20Fidelity%20November%202022.pdf https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/What%20is%20Fidelity%20November%202022.pdf

3 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Table 2: Fidelity Table

Example indicators

Definition Unit of implementation Data source(s) Data collection (who, when)

Score for levels Roll-up to next higher level Expected sample

Key Outcome Measure 1 – Partner Engagement

RPAC Partner attendance at meetings

Of all consortium partners, percentage at meetings

Number of invited RPAC Partners

RPAC

meeting attendance records

RPAC lead and/or staff, after each scheduled meeting

1 = 60%-100% of invited partners attend

0 = Less than 60% of invited partners attend

1 = 60%-100% of RPAC meetings achieve a score of

0 = Less than 60% achieve a score of

All RPAC meeting attendance divided by number of invited RPAC Partners to meetings

RPAC Partner attendance at trainings

Of all consortium partners, percentage a meeting

Number of invited RPAC Partners

RPAC training attendance records

RPAC lead and/or staff, after each scheduled training

1 = 60%-100% of invited partners attend

0 = Less than 60% of invited partners attend

1 = 60%-100% of RPAC trainings achieve a score of

0 = Less than 60% achieve a score of

All RPAC training attendance divided by invited RPAC Partners to trainings

Partners who complete their action steps

Of all consortium partners, percentage who complete actions steps

Number of action steps

RPAC

meeting minutes

RPAC lead and/or staff, after each scheduled meeting

1 = 60%-100% of partners completed their action steps

0 = Less than 60% of partners complete their action steps

1 = 60%-100% of partners achieve a score of 1

0 = Less than 60% of partners achieve a score of 1

All RPAC partners divided by partners who complete action steps

Place to add additional indicators

Scores 1; 0 1; 0

Composite score for all indicators 1 1 1

4 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Example indicators

Definition Unit of implementation Data source(s) Data collection (who, when)

Score for levels Roll-up to next higher level Expected sample

Key Outcome Measure 2 - Family Engagement

Eligible families receiving Case Management

Of families eligible for and requesting Case Management, percentage of families receiving Case Management

Number of families receiving Case Management

Agency data collection

Case managers, ongoing

1 = Family receives Case Management

0 = Family did not receive Case Management

1= 50%-100% of families receive Case Management

0 = Less than 50% of families receive Case Management

Families eligible for and requesting Case Management divided by families receiving Case Management

Eligible families attend support groups

Of families eligible for and requesting support groups, percentage of families attending support groups

Number of families attending support groups

Agency data collection

Case managers, ongoing

1 = Family attends support group

0 = Family did does not attend support group

1= 50%-100% of families attend support groups

0 = Less than 50% of families attend support groups

Families eligible for and requesting support groups divided by families attend support groups

Place to add additional indicators

Scores 1; 1 1; 1

Composite score for all indicators 2 2

5 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Additional resources on measuring fidelity The resources below offer additional information and checklists that can be used to track and measure fidelity to a service model when there are no predeveloped checklists available.

Evaluation Brief Measuring Implementation Fidelity

A tip sheet on why and how to measure fidelity.

Implementation and Fidelity PowerPoint Presentation

Information about key process related to implementation of evidence-based practices in child welfare and how to use and monitor fidelity.

Working with Practitioners to Develop Measures of Implementation Fidelity Evaluation tools that can help researchers understand why a program did or did not work, and practitioners make improvements in their practice.

Fidelity Monitoring Tip Sheet

Information on identifying data to collect to monitor fidelity, establishing a monitoring process, and using data collected.

Quality assurance and continuous quality improvement

RPAC Leads and their partner agencies need robust quality assurance (QA) and continuous quality improvement (CQI) processes for effective program functioning. This includes ongoing monitoring and maintenance of quality, especially in light of staff turnover, changes in community composition or characteristics, additions or changes in services and programs offered, as well as changes in the resources and funding available to PAS providers. This is also one of the areas of practice they are required to report to VDSS.

QA and CQI strategies

In this section we share strategies for creating standards for QA and CQI and highlight areas of focus for robust CQI.

https://www.jbassoc.com/wp-content/uploads/2018/03/Measuring-Implementation-Fidelity.pdf https://www.jbassoc.com/wp-content/uploads/2018/03/Measuring-Implementation-Fidelity.pdf https://www.acf.hhs.gov/sites/default/files/documents/opre/Implementation%20and%20Fidelity-oct-2021.pdf https://www.acf.hhs.gov/sites/default/files/documents/opre/Implementation%20and%20Fidelity-oct-2021.pdf https://www.mdrc.org/sites/default/files/ImplementationFidelity.pdf https://www.mdrc.org/sites/default/files/ImplementationFidelity.pdf https://opa.hhs.gov/sites/default/files/2021-03/fidelity-monitoring-tip-sheet-2020.pdf

6 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Creation and use of performance standards

• Have clearly articulated standards for staff, including benchmarks for successful staff performance and criteria for promotion, as well as performance improvement requirements

• Build in training for staff to maintain high performance

• Ensure service usage and outcome data gathered from service providers and recipients are used to inform practice

Consistent PAS performance across all regions

• Ensure consistent expectations and standards that are reviewed and adapted as necessary

Funding and budget monitoring

• Track expenditures of grant funds (e.g., rate at which funds are spent, purpose of the spending, and efficiency and value of use of funding)

• Explore other funding sources to offset costs

Family engagement

• Focus on effective recruitment and retention strategies o Maintain robust outreach including email, phone calls, newsletters, websites, social media, and in-person events, to help ensure families are aware of the program o Foster strong relationships with other service providers to facilitate appropriate referrals o Provide services that are respectful of families and responsive to their varied needs to encourage continued use of services o Continuously monitor rates of enrollment and quality of services to increase the chances that families will access and use needed servicesi

• Regularly collect satisfaction surveys from adoptive parents and adopted youth o Address any barriers to engagement families may face in a timely manner

• Include adoptive parent and youth/young adult input into program development and

CQI

o Offer adoptive parents and youth leadership roles on advisory councils and decision-making boards o Engage adoptive parents and youth as advocates, consultants, and staff o Incorporate those with lived experience—please refer to Toolkit #2 for examples

Additional resources on family engagement Below are resources that explain the benefits of engaging families and offer resources for enhancing family engagement.

7 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Family Engagement: Partnering With Families to Improve Child Welfare Outcomes

This resource explains the benefits and challenges of engaging families, and provides strategies and other resources related to family engagement.

Engagement and Persistence in Child Welfare Services: Implications for Program Effectiveness

This resource includes reasons for low participation in services and suggestions for increasing family engagement in services.

Successfully Engaging Families Formed by Adoption: Strategies for Residential Leaders

This resource provides strategies for residential staff who are caring for children who are adopted and includes strategies and tips relevant for other providers trying to improve or increase family engagement in post-adoption services.

Five Strategies for Successful Recruitment and Retention of Children and Families in Human Service Programs

A toolkit designed to increase participation in programs, with strategies for recruiting and retaining children and families in these programs, including developing effective recruitment and retention plans.

Additional resources for QA and CQI Below are additional information and resources on QA and CQI, including workbooks, videos, guidance, and frameworks that can be adapted for specific programs.

Focused CQI Services: A Guide for Agencies

Provides an overview of the CQI system assessment and implementation process, tips for identifying an assessment team, guidance on key planning activities, and information about how the Capacity

Building Collaborative can help agencies build CQI capacity.

Northern California Training Academy's Communities of Practice Hub

Continuous Quality Improvement in Child Welfare Services

Developed in partnership between the Northern California Training Academy, the Center for Social Services Research at the School of Social Welfare, the University of California, Berkeley, and Chapin Hall at the University of Chicago, the Continuous Quality https://www.childwelfare.gov/pubpdfs/f_fam_engagement.pdf https://www.childwelfare.gov/pubpdfs/f_fam_engagement.pdf https://www.urban.org/sites/default/files/2022-10/Engagement%20and%20Persistence%20in%20Child%20Welfare%20Service.pdf https://www.urban.org/sites/default/files/2022-10/Engagement%20and%20Persistence%20in%20Child%20Welfare%20Service.pdf https://www.urban.org/sites/default/files/2022-10/Engagement%20and%20Persistence%20in%20Child%20Welfare%20Service.pdf https://www.buildingbridges4youth.org/sites/default/files/Successfully%20engaging%20families%20formed%20by%20adoption(1).pdf https://www.buildingbridges4youth.org/sites/default/files/Successfully%20engaging%20families%20formed%20by%20adoption(1).pdf https://www.rand.org/content/dam/rand/pubs/tools/TL200/TL242/RAND_TL242.pdf https://www.rand.org/content/dam/rand/pubs/tools/TL200/TL242/RAND_TL242.pdf https://www.rand.org/content/dam/rand/pubs/tools/TL200/TL242/RAND_TL242.pdf https://capacity.childwelfare.gov/states/resources/focused-cqi-services-a-guide-for-agencies https://cpeonline.ucdavis.edu/courses/705/pages/continuous-quality-improvement-in-child-welfare-services https://cpeonline.ucdavis.edu/courses/705/pages/continuous-quality-improvement-in-child-welfare-services

8 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Improvement in Child Welfare Services series is a hybrid learning experience comprised of several interactive workbooks and video-based modules that focus on a number of topics that relate to the development of the CQI process.

A Framework for Child Welfare QA Systems

In this guide, the National Child Welfare Resource Center for Organizational Improvement (NCWRCOI) presents a framework for child welfare QA systems. While specific QA activities often will vary across agencies, the framework includes the broad elements all agencies should consider in creating new or energizing existing QA systems. The framework also presents the main elements in the form of implementation steps and shares examples to assist readers in conceptualizing application within their agencies.

Assessment activities

Assessment activities should enable the case manager, together with input from the family, to select goals and design services that effectively meet the needs of each family. In order to assess the extent to which the services are meeting the families’ needs, and to measure progress towards goals, it is important to ascertain what the desired outcome is and select an assessment activity that can measure progress toward that particular outcome. The table below describes the array of services funded by VDSS and delivered by the RPAC members, a description of the service, the desired outcome, and suggested activities for measuring progress towards these outcomes.

Table 3: Assessment activities to measure service outcomes

Service Description Outcome Assessment activity

Case Management, using a model

• Outreach and engagement

• Intake and screening

• Assessment

• Crisis prevention and management

• Goal/Service Planning

• Coordination of services

• Case Manager understands what services and benefits the family needs and is eligible for

• Case Manager has relevant information to

• Intake forms designed to capture relevant demographic, need for financial assistance, and social/emotional information http://muskie.usm.maine.edu/helpkids/rcpdfs/QA.pdf

9 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Service Description Outcome Assessment activity such as Success Coach3

• Resources and referrals

• Implement the service plan, including skill building, reinforcement, education, training, and support

• Advocacy

• Progress assessment and follow-up

• Case closure determine the need for Case Management

• Family receives appropriate referrals to needed services

• Family participates in development of and execution of an individualized and appropriate action plan, which may include:

o Knowledge or understanding of a topic or child’s behavior o Improved attachment-based parenting skills o Improved communication, functioning, or relationships o Greater success in conflict resolution o Knowledge or understanding of core adoption issues and trauma on the brain

• Reduction in crisis events

• Appropriate use of referral services

• Enrollment in and use of appropriate benefits

• Support and advocacy as needed

• List of questions to ask to assess level of need of parents and youth

• Tracking of service provided. Items to include (not an exhaustive list):

o Start and end dates of service o Dates for initial completion and periodic revision of the action plan o Trainings provided o Frequency, intensity, and resolution of crisis events o Referrals made, referrals accessed, and reasons for use or discontinuation of the service o Number, type, and results of outreach efforts

• Pre- and post-services questionnaires regarding parenting skills, relationships, conflict resolution, etc.

o Consider using validated tools o Consider using a widely used tool such as Family Adaptability and Cohesion Evaluation Scales IV

(FACES-IV)4

Education, Training, Support

• Education and training for staff, community, and families in adoption competency

• Events and other supportive activities for families

• Increase in staff, community, and families’ knowledge around adoption competence

• Pre- and post-training evaluations

• Attendance rates and satisfaction surveys after events

3 VDSS plans to adopt the Catawba County, North Carolina, Success Coach model and will include that in their Fiscal Year 2025 contract with RPAC leads. For more information about case management components and models, please see “Rushovich, B. (2024). Post-Adoption Case Management: Common Elements and Models. Child Trends.

4 A shorter version of the FACES –IV questionnaire is also available: https://onlinelibrary.wiley.com/doi/epdf/10.1111/jmft.12423 https://media.gopll.com/documents/FACES%20IV.pdf https://media.gopll.com/documents/FACES%20IV.pdf https://media.gopll.com/documents/FACES%20IV.pdf https://onlinelibrary.wiley.com/doi/epdf/10.1111/jmft.12423

10 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Service Description Outcome Assessment activity

Community building activities and events “R2 – Rest and Reconnect”

• Planned events to build community and give parents and children opportunity to rest and re/connect with others in similar situations

• Available activities that meet the needs and wants of families

• Greater sense of support and connection among adoptive parents and youth who are adopted

• Attendance rates to measure family’s acceptability of activities

• Satisfaction survey post R2 - rest and reconnect event

Support/ discussion groups

• Peer support activities for adoptive parents and youth who are adopted

• Greater sense of support and connection among adoptive parents and youth who are adopted

• Satisfaction surveys after peer support activities

Respite care/events

• Planned 8-hour respite to provide breaks for parents and children

• Families experience reduced stress and greater ability to cope with challenges

• Use of assessment instrument to measure stress before and after a respite event

24-hour telephone support

• Warmlines for information and support

• 24-hour hotline staff with trained professionals who can make emergency referrals

• Decrease in family stress

• De-escalation of situations, stabilization of placement, and reduction in use of emergency services

• Appropriate and timely referrals to emergency services when needed

• Warmline and hotline phone logs

• Review of hospital admissions

Education, Training

• Education and training for staff, community, and families in adoption competency

• Increase in staff, community, and families’ knowledge around adoption competence

• Pre- and post-training evaluations

Family assessment instruments

Standardized instruments provide additional information that can be used to measure outcomes pre- and post-service, and gauge progress toward treatment goals. In the table below are a few commonly used instruments for effective family assessments, according to the factor being assessed. Some instruments measure more than one factor.

Table 4: Examples of family assessment instruments

Assessment factor Assessment instrument

Patterns of Social Interaction

Family Assessment Device (FAD)

This tool uses the McMaster Model of Family Functioning and consists of 60 family-related statements to assess global family functioning. The respondents rate how well each item describes their own family. The assessment is free to use.

https://www.nctsn.org/measures/family-assessment-device https://www.nctsn.org/measures/family-assessment-device

11 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Assessment factor Assessment instrument

Assessment of Strategies in Family Effectiveness (ASF-E) This tool is used in clinical settings to assess the need for therapy and track effectiveness of family therapy. It consists of 20 items. The tool is free to use.

North Carolina Family Assessment Scale for General Services and Reunification (NCFAS-G+R).

The NCFAS-G+R measures family functioning from the perspective of the worker most involved with the family. Workers can use the tool to decide whether to take action within several domains. This tool includes the following domains: Environment, Parental Capabilities, Family Interactions, Family Safety, Child Well-Being, Youth Well-Being, Social/Community Life, Self-Sufficiency, Family Health, Caregiver/Child Ambivalence, and Readiness for Reunification.

Parenting Practices

Adult Adolescent Parenting Inventory (AAPI-2.1) This tool was created to recognize high-risk parenting behaviors and activities and is a 45 item self-report questionnaire using a rating scale.

It can be administered as a test-retest tool to measure change in parenting practice. The tool is purchased online, and cost is dependent upon the number of measures purchased.

Parenting Stress Index

This tool is used to screen for risk of abusive or neglecting parenting behaviors and practices. It evaluates social interaction traits that may have an impact on the quality of family functioning through a 120 item self-report questionnaire. There is also a short form (PSI-SF) available, with only 36 items. The cost differs by version of the instrument.

Protective Factors Survey

This is a self-administered survey that measures protective factors in five areas: family functioning/resiliency, social support, concrete support, nurturing and attachment, and knowledge of parenting/child development. Participants are asked to respond to 20 statements about their family, using a seven-point frequency or agreement scale. It is free to use.

North Carolina Family Assessment Scale for General Services and Reunification (NCFAS-G+R).

See “Patterns of Social Interaction” above.

https://friedemm.info/index.php/assessment-of-strategies-in-nursing/instrumental https://www.nfpn.org/assessment-tools/ncfas-g-r-package/ https://www.nfpn.org/assessment-tools/ncfas-g-r-package/ https://edinstruments.com/instruments/adult-adolescent-parenting-inventory-aapi-21#:~:text=The%20Adult%20Adolescent%20Parenting%20Inventory,of%20child%20abuse%20and%20neglect.

https://effectiveservices.force.com/s/measure/a007R00000v8QbCQAU/parenting-stress-index https://friendsnrc.org/evaluation/protective-factors-survey/ https://www.nfpn.org/assessment-tools/ncfas-g-r-package/ https://www.nfpn.org/assessment-tools/ncfas-g-r-package/

12 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Background and History of Parents or Caregivers

Adverse Childhood Experience Questionnaire for Adults instrument is often used to assess the effects of child abuse and neglect.

It is a 10-item self-report scale that is free to use.

Accessing the Necessities

Home Observation for the Measurement of Environment (HOME) This instrument is used to gauge a family’s ability to meet basic needs.

Several versions of the tool are available for different age groups. The tool was developed in the 1960s but was revised in 2016. It can take an hour to administer and may be purchased from HOME Inventory LLC, lrcoulson@ualr.edu and the cost is dependent on the version and number of forms purchased.

Protective Factors Survey See “Parenting Practices” above.

Domestic Violence

The Danger Assessment Tool (DA) This instrument assists in assessing the risk that an abused individual would be killed by an intimate partner. There are two parts to the instrument: a calendar to help determine the frequency and cruelty of the abuse and a 20-item scoring instrument. The tool is available online for free and in multiple languages.

Idaho Risk Assessment of Dangerousness Tool (IRAD) This tool determines both signs of lethality and potential future harm.

Seven factors are explored to identify risks related to history of domestic violence threats to kill victims and/or children; threats to harm others (e.g., parents, friends); threats of suicide; recent separation;

obsessive, coercive, or controlling behavior; prior police involvement;

and alcohol or drug misuse by the suspect. The assessment is free to use.

Mental Health, Physical Health, Physical, Intellectual, and Cognitive Functioning

Center for Epidemiologic Studies Depression Scale Revised (CESD-R)

The 20 items in CESD-R scale measure symptoms of depression in nine different groups as defined by the American Psychiatric Association Diagnostic and Statistical Manual, fifth edition: sadness, loss of interest, appetite, sleep, thinking/concentration, guilt, tired, movement, and suicidal ideation. It is free to use.

Casey Life Skills (CLS) https://www.traumainformedcare.chcs.org/resource/original-ace-questionnaire/ https://effectiveservices.force.com/s/measure/a007R00000v8QXtQAM/home-observation-for-measurement-of-the-environment https://friendsnrc.org/evaluation/protective-factors-survey/ https://www.dangerassessment.org/uploads/pdf/DAEnglish2010.pdf https://idvsa.org/initiatives/idaho-coordinated-response/ https://cesd-r.com/download/ https://www.casey.org/casey-life-skills/

13 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

CLS is a set of tools that assess the independent skills youth need to achieve their long-term goals. It aims to guide youth toward developing healthy, productive lives. It is free to use.

Substance Use

Addiction Severity Index (ASI) This tool is used to identify the services a person needs and how substance abuse affects their life. The assessment highlights seven potential areas of challenges. They are: medical status, employment and support, drug use, alcohol use, legal status, family/social status, and psychiatric status. It can take up to an hour to complete. The ASI is scored using composite scores and interviewer severity ratings and is free.

Additional resources for family assessment Below are resources with further information about the importance of using family assessment tools, as well as lists of tools and comparisons to help you select the most appropriate tool for what you want to measure.

Comprehensive Family Assessment

Information and resources on why and how to measure family assessment, including resources on identification, screening and assessment of child abuse and neglect, safety and risk screening, needs and strengths of parents and children, family progress, cultural issues, and sources for tools, instruments, and measures.

Family Assessment Tools

A list of family assessment tools, guidelines, comparisons of assessment systems, and inventory of measures for different aspects of family functioning.

Selected Assessment and Screening Tools for Use in the Family Development and Self- Sufficiency Program

This is a resource and guide for selecting and implementing assessment tools. The document includes information about the tool’s general purpose, target population, cost, training

Family Functioning Measures

This site has 40 validated family functioning tools organized into a family measurement tool matrix. These tools assess different aspects of family measurement such as organizational processes, belief systems, and communication processes.

https://adai.washington.edu/instruments/pdf/addiction_severity_index_baseline_followup_4.pdf https://www.childwelfare.gov/topics/systemwide/assessment/family-assess/ https://peerta.acf.hhs.gov/sites/default/files/public/uploaded_files/SFS%20Family%20Assessment%202-22-16_v3_508c.pdf https://humanrights.iowa.gov/sites/default/files/media/FaDSS_Menu_Selected_Assessment_Screening_Tools_FINAL_FY19_0.pdf https://humanrights.iowa.gov/sites/default/files/media/FaDSS_Menu_Selected_Assessment_Screening_Tools_FINAL_FY19_0.pdf https://humanrights.iowa.gov/sites/default/files/media/FaDSS_Menu_Selected_Assessment_Screening_Tools_FINAL_FY19_0.pdf https://lcirn.ucla.edu/resources/familyfunctioning/

14 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation requirements, and accessibility. All the tools have been researched and many have been shown to be valid and reliable.

Evaluation of program outcomes Below we discuss considerations around evaluation of effects of case management and direct services to families on post-adoption disruptions or dissolutions and offer some resources for evaluation planning. You need:

• A strong foundation, which includes ensuring your evaluation plan identifies the correct outcomes to measure and is well-designed and executed.

o This includes a clear and coherent theory of change, and a related logic model that articulates how your program leads to the outcomes you hope to achieve.

o Please note that each organization would develop its own logic model and theory of change for how it conceptualizes its own offerings.

• High quality materials, such as valid and reliable assessment and measurement tools— see Table 4 for suggestions.

Evaluation fundamentals

• Identify research questions, such as, do you want to:

o Find out if outcomes have improved over time?

o Compare outcomes across subgroups of families?

• Describe contextual factors such as:

o Family characteristics o Regional availability and access to services that may affect outcomes

• Identify how to measure your outcomes including:

o Contextual factors o Family characteristics

• Select data sources, collection approaches, and tools relevant to the needed data

• Develop an approach for analyzing the data

• Decide on the best way to report and disseminate your findings to the appropriate target audience/s

Additional resources for evaluation planning Below are resources that can help you write and implement a strong evaluation plan for your program.

15 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

The Program Manager’s Guide to Evaluation Second Edition

This resource lays out guidelines for planning and implementing a program evaluation. The specific steps in the evaluation process are described and guidance is provided on how to tailor an evaluation to your program's needs.

Plan For Program Evaluation from the Start

This resource describes the reasons for including an evaluation plan early on, and the need for involving all relevant program stakeholders in the development of the plan.

It also describes how an evaluation plan lays out the evaluation’s goals and purpose, the research questions, and information to be gathered.

Conclusion There are many factors related to how best to track, measure, and report on services provided to support families after adoption. The information in this toolkit can help PAS providers and funders, however, will need to be balance against other service priorities. We offer some recommendations for your consideration:

• Develop and use quality assurance and continuous quality improvement processes and track and measure program improvements.

• Assess the level of fidelity to the intended program model and measure adaptations needed and made.

• Engage families in service planning and provision.

o Incorporate a culturally responsive approach, and address issues related to

CQI/QA, fidelity, and outcomes.

• Use assessment tools to develop family-specific and effective service plans.

• Effectively document services, using clearly defined recordkeeping procedures, in a way that allows for quality assurance and can lead to evaluation of outcomes.

• Develop and implement an evaluation plan to assess outcomes.

i Barnes-Proby, D., Schultz, D., Jaycox, L. H., & Ayer, L.(2017). “Five Strategies for Successful Recruitment and Retention of Children and Families in Human Service Programs.” Santa Monica, CA: RAND Corporation https://www.acf.hhs.gov/sites/default/files/documents/opre/program_managers_guide_to_eval2010_508.pdf https://www.acf.hhs.gov/sites/default/files/documents/opre/program_managers_guide_to_eval2010_508.pdf https://nij.ojp.gov/topics/articles/plan-program-evaluation-start

16 Toolkit #1: Fidelity, Quality Assurance; Assessment; and Evaluation

Appendix A – Logic Model

Assumptions: The prevalence of trauma prior to adoption, the normative stressors of adoption, and other biological and experiential (e.g., physical health, cultural, and/or racial) challenges that may have existed prior to and apart from the adoption, are all possible reasons families may need assistance. To reduce the likelihood of adoption instability, parents must deeply understand the child’s history and its implications for realistic expectations, appropriate parenting strategies, and types of support the child and family are likely to need in the future; children must understand their stories, process related emotions, and form new healthy attachments; and post-adoption service providers need to understand the complexity of factors influencing well-being, so they can appropriately assess and counsel families and provide referrals to community service providers as needed.

Resources Available è

Post Adoption Activities and Inputs è

Outputs/Short-Term è

Intermediate/ Long-Term

• VDSS Post-Adoption

Grants

• Adoption Financial

Assistance

• Regional RPAC Partners

(RPAC leads and partner agencies)

• Local DSS services

• Medicaid

• Faith Community

• Private Agencies

• Health Care Providers

• Educational Institutions

• Juvenile Court

• CSA Policy and

Management

• Community Service

Providers

Staff

• Training in adoption competency and culturally appropriate service provision

• Outreach to adoptive families

• Promotion to families of available resources

• Education for community about adoption to reduce stigma and improve support array

• Coordination among services providers to facilitate referrals

Families

• Training/Preparation

• Ongoing Education

• Parent and Youth Peer Support

• Case Management

• Assistance with Birth Family Contact

• Clinical Services

• R2 – rest and reconnect

• Crisis Support

Outputs

• # of staff trained in adoption competence and culturally appropriate service provision

• # of, types, and attendance at outreach activities

• # of, types, and parents receiving promotional materials

• # of, types, and attendance at community education activities

• # and types of referrals among providers

• # of families receiving needed services

Short-term

• RPAC and LDSS staff are competent to provide service to meet the needs of adoptive families, and know where to refer families as needed

• Families know what services and support are available and reach out to providers when needed

• Families receive appropriate services from trained competent professionals

Intermediate Quality of services available to families

Gaps in service availability

Adoption competent service providers

Families access to high quality adoption competent services as needed

Long-term

• Adoptive families are stable, safe, nurturing, and permanent

• Adopted children thrive and meet their full potential as functioning adult