ARFQ EDD2700000004 ADDENDUM No 2.pdf

Proc Folder: 2087372

State of West Virginia Agency Request for Quote

Educational

Doc Description: Statewide Evaluation for the 21st CCLC Program

Proc Type: Agency Contract - Fixed Amt

Date Issued Solicitation Closes

2026-10-30 2026-11-12

BID RECEIVING LOCATION

PROCUREMENT OFFICER

DEPARTMENT OF EDUCATION

BLDG 6, RM 700

1900 KANAWHA BLVD E

CHARLESTON

us

VENDOR

WV 25305

Vendor Customer Code:

Vendor Name :

Address:

Street:

City:

State:

Principal Contact :

Vendor Contact Phone:

10:30

FOR INFORMATION CONTACT THE BUYER

Jo Ann Adkins

(304) 558-2686 joann.adkins@k12.wv.us

Vendor Signature X

Solicitation No

ARFQ 0402

Country:

FEIN#

EDD2700000004

Extension:

All offers subject to all terms and conditions contained in this solicitation

Date Printed: Oct 30, 2026 Page 1

Reason for Modification:

Addendum No. 2

To provide Answer to Vendor Questions

Version

Zip:

DATE

FORM ID: WV-PRC-ARFQ-002 2020/05

ADDITIONAL INFORMATION

The West Virg inia Department of Education (WVDE) is soliciting bids on behalf of the Office of ESEA, Improvement and Support to establish a contract for a Statwide Evaluation for the 21st Century Community Learning Center Program, per the attached

Instructions to Bidders, Terms and Conditions and specifications.

IINVOICETO

DEPARTMENT OF

EDUCATION

BLDG 6, RM 700

1900 KANAWHA BLVD E

CHARLESTON WV

us

Line Comm Ln Desc

Evaluation For 21st CCLC YEAR ONE - Monthly Payment Amount

Comm Code Manufacturer

86132200

Extended Description:

Evaluation For 21st CCLC YEAR ONE

MONTHLY PAYMENT AMOUNT

ISHIPTO

DEPARTMENT OF

EDUCATION ·

OFFICE OF ESEA &

IMPROVEMENT SUPPORT

1900 KANAWHA BLVD E,

BLDG 6 RM 700

CHARLESTON WV

us

Qty Unit Issue Unit Price Total Price

12.00000 MO

Specification Model#

Inclusive of all costs, travel expenses, lodging, meals, mileage, materials, supplies, equipment and technology, administrative expenses, overhead and profit, taxes, fees, permits, and insurance. And any other costs necessary to fully perform the contract requirements.

No additional fees, surcarges, travel reimbursements, or other expenses will be paid beyond the awarded contract amount.

Date Printed: Oct 30. 2026 Page 2 FORM ID: WV-PRC-ARFQ--002 2020/05

IINVOICETO

DEPARTMENT OF

EDUCATION

BLDG 6, RM 700

1900 KANAWHA BLVD E

CHARLESTON WV

us

Line Comm Ln Desc

2 Evaluation For 21st CCLC YEAR TWO - Monthly Payment Amoun

Comm Code Manufacturer

86132200

Extended Description:

Evaluation For 21st CCLC YEAR TWO

MONTHLY PAYMENT AMOUNT

ISHIP TO

DEPARTMENT OF

EDUCATION

OFFICE OF ESEA &

IMPROVEMENT SUPPORT

1900 KANAWHA BLVD E,

BLDG 6 RM 700

CHARLESTON WV

us

Qty Unit Issue Unit Price Total Price

12.00000 MO

Specification Model#

Inclusive of all costs, travel expenses, lodging, meals, mileage, materials, supplies, equipment and technology, administrative expenses, overhead and profit, taxes, fees, permits, and insurance. And any other costs necessary to fully perform the contract requirements.

No additional fees, surcarges, travel reimbursements, or other expenses will be paid beyond the awarded contract amount.

IINVOICETO

DEPARTMENT OF

EDUCATION

BLDG 6, RM 700

1900 KANAWHA BLVD E

CHARLESTON WV

us

Line Comm Ln Desc

3 Evaluation For 21st CCLC YEAR THREE - Monthly Payment Amoun

Comm Code Manufacturer

86132200

Extended Description:

Evaluation For 21st CCLC YEAR THREE

MONTHLY PAYMENT AMOUNT

lsHIPTO

DEPARTMENT OF

EDUCATION

OFFICE OF ESEA &

IMPROVEMENT SUPPORT

1900 KANAWHA BLVD E,

BLDG 6 RM 700

CHARLESTON WV

us

Qty Unit Issue Unit Price Total Price

12.00000 MO

Specification Model#

Inclusive of all costs, travel expenses, lodging, meals, mileage, materials, supplies, equipment and technology, administrative expenses, overhead and profit, taxes, fees, permits, and insurance. And any other costs necessary to fully perform the contract requirements.

No additional fees, surcarges, travel reimbursements, or other expenses will be paid beyond the awarded contract amount.

Date Printed: Oct 30. 2026 Page 3 FORM ID: WV-PRC-ARFQ-002 2020/05

I INVOICE TO

DEPARTMENT OF

EDUCATION

BLDG 6, RM 700

1900 KANAWHA BLVD E

CHARLESTON WV

us

Line Comm Ln Desc

4 Evaluation For 21st CCLC YEAR FOUR - Monthly Payment Amou

Comm Code Manufacturer

86132200

Extended Description:

Evaluation For 21st CCLC YEAR FOUR

MONTHLY PAYMENT AMOUNT

lsHIPTO

DEPARTMENT OF

EDUCATION

OFFICE OF ESEA &

IMPROVEMENT SUPPORT

1900 KANAWHA BLVD E,

BLDG 6 RM 700

CHARLESTON WV

us

Qty Unit Issue Unit Price Total Price

12.00000 MO

Specification Model#

Inclusive of all costs, travel expenses, lodging, meals, mileage, materials, supplies, equipment and technology, administrative expenses, overhead and profit, taxes, fees, permits, and insurance. And any other costs necessary to fully perform the contract requirements.

No additional fees, surcarges, travel reimbursements, or other expenses will be paid beyond the awarded contract amount.

I SCHEDULE OF EVENTS

Event Event Date

Date Printed: Oct 30, 2026 Page 4 FORM ID: WV-PRC-ARFQ-002 2020/05

Document Phase Document Description Page

EDD2700000004 Draft Statewide Evaluation for the 21st CCLC Program

ADDITIONAL TERMS AND CONDITIO S

See attached document(s) for additional Terms and Conditions

SOLICITATION NUMBER: ARFQ EDD2700000004

Addendum Number:02

The purpose of this addendum is to modify the solicitation identified as ARFQ_EDD2600000006("Solicitation") to reflect the change(s) identified and described below.

Applicable Addendum Category:

[] Modify bid opening date and time

[ ] Modify specifications of product or service being sought

[X ] Attachment of vendor questions and responses

[ ] Attachment of pre-bid sign-in sheet

] Correction of error

[ ] Other

Description of Modification to Solicitation: To provide the mandatory pre-bid sign in sheets.

1. To provide answers to questions submitted by vendors during the solicitation process. Question and Response Session is closed.

2. No other changes.

Terms and Conditions:

1. All provisions of the Solicitation and other addenda not modified herein shall remain in full force and effect.

2. Vendor should acknowledge receipt of all addenda issued for this Solicitation by completing an Addendum Acknowledgment, a copy of which is included herewith. Failure to acknowledge addenda may result in bid disqualification. The addendum acknowledgement should be submitted with the bid to expedite document processing.

ADDENDUM ACKNOWLEDGEMENT FORM

SOLICITATION NO.: ARFO EDD2700000004

Instructions: Please acknowledge receipt of all addenda issued with this solicitation by completing this addendum acknowledgment form. Check the box next to each addendum received and sign below.

Failure to acknowledge addenda may result in bid disqualification.

Acknowledgment: I hereby acknowledge receipt of the following addenda and have made the necessary revisions to my proposal, plans and/or specification, etc.

Addendum Numbers Received:

(Check the box next to each addendum received)

[ ] Addendum No. I [ ] Addendum No. 6

[ ] Addendum No. 2 [ ] Addendum No. 7

[ ] Addendum No. 3 [ ] Addendum No. 8

[ ] Addendum No. 4 [ ] Addendum No. 9

[ ] Addendum No. 5 [ ] Addendum No. I 0

I understand that failure to confirm the receipt of addenda may be cause for rejection of this bid. I further understand that that any verbal representation made or assumed to be made during any oral discussion held between Vendor' s representatives and any state personnel is not binding. Only the information issued in writing and added to the specifications by an official addendum is binding.

Company

Authorized Signature

Date

NOTE: This addendum acknowledgement should be submitted with the bid to expedite document processing.

ARFQ EDD2700000004 - Statewide Evaluation for the 21 st CCLC Program

ADDENDUM NO.2

Vendor Questions and Agency Responses

1Q. Can required sample reports and work products be provided via electronic links within the proposal? If electronic links are not acceptable and the full work products must be included with the proposal, can they be provided in an appendix?

1R. Required sample reports and work products must be included with the proposal and may be provided in a clearly labeled appendix. Electronic links may be included as supplemental references but may not substitute .

for the required samples.

The proposal should identify the RFQ requirement{s) each sample addresses and the proposed lead evaluator's role in developing the work product. Samples must be complete and sufficiently detailed to demonstrate the required qualifications. Proprietary or personally identifiable information may be redacted, provided sufficient content remains to evaluate the work.

2Q. Addendum 1, Q13: Does 'two in-person site visits' mean two of the five visits in Spec 4.1 .2.1, or two additional visits?

2R. The two in-person visits referenced in Addendum 1, Q13, are not additional to the five required site visits. The Contractor may complete the five required site visits over the course of two in-person trips, visiting multiple grantee programs during each trip to observe program operations and fulfill the requirements of Specification 4.1.2.1.

3Q. Can you share the 2023-2024 Evaluation Report?

3R. Yes, it is attached.

Nita M. Lowey 21 st Century Community Learning Centers 2023-2024 Evaluation

West Virginia Board of Education 2023-2024

Nancy J. White , President Victor L. Gabriel, Vice President

F. Scott Rotruck, Secretary L. Paul Hardesty, Legislative Liaison

Robert W. Dunlevy, Member Daniel D. Snavely, M.D. , Member

Christopher A. Stansbury, O.D., Member Debra K. Sullivan , Member

Gregory F. Wooten , Member

Sarah Armstrong Tucker, Ph.D., Ex Officio Chancellor

West. Virginia Higher Education Policy Commission West. Virginia Council for Community and Technical College Education

Michele L. Blatt , Ex Officio State Superintendent of Schools

West. Virginia Department of Education

Table of Contents

Program Description

Backgrou nd and Context Evaluation History

Evaluation Overview and (ll.lestions

Eva luation Rationale and Philosophy

Eva luation Synopsis Eva luation Questions

Standards Use and Metaevaluation

Data Sources and Collection

Data Collection Procedures

Sch oo l-Day Teacher Survey

Program Director Survey Artifacts and Extant Data

Methods

Overview

Procedures fo r Creating Matched Samples Appro ach

Data Inclus ion Criteria

Program Operations

EQ1

EQ2

Social-Emotional Skills and Behavior

Resilience

EQ3

EQ4

EQS

EQ6

Findings

Program Operations

EQ1

EQ2

Social-Emotional Skills and Behavior

Resilience

EQ3

EQ4

Supplemental Funding

EQS

Vo lunteer Hours

Best Practices

EQ6

End-of-Year Recommendations

Commendations

Recom mendations References

Programmatic, Theoretical, and Methodological Software

Appendix A. Evaluation Logic Model

Appendix B. Teacher survey (with Consent Language)

Informed Consent of Parents/ Guardians

E-mail Notifications for Teacher Survey

Teacher Survey Questions Appendix C. Program Director Survey

Section 1. Literacy and Numeracy Section 2. Quality of programs, safe and supportive environments

Section 3. Program sustainability Section 4. Community and Family lnvolvement

Section 5. Program Quality Improvement Processes

Appendix D. Details for Technical Analyses

Evaluation Question 1: ELA and Math lmprovement Equation Set 1: Absence Rates

Appendix E. Proposed Rubric for Local Program Evaluation Reports

Evaluative Guidelines for Score Point Descriptors

Scoring Rubric

Program Description

Background and Context

The Nita M. Lowey 21 st Century Community Learning Centers (21 st CCLC) program is a formula grant program maintained by the Office of Academic Improvement within the Office of Elementary and Secondary Education at the United States Department of Education (USED) to support the "creation of community learning centers that provide academic enrichment opportunities during non-school hours for children, particularly students who attend high poverty and low-performing schools" (USED, 2022). The 21 st CCLC program was authorized under Title IV, Part B, of the Elementary and Secondary Education Act (ESEA; 20 U.S.C. 7171), which transferred administration of the program from the USED to state education agencies when No Child Left Behind was enacted. State Education Agencies (SEAs) receive the federal funding from the USED, which is then distributed to subgrantees based upon a competitive grant process.

In line with federal legislation, the West Virginia Department of Education (WVDE) administers 21 st CCLC in the state to provide opportunities for communities to establish or expand activities that:

1. provide opportunities for academic enrichment, including providing tutoring services to help students-particularly students who attend low-performing schools-to meet state and local student academic achievement standards in core academic subjects, such as reading and mathematics (ESEA § 4201(a)(1));

2. offer students a broad array of additional services, programs, and activities-such as youth development activities; drug and violence prevention programs; counseling programs; art, music, and recreation programs; technology education programs;

mathematics, science, career and technical programs; and character education programs-that are designed to reinforce and complement the regular academic program of participating students (ESEA § 4201(a)(2)); and

3. offer families of participating students opportunities for literacy and related educational development (ESEA § 4201(a)(3)).

A summary of the federally-based funding provided to the WVDE for 21 st CCLC over the past 15 years is displayed in Figure 1 for informational purposes. Using available federal funding, the WVDE provides competitive local grants to eligible organizations for supporting the implementation of community learning centers that aid student learning and development.

Eligible applicants are public and private agencies, city and county governmental agencies, faith-based organizations, institutions of higher education, and for-profit corporations.

During the 2023-2024 school year, there were a total of 43 subgrantees with active projects spanning 133 physically unique sites (with 44 public school locations, 66 community-based locations, and 23 faith-based locations). Identifying information for these subgrantees by grant type is contained in Table 1.

Funds for State Formula-Allocated Programs 21st CCLC Program for West Virginia

$9,000,000

$8,000,000 $7,805,802

$7,450,724 $7,353,442 7,032,899 $7,098,334 $6,972,211

$7,000,000 $7,313,015 $7,501,564

$6,893,813 $6,849,474 $6,814,866 $6,000,000

$5,000,000

$4,000,000

$3,000,000

$2,000,000

$1,000,000

$0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024

Figure 1. Federal funding made available to WVDE for 21 st CCLC. FY22 includes funds provided under the Consolidated Appropriations Act (2022) and the Bipartisan Safer Communities Act.

Table 1. List of grantees.

Grant Type Community Based

Grantee/County Boys and Girls Club of Parkersburg - Wood Boys and Girls Club of the Eastern Panhandle - Berkeley Coda Mountain Academy - Fayette Mountaineer Connections (Mountaineer Boys and Girls Club)* - Monongalia PATCH 21 - Wirt Playmates Child Development Centers (Soaring to the Future) - Wayne Playmates Child Development Centers (Recovery)- Wayne Playmates Child Development Centers Recovery (VEX and Reading) - Wayne Save the Children - Calhoun Save the Children - Fayette Save the Children - Gilmer Save the Children - Nicholas Southern Educational Services Cooperative (BLAST) - Fayette, Summers, Webster Southern Educational Services Cooperative -Summers Southern Educational Services Cooperative (WEST) - Webster Step by Step (Kanawha Dreamers) - Kanawha Step by Step (Chapmanville-Big Ugly Community Dreamers) - Lincoln/Logan Step by Step (Man-Buffalo Community Dreamers) - Logan Step By Step (Kanawha Dreamers Teen Centers) - Kanawha Step By Step (Logan Community Dreamers) - Logan Tygart Vally United Way (Flipside) - Marion/Taylor

Grant Type Faith Based

Grantee/County Bob Burdette Center, Inc. (Kanawha) Partnership of African American Churches - Kanawha World Vision, Inc. (KidREACH FY23) - Barbour World Vision, Inc.

(KidREACH)- Barbour

School District Cabell County Schools Clay County Schools (REACH 1) Clay County Schools (REACH 2) Greenbrier County Schools (Building E-STEAM) Greenbrier County Schools (Go Greenbrier) Greenbrier County Schools (Power: Mind. Body. Spirit) Harrison County Schools (Project ISAAC Harrison) Harrison County Schools (Project ISAAC Monongalia/Randolph) Lincoln County Schools Marion County Schools Morgan County Schools (MC After 3) Morgan County Schools (Berkeley County High School) Nicholas County Schools Preston County Schools (After School Explorers 1) Preston County Schools (After School Explorers 2)

* Fiscal agent transferred to Monongalia County Schools.

Within West Virginia, there are two platforms that support 21 st CCLC data collection activities and serve as repositories of information across each of the program sites. The first is the WVDE 21 st CCLC data application, which houses fields for data entry (e.g., program details, site information, activities, student information, attendance) and contains the ability to export reports by topic (e.g., activities, participation, staffing, outcomes). Use of this platform is primarily to collect and summarize information that is subsequently uploaded into the federal data reporting system, 21APR. The second West Virginia-specific application is the WVDE 21 st

CCLC monitoring application, which is used to meet required compliance monitoring and allows for program directors to upload evidence in support of various critical elements that demonstrate fidelity to implementation .

Online professional learning and technical assistance for 21 st CCLC is made available by USED through the You for Youth (Y4Y) professional learning community. Resources and networking opportunities exist for state coordinators, program directors, site coordinators, and other 21 st

CCLC practitioners. However, this resource site has been archived, though it is still accessible.

Additionally, the 21 st CCLC coordinators at the WVDE provide similar support for all the subgrantees throughout the state, including a robust and expansive OneNote repository that contains documentation and resources for all subgrantees.

Evaluation History

A cursory overview of the evaluation history is provided in Table 2. The highlights/recommendations contained within the table are either direct quotes or paraphrased excerpts from the final evaluation reports for the relevant years.

Revisiting the history of program evaluation activities is important for setting the expectations under which the stakeholders have interacted with the WVDE evaluator(s). The use of teacher surveys and program director surveys has been a component of all evaluations listed in Table

2. The exigency for the surveys in the current evaluation plan is continued based on their historical usefulness in making claims related to program effectiveness from critical stakeholder groups.

In general, the evaluation described for 2023-2024 builds upon program evaluation designs from more recent years. The evaluation questions and logic model were developed in 2016 and are still in use for the current evaluation due to their utility. The evaluation logic model (see Appendix A; also, see Hammer, 2018), for example, only has minor updates from previous years' evaluations. Maintaining this continuity in the evaluation studies, as desired by the Division of Federal Programs & Support at the WVDE, has allowed for trend analyses and a more complete understandingofimplementationfidelityovertime. In building upon previous years' evaluations, the methodology of the 2023-2024 evaluation now incorporates a sixth evaluation question that addresses program quality improvement processes.

Table 2. Highlights and recommendations from the past five years of program evaluation activities.

Year 2018-2019

2019-2020

Highlights/Recommendations

• Site management, as it supports quality of programs with respect to safe and supportive environments, was reported to have a small correlation with year of grant (i.e., year of program implementation), while other factors (i.e., family programming/engagement, breadth and depth of enrichment programs offered to students, and use and influence of advisory council) were reported to have moderate correlations. Primarily, t his effect could be attributed to some sites having stronger site management observed in just years one or two of grant implementation, while the other factors tend to take longer to develop.

• The results of the teacher survey, whi le parsimonious, cannot be readily interpreted as to whether or not the observed changes in the noted indicators (i.e., homework completion and class participation, and student behavior) actually led to positive outcomes (just changes).

Therefore, a modification to the instrument is needed for the 2019 -2020 program evaluation in orderto simultaneously gather information about perfo rmance and progress.

• A series of analyses aimed at the usability of the WVDE 21 st CCLC data application were conducted. In short, a new data entry system was recommended to better ensure long-term sustainability and interpretability of the data, documentation practices with in the ~ ata

Year

2020-2021

2021-2022

2022-2023

High lights/ Recomme ndations system itself, and ease of use and access for subgrantee program directors and their staff.

• Increased availabil ity of data summaries for the 21 st CCLC State Education Agency (SEA) coordinators as well as subgrantee program directors led to improvements in the fidelity of data entry, as well as enhanced data quality ver ifications. Similar summaries were recommended for standard operational use in future implementation years.

• A new data entry system was used operationally, which was widely regarded by all stakeholder groups as a substantial improvement.

• The response of the WVDE 21 st CCLC team to the coronavirus pandemic continued to be flexible and adaptive to unique needs that have arisen because of the pandemic, particularly with the renewability policy, summer addendum opportunity, and allowability of grant rollovers and rejection of grant funds.

• While the program celebrated successes and has many strengths, one recommendation was that the WVDE should consider including criteria within the RFP scoring process that considers the fidelity of implementation of previous grantees who are reapplying for a 21st CCLC grant.

• Recovery from the pandemic was reported to be a daunting task for many subgrantees. Many students spent a great deal of time isolated in ways that impacted their social and emotional learning skills, as well as their academic engagement. 21 st CCLC afterschool program grantee staff were able to provide individualized attention to help students successfully re adjust to social settings and behave in appropriate ways.

• Family members benefited from the increased communication protocols that were started initially as a result of the pandemic. Blended modality /formats for program activities were reported to have increased family involvement in 21 st CCLC activities.

• High school participants in 21 st CCLC programming were reported to have benefited with credit recovery.

• A quasi -experimenta l analysis of end-of-year assessment data found that 21 st CCLC participation led to the greatest gains in grades 04 -05 math test performance.

• Students with more than 90 days of participation were more likely to have no discipline referrals after they began participating in 21 st CCLC programming.

• In addition to academic benefits, various whole -child supports were provided . Subgrantees used evidence-based practices to help students with social and emotional issues, leading to significant behavior improvements, such as reductions in violence and substance abuse.

Students with disabilities found safer environments to build social connections. Overall, students gained confidence, improved self expression through project-based learning, and found basic security.

Many transitioned from being shy to confident, disruptive to leaders, and benefited from healthy eating habits and access to food through afterschool programs.

Evaluation Overview and Questions

Evaluation Rationale and Philosophy

Evaluation is typically defined as judging the merit or worth of a program, policy, or system (Scriven, 1991). The purpose of this program evaluation is to provide information about the implementation and outcomes of the 21 st CCLC program in West Virginia, during the period from July 2023 through June 2024. For clarification purposes, the evaluand (which is the subject/program which is being evaluated) is the 21 st CCLC program in West Virginia. The key stakeholders invested in the success of the program are the WVDE Federal Programs staff, subgrantee program directors and site coordinators, community partners, and the students and their families who are involved with the 21 st CCLC sites outside of normal school hours.

Ultimately, the evaluation serves a summative interpretation and use for the implementation of the 21 st CCLC program statewide during the 2023-2024 school year. Though, after initial client meetings with the WVDE Division of Federal Programs & Support, it has been determined that it is appropriate to include a formative component of the evaluation that addresses the desire to improve program implementation and sustainability even within the current school year.

Formative evaluations "provide information for program improvement [and] often, such evaluations provide information to judge the merit or worth of one part of a program" (Fitzpatrick, Sanders, & Worthen, 2011, p. 20). Process evaluation provides information about how to modify 21 st CCLC implementation according to how well it is currently being implemented across all sites, the barriers to success, supports for implementation, etc.

Altogether, the evaluation provides information that can be used in program decision -making such as how to increase the effectiveness and/or efficiency of program implementation at schools that have already adopted the program, as well as potential types of technical assistance to provide with future groups of subgrantees.

The evaluator for this project was Dr. Jonathan D. Rollins Ill, who works as a coordinator within the Office of Data Analysis and Research in the Division of Data Management & Information Systems at the WVDE. For this reason, the scope of any analyses was part of an internal evaluation in that it was conducted by an evaluator employed within the same SEA. However, the evaluator is not part of the Division of Federal Programs and Support, and reports to a different set of supervisors in a separate part of the organizational structure, which may reduce bias with respect to 21st CCLC.

The evaluation is defined in part by including aspects of a utilization -focused evaluation approach that centered on increasing the use of the evaluation findings. Utilization -focused evaluation is a decision-oriented approach based on two primary assumptions: (1) "The primary purpose of evaluation is to inform decisions," and (2) "Use is most likely to occur if the evaluator identifies one or more stakeholders who care about the evaluation and are in a position to use it" (Fitzpatrick et al., 2011, p. 179). Additionally, according to Patton (1997), "Intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process and findings; they are more likely to understand and feel ownersh ip if they've been actively involved" (p. 22). The evaluator has met with the 21 st CCLC WVDE coordinators to obtain their perspectives and expectations for what the evaluation will accomplish; doing so will hopefully increase use of the evaluation findings.

For purposes of this evaluation, the evaluator's role is "evaluator as teacher" (Patton, 1997).

This responsibility means that the evaluator pays careful attention to ensuring that the clients (the WVDE Federal Programs staff), as well as other stakeholders, are able to understand the findings by taking as much time as necessary to explain any findings from the evaluation.

Weekly to monthly evaluation meetings are currently being implemented to promote transparency and understanding of the program evaluation process.

Evaluation Synopsis

In fall 2016, an evaluation logic model was developed by the former program evaluator.

Stakeholder input on the logic model was gathered at the October 2016 multi-state conference in Chattanooga, Tennessee. The final logic model delineated the following outcomes:

1. High quality enrichment programs that increase students' literacy and numeracy skills;

2. High quality enrichment programs that increase students' social/ emotional skills, behavior, and resilience;

3. All programs operating as high quality, safe, and supportive environments;

4. Increasing sustainability; and

5. Increasing family and community involvement

The 2023-2024 program evaluation continues to be based on this evaluation logic model, with a new row added to address program quality improvement processes. The evaluation logic model can be found in Appendix A.

Evaluation Questions

A total of six questions were specified for this evaluation. Evaluation questions, like research questions, guide the inquiry and methodology and are tightly coupled with the overarching purpose.

The following evaluation questions (EQs) were investigated:

To what extent did:

EQl. Participation in 21& ca.c enrichment programs increase students' literacy and numeracy skills?

EQ2. Participation in 21& ca.c enrichment programs improve students' social/emotional skills, behavior, and resilience?

EQ3. Programs operate as high quality, safe, and supportive environments?

EQ4. The sustainability of programs improve?

EQS. Community and family involvement increase?

EQ6. Processes supporting program quality improvement occur?

Standards Use and Metaevaluation

The guiding practices of the evaluation will be based upon the Joint Committee on Standards for Educational Evaluation (Yarbrough, et. al., 2011). Essentially, the standards are typically used to ensure that (1) the evaluation is geared toward meeting the stakeholders' needs, (2) the process is accurate and timely, (3) the practices are ethical, (4) the evaluation is honest and trustworthy, and (5) the evaluation will have proper documentation and accountability.

The evaluator reviewed the standards to ensure that all applicable standards were addressed appropriately in design, implementation, analysis, and reporting. Limitations of the evaluation were considered as well.

As it relates to the quality of the evaluation, a metaevaluation will also be performed.

Stufflebeam (2001) provides a fairly comprehensive checklist which will be used to ensure that all reasonable steps are being taken to structure and help ensure a fair and accessible evaluation. A more recent metaevaluation checklist provided by the Joint Committee on Standards for Educational Evaluation (2018) will also be used.

The metaevaluation will be performed alongside the evaluation to verify findings and retain objectivity. Any discrepant findings will be discussed and resolved prior to reporting the results. The metaevaluation is being performed to maintain a bias-free and fair interpretation of the findings (Datta, 2000). This step is particularly important in the context of internal evaluations. For instance, trying to de-emphasize the values of the evaluator also implies not placing value on one stakeholder group over another, such as colleagues in an SEA (King & Stevahn, 2002). The benefits are not only for objectivity in reporting the findings, but also in boosting accountability and client and stakeholder protection within the evaluation. A final consideration is that the use of the Joint Committee on Standards for Educational Evaluation (2011) also brings in third-party objective guidelines, which help bolster the validity of reaching more objective conclusions.

Additionally, ethical implications of the findings must be clarified. Although an unintended consequence, the evaluation might have lead program directors to become more acutely aware of family involvement and engagement as a result of a survey and focus group discussion. While this consequence is not a bad outcome, it is important to note as a possibility. Furthermore, admissions of bias are important considerations for understanding how metaevaluation can be specifically applied to the evaluation results. The evaluator may be a single person for an unspecified period of time, which could allow the possibility of a narrower perspective in the interpretation of the findings. However, reviews of documents by other WVDE staff and the use of the metaevaluation techniques outlined above will largely mitigate any unintended outcomes.

Data Sources and Collection

Data Collection Procedures

Various data collection procedures were employed to answer each of the evaluation questions. Survey data addressing implementation questions were analyzed using qualitative and descriptive data analysis. Analysis of extant data, using matching groups of participating and nonparticipating students, allowed for comparisons of student outcomes. Details about the output implementation indicators and the outcome impact indicators can be found in the logic model in Appendix A.

The evaluation design included both quantitative and qualitative methods. Using a combination of the two methodological approaches produces a better representation of the program as a whole. Data collection consisted of multiple data sources, including surveys, focus groups, and artifacts and secondary data, as a way to increase the validity and accuracy of the evaluation find ings.

All surveys described herein were based upon a cross-sectional design (i .e., collecting data at one point in time). Surveys were administered to two stakeholder groups: school-day teachers and subgrantee program directors. Since the entire population of relevant school-day teachers and program directors received copies of their respective surveys, no sampling considerations were needed. Survey implementation considerations consisted primarily of strategies to bolster response rates in the respective populations.

The formal evaluation plan was submitted to the WVDE IRB and approved with Expedited status (IRB-WVDE-048).

School-Day Teacher Survey

Surveys were used to collect information regarding student engagement in learning from school-day teachers of students who attended the 21 st CCLC program for any duration of time, were enrolled in grades 1- 5, and whose parents had not denied consent for their participation in the study. Consent forms were given to parents at the time they registered their children;

parents were instructed that if they agreed to allow their children be part of the evaluation, no action was necessary. If they denied consent, they returned the signed form, which was kept in program directors' offices until March of each year, and then sent to the WVDE Office of Data Management and Information Systems. Denial of consent was logged into the 21 st CCLC database maintained at the WVDE by the data application developer.

The survey given to school-day teachers asked the question about student engagement in learning with a pre-post retrospective format. That is, teachers were asked to provide a rating for the students' level of engagement at the time they began working with the student versus the end of the year when they are currently answering.

Each participant received an introductory email outlining the purpose of the evaluation, the methods of data co llection in which they are being asked to participate, the timeline for the data collection, and contact information for follow-up questions. Teachers were then ema iled an anonymous link to the survey and were able to give electronic consent at the start of the survey.

A copy of the teacher survey instrument, accompanied by the consent language, can be found in Appendix B.

Program Director Survey

The overarching purpose of the program director survey was to capture self-reported measures of program effectiveness, program sustainability, and areas for improvement that the program directors have observed across each of their sites. The survey contained items that align with each of the five evaluation questions. Sections of the program director survey remain largely unchanged from the previous year's evaluation, but an effort was made in 2023- 2024 to streamline the data collection to remove similar/duplicate inquiries that are also made in annual continuation reports.

A copy of the program director survey instrument can be found in Appendix C.

Artifacts and Extant Data

Data previously collected by the program were also analyzed. Relevant items include program fliers, banners, and other written or visual representations of program activities. Additionally, existing documentation such as annual continuation reports and local evaluation reports were analyzed. As outlined in the next section in more detail, additional data were queried (for one or more years, when appropriate) and used to answer evaluation questions as they relate to the evaluation logic model: demographic information, summative assessment scores, attendance during regular school day, attendance at 21 st CCLC program site (i.e., dose strength), and teacher survey data.

Methods

Overview

The below table provides a mapping between the EQs and general aspects of the methodology, including the data sources, sample characteristics, and analyses involved. Extant data came from both the 21 st CCLC database as well as the West Virginia Education Information System (WVEIS, the statewide longitudinal data system for regular school day data). Collected data included surveys and artifacts.

More detailed information for each EQ is provided in the narrative that follows the table.

Table 3. General methodology details by evaluation question.

I Evaluation Question I Sample

I Data Source(s) Characteristics Analyses EQ1. ELA and Math • End of Year (EOY) • Matched sample of • Quasi-experimental

Improvement assessment data 21st CCLC participants design looking for

• Program director and other students within - and between -survey • Current program group differences directors as of Spring • Descriptive statistics

EQ2. Social- • EOY attendance data • Matched sample of • Quasi-experimental

Emotional (regular school day) 21•1 CCLC participants design looking for

Improvement • EOY Discipline and other students within- and between -

• Program director group differences survey

• Teach er Survey

EQ3. Safe and • Continuation/ Progress • Current program • Content and Supportive Reports di rectors as of Spring thematic analysis of

Environment • Artifacts (e.g., media 2024 narratives articles) • Artifact analysis

EQ4. Program • Program di rector • Current program • Descriptive analyses

Sustainability survey directors as of Spring • Content and

• Continuation/Progress 2024 thematic analysis of

Reports narratives

EQS. Community/ • Progr,am director • Current program • Descriptive analyses Family survey directors as of Spring • Content and Involvement • Continuation/Progress 2024 thematic analysis of

Reports narratives

EQ6. Program Quality • Program director • Current program • Descriptive statistics

Improvement survey di rectors as of Spring • Artifact analysis

• Continuation/ Progress 2024

Reports . Local Evaluation Reports

Procedures for Creating Matched Samples

Approach

Coarsened Exact Matching (CEM; lac us, King, & Porro, 2012; lacus, King, Porro, & Nielsen, 2021) is a statistical procedure that creates a matched comparison set for a given data set. For examp le, 21st CCLC is considered a "treatment," in research terms, in that it provides an intervention that is intended to increase student performance among various indicators. CEM creates a matched data set for a "control" group who did not receive an intervention through 21 st CCLC. The data are matched on an individual student basis, with the option of allowing a one-to -many match (i .e., for each 21 st CCLC student, allowing for multiple non -21 st CCLC matches t hat are weighted accordingly to account for the amount of data).

A total of seven categorical variables were used in the matching process in the below quasi experimental analyses. For categorical variables, CEM conducts an exact match across all variables. Fo r continuous variables, CEM "coarsens" it into categories/bins that can either be specified in advance of the analysis or during it. For example, grade-levels were collapsed into grade spans (i .e., K- 02, 03-05, 06-08, 09-12), and race was collapsed to be dichotomous (i.e., white, non-white). Previous year absence status was collapsed into three categories (i .e., chronically absent, needs attention, and no absence issues).

Table 4. Seven categorical variables used in the CEM procedure.

Variable School ID Grade-Span (Collapsed Categories) Gender Race (Collapsed Categories) Special Education Status Direct Certification (Proxy for low SES status) Previous Year Absence Status (Three Categories)

Data Inclusion Criteria

Variable Type Categor ical Categorical Categorical Categorical Categorical Categorical Categorical

Students were included in the "treatment" group when the following criteria were met:

• had WVEIS IDs;

• participated in 21 st CCLC in SY 2023-2024; and,

• had WVEIS outcome data during 2022-2023 and 2023-2024.

Students were excluded from the "control" group when the following criteria were met:

• did not have a WVEIS ID; and,

• participated in 21 st CCLC in either SY 2022-2023 or SY 2023-2024.

For the below analyses, the data inclusion criterion and CEM procedure yielded a total of 6,987 students who participated in 21 st CCLC programming in SY 2023-2024 and a corresponding match of 28,434 students who did not (see Table 5). Of the remaining 506 records with 21 st CCLC attendance for which no matches could be found, the primary reason was because an exact match could not be found across the list of demographic variables within each relevant school. While additional follow-up procedures could have been conducted to find matches for those records (e.g., propensity score matching, Mahalanobis distance matching), there was concern that doing so would further contaminate the more precise controls used to create the original matched sample.

Table 5. The breakdown of matched and unmatched records yielded by the CEM procedure.

All Records Matched Records Unmatched Records

Program Operations

Control Group 197,076 28,434

168,642

Treatment Grau p 7,493 6,987

Descriptive statistics are presented in tabular and graphical form to better understand 21 st

CCLC attendance as a function of grade-span as well as time of year.

EQ1. To what extent did participation in 21st CCLC enrichment programs increase students' literacy and numeracy skills?

The evaluator conducted a quasi-experimental examination of existing student general summative assessment data obtained from WVEIS in English/language arts (ELA) and mathematics for students who participated at a 21 st CCLC site during 2023-2024 compared with a matched group of students identified as non-participants. Using two years of data from the West Virginia General Summative Assessment (WVGSA), students in fourth grade through eighth grade were compared in terms of growth.

In order to determine if a given student in grades 04-08 maintained or improved (i .e., gained) their performance, scale score ranges commensurate with those used in the WVDE's accountability system, the West Virginia Balanced Scorecard, was used (see Table D1 in Appendix D). The weighted percentages of students who maintained or gained, as well as the weighted percentage of students who gained, were calculated. The percentages were weighted using the individual record weights from the CEM procedure specific to EQ1.

Specifically for this evaluation question (EQ1), the coarsened exact matching procedure was expanded to include two additional matching variables, namely prior year ELA achievement increment level (e.g., 1.1, 2.3, 3.2, etc.) and prior year mathematics achievement increment level.

This further refining of the sample meant that students were closely matched not only on the seven variables in Table 4, but also academic history variables. This further matching entailed a control group sample size equivalent to 2,309 students and a treatment group of 1,368 students. One limitation is that the analysis did not find matches for the majority of the 21 st

CCLC participants that are considered under the other evaluation questions due to the strict matching cr iteria, so the results may not be readily generalizable to the full population of students participating in 21 st CCLC.

Inferences between groups and relative differences were also calculated. The outcome measures of interest were the WVGSA scores from spring 2024 for ELA and mathematics. A quasi-experimental approach was used to compare proficiency rates and effect sizes of program participants by attendance amount for spring 2024 WVGSA scores to corresponding matches in the control group.

Additionally, self-reported data from Section 1 of the program director survey were analyzed to determine the cadence of instructional methods/ approaches that were used in afterschool to build students' ELA and mathematics competencies.

EQ2. To what extent did participation in 21st CCLC enrichment programs improve students' social/ emotional skills, behavior, and resilience?

Social-emotional skills and behavior were analyzed using four primary sources of data:

teacher survey responses, program director survey data, End of Year (EOY) discipline data, and attendance data (both regular school day and 2pt CCLC).

Social-Emotional Skills and Behavior

Similar to EQ1, descriptive statistics were used to summarize the proportion of students whose teachers perceived improvement in engagement from the start to the end of the school year by 21 st CCLC attendance bands. One limitation of the WVEIS discipline data is that many minor misbehaviors in the classroom are corrected through routine classroom management techn iques and do not require the formality of referrals. Particularly, elementary students do not receive as many referrals as secondary students, and hence the results have limited utility on their own. For the teacher survey responses, the quasi -experimental approach was not possible because there were not survey responses for non -participants. Additionally, the lowest amount of participation cons idered for the survey was 15 days of attendance in 21 st

CCLC programming(which differs intentionally from the30-day requirement which is required for reporting purposes in 21APR). Descriptive statistics were used to summarize the proportion of students whose teachers perceived improvement from the start to the end of the school year and split by 21 st CCLC attendance bands. Results are displayed in a side-by-side bar plot Historically, reported attendance bands for 21 st CCLC attendance (i.e., 1-29 days, 30-59 days, 60- 89 days, and 90+ days) were used as the grouping variable for program participants.

EOY discipline data was analyzed for those students who attended 21 st CCLC within the matched sample. For each given grade-band and 21 st CCLC attendance band, the percentage of students who did not receive any refe rrals after their first day of 21 st CCLC attendance was calculated . One limitation in these calculations is that the practice of issu ing discipline referrals often differs among schools, as well as grade -span. Furthermore, the calculations are simple in that they do not directly account for the amount of regular school day attendance and enrollment with respect to when a student began the ir 21st CCLC attendance. However, given the larger sample sizes, this lim itation may be somewhat mitigated. To provide a po int of triangulation, and to probe for what supports may be needed, program director survey data was used to provide additional context. Descriptive statistics are reported.

Resilience

Attendance data were used as a single proxy indicator for resilience. The reason for doing so is because students who are growing in resilience are anticipated to adapt in response to adverse scenarios and be integrated within the school environment on a more regular basis.

Furthermore, although attendance is an indirect indicator, it is not dependent upon subjective interpretation (either self-reported or via observation).

A linear mixed model was specified to analyze and isolate the impact of 21 st CCLC attendance on improving absence rates during the regular school day (see Equation Set 1 in Appendix D for supporting technical details). The model allowed for variability in attendance and absence rates across 21 st CCLC sites, accounted for students' regular-day school, and controlled for students' previous school year absence rates. A brief narrative is used to capture the primary findings in the Results section, but an extended set offindings is available in Appendix D for interested readers.

EQ3. To what extent did programs operate as high quaUty, safe, and supportive environments?

A thematic analysis was done of the Continuation/Progress reports submitted by grantees in the Grants and Planning System (GPS). These reports are required annually by all grantees at the conclusion of years 1-4 of their grants. Narratives presented within the Continuation/Progress reports were screened for success stories that were participant oriented and provided granular details about specific obseNable or measurable changes that occurred in behavior and/or performance. The goal of the thematic analysis was to identify convergent and divergent topics between the Continuation/Progress reports, as well as analyze how the reported successes differed across different stakeholder groups.

EQ4. To what extent did the sustainability of programs improve?

The sustainability of 21 st CCLC programming is much more involved than just securing additional funding streams. In order to provide a more complete picture, program director survey data related to professional learning and technical assistance with grant writing and fund raising was analyzed. Furthermore, descriptive statistics were calculated on self-reported supplemental funds.

Volunte…