SFY2627_Budget_OSU_RESEA.docx

Ohio RESEA Evaluation Statement of Work

FY26-27

Tian Lou, Ph.D.

The Ohio State University Ohio Education Research Center

Joshua Hawley, Ed.D.

The Ohio State University Ohio Education Research Center

Table of Contents

Glossary ADRF: Administrative Data Research Facility APPAM: Association for Public Policy Analysis and Management DOL: Department of Labor EDR: Evaluation Design Report ETA: Employment and Training Administration IRB: Institutional Review Board RESEA: Reemployment Services and Eligibility Assessment RCT: Randomized Controlled Trial SNAP: Supplemental Nutrition Assistance Program SWIS: State Wage Interchange System ODJFS: Ohio Department of Jobs and Family Services OLDA: Ohio Longitudinal Data Archive OMJ: Ohio Means Jobs Q2 Employment: Employment two quarters after receiving UI benefits UI: Unemployment Insurance UI weeks: number of weeks claimants staying on UI benefits WPRS: Worker Profiling and Reemployment Services WIOA: Workforce Innovation and Opportunity Act

1. Project Description The Ohio State University (OSU) is collaborating with Ohio Department of Job and Family Services (ODJFS) to conduct a statewide randomized controlled trial (RCT) to evaluate the effectiveness of the Reemployment Services and Eligibility Assessment (RESEA) program. The program provides job search assistance to Unemployment Insurance (UI) claimants who are likely to exhaust their benefits. Our major goal is to meet the federal requirements for RESEA program evaluation, which includes examining whether the RESEA program can reduce the number of weeks UI claimants stay on UI benefits and whether it can increase UI claimants’ employment rate two quarters after receiving the first UI benefits. Through the current work with ODJFS, we have identified additional outcome metrics, such as earnings and workforce program participation, and the need of supplementing the quantitative analyses with qualitative analyses, which will help ODJFS and the evaluation team understand how the RESEA program is implemented at the county level.

In FY24-25, OSU has developed a data flow process and an algorithm for conducting the randomization on a weekly basis. The process was tested in a pilot study in three workforce areas (Area 1, Area 11, and Area 19) between April and July 2024. OSU has also facilitated focus groups with the three workforce areas to learn how the randomization process was working at the county level. The statewide randomization officially started in mid-July 2024. OSU delivers a monthly report to ODJFS to track the total number of claimants involved in the randomization and their demographic characteristics and job histories. By mid-February 2025, over 20,000 eligible claimants have been included in the randomization. As part of the implementation study, the OSU evaluation team visited four representative workforce areas (Area 4, Area 11, Area 12, and Area 16) to interview RESEA program managers and advisors to gather information about RESEA service delivery.

OSU proposes to renew the RESEA contract with ODJFS for FY26-27 to continue the evaluation work. Specifically, OSU expects to maintain the ongoing statewide randomization throughout the calendar year 2025 to ensure the randomization sample is large enough for detecting statistically meaningful results. The current randomization is a three-arm design, in which we equally assign eligible claimants to three groups: 1) treatment group 1: two one-on-one appointments with RESEA advisors; 2) treatment group 2: two one-on-one appointments and one workshop; 3) control group: no RESEA services. The three-arm design requires a larger sample size than the two-arm design in the original proposal. Based on the historical data, we plan to randomize around 40,000 claimants but will update the power analysis to get a more precise number whenever outcome data becomes available. After the randomization is done, OSU will analyze the causal impacts of RESEA services on participants’ outcomes and produce reports to document the randomization process and results.

In addition, OSU will continue the implementation study to further ODJFS’s understanding of the RESEA service delivery model, RESEA staff onboarding and turnover, and challenges faced by both claimants and RESEA advisors. The OSU evaluation team will hold quarterly check-in meetings with the four representative workforce areas, provide timely synthesis pieces, and write a final report by the end of the contract period, including all the detailed information gathered throughout the study.

Finally, ODJFS has received additional funding to expand the evaluation work to assess whether ODJFS should increase RESEA service capacity to include Supplemental Nutrition Assistance Program (SNAP) recipients who are eligible for UI benefits but are not in the RESEA selection pool (referred to as SNAP-UI project throughout the rest of the proposal). The analytic work is expected to be done in the Administrative Data Research Facility (ADRF). OSU will assist with the research design, weekly randomization, data transfer between ODJFS, CHRR, and ADRF, data analysis, and report writing for this expanded work.

2. Project Activities The entire RESEA evaluation work has four stages, including pre-evaluation assessment, pilot study, implementation study, and impact study. There are also administrative activities during each stage. The project activities listed in this section focus on the work during FY26-27, as well as the evaluation work for the SNAP-UI project.

2.1. Pilot study report

Between late April and early July 2024 (11 weeks in total), the pilot study randomization has been implemented in three workforce areas, which represent rural (Area 1), urban (Area 11), and Appalachian (Area 19) counties in Ohio. During the pilot study, we streamlined the weekly randomization data transfer and the biweekly service outcome data transfer between OSU and ODJFS, troubleshooted data format, such as removing claimants without valid identifiers and county information, and excluded certain population from the randomization, such as claimants with 100 Worker Profiling and Reemployment Services (WPRS) scores.[footnoteRef:2] This practice ensured that the randomization algorithm was tested before used at the statewide level, making the process a lot smoother and more efficient. [2: Claimants with 100 WPRS scores are required to receive RESEA services and therefore cannot be included in the randomization pool.]

Over one thousand claimants were included in the pilot study. The service outcome data indicates that all three pilot areas delivered service accurately based on the randomization assignments. Almost no one in the control group received RESEA services, ensuring that the control group provides a great counterfactual for the treatment groups. While nearly 35% of the claimants in each treatment group did not receive services or received partial services, this was expected, as some claimants may have returned to work before they were scheduled for or completed RESEA services. This should not influence how we estimate the impacts of the RESEA program, as we treat each treatment group as a whole and estimate the overall impacts of RESEA services.

While the sample size of the pilot study is not large enough to detect statistically significant results, we can still utilize the statistics obtained in this process, such as the average number of weeks on UI, employment rates, and their standard deviations, to preview the potential differences in outcomes between the treatment and the control groups. Additionally, these statistics can help update the power analysis, providing a more precise estimate of the minimum sample size required for the statewide randomization.[footnoteRef:3] [3: We also need to interpret the outcomes from the pilot study with caution. The pilot study randomization was implemented during the late Spring and early Summer. It’s likely that some of the UI claimants in the sample had seasonal jobs and would return to work faster or slower than the claimants from other quarters, depending on their industries and occupations. We will discuss more details in the pilot study report.]

UI claimants in Ohio can typically receive benefits for up to 26 weeks. The last cohort in the pilot study is expected to exhaust their regular benefits by the end of 2024. The UI claims data is delivered to OSU on a weekly basis. Therefore, the OSU evaluation team will be able to complete analysis related to UI reliance, such as the number of weeks on UI and UI exhaustion rates, in spring 2025. Employment and earnings outcomes rely on UI wage records, which typically has a six- to nine-month lag.[footnoteRef:4] Therefore, we expect to complete the employment and earnings analysis by winter 2026. [4: As of the time of writing the proposal (end of 2025Q1), the most recent UI wage records available to OSU are from 2024Q3.]

2.2. Implementation study

The OSU evaluation team developed the implementation study protocol during the spring and summer 2024 and completed the initial site visits in four representative workforce areas (Area 4, Area 11, Area 12, and Area 16) during fall 2024. The initial site visits focused on gathering information regarding the four workforce areas’ workforce and labor market conditions, organizational structures, staff educational and training backgrounds, RESEA service delivery models, data entry process, and staff’s feedback regarding the additional support needed. The team developed a protocol for quarterly check-ins during winter 2024 and conducted quarterly check-ins with program staff during spring 2025. The evaluation team plans to deliver a report to summarize the initial implementation study findings by the end of summer 2025.

ODJFS is interested in continuing the implementation study to identify and understand the onboarding process for RESEA advisors and examine how staff turnovers are managed locally and impact service delivery. The evaluation team will continue to hold quarterly check-in meetings with the four workforce areas to keep track of any changes in staff, service delivery model, and data entry. In consultation with ODJFS and if deemed necessary, the team may add one additional workforce area to the implementation study. OSU will consistently revise and update the research protocol following each round of data collection and based on feedback from ODJFS. The team will deliver synthesis pieces to ODJFS regularly, which summarize key information from each quarterly check-ins and provide timely information. The final implementation findings will be included as part of the overall randomization report.

2.3. Impact Study (Statewide Randomization)

The statewide randomization started in mid-July 2024. As of mid-February 2025 (approximately seven months later), we have over 20,000 eligible claimants in the pool.[footnoteRef:5] The fast accumulation of claimants in the pool is partially because of the surge of claimants during the wintertime. Workers in industries that have more seasonal jobs, such as Retail and Construction, typically file UI claims between December and February. During the peak weeks, we have 900 to 1,000 eligible claimants each week, compared to the 500 to 600 eligible claimants each week between April and June. Assuming the labor market conditions and county service capacity in 2025 are similar to 2024, even if the number of eligible claimants becomes lower during the spring and summer months, we should still be able to randomize over 40,000 claimants by the end of 2025. [5: The number already excludes claimants who cannot be selected due to county service constraint. ]

The current target sample size is larger than the originally planned 30,000 claimants because of the change in the evaluation design and the uncertainty of the program effects. After discussing with ODJFS, we decided to adopt a three-arm design, which not only allows us to evaluate the impacts of the traditional RESEA services (i.e., two mandatory one-on-one appointments with RESEA advisors), but also allows us to further examine the values of the new RESEA workshop that was launched in the summer of 2024. The results will help ODJFS assess whether the additional resources spent on the development and implementation of the workshop are cost effective. In addition, while we can estimate the expected program effects based on prior studies and the pilot study, the real program effects may be larger or smaller, depending on the overall economic conditions and claimants’ compositions during the statewide randomization. Therefore, we need to ensure that we have sufficiently large sample size in case the real program effects are small.[footnoteRef:6] [6: When the program effects are small, we need a large sample size to minimize the likelihood of Type II errors, i.e., reducing the likelihood of failing to rejecting the null hypothesis (no program effects) when a real effect exists. ]

Moreover, another key activity that determines the success of the randomization is the monitoring of the sample throughout the randomization. The OSU evaluation team develops a monthly report that tracks the number of claimants in the two treatment and control groups, as well as their service outcomes (whether attended the two one-on-one appointments and/or the workshop), demographic characteristics, and job histories. Because the control group should serve as the counterfactual of the treatment groups, claimants in the control group should be very similar to claimants in the treatment group in characteristics. The only differences between the three groups should be the RESEA services they receive. As of mid-February 2025, the monthly report shows that the number of claimants in each group are roughly the same and their characteristics are not statistically different. Moreover, county RESEA advisors have generally deliver service according to the randomization assignments. There are very few non-conforming cases, which should not influence the evaluation work.

2.4. SNAP-UI Work

ODJFS is interested in expanding the RESEA program to include people who are SNAP recipients and are also eligible for UI benefits (referred to as SNAP-UI claimants). The goal is to improve SNAP recipients’ job search skills and reduce their reliance on both SNAP and UI benefits. OSU will work closely with ODJFS to conduct a time-limited RCT for SNAP-UI claimants in selected counties where there is capacity to serve additional RESEA clients. We will adopt the same algorithm as the RESEA statewide randomization, except that instead of a three-arm design, we will randomize SNAP-UI claimants to one treatment group and one control group.

In addition, depending on the number of SNAP-UI claimants that are available for the randomization and the capacity of expanding the service at county level, it may be necessary to develop a new scoring system using machine learning models, which will enable ODJFS to better prioritize SNAP-UI claimants who are in greatest need of job assistance services. For example, the randomization may be limited to SNAP-UI claimants with scores in the top 20th percentile.

Currently, there is limited research regarding how reemployment services can affect SNAP recipients’ labor market outcomes. Prior studies have examined how SNAP work requirements influence participants’ SNAP enrollment, employment, and earnings, but generally found that the work requirements only reduce SNAP participation and had little impact on labor market outcomes. One possible explanation of the overall lack of impacts on employment and earnings is that the work requirements only force SNAP participants to look for jobs and leave the SNAP program due to eligibility concerns, rather than providing active job search support. The job search services offered through the RESEA program may help fill this gap and provide meaningful benefits to SNAP-UI claimants.

The work is required to take place in ADRF. After ODJFS completes updating the data sharing agreement with Coleridge Initiative, the OSU team can assist with the data transfer. The OSU team has been working with Coleridge Initiative since 2018 and have transferred multiple ODJFS’s data to ADRF on a regular basis, including the UI wage records and the UI claims data. The entire SNAP-UI project will last two years (calendar year 2025 and 2026). The OSU team will work closely with ODJFS to determine the randomization population, complete research design, update the data sharing agreement, establish the data transfer process between ODJFS, OSU, and ADRF during the spring and summer of 2025, conduct data analysis, and write reports. We expect to start the time-limited randomization by the end of summer 2025 and the process will last until the end of 2025. The data analysis and report drafting timeline should be similar to the timeline for the statewide randomization.

2.5. Administrative Activities

There will be administrative activities throughout the project. First, Ohio Longitudinal Data Analytics (OLDA) staff will work with ODJFS on updating the data sharing agreement, contract, and data delivery process. Second, while the initially planned analysis did not need an Institutional Review Board (IRB) application because the evaluation metrics are required by the federal, the additional outcome metrics and the SNAP-UI works may require us to file an IRB at The Ohio State University to ensure that the proposed research complies with the regulations on human subjects' research. Finally, the research team will meet with state and county RESEA program staff to learn the details of the program and to coordinate the randomization process. We will also meet with ODJFS regularly to communicate and report on the evaluation progress, and we will participate in monthly meetings with ABT associates and bi-weekly meetings with MDRC.

3. Project Timeline and Deliverables Assuming both statewide randomization and the SNAP-UI randomization can be completed by the end of 2025, our best estimate of the timeline at the time of writing (March 2025) is:

Tasks & Deliverables

1. EDR

2. Implementation Study

Quarterly Synthesis report

3. Pre-Evaluation Assessment

4. Pilot Study

5. Impact Study

6. SNAP-UI Project

Conditional on finalizing the data delivery process and randomization process by the end of 2025

4. Dissemination The project will initially be registered as an RCT in the OSF or the AEA Registry for trials. Following that, we will provide the initial evaluation design report, the pre-evaluation design and final study results to the OSF project associated with this trial. Secondly, we expect to disseminate the results on an annual basis through our state and national research network, such as through the Association for Public Policy Analysis and Management (APPAM) or other research networks. Finally, we are open to working with the JFS staff to prepare regular updates for the team working on RESEA, and to state-wide communities of practice with an interest in Unemployment Insurance.

5. Project Administration The project will be administered through Ohio State University according to rules set up by the Office of Sponsored Projects. Dr. Tian Lou and Josh Hawley will serve as co-principal investigators on all tasks for the project. Each quarter we will prepare a short briefing and report for the agency showing progress and deliverables.

The data management for the project will occur in collaboration with the CHRR at The Ohio State University. CHRR is a large social science research unit at OSU. Dr. Hawley is the Associate Director, and Dr. Lou is a Research Scientist. While IOP security rules are maintained by the Department of Administrative Services, the CHRR systems are under separate security rules. This will come into play when data are analyzed under tools that are required but unavailable from the IOP. The following paragraph briefly summarizes our IT security system at the center.

CHRR maintains an enterprise security program for all components of its information systems that is compliant with NIST 800-53. CHRR's security program is based on a risk management framework that addresses security control selection based on a FIPS 199 worst-case impact analysis. CHRR information systems are considered ‘Moderate’ under the NIST impact categorization system. Therefore, CHRR has implemented the applicable security controls under the moderate baseline from the NIST Special Publication 800-53B, revision 5. The security program covers the 19 operational, managerial, and technical NIST control families. CHRR maintains comprehensive security and privacy procedures in its Security Manual, including, but not limited to, the following major topics.

6. Budget

Personnel Tian Lou, Research Scientist (Principal Investigator), $114,075 in FY26 and $117,497 in FY27. We have budgeted 75% Annual FTE for each year of the award for Tian Lou. She will be the Principal Investigator and will provide overall direction and oversight of the project. She will lead the Pilot Study, Impact Study and SNAP-UI project. She will be responsible for coordinating with ODJFS and RESEA staff, develop research design, conduct econometric analysis, draft report, and disseminate results. Fringe benefits are calculated at 30% are applied to salary.

Xiangyu Ren, Data Scientist, $97,500 in FY26 and $100,425 in FY27. We have budgeted 100% of Xiangyu Ren’s time for each year of the award. She will be the research analyst for this project. Her major responsibilities include data analyses, and draft report under the supervision of PI. Fringe benefits are calculated at 30% are applied to salary.

Jess Collura, Senior Research Associate $58,898 in FY26 $60,665 in FY27. We have budgeted 50% Annual FTE for each year of the award for Jess Collura. She will be the lead analyst for the implementation study. She will conduct interviews and focused groups with RESEA program staff, conduct qualitative analysis, and draft implementation study report. Fringe benefits are calculated at 30% are applied to salary.

All salaries are escalated at a rate of 3% annually.

Travel $5,750 in FY26 and $5,750 FY27. We estimate that the study team will conduct in-state travel annually to support the implementation study as well as one trip to participate in the UI/SNAP cross-state convening. Travel consists of airfare (out of state), milage (in state), per diem and lodging costs, and is consistent with prior years’ travel expenses.

Other Direct Costs CHRR Computer Fee. $36,113 in FY26 and $36,113 in Year 2. Staff working at CHRR who access the CHRR computer system are charged an $11/hour computer usage fee. We estimate 3,283 hours in FY26 and 3,283 hours in FY27 at $11 per hour.

Indirect Costs $179,593 in FY26 and $184,259 in FY27. Indirect Costs are calculated at The Ohio State University’s federally negotiated rate of 57.5%.