WIC 2020 Eligibility Estimates_Recommendation Memo_REVISED FINAL 2.pdf

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 1

Memorandum

Date: August 11, 2022

To: Grant Lovellette, Food and Nutrition Service

From: Kelsey Gray, Courtenay Kessler, Josh Rozen, Andrew Bryant, Richard Griffiths, and Breanna Wakar, Insight Policy Research

Subject: 2020 WIC Eligibility Estimates: Final recommendation memorandum for selecting revised methodology for preparing the estimates – Revised 2 his memorandum presents the final recommendations for updating the methodology used to estimate eligibility for the Special Supplemental Nutrition Program for Women, Infants, and Children

(WIC) and includes estimates for 2016 through 2020. This work is part of the 2020 WIC Eligibility Estimates project conducted by Insight Policy Research and the Urban Institute for the U.S. Department of Agriculture’s Food and Nutrition Service (FNS). The overall objective of this study is to identify improved methods for providing FNS with accurate, reliable, timely, and cost-effective estimates of WIC eligibility and program reach.

The estimation procedures used to develop the 2005–2019 eligibility estimates build on a methodology originally developed by the Committee on National Statistics (CNSTAT) of the National Research Council in 2003 (Ver Ploeg & Betson, 2003). The CNSTAT panel recommended using data from the U.S. Census Bureau’s Current Population Survey Annual Social and Economic Supplement (CPS-ASEC) for the initial counts of eligible infants and children. These counts are then refined through a series of adjustment factors to estimate the number of pregnant and postpartum (breastfeeding and non-breastfeeding) women1 and more closely reflect WIC eligibility requirements.

Over the past 20 years, the underlying data sources have evolved, and the adjustment factors have been enhanced. However, a comprehensive review of the underlying methodologies used to produce the estimates was never conducted. Limitations of the existing methodologies and outdated adjustment factors created challenges for reliably estimating the number of individuals eligible for WIC. As a result, some estimated coverage rates are over 100 percent. Recent estimates of ineligible individuals participating in WIC are quite low (2.5 percent overall), suggesting multiple coverage rates over 100 percent are somewhat implausible (Magness et al., 2021). FNS requires accurate estimations of the number of individuals eligible for WIC and the number likely to participate to better predict future funding needs, measure WIC performance, and identify potentially unmet nutrition assistance needs.

The study team met with FNS and a panel of methodological experts between October 2021 and January 2022 to identify potential improvements to the existing methodologies. Based on feedback from these meetings and additional research, the study team identified four new methodologies to produce eligibility estimates by participant category (infants; children aged 1–4; pregnant women; and breastfeeding and non-breastfeeding postpartum women); racial and ethnic group; and at the national, 1 Throughout this memorandum, the term postpartum women refers to both breastfeeding and non-breastfeeding women.

T

Coverage Rate

Percentage of women, infants, and children eligible for WIC who received WIC benefits in an average month

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 2 regional, and State levels (including the District of Columbia and Puerto Rico).2 Results were largely consistent across the methodologies but also indicate specific areas for improvement. Based on the results from each method, the study team recommends adopting Method 3 as the base method for future eligibility estimates. Several possible modifications are discussed in section F of the memorandum.

The study team made several changes to the estimates presented in this final memorandum compared to the revised draft memorandum.3 The study team implemented refinements to Method 3 after receiving input from FNS on the recommendations presented in the revised draft memorandum (e.g., pooling 2 years of ACS data to calculate State-level coverage rates). The study team updated the factors used to adjust the estimates of eligible women for all new methods based on newly released data on fetal deaths and maternal mortality. The study team updated the alternate weights provided by the U.S.

Census Bureau for the 2020 CPS-ASEC (i.e., 2019 estimates) and incorporated the new weights for the 2021 CPS-ASEC (i.e., 2020 estimates) for Methods 1 and 3.4 The study team recommends using the alternate weights provided by the U.S. Census Bureau for the 2019 and 2020 estimates.

Section A provides an overview of the recent estimation challenges and FNS’s priorities for producing the estimates in the future. Section B describes the four new methodologies the study team developed.

Section C explains key findings and the strengths and limitations of each methodology. Section D describes the approaches the study team explored to improve coverage rates at the State level and incorporate Puerto Rico in the national-level standard errors. Section E describes the rationale for the study team’s recommendations to produce the estimates using Method 3. Section F describes remaining estimation challenges and recommendations for future research. Section G presents next steps.

Appendix A summarizes the data sources and adjustment factors for each new method, and appendix B displays the step-by-step adjustments on the estimates by participant category.

A. Study Objectives to Enhance the Estimation Methodology

The purpose of identifying the new methodologies is to address issues regarding the reliability and validity of the estimates and challenges associated with the production of the estimates. The four methodologies developed by the study team were informed by feedback from the expert panel members on improving the reliability of the estimates and conversations with FNS. The study team also considered goals and priorities identified by FNS for producing the estimates.

Improving the reliability and validity of the estimates. The expert panel recommended focusing on the biggest “problem areas” for producing reliable and valid estimates:

Undercount of infants. The undercount of infants in survey data is a challenge across all the proposed primary data sources for estimating WIC eligibility and must be addressed to produce more reliable estimates of WIC eligibility.

Undercount of pregnant women. Pregnant women are currently identified using the estimates of eligible infants. The panel felt this approach was reasonable, but if the undercount of infants is not resolved, identifying pregnant women independently of infants could lead to more accurate estimates of pregnant women.

2 See the Task 1.3 final analysis plan for a more detailed summary of each method (Gray et al., 2022).

3 See the Task 2.2. revised draft memorandum delivered to FNS on March 10, 2022.

4 Alternate weights for the 2021 CPS-ASEC and updated weights for the 2020 CPS-ASEC were released in March 2022.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 3

Identification of postpartum breastfeeding and non-breastfeeding women. The National Immunization Survey (NIS) is likely a good source for breastfeeding data, but the expert panel recommended the study team revisit how breastfeeding and non-breastfeeding women are identified in the data. The estimated counts of non-breastfeeding women are also often lower than expected.

Undercount of adjunctive income eligibility. Program participation is underreported in the proposed primary data sources for estimating WIC eligibility. Some infants and children may also be eligible for WIC through the Children’s Health Insurance Program (CHIP) implemented via Medicaid expansion but are not identified as eligible under the current estimation methodology.

Exploration of coverage rates over 100 percent. The expert panel recommended using coverage rates estimated to be over 100 percent (e.g., infants, postpartum non-breastfeeding women) to guide the identification of potential problems with the current methodology.

Improving the production of the estimates. FNS identified several priorities for producing the revised estimates under the current project:

Robustness of the estimates by participant category and race and ethnicity at the national, State, and regional levels

Timeframe for producing the estimates (from the time the primary data are available)

Ability to reproduce the estimates on a regular (ideally annual) basis

Financial feasibility of producing the estimates within the project budget

The study team’s recommendations proposed in this memorandum were informed by how well each method improves (1) the reliability and validity of the estimates and (2) the processes required to produce the estimates.

B. Proposed Methodologies to Produce the Estimates

The study team developed four new methodologies to produce the eligibility estimates. All four estimation methodologies build on the original methodology recommended by the CNSTAT panel (Ver Ploeg & Betson, 2003). The CNSTAT panel recommended using CPS-ASEC data for the initial counts of eligible infants and children in the United States. Under the proposed methodologies, two of the methods (Methods 1 and 3) continue to use the CPS-ASEC data as the primary data source to produce the initial counts of eligible infants and children; the other two methods (Methods 2 and 4) use the American Community Survey (ACS) data to estimate these counts. Each method was then refined through a series of new and enhanced adjustment factors designed to reflect WIC eligibility requirements more closely. This memorandum summarizes the results from each method compared with Method 0, the methodology used to compute the 2016–2019 estimates presented in the 2019 WIC eligibility estimates report (Farson Gray et al., 2022).

Table 1 summarizes each proposed methodology, the primary data sources, and the approach for implementing adjustments to the estimates. Adjustments are made to the preliminary estimates to better reflect WIC eligibility criteria (for example, breastfeeding status). Method 0 uses the methodologies from the National- and State-Level Estimates of WIC Eligibility and WIC Program Reach in 2019 report (Farson Gray et al., 2022) and provides a comparison for each revised methodology. The first two methods (Methods 1 and 2) implemented enhancements to existing adjustment factors by

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 4 either (1) updating the adjustment factors using more recent data or (2) changing the data sources used to calculate the adjustment factors. Methods 3 and 4 used a combination of statistical modeling to adjust the preliminary estimates and created new adjustment factors or used enhanced adjustment factors. The primary differences in the proposed methodologies are the national-level data sources and the approach for developing the adjustment factors. Appendix A describes the data sources and adjustment factors used for each method; enhancements to the adjustment factors are described in detail in the final analysis plan (Gray et al., 2022).

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 5

Table 1. Proposed Methods to Develop WIC Eligibility Estimates

Method 0 (Baseline) Method 1 Method 2 Method 3 Method 4

Primary Data Sources

National-level estimates CPS-ASEC CPS-ASEC ACS CPS-ASEC ACS

State-level estimates ACS ACS ACS ACS ACS

Territory-level estimates

PRCS + IDB PRCS + IDB PRCS + IDB PRCS + IDB PRCS + IDB

Population Estimates

Infants and children Census Population estimates

Census Population estimates

Census Population estimates

Vital Statistics Vital Statistics

Pregnant women Starting point is infants Starting point is infants Starting point is infants Estimates of pregnancy among women (15–44)

Estimates of pregnancy among women (15–44)

Postpartum women Starting point is infants Starting point is infants Starting point is infants Starting point is infants Starting point is infants

Adjustment Factors

Adjunctive eligibility Not applicable

Add infants and children participating in CHIP in 4 States identified as allowing CHIP to confer adjunctive eligibility

Not applicable

Apply proportion of infants and children identified as adjunctively income-eligible from ACS

Not applicable

Annual-to-monthly

Use SIPP to account for monthly fluctuations in income and certification periods (from 2014 SIPP)

Use SIPP to account for monthly fluctuations in income and certification periods (from 2014 SIPP)

Use SIPP to account for monthly fluctuations in income and certification periods (from 2014 SIPP)

Use SIPP to account for monthly fluctuations in income and certification periods (from 2014 SIPP)

Use SIPP to account for monthly fluctuations in income and certification periods (from 2014 SIPP)

Nutritional risk adjustment factor

Use participant category-specific adjustment factors recommended by CNSTAT:

infants (0.97), children (0.99), pregnant women (0.97), postpartum women (1.00)

Set to 1.00 across all participant categories

Set to 1.00 across all participant categories

Set to 1.00 across all participant categories

Set to 1.00 across all participant categories

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 6

Method 0 (Baseline) Method 1 Method 2 Method 3 Method 4

Multiple births; fetal, infant, and maternal deaths; and abortions

Adjust count of eligible women to account for impact of multiple births, fetal deaths, and infant deaths

Adjust count of eligible women to account for impact of multiple births, fetal deaths, and infant deaths

Adjust count of eligible women to account for impact of multiple births, fetal deaths, and infant deaths

Adjustment varies by category:

Pregnant: not applicable Breastfeeding: multiple births and maternal deaths

Non-breastfeeding:

multiple births; fetal, infant, and maternal deaths; and abortions

Adjustment varies by category:

Pregnant: not applicable Breastfeeding: multiple births and maternal deaths

Non-breastfeeding:

multiple births; fetal, infant, and maternal deaths; and abortions

Income during pregnancy

Adjust for lower likelihood of financial eligibility during pregnancy versus after birth (average data from SIPP 2004, 2008, and 2014)

Adjust for lower likelihood of financial eligibility during pregnancy versus after birth (average data from SIPP 2004, 2008, and 2014)

Adjust for lower likelihood of financial eligibility during pregnancy versus after birth (average data from SIPP 2004, 2008, and 2014)

Not applicable (income eligibility determined separately from infant eligibility)

Not applicable (income eligibility determined separately from infant eligibility)

Breastfeeding CDC NIS breastfeeding rates computed by CDC

CDC NIS breastfeeding rates computed by CDC updated to include adjunctive income eligibility via Medicaid

CDC NIS breastfeeding rates computed by CDC updated to include adjunctive income eligibility via Medicaid

CDC NIS breastfeeding rates computed by CDC updated to include adjunctive income eligibility via Medicaid

CDC NIS breastfeeding rates computed by CDC updated to include adjunctive income eligibility via Medicaid

Note: ACS = American Community Survey; CDC = Centers for Disease Control and Prevention; CHIP = Children’s Health Insurance Program; CNSTAT = Committee on National Statistics; CPS-ASEC = Current Population Survey Annual Social and Economic Supplement; IDB = International Data Base; NIS = National Immunization Survey; PRCS = Puerto Rico Community Survey; SIPP = Survey of Income and Program Participation

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 7

C. Key Findings

The new methodologies were generally consistent with findings from previously reported estimates.

Specifically, the relative order of coverage rates by participant category remained consistent across all methods. Coverage rates were highest for infants, followed by those for postpartum women and pregnant women. Coverage rates for children were consistently the lowest. Coverage rates for Hispanic/Latino and Black-only non-Hispanic individuals were also generally higher than for White-only non-Hispanic individuals.

While general trends in coverage rates remained consistent across all methods, the eligibility estimates and coverage rates varied across each method. Some of the adjustments to the estimates led to increases in counts that were counteracted by decreases related to other adjustments. Appendix B provides a step-by-step summary of how and where methodological changes led to differences in the estimates for each method by participant category (i.e., infants, children, pregnant women, and postpartum women). To focus on differences between methods rather than trends over time, the study team compared the eligibility estimates and coverage rates for 1 year. The study team used the results for 2018 because it was the most recent year of data without data collection challenges (e.g., data quality for the 2020 CPS-ASEC; data availability and low response rates for 2020 related to the COVID-19 pandemic).5

1. Estimation Results From Each Methodology

All methods produced relatively consistent population estimates for each participant category (table 2).

Methods 0, 1, and 2 use the Census Population estimates to weight the number of infants and children.

The study team used updated Census Population estimates (Vintage 2020) to weight the estimates for 2018 in Methods 1 and 2, which explains some of the discrepancies between Method 0 and Method 1 (other discrepancies are based on updated adjustment factors). Methods 3 and 4 use Vital Statistics data adjusted for infant deaths and immigration to estimate the total number of infants and children. The changes in estimates for pregnant and postpartum women in Methods 3 and 4 are largely a result of (1) estimating pregnant women based on women of reproductive age rather than infants and (2) calculating separate adjustment factors for postpartum breastfeeding and non-breastfeeding women.

Table 2. Population Estimates by Participant Category and Method, 2018

Participant Category Method 0 Method 1 Method 2 Method 3 Method 4

Total 26,526,035 26,435,472 26,418,833 26,431,204 26,349,121

Infants 3,817,508 3,786,009 3,779,903 3,743,398 3,732,336

Children 16,063,431 16,059,224 16,059,378 15,788,027 15,733,161

1-year-old children 3,924,580 3,938,230 3,937,537 3,831,257 3,841,889

2-year-old children 3,974,744 3,957,088 3,996,792 3,919,224 3,918,703

3-year-old children 4,050,942 4,042,265 4,066,103 3,980,231 3,986,419

4-year-old children 4,113,165 4,121,641 4,058,946 4,057,316 3,986,150

Women 6,645,097 6,590,238 6,579,551 6,899,779 6,883,624

Pregnant women 2,847,899 2,824,388 2,819,808 2,923,104 2,918,808

Postpartum women 3,797,198 3,765,851 3,759,743 3,976,675 3,964,815

5 The 2016–2020 estimates produced under each method were provided under the Task 3.3 deliverable.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 8

Participant Category Method 0 Method 1 Method 2 Method 3 Method 4

Breastfeeding women 2,222,934 2,014,351 2,012,005 1,967,357 1,962,439

Non-breastfeeding women 1,574,264 1,751,500 1,747,738 2,009,319 2,002,376

Overall, the number of eligible individuals increased from the baseline method (Method 0; table 3). The methodologies based on the CPS-ASEC (Methods 0, 1, and 3) had consistently lower estimates of the population eligible for WIC than methodologies based on the ACS (Methods 2 and 4), despite general population estimates being similar across all methods.

The ACS-based estimates (Methods 2 and 4) likely produce higher estimates of WIC eligibility for several reasons:

1. The data collection methodology for the ACS and CPS-ASEC is substantially different (U.S. Census Bureau, 2021a). The CPS-ASEC is conducted by interviewers via Computer Assisted Phone Interviewing (CAPI), while the ACS collects data through self-response mail-in questionnaires followed up by interviews using CAPI. These differences in how data are collected may contribute to discrepancies in reported income information. Differences in the recall period may also contribute to faulty information: Survey respondents may underreport their income because of the passage of time between receipt of that income and reporting on that income (Moore et al., 1997).

2. The ACS collects information on all the major income sources, whereas the CPS-ASEC income questions are “much more detailed and provide more comprehensive coverage of all potential income sources” (U.S. Census Bureau, 2021a). As a result, the CPS-ASEC provides more accurate information on a household’s income eligibility for WIC. The U.S. Census Bureau notes that because of its detailed questionnaire and experienced interviewing staff, the CPS-ASEC is the survey used to produce the official estimates of poverty in the United States (U.S. Census Bureau, 2021a).

3. While the underreporting of program participation in both the ACS and the CPS-ASEC is a known issue, the ACS tends to have more accurate information on program participation. A 2020 study that linked ACS and CPS-ASEC data to administrative data from two State agencies found a 24 percent underreport of Supplemental Nutrition Assistance Program (SNAP) participation in the ACS and a 40 percent underreport of SNAP participation in the CPS-ASEC (Meyer et al., 2020). As a result, a higher percentage of non-income-eligible individuals are found to be adjunctively income-eligible in the ACS compared with the CPS-ASEC. Eighteen percent of non-income-eligible individuals in the CPS-ASEC under Method 1 were identified as adjunctively income-eligible compared with 25 percent of non-income-eligible individuals in the ACS under Method 2.

Table 3. Estimates of Eligible Individuals by Participant Category and Method, 2018

Participant Category Method 0 Method 1 Method 2 Method 3 Method 4

Total 11,992,617 12,208,042 14,086,942 13,042,077 13,664,551

Infants 1,778,074 1,817,678 2,157,235 2,000,830 2,123,082

Children 7,827,066 7,920,128 9,000,285 8,457,986 8,749,990

1-year-old children 1,925,732 1,961,784 2,238,907 2,071,319 2,168,926

2-year-old children 1,931,721 1,947,838 2,244,638 2,113,324 2,184,342

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 9

Participant Category Method 0 Method 1 Method 2 Method 3 Method 4

3-year-old children 2,051,732 2,070,678 2,262,780 2,162,163 2,202,444

4-year-old children 1,917,881 1,939,827 2,253,960 2,111,181 2,194,279

Women 2,387,477 2,470,236 2,929,422 2,583,261 2,791,479

Pregnant women 1,172,560 1,235,609 1,465,821 1,160,205 1,281,518

Postpartum women 1,214,917 1,234,627 1,463,601 1,423,056 1,509,961

Breastfeeding women 806,186 815,248 964,328 891,825 944,902

Non-breastfeeding women 408,731 419,379 499,274 531,231 565,059

Table 4 shows the difference between the baseline WIC eligibility estimates and the new methods.

Estimates for pregnant women were lower in Methods 3 and 4 compared with Methods 1 and 2 because pregnant women were estimated independently of eligible infants. The results from this new method were strikingly similar between Methods 3 and 4 and the baseline method.

Table 4. Difference Between Baseline WIC Eligibility Estimates (Method 0) and New Methods (Methods 1–4), 2018

Participant Category Method 0 Method 1 –

Method 0

Method 2 –

Method 0

Method 3 –

Method 0

Method 4 –

Method 0

Total 11,992,617 215,425 2,094,325 1,049,460 1,671,934

Infants 1,778,074 39,604 379,161 222,756 345,008

Children 7,827,066 93,062 1,173,219 630,920 922,924

1-year-old children 1,925,732 36,052 313,175 145,587 243,194

2-year-old children 1,931,721 16,117 312,917 181,603 252,621

3-year-old children 2,051,732 18,946 211,048 110,431 150,712

4-year-old children 1,917,881 21,946 336,079 193,300 276,398

Women 2,387,477 82,759 541,945 195,784 404,002

Pregnant women 1,172,560 63,049 293,261 -12,355 108,958

Postpartum women 1,214,917 19,710 248,684 208,139 295,044

Breastfeeding women 806,186 9,062 158,142 85,639 138,716

Non-breastfeeding women

408,731 10,648 90,543 122,500 156,328

Figure 1 shows the trends in the overall number of eligible individuals by method for 2016 through 2020.

The baseline method (Method 0) consistently had the lowest estimates of WIC eligibility; Method 2 consistently had the highest estimates of WIC eligibility. Methods 0, 1, and 3 use the alternate weights provided by the U.S. Census Bureau for the 2020 CPS-ASEC file (see Farson Gray et al., 2022, for more information) and for the 2021 CPS-ASEC file.6

The decrease in the number of individuals eligible for WIC between 2018 and 2019 in Methods 0 and 1 is likely attributable to changes in the economy and a decline in SNAP and Medicaid participation. The decline in the number of eligible individuals between 2018 and 2019 is less exaggerated in Methods 2, 3, 6 The U.S. Census Bureau updated the alternate weights provided for the 2020 CPS-ASEC file (i.e., 2019 estimates) in February 2022. Method 0 uses the original alternate weights provided by the U.S. Census Bureau in March 2021. Methods 1 and 3 use the updated alternate weights for the 2020 CPS-ASEC file and the new alternate weights for the 2021 CPS-ASEC file.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 10 and 4 compared to Methods 0 and 1, likely because these methods draw from the ACS data for adjunctive income eligibility (which had less of a decline in 2019).

Figure 1. Trends in WIC Eligibility by Method, 2016–2020

The larger estimates of WIC eligibility found in the ACS-based methods (Methods 2 and 4) led to much lower coverage rates for each participant category. However, the general order of magnitude by participant category remained consistent across all methods. Table 5 presents coverage rates by participant category for each method. Rates over 100 percent are noted in red.

Table 5. Coverage Rates by Participant Category and Method, 2018

Participant Category Method 0 Method 1 Method 2 Method 3 Method 4

Total 56.3 55.3 47.9 51.7 49.4

Infants 94.9 92.8 78.2 84.3 79.5

Children 44.2 43.7 38.4 40.9 39.5

1-year-old children 61.4 60.2 52.8 57.1 54.5

2-year-old children 48.4 48.0 41.6 44.2 42.8

3-year-old children 40.7 40.4 36.9 38.7 37.9

4-year-old children 26.5 26.2 22.5 24.1 23.2

Pregnant women 51.7 49.1 41.4 52.3 47.3

Postpartum women 81.9 80.6 68.0 69.9 65.9

Breastfeeding women 66.6 65.9 55.7 60.2 56.8

Non-breastfeeding women 112.0 109.2 91.7 86.2 81.0

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 11

Table 6 presents the difference in coverage rates from the baseline method (Method 0) to Methods 1 through 4. The study team used separate adjustment factors for postpartum non-breastfeeding women and breastfeeding women in Methods 3 and 4, which contributed to the large decrease in the coverage rate for postpartum women, specifically postpartum non-breastfeeding women. These changes increased the estimated number of eligible postpartum women and decreased the coverage rates for breastfeeding and non-breastfeeding women. Applying the ACS-based percentage of adjunctively income-eligible individuals to the CPS-ASEC also decreased the coverage rates in Method 3 compared with Method 1.

Table 6. Difference in Coverage Rates by Participant Category and Method, 2018

Participant Category Method 0 Method 1 –

Method 0

Method 2 –

Method 0

Method 3 –

Method 0

Method 4 –

Method 0

Total 56.3 -1.0 -8.4 -4.6 -6.9

Infants 94.9 -2.1 -16.7 -10.6 -15.4

Children 44.2 -0.5 -5.8 -3.3 -4.7

1-year-old children 61.4 -1.2 -8.6 -4.3 -6.9

2-year-old children 48.4 -0.4 -6.8 -4.2 -5.6

3-year-old children 40.7 -0.3 -3.8 -2.0 -2.8

4-year-old children 26.5 -0.3 -4.0 -2.4 -3.3

Pregnant women 51.7 -2.6 -10.3 0.6 -4.4

Postpartum women 81.9 -1.3 -13.9 -12.0 -16.0

Breastfeeding women 66.6 -0.7 -10.9 -6.4 -9.8

Non-breastfeeding women 112.0 -2.8 -20.3 -25.8 -31.0

Table 7 presents coverage rates by race and Hispanic ethnicity for each method. Across all methods, Hispanic/Latino individuals had the highest coverage rates, while White-only non-Hispanic individuals had the lowest coverage rates.

Table 7. Coverage Rates by Race and Hispanic Ethnicity and Method, 2018

Race and Hispanic Ethnicity Method 0 Method 1 Method 2 Method 3 Method 4

Hispanic/Latino 62.4 61.9 57.6 64.4 64.2

White only, not Hispanic 47.9 47.4 37.7 39.8 35.8

Black only, not Hispanic 57.8 56.3 52.0 50.3 49.9

All other races and ethnicities, not Hispanic 60.1 56.1 44.9 60.4 59.1

One common concern about the WIC eligibility estimates is the number of coverage rates over 100 percent. The study team evaluated how the revised methods affected coverage rates over 100 percent.

Table 8 displays counts of the number of coverage rates over 100 percent by estimation year and method. These counts represent all the coverage rates (approximately 1,000 rates) currently produced in the WIC eligibility estimates report (Farson Gray et al., 2022)—for example, coverage rates by State and race and ethnicity, State and participant category, and participant category by race and ethnicity.

Importantly, a reduction in the number of coverage rates over 100 percent does not necessarily indicate a method is more accurate than a method with higher numbers of coverage rates over 100 percent.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 12

Table 8. Count of Coverage Rates With Lower Bound Over 100 Percent by Estimation Year and Method

Estimates Year Method 0 Method 1 Method 2 Method 3 Method 4

2020 26 23 10 8 4

2019 34 27 11 6 4

2018 35 31 9 5 8

2017 32 27 7 4 2

2016 35 28 10 8 4

Note: All coverage rates with a lower bound above 100 percent would be suppressed in reports. The counts provided for Method 3 include the recommended small area estimation techniques to improve the State-level coverage rates described in section D.

2. Strengths and Limitations of Each Methodology

Table 9 summarizes the relative strengths and limitations of each methodology based on each methodology’s ability to (1) improve the reliability and validity of the estimates and (2) improve the production of the estimates. Section D describes how each method could be strengthened using small area estimation techniques, which would lead to more reliable and robust estimates by State and race and Hispanic ethnicity.

Table 9. Summary of Strengths and Limitations of Each Methodology

Method Strengths Limitations

Reliability and validity: Improves identification of eligible breastfeeding women using updated NIS breastfeeding rates; most similar to prior estimates (Method 0)

Production: Data reliably provided annually; code has been tested over many years

Reliability and validity: No changes to address undercount of infants, pregnant women, or postpartum non-breastfeeding women;

highest number of coverage rates over 100 percent (see table 8)

Production: SIPP data used for two adjustment factors (annual-to-monthly and income during pregnancy) are outdated

Reliability and validity: Uses ACS to increase number of eligible individuals and subsequently lowers coverage rates (leaving fewer rates over 100 percent, see table 8); more valid assessment of adjunctive income eligibility; improves identification of eligible breastfeeding women using updated NIS breastfeeding rates

Production: While code requires changes, data sources already used by prior eligibility estimates

Reliability and validity: Use of ACS for income eligibility may overestimate the number of eligible infants and children

Production: In 2020, ACS had low response rates, which led to release of experimental data; it is unclear whether it will be a one-time event (related to COVID-19) or if ACS will continue to struggle; data quality and availability (including release dates) may continue to be an issue; SIPP data used for two adjustment factors (annual-to-monthly and income during pregnancy) are outdated

Reliability and validity: Weights the number of infants and children based on more recent Vital Statistics data for appropriate birth year (compared with Census Population estimates that have 2-year lag); independently estimates number of eligible pregnant women rather than relying on infant estimates; adjustment factors are applied separately for postpartum non-breastfeeding and breastfeeding women; uses strength of CPS-ASEC to estimate income eligibility and ACS to estimate adjunctive income eligibility;

Reliability and validity: Independent estimate of women yields lower coverage rates

Production: Data for maternal mortality are slightly outdated; requires gathering additional information to calculate separate adjustment factors for pregnant and postpartum women

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 13

Method Strengths Limitations decreases number of coverage rates over 100 percent (see table 8)

Production: Change in estimation of women reduces reliance on outdated SIPP panel data, potentially reducing time needed to produce estimates

Reliability and validity: Weights the number of infants and children based on more recent Vital Statistics data for appropriate birth year (compared with Census Population estimates that have 2-year lag); independently estimates number of eligible pregnant women rather than relying on infant estimates; adjustment factors are applied separately for postpartum non-breastfeeding and breastfeeding women; decreases number of coverage rates over 100 percent (see table 8)

Production: Change in estimation of women reduces reliance on outdated SIPP panel data, potentially reducing time needed to produce estimates; using one data source to estimate income and adjunctive income eligibility is a simpler process

Reliability and validity: Use of ACS for income eligibility may overestimate number of eligible individuals; independent estimate of women yields lower coverage rates

Production: Data for maternal mortality are slightly outdated; requires gathering additional information to calculate separate adjustment factors for pregnant and postpartum women

Note: ACS = American Community Survey; CPS-ASEC = Current Population Survey Annual Social and Economic Supplement; NIS = National Immunization Survey; SIPP = Survey of Income and Program Participation

D. Small Area Estimation

The study team explored multiple options for (1) producing State-level estimates and coverage rates for each of the eight WIC participant categories and by race and ethnicity, (2) reporting on State-level coverage rates in future reports, and (3) including Puerto Rico in the national-level standard errors. The study team’s final recommendations are provided in section E.

1. State-Level Estimates and Coverage Rates

FNS would like to display State-level coverage rates by participant category and race and ethnicity.

However, the expert panel could not recall any known examples of studies that are producing estimates of program participation (e.g., Medicaid, SNAP) at such granular levels (e.g., participant category and race/ethnicity), largely because of the sample size limitations in the primary data sources (CPS-ASEC and

ACS).

The expert panel suggested exploring several options to improve the robustness of the State-level estimates and coverage rates by WIC participant category and race and ethnicity. The study team selected two options for comparison: (1) pooling several years of ACS data to increase the sample size and (2) collapsing States to increase the sample size. For both options, the study team used Method 3 (recommended estimation methodology; see section E) to calculate the estimates and rates. The study team selected seven small States (New Hampshire, North Dakota, Maine, Montana, South Dakota, Vermont, and Wyoming) with historically high coverage rates to assess the impact of each option on eligibility estimates and participation rates (the percentage of WIC participants among the number of individuals in the demographic population targeted by WIC). For each option, the study team calculated the variance, standard error, and coefficients of variation.7 The coefficients of variation enable easy comparison between the two options. The U.S. Census Bureau publishes quality standards for producing

7 The coefficient of variation is the ratio of the standard error to the estimate.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 14 estimates derived from its surveys; these publication precision standards are based on the coefficient of variation. The standards cite a coefficient of variation of 0.30 as a threshold for low-quality estimates (U.S. Census Bureau, 2013, Requirement F1-6). Estimates with a coefficient of variation above 0.30 are considered lower quality estimates.

Pool several years of data. The expert panel recommended pooling several years of data from the ACS to increase the sample size for the State-level coverage rates. For the 2018 estimates, the study team pooled 3 years of ACS data (see table 10.a) and 2 years of ACS data (see table 10.b) to calculate eligibility estimates and participation rates by participant category and race and ethnicity. Tables 10.a and 10.b report the average and maximum coefficients of variation for the estimated number of eligible individuals and participation rates across all participant categories (infants, children aged 1–4, pregnant women, postpartum breastfeeding women, and postpartum non-breastfeeding women) and race and ethnicity under Method 3 before and after pooling the ACS data.8 The tables separate the States by FNS region (Northeast and Mountain Plains) to facilitate easier comparisons with table 11, which collapses States into clustered groups. The tables also report the percentage of coefficients of variation less than the U.S. Census Bureau-recommended threshold of 0.30. Across all States, the pooled approach improved or maintained the average coefficients of variation, the maximum coefficients of variation, and the coefficients of variation below the 0.30 threshold.9

Table 10.a. Coefficients of Variation for States After Pooling 3 Years of Data Under Method 3, 2018

State

Coefficients of Variation

Eligibility Estimates Participation Rates

Mean Max Percent

< 0.30 Mean Max

Percent

< 0.30

Without pooling

Vermont 26.1 67.3 77.8 18.8 61.8 83.3

New Hampshire 18.7 42.8 88.9 11.1 39.7 94.4

Maine 20.0 77.9 83.3 12.8 43.4 88.9

With pooling

Vermont 15.7 44.3 88.9 11.3 36.4 94.4

New Hampshire 11.5 26.8 100.0 6.7 24.4 100.0

Maine 10.4 23.7 100.0 6.1 17.5 100.0

Without pooling

Wyoming 17.9 25.0 100.0 17.1 114.5 94.4

North Dakota 21.5 43.6 88.9 13.3 40.5 94.4

South Dakota 21.8 78.8 88.9 12.1 35.8 94.4

Montana 19.2 90.2 88.9 14.7 90.2 94.4

With pooling

Wyoming 12.1 18.9 100.0 9.0 42.0 94.4

North Dakota 12.8 25.0 100.0 7.4 21.2 100.0

South Dakota 11.5 43.7 94.4 7.0 27.9 100.0

Montana 12.8 90.1 94.4 10.0 90.1 94.4

8 Coefficients of variation for coverage rates are excluded because coverage rate coefficients always equal eligibility estimates coefficients.

9 As a statistical rule, when the sample size is tripled, the standard error and coefficient of variation will be reduced by 43 percent. Table 10.a shows the average coefficients of variation generally decreased by 43 percent.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 15

Table 10.b. Coefficients of Variation for States After Pooling 2 Years of Data Under Method 3, 2018

State

Coefficients of Variation

Eligibility Estimates Participation Rates

Mean Max Percent

< 0.30 Mean Max

Percent

< 0.30

Without pooling

Vermont 26.1 67.3 77.8 18.8 61.8 83.3

New Hampshire 18.7 42.8 88.9 11.1 39.7 94.4

Maine 20.0 77.9 83.3 12.8 43.4 88.9

With pooling

Vermont 18.2 44.8 88.9 12.8 37.0 94.4

New Hampshire 14.4 36.1 94.4 8.1 30.1 94.4

Maine 12.4 26.0 100.0 7.8 20.4 100.0

Without pooling

Wyoming 17.9 25.0 100.0 17.1 114.5 94.4

North Dakota 21.5 43.6 88.9 13.3 40.5 94.4

South Dakota 21.8 78.8 88.9 12.1 35.8 94.4

Montana 19.2 90.2 88.9 14.7 90.2 94.4

With pooling

Wyoming 13.9 20.3 100.0 10.8 54.1 94.4

North Dakota 14.8 31.5 88.9 9.4 25.6 100.0

South Dakota 13.8 47.1 94.4 8.5 31.1 94.4

Montana 14.4 90.2 94.4 11.3 90.2 94.4

Collapse States to create clusters of States. The expert panel also recommended combining one or more States to increase the sample size of individual States. The study team explored collapsing two or more States and assessed the impact on the coefficients of variation. For the States in the Northeast Region, the study team explored collapsing (1) Vermont and New Hampshire and (2) Vermont, New Hampshire, and Maine. In the Mountain Plains Region, the study team explored three combinations: (1) Wyoming and North Dakota; (2) Wyoming, North Dakota, and South Dakota; and (3) Wyoming, North Dakota, South Dakota, and Montana. Table 11 shows the individual State coefficients of variation compared with the collapsed coefficients of variation under Method 3. The percentage of coefficients of variation meeting the threshold of 0.30 improves as more States are collapsed.10

10 As a statistical rule, when the sample size is doubled, the standard error and coefficient of variation will be reduced by 29 percent. Table 11 shows the average coefficients of variation generally decreased by 29 percent when two States with similar sample sizes were collapsed.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 16

Table 11. Coefficients of Variation for States and Collapsed States Under Method 3, 2018

State/Collapsed State

Coefficients of Variation

Eligibility Estimates Participation Rates

Mean Max Percent

< 0.30 Mean Max

Percent

< 0.30

Individual Northeastern States

Vermont (VT) 26.1 67.3 77.8 18.8 61.8 83.3

New Hampshire (NE) 18.7 42.8 88.9 11.1 39.7 94.4

Maine (ME) 20.1 78.0 83.3 12.8 43.5 88.9

Pooled Northeastern States

VT-NH 15.7 36.9 94.4 9.9 34.1 94.4

VT-NH-ME 11.7 26.7 100.0 7.5 25.9 100.0

Individual Mountain Plains States

Wyoming (WY) 17.9 25.0 100.0 17.1 114.5 94.4

North Dakota (ND) 21.5 43.6 88.9 13.3 40.5 94.4

South Dakota (SD) 21.7 78.8 88.9 12.1 35.8 94.4

Montana (MT) 19.2 90.2 88.9 14.7 90.2 94.4

Pooled Mountain Plains States

WY-ND 15.1 43.6 94.4 9.6 39.3 94.4

WY-ND-SD 12.3 41.6 94.4 7.8 28.6 100.0

WY-ND-SD-MT 10.6 38.5 94.4 6.9 27.1 100.0

Note: The numbers presented in this table use the estimates and rates calculated under Method 3 for the revised draft recommendation memorandum (Task 2.2). Because of the run times required to reproduce the estimates, the study team did not update the numbers presented in this table.

2. Presentation of Coverage Rates

The expert panel recommended displaying the confidence intervals at the 95-percent confidence level for State-level coverage rates to help users understand the uncertainty surrounding certain State-level estimates. The study team explored three options for displaying coverage rates to help users interpret the point estimates:

Example 1: For each State-level coverage rate, present the upper and lower bounds of the point estimate in a tabular view.

Table 12. Example 1: Layout for Coverage Rate Confidence Interval

State

Coverage Rate Confidence Interval

Lower Bound Estimate Upper Bound

State A 47 50 53

The text describing this table would state: “The actual coverage rate for State A is estimated to be between 47 percent and 53 percent. The likelihood of the actual coverage rate being nearer the outer bounds of this interval is less than the likelihood of the coverage rate being 50 percent.”

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 17

Example 2: Instead of showing the upper and lower bounds of the confidence interval in a table, present the confidence interval as a box plot.

Figure 2. Example 2: Layout for Coverage Rate Confidence Interval

The text describing this figure would state: “The actual coverage rate for State A is estimated to be between 47 percent and 53 percent. The likelihood of the actual coverage rate being nearer the outer bounds of this interval is less than the likelihood of the coverage rate being between 49 and 51 percent, as shown in the figure.”

Example 3: To visually represent the decreased likelihood of the actual rate being at the outer bounds of the point estimate, present the confidence interval using a heatmap.

Figure 3. Example 2: Layout for Coverage Rate Confidence Interval

The text describing this figure would state: “The actual coverage rate for this State is estimated to be between 47 and 53 percent. The colors in the heatmap depict the likelihood of the actual coverage rate being a particular value. Values most likely to be close to the actual coverage rate appear in dark green. The likelihood of any particular value being the true coverage rate is reduced as the dark green becomes lighter.”

The study team recommends FNS explore these options during the next iteration of the WIC eligibility estimates report to help readers better interpret the uncertainty surrounding the coverage rate estimates. It may be useful to show both a tabular and a graphical view of the coverage rates in future WIC eligibility estimate reports.

3. Incorporating Puerto Rico in National-Level Standard Errors

FNS would like to include Puerto Rico in the national estimates and the national standard error estimates. Discussions with the panel members identified several possible approaches. The proposed methods vary depending on the primary data source for the national estimates (CPS-ASEC or ACS) and are described in the final analysis plan (Gray et al., 2022). For CPS-based estimates (Methods 1 and 3), the study team explored two options to incorporate Puerto Rico:

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 18

Option 1: Calculate State ratio-adjusted estimates using baseline methodology and add Puerto Rico to both national and State estimates separately.

Option 2: Add Puerto Rico to CPS-ASEC national estimates and include estimates from the PRCS using a ratio-adjusted State shares method.11

Based on the results presented in the revised draft recommendation memorandum, under option 1 for Method 3, the estimated number of eligible individuals in Puerto Rico in 2018 is 138,861; under option 2, it is 129,941. While the national estimate remains the same under both methods, the difference in the Puerto Rico estimate of about 9,000 individuals is distributed among the 50 States and the District of Columbia under option 2. On average, the standard errors of the State eligibility estimates decreased by 6 percent under option 2. The standard errors for Puerto Rico under options 1 and 2 (2,666 and 3,877, respectively) was higher under option 2. Similarly, for the ACS-based estimates (Methods 2 and 4), the standard errors remained constant with and without Puerto Rico under option 1 and varied under option 2.12

E. Recommendations

The study team recommends FNS use Method 3 to produce the estimates of WIC eligibility. This recommendation is based on the data sources, population estimates, adjunctive income eligibility estimates, and adjustment factors. For small area estimation, the study team recommends (1) improving State-level estimates and coverage rates by pooling 2 years of ACS data for all States and (2) including Puerto Rico in the national-level standard errors using option 1 for the CPS-based estimates.

1. Data Sources

Recommendation for FNS: The study team recommends that FNS produce the annual estimates of WIC eligibility using the CPS-ASEC as the primary data source.

Rationale: The study team explored producing the national-level estimates using the CPS-ASEC (Methods 1 and 3) and the ACS (Methods 2 and 4). While the ACS identifies more eligible individuals, it may overestimate the number of eligible individuals for several reasons. First, the ACS contains slightly less detailed information on household income (Czajka & Denmead, 2012)13, which can lead to overestimating the number of individuals identified as eligible. The ACS income data also represent 23 months of data. The survey is conducted on a rolling basis throughout the year and asks income questions for the 12 months prior (not the prior calendar year). For example, the 2019 ACS would include income information from January 2018 through November 2019 (Czajka & Denmead, 2012). Second, data on adjunctive income eligibility via Medicaid and SNAP are reported for different periods in the ACS. Survey respondents are asked whether they are receiving Medicaid, medical assistance, or any other kind of government assistance plan for those with low incomes or a disability. The study team is unable to differentiate whether a respondent is receiving Medicaid or CHIP, which may overestimate the

11 Option 2 numbers are based on the data presented in the revised version of the final recommendation memorandum. Because of the run times required to reproduce the estimates, the study team did not update these estimates for the second revision.

12 Because of the run times required to reproduce each small area estimation technique, the results presented in this section remain unchanged from the revised draft recommendation memorandum. The minimal changes made to the estimates between the revised draft and final recommendation memorandum would likely result in small (if any) changes to the findings presented. The study team expects these differences would not change the final recommendation for incorporating Puerto Rico in the national-level standard errors.

13 This study found that the CPS-ASEC captured the most income information. However, the ACS captured 98 percent of the income data reported on the CPS-ASEC.

Insight ▪ 2020 WIC Eligibility Estimates: Final Recommendation Memorandum 19 number of individuals adjunctively income-eligible for WIC. However, given the underreporting of program participation in the ACS, the study team still recommends using the ACS to adjust for adjunctive income eligibility. For SNAP, respondents are asked whether they received SNAP at any point in the past 12 months.

The study team also recommends using the CPS-ASEC as the primary data source to be consistent with other reports published by FNS estimating program eligibility.