Attachment_II__PWS_for_Line_Item_6.pdf

CACFP Improper Payments

ATTACHEMENT II - CLIN 6

PERFORMANCE WORK STATEMENT

C.1 GENERAL OVERVIEW

Under “The Improper Payments Elimination and Recovery Act (IPERA) of 2010” (Public Law 111-

204) the Department of Agriculture (USDA) is required to identify and reduce erroneous payments including overpayments and underpayments in various programs including the Child and Adult Care Food Program (CACFP).

The CACFP has a complicated administrative and operational structure, with risk of improper payments present at multiple stages in the benefit determination, benefit delivery, reimbursement claiming, payment, and program oversight process. Erroneous payments in the CACFP can result from misclassification of the meal eligibility (free, reduced-price, paid) status of participating children or improper meal counting and claiming of reimbursable meals. Misclassification of eligibility status can occur in several ways: (1) Childcare center staff may make administrative errors during the approval or verification processing of income eligibility statements, or the processing of categorical eligibility information, (2) Households can provide incorrect information (either unintentional or intentional) with respect to household size, household income, or participation in other means-tested programs at the time of application or verification. An over- or under-payment results when a misclassified child receives a reimbursable meal that is claimed for reimbursement that does not correctly reflect the child‘s true eligibility status. Improper meal counting and claiming can occur when: (1) a childcare center records the meals served in the incorrect meal reimbursement classification; (2) the center and/or sponsor incorrectly consolidates recorded meal counts; and (3) the center and/or sponsor claims and receives reimbursement for meals that were not served.

The following table displays the levels of reimbursement in CACFP for 2013-2014.

CHILD AND ADULT CARE FOOD PROGRAM (CACFP)

[Per Meal Rates in Whole or Fractions of U.S. Dollars Effective from July 1, 2013–June 30, 2014] Centers Breakfast Lunch and

Supper 1 Snack

CONTIGUOUS STATES:

PAID

REDUCED PRICE

FREE

ALASKA:

PAID

REDUCED PRICE

FREE

HAWAII:

PAID

REDUCED PRICE

FREE

$0.28 1.28 1.58

0.41 2.23 2.53

0.31 1.55 1.85

$0.28 2.53 2.93

0.45 4.34 4.74

0.32 3.02 3.42

$0.07 0.40 0.80

0.11 0.65 1.30

0.08 0.47 0.94

1 These rates do not include the value of USDA foods or cash-in-lieu of USDA foods which institutions receive as additional assistance for each CACFP lunch or supper served to participants. A notice announcing the value of USDA foods and cash-in-lieu of USDA foods is published separately in the FEDERAL REGISTER.

SCOPE

The Government seeks a qualified contractor to assist the Office of Policy Support (OPS), United States Department of Agriculture (USDA), Food and Nutrition Service (FNS) to design, conduct, and report on the level of erroneous payments (dollars and rates) to childcare centers and center sponsors participating in CACFP. It builds on the methods developed for school meals in the Access, Participation, Eligibility and Certification (APEC) study series and complements the annual measure of reimbursement “tiering” errors in family day care homes for reporting on CACFP.

C.2 BACKGROUND

Need for the Study

On November 26, 2002, Congress enacted the Improper Payments Information Act of 2002, P.L. 107- 300 (http://www.whitehouse.gov/omb/memoranda_m03-13/ ). This Act required the Department of Agriculture (USDA) to report annually on the extent of erroneous payments in various programs that might be susceptible to significant erroneous payments (exceeding $10 million and 2.5 percent of benefits paid) and report annually on the actions they were taking to reduce these erroneous payments. An OMB directive issued May 21, 2003, provided specific guidance for Agencies to implement the provisions of this Act (http://www.whitehouse.gov/omb/memoranda_m03-13/ ).

This directive stated that annual erroneous payment estimate is the gross (not net) total of both overpayments and underpayments, i.e. it is the sum of the absolute value of overpayments and underpayments.

On November 20, 2009, President Obama issued Executive Order 13520 aimed at reducing improper payments (http://edocket.access.gpo.gov/2009/pdf/E9-28493.pdf). The purpose was to reduce improper payments by intensifying efforts to eliminate payment error, waste, fraud, and abuse in the major programs administered by the Federal Government, while continuing to ensure that Federal Programs serve and provide access to their intended beneficiaries.

On July 22, 2010, President Obama signed into public law “The Improper Payments Elimination and Recovery Act (IPERA) of 2010”, P.L. 111-204 (http://www.whitehouse.gov/sites/default/files/omb/financial/_improper/PL_111-204.pdf) . The act was aimed at reducing wasteful payments by $50 billion by 2012. The law was intended to assist agencies in reducing and reversing improper payments by: (1) requiring annual risk assessments and the measurement of improper payments; (2) expanding the type of programs that are required to conduct recovery audits; (3) providing agency heads with more options to use/reallocate recovered funds; and (4) providing a list of actions to ensure agency compliance and corrective actions are taken when non-compliance is found.

To comply with the IPERA 2010, USDA needs a reliable measure to estimate erroneous payments for the CACFP. The program is multi-faceted and consists of various administrative venues that serves both children and adults in both center-type facilities and private homes. FNS currently conducts an annual study of childcare provided in homes and the aim of this study will only include childcare centers (sponsored and independent). A large-scale data collection for a nationally representative study for estimating erroneous payments in childcare centers could be conducted periodically but it would be cost-prohibitive to conduct such a study annually given the number of participating centers. This solicitation seeks to leverage the methodology used to conduct the Access, Participation, Eligibility, and Participation (APEC) study series in both sponsored and independent childcare centers operating under CACFP. Thus USDA also needs reliable estimation http://www.whitehouse.gov/omb/memoranda_m03-13/ http://www.whitehouse.gov/omb/memoranda_m03-13/ http://edocket.access.gpo.gov/2009/pdf/E9-28493.pdf http://www.whitehouse.gov/sites/default/files/omb/financial/_improper/PL_111-204.pdf models that could be used by USDA staff for updating erroneous payment estimates annually. These estimation models would rely on extant data sources for the annual updates. Finally, as an option, FNS would like a white paper describing methodologies that could generate State-level estimates of erroneous payments.

Overview of the Child and Adult Care Food Program

FNS is the primary Federal agency responsible for providing a nutrition safety net for low-income people across the United States. FNS administers 15 nutrition assistance programs, including the CACFP. CACFP plays a vital role in providing children and the elderly with access to adequate food while improving the quality and affordability of day care for low-income families. The program subsidizes nutritious meals and snacks served to children and adults in participating day care facilities. Meals are also provided to children in emergency shelters and eligible after school programs. Providers are reimbursed for each qualifying meal or snack they serve to program participants.

Information from the USDA FNS National Data Bank1 reveals the following regarding the CACFP sponsored and independent childcare centers based on administrative reporting:

• In FY13, more than 1.3 billion nutritious meals and snacks were served in CACFP centers.

• Of the meals served in centers, over 80% were reimbursed as free or reduced-price meals.

• The total reimbursed meal costs for CACFP centers in FY 13 were more than $1.8 billion.

• There were more than 20,000 sponsored childcare centers in FY 13 with an average daily attendance of more than 2.7 million children.

In Centers, children from families with incomes at or below 130 percent of the federal poverty level (FPL) receive free meals upon certification as free-eligible. Children from families who receive benefits under the Supplemental Nutrition Assistance Program (SNAP), Temporary Assistance to Needy Families (TANF), the Food Distribution Program on Indian Reservations (FDPIR), and Head Start or Even Start are also eligible to receive free meals. Children from families with incomes between 130 percent and 185 percent of the FPL are eligible for reduced-price meals. For the period July 1, 2013 through June 30, 2014, 130 percent of the FPL is $30,615 for a family of four;

185 percent is $43,568.2 Children from families with incomes over 185 percent of the FPL are categorized as full price (paid), though centers receive a subsidy for full price (paid) meals as well.3

Program Regulations and Administration

CACFP is authorized at section 17 of the National School Lunch Act (42 U.S.C. 1766). Program regulations are issued by the USDA under 7 CFR part 226.4

FNS administers CACFP through payments made to FNS Regional Offices or State agencies in order to reimburse institutions for enrolled food service participants. However, in certain States, CACFP is

1 Data from the FNS National Data Bank (NDB). Last accessed January 28, 2014.

2 http://www.fns.usda.gov/sites/default/files/IEG_Table-032913.pdf

3 http://www.fns.usda.gov/school-meals/income-eligibility-guidelines

4 http://www.fns.usda.gov/cnd/Care/Regs-Policy/CFR226.pdf http://www.fns.usda.gov/sites/default/files/IEG_Table-032913.pdf http://www.fns.usda.gov/school-meals/income-eligibility-guidelines http://www.fns.usda.gov/cnd/Care/Regs-Policy/CFR226.pdf administered by an alternate agency such as the State health or social services department. The State agencies or alternate agencies enter into agreements with independent centers and sponsoring organizations to assume administrative and financial responsibility for CACFP operations. The eligible institutions5 must be licensed or alternatively approved according to Federal, State or local standards. The State agencies then support the sponsoring organizations and independent centers by processing applications from institutions, reviewing on-site operations, training personnel, auditing program records, processing payments for meals served to participants, and monitoring compliance with program regulations. The sponsoring organizations provide payment to sponsored day care providers for meals and snacks that meet the requirements. Figure 1 provides an overview of the administrative structure of CACFP.

Figure 1: Overview of CACFP Administration

Past Relevant Research

Research on the existence of improper payments in the childcare center setting is scarce. An audit report by USDA’s Office of Inspector General (OIG) found that one-third of the sites (inclusive of all of the CACFP operating modalities)visited in Florida and Michigan were claiming more meals than were observed being served. The report recommended that FNS issue guidance on the required 5-day reconciliation to ensure consistency across monitoring visits.6 An earlier audit report identified similar findings regarding over-claiming and recommended improved financial reviews as a means to address this issue. 7

The Institute of Medicine (IOM) was asked by FNS to convene an expert panel to assess and

5 Institution means a sponsoring organization, childcare center, at-risk afterschool care center, outside-school-hours care center, emergency shelter or adult day care center, which enters into an agreement with the State agency to assume final administrative and financial responsibility for program operations.

6 http://www.usda.gov/oig/webdocs/27099-0001-DA.pdf

7 http://www.usda.gov/oig/webdocs/27601-0012-SF.pdf

Independent Centers

Centers Sponsoring Organization

State CACFP Agency

Regional FNS Office

USDA, FNS

Headquarters

Family or Group Homes http://www.usda.gov/oig/webdocs/27099-0001-DA.pdf http://www.usda.gov/oig/webdocs/27601-0012-SF.pdf recommend revisions to the CACFP meal patterns in 2008. A final report was published in 2010 and while the primary focus of the recommendation was on the meal patterns, the report also recommended that FNS evaluate and assess the monitoring and reimbursement system.8 A follow up workshop in 2012 addressed the future research needs for the program. Workshop presentations explored a variety of topics and one session focused on exploring a variety of topics that may inform this study, specifically an overview of program administrative data9. A list of available workshop presentations by presenter as well as a conference agenda may be found here:

http://www.iom.edu/Activities/Nutrition/ChildAdultCareFood/2012-FEB-07.aspx.

While little research has been conducted on improper payments in childcare centers, extensive research has been conducted on the primary sources of improper payments in the FDCH component of CACFP and provides some context for this study. FNS conducts an annual series on errors in tiering for Family Day Care Homes (FDCH). The 2012 study found an erroneous payment rate of

1.09 percent which corresponds to an estimate of $8.54 million in erroneous payments.10

In addition to the annual studies on improper payments in FDCHs, FNS conducts a study series called Access, Participation, Eligibility, and Certification (APEC) that examines improper payments in the National School Lunch Program (NSLP) and the School Breakfast Program (SBP). FNS expects the contractor to use and modify the data collection and analysis methods from the APEC studies and apply them to CACFP centers for this study. The APEC series utilizes a variety of methods (such as application abstractions, meal observations, surveys, and review of administrative data) to produce national estimates of erroneous payments for NSLP and SBP. A copy of the full report and supporting materials from the first iteration of the study are located here:

http://www.fns.usda.gov/nslpsbp-access-participation-eligibility-and-certification-study- %E2%80%93-erroneous-payments-nslp-and-sbp. The approved study plan for the second APEC study, currently underway, will be provided to the contractor selected for this study after award. In addition, the Office of Management and Budget package for the most recent APEC study is available by visiting: http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201208-0584-008.

Definition of Erroneous Payment Error and Types of Errors

The focus of this study is to determine the annual estimated amount of “erroneous payments” in CACFP-participating childcare centers (both Independent and Sponsored) as a result of certification and non-certification errors. For this study “erroneous payment” is defined as the gross of overpayments and underpayments of payments made in error for reimbursable meals served to children.

Certification errors are those that occur when a child is incorrectly certified as free or reduced price (eligible). These errors may occur as a result of household misreporting or administrative errors at the center or with the sponsor. Examples of errors associated with households include:

misreporting income, household size or program case numbers. Examples of administrative errors at either the center or sponsor levels include: incorrect determination of a child’s eligibility status and errors associated with recording or transmitting the eligibility status of children. All reimbursable meals served to enrolled children at participating centers are eligible for

8 http://www.iom.edu/Reports/2010/Child-and-Adult-Care-Food-Program-Aligning-Dietary-Guidance-for-All.aspx 9http://www.iom.edu/~/media/Files/Activity%20Files/Nutrition/CACFP/2012-FEB-07/Glantz%20Presentation%20020712_1.pdf 10 http://www.fns.usda.gov/sites/default/files/CACFPTiering12.pdf http://www.iom.edu/Activities/Nutrition/ChildAdultCareFood/2012-FEB-07.aspx http://www.fns.usda.gov/nslpsbp-access-participation-eligibility-and-certification-study-%E2%80%93-erroneous-payments-nslp-and-sbp http://www.fns.usda.gov/nslpsbp-access-participation-eligibility-and-certification-study-%E2%80%93-erroneous-payments-nslp-and-sbp http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201208-0584-008 http://www.iom.edu/Reports/2010/Child-and-Adult-Care-Food-Program-Aligning-Dietary-Guidance-for-All.aspx http://www.iom.edu/~/media/Files/Activity%20Files/Nutrition/CACFP/2012-FEB-07/Glantz%20Presentation%20020712_1.pdf http://www.fns.usda.gov/sites/default/files/CACFPTiering12.pdf reimbursement at the “paid eligible” rate. Examples of over and underpayments are provided in the table below. Children whose families receive benefits from SNAP, FDPIR, or State programs funded through TANF are categorically eligible for free meals. Children who are participants of Head Start or Even Start programs, in foster care, or experiencing homelessness are automatically eligible for free meals, without further application or eligibility determination.

Certification Status Actual Eligibility Program Participant Type of Payment Error Free Reduced Price Yes Overpayment Free Paid Yes Overpayment

Reduced Price Paid Yes Overpayment Reduced Price Free Yes Underpayment

Erroneous payments also result when centers or sponsors submit improper meal counts and claims for reimbursable meals and are classified as non-certification errors. A meal count may be incorrect because a meal is not counted and claimed in the category which reflects the child’s eligibility, because meals counts are improperly totaled or transmitted, or because reimbursement is claimed for meals which are not served or which are served to ineligible participants (e.g. adults/staff members). Therefore, improper meal counts can result in an erroneous payment for either the free/reduced price component of the reimbursement (if the meal is claimed in the improper category) or for the entire amount of the reimbursement (if a claim is paid for a meal which was not served or served to an ineligible participant). Error rates may differ based on the reimbursement method (actual count, claiming percentage, or blended rate) and this should also be considered when designing the study.11

Sources of Certification Errors Sources of Non-Certification Errors Administrative (i.e., misclassifying children either through application or categorical eligibility)

Meal Counts

Misreporting by households Meal Claims

FNS believes that determining how the interaction between certification and non-certification errors may affect the overall level of erroneous payments in CACFP sponsored and independent childcare centers presents difficult technical and methodological issues. Therefore, for this study, FNS is requesting separate national estimates for CACFP of the erroneous payments associated with certification and with non-certifications errors as well as the subcategories within each.

Subcategories of Certification Errors Subcategories of Non-Certification Errors Administrative:

1. Error in eligibility determination

2. Error in recording eligibility status

3. Error in transmitting eligibility status

4. Missing Income Statement/Case Number

Center Records:

1. Center record of totals

2. Sum of verified counts from individual classroom (or centers if multi-site)

Household Reporting:

1. Incomplete Income Statement/Case

Number

2. Categorical Eligibility

Center to Sponsor:

1. Center’s record of the report compared to the Sponsor’s record

2. Sponsor’s record of what center

11 http://origin.www.fns.usda.gov/cnd/Care/CACFP/aboutcacfp.htm http://origin.www.fns.usda.gov/cnd/Care/CACFP/aboutcacfp.htm

3. Household Income

4. Household Size reported

Sponsor to State Agency:

1. Sponsor’s record of what was reported to the State Agency

We are also requesting an evaluation of methods to adjust the estimate for erroneous payments based on misclassification to create a combined erroneous payments rate that would reflect the overall effect of certification and non-certification errors in the CACFP.

Important Considerations

While FNS expects that the methodology used for the APEC study series will provide some insight into approaches for studying improper payments within the CACFP, these programs are distinct in their respective operations. The APEC series focuses on meals that are served primarily in a lunchroom setting; whereas centers participating in the CACFP may use a variety of meal modalities (family-style, classroom, lunchroom, etc). Another distinction is that children may receive multiple servings upon request during the meal service or students may receive their meal/snack outside of the published meal hours. It should also be noted that the processes for meal counting and claiming for CACFP providers is dependent upon how the center is reimbursed (actual count, blended rate, or claiming percentage). These are just a few of the unique characteristics of the CACFP that will need to be considered when developing the appropriate methodologies for the successful execution of the study being requested. There are a number of flexibilities afforded to participating CACFP centers that may also impact the design of this study that offerors should consider and demonstrate that the project staff exhibit expertise in these areas. Some of these flexibilities are summarized here: http://www.fns.usda.gov/sites/default/files/CACFP15-2013.pdf.

These flexibilities could impact both the errors associated with administrative procedures as well as meal counting and claiming and should be thoroughly considered when designing the study.

C.3 STUDY OBJECTIVES AND RESEARCH QUESTIONS

FNS is seeking a contractor to conduct a study of improper payments in CACFP-participating childcare centers (both Independent and Sponsored). The purpose of this study is to determine the annual estimated amount of erroneous payments (both as a dollar amount and rate) in CACFP-participating childcare centers (both Independent and Sponsored) as defined above (overpayments, underpayments, and gross total of overpayments and underpayments) as a result of certification and non-certification error. Due to the various types of providers within CACFP, FNS is conducting IPERA studies on the types of providers separately and this solicitation focuses strictly on Independent and Sponsored childcare centers. This study will serve as a complement to the tiering studies that occur in family day care homes (FDCHs).

The study also has an additional task: The contractor shall provide estimation models for use by FNS Staff for annually updating erroneous payment estimates for CACFP using available extant data (such as routine administrative records), i.e. without the need for any additional data collection similar to those developed in APEC-I. The current aging models for the APEC study series use data collected during the sample school year as well as data that is readily available on an annual basis to predict erroneous payments in subsequent years. The approach is similar to those used in forecasting models; however the variables used to predict the outcomes of interest are based on http://www.fns.usda.gov/sites/default/files/CACFP15-2013.pdf actual rather than predicted data. These models were designed to capture associations between district characteristics and certificate error rates. For this solicitation, FNS is looking for models that might capture similar associations within CACFP.

The study objectives and research questions presented below are intended to meet the Congressional requirements under “The Improper Payments Elimination and Recovery Act (IPERA) of 2010” (P.L. 111-204) for estimating the extent of overpayments, underpayments and overall erroneous payments within CACFP-participating childcare centers.

All error estimates shall provide an overall estimate, and a breakdown of the contribution to the estimate of overpayments and underpayments. All error estimates should also be presented as a percentage, case error rate, and dollar amount of overall CACFP-program spending and program spending in CACFP-participating centers only. All estimates shall provide a gross and a net error rate and a separate assessment of under- and over-charges. The Offeror’s proposal shall include in an appendix not to be included in the page limit, a table presenting which tables from APEC-I the Offeror will and will not replicate, with an explanation for the latter. 12 Mean and distribution should be provided for all variables.

OBJECTIVE 1: Generate national estimates of Certification error and, separately, Non-certification error.

• The contractor shall provide breakdowns of the error rate by the following subcategories:

o Certification Error

National Estimate

• Underpayment rate estimate

• Overpayment rate estimate

• Total certification error rate

Household Reporting Error

• Incomplete Application

• Income

• Household size

• Categorical Eligibility

Administrative Reporting Error

• Eligibility Determination Error [Percentage of children misclassified and identification of what their proper classification should have been (i.e., x% of free children should have been paid, x% of paid children should have been free)]

• Eligibility Recording Error (centers and sponsors)

• Transmittal Error (centers and sponsors) o Non-certification Error Meal categorization: To what extent do center based errors in identifying a meal as reimbursable or non-reimbursable contribute to the error rate.

Center Records

• Center records of total

• Verified Counts

Center to Sponsor Records

12 http://www.fns.usda.gov/nslpsbp-access-participation-eligibility-and-certification-study-%E2%80%93-erroneous-payments-nslp-and-sbp http://www.fns.usda.gov/nslpsbp-access-participation-eligibility-and-certification-study-%E2%80%93-erroneous-payments-nslp-and-sbp

• Center’s report to Sponsor

• Sponsor’s record of Center’s report

Sponsor to State Agency Records

• Sponsor’s record of report to State Agency

OBJECTIVE 2: Develop estimation models for aging the National Certification Error and National Non-Certification Error data annually, including estimates of over-and under-payment.

• What modeling strategy will maximize accuracy in predicting errors?

• What are the potential problems and issues surrounding the use of existing data sources on the existing methodology? How does the proposed methodology for the refined models address these concerns?

• How do changes in the verification system (such as changes in verification requirements, shifts in the proportion of applications selected for random and focused sampling) affect the erroneous payment estimates?

• How do changes in the number of children categorically eligible affect the erroneous payment estimates?

• What are the standard errors and confidence intervals associated with these erroneous payment estimates?

• How reliable will the estimation model-generated annual estimates be in future years?

• What additional data could help improve the estimates generated by the estimation models? Assume FNS could modify the reporting requirements. What level of burden would new data collection impose on stakeholders?

OBJECTIVE 3: Describe Methodologies to generate State-Level estimates of erroneous payments

• Can econometric models similar to those used for national estimates of erroneous payments be developed at the state-level with existing administrative data? Which administrative data could be used?

• What resources would be needed to conduct on-site data collection to produce state-level estimates of erroneous payments due to administrative error? Due to meal counting and claiming? Due to a combination of administrative error and meal counting and claiming?

• Could a quality control system similar to the one used in the Supplemental Nutrition Assistance Program (SNAP) be used to develop State-level estimates for CACFP?13 If so, how?

• Could FNS implement a State-level review of applications similar to the Regional Office Review of Applications (RORA) currently conducted in the NSLP and SBP?14 If so, how?

13 http://www.fns.usda.gov/snap/quality-control

14 http://www.fns.usda.gov/es/regional-office-review-applications-rora-school-meals-2012 http://www.fns.usda.gov/snap/quality-control http://www.fns.usda.gov/es/regional-office-review-applications-rora-school-meals-2012

CLIN 1 Generate national estimates of Certification error and Non-certification error

CLIN 2 Develop estimation models for aging the National Certification error and National Non-Certification error data annually

CLIN 3 Describe Methodologies to generate State-Level estimates of erroneous payments

C.4 TASK AND DELIVERABLES

The awarded contractor shall perform all of the below to address the study objectives and provide the major deliverables below in a final form acceptable to FNS. In addition, the contractor shall arrange for external review of every draft report by organizing a peer review panel that consists of persons with subject matter expertise in CACFP, the IPIA/IPERA laws, and (modeling and estimation)(see F.5 Performance Measures below).

Task 1: Study Plan

The contractor shall submit a draft, revised and final study plan to incorporate all study objectives, as listed in section C.3. The plan shall explicitly address any implications for each of the study objectives. This study plan shall apply the methodology used for the APEC study series as well as insights from the CACFP Tiering study series. The study plan shall also address the pros and cons of using the APEC study series methodology as well as proposed modifications to strengthen the study.

The contractor shall meet with FNS staff to assess the challenges to ensure successful implementation of the study presented in the contractor’s proposal. There shall also be a discussion of how missing data will be handled.

In addition, the study plan will include a discussion of how the contractor will develop the sample design in order to obtain a nationally representative sample of CACFP sponsored and independent childcare centers. The plan must include a rationale for the selection criteria. The contractor shall take into consideration the need to contact the Regions and States in order to obtain lists of CACFP sponsors and centers within the States since these lists do not reside at the federal level. The contractor shall follow FNS approved procedures (Regions are contacted first through FNS and will give the go ahead for contact with States and other entities) for contacting their Regions and States in requesting lists of sponsors and independent centers as well as providers.

If stratified sampling is proposed the contractor shall describe the anticipated recruitment strategy.

The contractor shall be required, at a minimum, to contact: States to obtain lists of sponsors and independent centers within a sampled State; Sponsors to obtain lists of childcare centers reporting to them, contact information for the centers, and other administrative information relevant to the study;

and Childcare Centers to obtain any available additional information needed for the successful execution of the study.

The nationally representative responses for this study shall be sufficient to provide national estimates at ±.05 level of precision with .95 level of confidence. The responses received must be sufficient to yield estimates for agreed upon minimum detectable differences applicable to the variables of interest and at 95% level of confidence (at 0.80 power), and 90% level of confidence for any proposed subgroup analyses.

The Offeror shall strive to achieve a minimum of an 80% response rate. The study plan must describe the plan for analyzing data for which an 80% response rate did not occur, including noting cut-off points at which the contractor will determine data is not usable. Response rates shall be calculated according to standards established by OMB that applies to federal agencies, non-response bias analyzed, and appropriate adjustments made if bias exists. 15

Deliverables:

1.1 Final Study Plan

The contractor shall submit a draft, revised and final study plan incorporating FNS comments by email for approval after each review. The study plan will provide details of the research methods and design, sampling methodology, and recruitment strategy to meet the study objectives, response rate, and precision requirements. The study plan shall include, but is not limited to, identifying data that will be collected and the sources of this data, how the data will be collected, including the process for contacting sponsors and centers, the timeline for data collection, data variables, and how the contractor will clean, analyze, interpret and report the data. The study plan shall include draft table shells, with proposed analysis methods that are cross referenced by the research objective and question being addressed. The contractor shall also include specifications for quality control and protection of confidential information, as appropriate.

Task 2: Develop and Pilot Test Data Collection Instruments

The contractor shall develop all data collection instruments and procedures to collect data required for the study. FNS requires a full cognitive pre-test of the revised draft of all data collection materials and a memorandum summarizing the results. No single instrument should be tested on more than nine (9) respondents. The respondents included in the pretest should not be included in the study. The results of the pre-test will be used to refine the instruments and materials with procedures on how these will be implemented prior to submission for Office of Management and Budget (OMB) approval. A copy of the finalized data collection instruments (to include screenshots of all web-based or electronic instruments) will be submitted to FNS and included in the OMB package.

The contractor shall provide FNS representatives the opportunity to test all instruments in the format in which they will be administered. This opportunity shall be provided at a location and time specified by FNS. The contractor is responsible for ensuring that FNS is provided all the necessary items to be able to review and test the content and functionality of all the instruments.

15 http://www.whitehouse.gov/omb/assets/omb/inforeg/statpolicy/standards_stat_surveys.pdf.

2.1 Final Instruments and Procedures

The contractor must provide the draft, revised and final instruments and procedures to collect data required for the study to meet the objectives in section C.3. The instruments must be submitted in an electronic version for approval by FNS.

2.2 Pretest Memorandum

The contractor shall provide the draft and final pretest memorandum incorporating FNS comments by email for approval. The Draft Pretest Memorandum shall include a question by question analysis of results from cognitive pretesting, paying particular attention to how the draft instrument was changed based on pretesting.

The Final Pretest Memorandum will address the cognitive pretesting results and incorporate FNS feedback and suggested revisions or provide rationale for not making changes.

2.3 Recruitment Materials

The contractor will provide a draft, revised, and final of all recruitment materials.

The final recruitment materials shall be submitted for OMB clearance. The contractor will need to develop recruitment materials, including letters to be sent to FNS Regional Offices and State agencies, program representatives, sponsors and centers announcing the study and data collection period. The letter and all materials used must go through the FNS Clearance process as well as OMB clearance and will not be considered accepted until OMB has approved

it. The contractor will need to draft recruitment materials appropriate to each audience from which it anticipates collecting data.

Task 3 : Obtain Office of Management and Budget (OMB) Clearance

Requirements for Obtaining OMB Approval for an Information Collection

The Offeror shall develop a 60-day Federal Register Notice for publication that meets all requirements of the Federal Register Office. In addition, the Offeror shall submit to FNS a package requesting OMB approval for all data collection activities. The Offeror should be aware that the OMB data collection package will be reviewed at FNS by the Office of Policy Support (OPS), the Planning and Regulatory Affairs Office (PRAO), USDA’s National Agricultural Statistics Service (NASS), and USDA’s Office of the Chief Information Officer (OCIO), prior to being submitted to OMB for review and approval. Revisions to the package may be required after each level of review. The Offeror shall be required to update the OMB package after each set of required revisions and edits. The package shall not be accepted by the OPS COR until OMB gives final clearance of the package.

When developing the project schedule, the Offeror shall allow, at a minimum, 52 weeks for OMB’s final approval from the submission of the 60-day notice to FNS to final OMB approval.

The approval process may take less than this amount of time; offerors are advised that the amount of time for clearance is directly related to the quality of the initial submission of the OMB package. The OMB data collection package shall contain copies of all recruiting materials and data collection instruments (to include screenshots, if applicable) and a supporting statement as set forth in the revised Standard Form No. 83a, “Instructions for

Requesting OMB Approval under the Federal Reports Act, as Amended.” Special note shall be taken that scripts are required to be prepared and included in the OMB package for all communications (either by telephone, mail, email, or internet) with all persons and organizations contacted in connection with the project outside of the Federal government. If required, versions of the data collection instruments translated into other languages, such as Spanish, must be included in the OMB package. The time required in communications with respondents and separately for non-respondents contacted in the project must be included in the burden table. The OMB package should be submitted in MS Word format for submission to PRAO. The 60-day notice shall not be considered approved until it is posted in the Federal Register. The OMB package shall not be considered approved until OMB provides final clearance.

United States Department of Agriculture (USDA) guidance and instructions for preparing an Information Collection Request (ICR) are provided by OCIO at: http://www.ocio.usda.gov/policy-directives-records-forms/information-quality-activities. The Offeror is advised that there may be differences in format and practices of ICRs used for obtaining OMB Clearance at other Federal agencies and/or at FNS on prior studies, therefore, the ICR package will be rejected as unacceptable unless they are fully compliant with FNS clearance requirements current at the time of submission.

3.1 Federal Register Notice

The contractor will provide a 60-day Federal Register Notice for publication that meets all requirements of the Federal Register Office.

3.2 OMB Clearance Package

The contractor will provide a clearance package that provides narrative addressing each of the supporting statements (Parts A and B) with appropriate appendices (which should be listed in the table of contents in the order discussed). Offerors should anticipate the need to make revisions after each level of review. The contractor will also provide all the information necessary for completing the 83-I form. The link for the 83-I form: http://www.whitehouse.gov/sites/default/files/omb/inforeg/83i-fill.pdf.

The OMB package shall provide an explicit, concise description of the direct links between the study objectives, research questions, variables, instrument items, data analysis plans and desired products.

It shall include a summary of public comments received in response to the 60-Day Federal Register Notice concerning the information collection and any actions taken in response to these comments. The contractor may be asked to participate in a conference call presentation of the study to OMB, if requested. The package is not considered final and accepted until OMB approval is received.

Task 4: RecruitStates, Sponsors, and Childcare Centers

The contractor shall draw a nationally representative sample of Independent and Sponsored CACFP childcare centers. A draft, revised, and final summary report shall be submitted to FNS at the completion of the recruitment process. Progress on recruiting the sample shall be described in the monthly progress reports.

http://www.ocio.usda.gov/policy-directives-records-forms/information-quality-activities http://www.ocio.usda.gov/policy-directives-records-forms/information-quality-activities http://www.whitehouse.gov/sites/default/files/omb/inforeg/83i-fill.pdf http://www.whitehouse.gov/sites/default/files/omb/inforeg/83i-fill.pdf

Once the site selection has been finalized, the contractor will meet with the selected sites to develop an Memorandum of Understanding (MOU) including, at a minimum: a schedule, description of what is being requested, and designation of an individual as the primary contact person responsible for ensuring that the study is implemented properly. The contractor and centers will establish a MOU that details the decisions reached.

4.1 Recruitment Materials for Regions ,States, Sponsors, and Childcare Centers The contractor will provide a draft, revised, and final of all recruitment materials.

The final recruitment materials shall be submitted as part of the OMB clearance package. The contractor will need to develop recruitment materials, including letters to be sent to FNS Regional Offices and State agencies, program representatives, sponsors and centers announcing the study and data collection period. The letter and all materials used must go through the FNS Clearance process as well as OMB clearance and will not be considered accepted until OMB has approved them. The contractor will need to draft recruitment materials appropriate to each audience from which it anticipates collecting data.

4.2 Glossy Fact Sheet

The contractor shall prepare a glossy fact sheet or brochure describing the study.

The brochure will be distributed to CACFP sponsors, centers, and other stakeholders for recruitment into the study.

4.3 D Summary Report on Recruitment

A draft and final summary report shall be submitted to FNS at the completion of the recruitment process. This summary report will include, problems encountered, solutions implemented, results of the recruitment process, and characteristics of the final sample.

4.4 Teleconferences or Webinars with Sampled Parties

Prior to onsite data collection the contractor shall contact the states, sponsors, and centers to conduct an orientation meeting via teleconference to inform the sampled parties about the nature and scope of the planned data collection activities. It will also offer the contractor an opportunity to discuss the availability of administrative records, timing of meal periods, operating hours, and reach some agreement on the data collection methods most suitable for each entity. Included in this discussion will be a determination of the type of parental consent required to obtain individual child records.

4.5 Memorandum of Understanding with Sampled Parties

A MOU that details the decisions reached between the contractor and each entity shall be signed by both parties. Discussions will also take place to designate an individual to serve as a primary contact person responsible for enabling the collection of administrative data and scheduling on site data collection. The contractor shall submit to FNS a summary memorandum summarizing each of the orientation meetings within one week of completing all the teleconferences. This summary memorandum shall include all issues discussed during the teleconferences, the conclusions reached, and any issues remaining to be resolved.

Task 5: Data Collector Training Package(s)

The contractor shall develop the data collector’s training materials (including training scripts and practice exercises) to accompany the data collection instruments. If multiple types of training are provided, the contractor shall provide FNS with each training package.

Deliverable:

5.1 Data Collector Training Package(s)

The draft, revised and final training package shall include, at a minimum, an overview of the study, data collection instruments (e.g. screenshots, instructions), privacy protection, simulated data collection exercises, conducting surveys and any other necessary information to facilitate the collection of complete and accurate data. The data collection training materials shall be submitted to FNS for review and approval. The contractor shall provide FNS with a final data collection training manual that includes FNS comments and any changes to the study plan or instruments as required by OMB.

Task 6: Select and Train Data Collectors

The contractor shall select and train qualified staff to collect the data required for the study. The contractor shall use the materials developed to prepare the staff for data collection. This training shall include, at a minimum, an overview of the study, data collection instruments, procedures for data collection, and simulated data collection exercises. For security purposes all on-site data collection at sites shall be conducted by contractor personnel who have passed a criminal background check. The staff shall demonstrate the ability and experience to administer the instruments reliably and consistently through certification tests, role plays, or other methods to provide an assurance of readiness. This training shall be scheduled with adequate lead notice so that FNS, at its discretion, may attend and participate either in person or via phone. FNS may, at its option, invite others to observe.

Deliverable:

6.1 Memorandum on Training

The contractor shall submit to FNS a memorandum summarizing the training sessions indicating data collector’s readiness to collect data. All trainees must demonstrate proficiency in the data collection activities to which they are assigned. The memorandum should also detail issues encountered during the training and how they were resolved.

Task 7: Schedule and Conduct Data Collection

Following receipt of OMB approval, the contractor shall arrange with the appropriate personnel at the selected sites to collect data in accordance with the plan as set forth in the approved study plan.

The contractor shall provide FNS access to a read-only version of the data management system used to monitor the status of the data collection (see below F.5 Performance Measures).

As data are collected, the contractor shall create a database consistent with the procedures described in the approved Data Collection and Analysis Plan sections of the Final Study Plan. At a minimum, the creation of these data files shall include a 100 percent manual edit, verified data entry procedures, machine editing, and the production of descriptive statistics to ensure the integrity of the study data set.

The nationally representative sample selected for this study shall be sufficient to provide national estimates at the level of precision as described in Task 1.

Deliverables:

7.1 Bi-weekly Data Collection Status Reports and Conference Calls

Throughout the data collection period the contractor shall keep FNS advised on the progress of data collection. The contractor will provide bi-weekly updates on the progress toward the data collection goals. Contractor will also identify and describe any problems encountered during data collection as well as provide solutions. The format of these reports will be in a format (in Word, Excel, or PDF) mutually agreed upon by FNS and the contractor. The calls will be scheduled based on a mutually agreed upon time and date by FNS and the contractor.

Task 8: Create Study Database and Analyze Data

Following the procedures described in the approved Final Study Plan, the contractor shall create analytic files and conduct the appropriate analyses. If changes are made to the analytic framework after the Final Study Plan was approved, the contractor shall submit a Memorandum of Intent outlining these changes for FNS approval. This shall not change the Scope of Work nor have a substantial impact on overall anticipated costs. No revisions or changes to the contract shall occur without the approval of the contracting officer. The contractor shall provide FNS with a copy of all the data files that they used for generating the first draft of the final report. The contractor shall also provide available documentation that would allow FNS to analyze these files.

After the final report is accepted by FNS, the contractor shall provide a copy of the data files used for generating the analyses presented in the final report and also provide the documentation that was used by the contractor when running these draft files. The contractor shall provide FNS with fully documented data files in SAS, SPSS or other suitable analytic software approved by FNS, containing all of the data collected and used in the analysis. The contractor shall prepare and submit to FNS two (2) sets of CDs that include the final data files and data analytic files used to produce all results that appear in the final report. Contractors should plan to submit two versions of the final analysis files: restricted- use files for FNS’ Office of Policy Support’s internal use, and public-use files. The public-use files should be cleaned of all Personally Identifiable Information (PII) and Confidential Information. FNS will post the public use data files to its website for use by the general research community in accordance with the President’s Open Data Policy16. Thus, the public use files should be cleaned of all PII and confidential information, and have documentation sufficient for a researcher to use the data without further guidance from FNS or the contractor.

Contractors shall o provide adequate…