DRAFT_SFSP_PC_MAQ_PWS_Final.pdf

DRAFT

SFSP PARTICIPANT CHARACTERISTICS AND MEAL ANALYSIS OF QUALITY

PERFORMANCE WORK STATEMENT (PWS)

I. INTRODUCTION

The contractor shall conduct a nationally representative study of Summer Food Service Program (SFSP) participant characteristics and meal quality. The study shall include a subsample of children who are eligible for, yet do not participate in, the program. The study shall provide key information on the successes and challenges with increasing participation in SFSP and serving nutritious meals that are well-liked and consumed by SFSP participants.

II. BACKGROUND

Overview of the Summer Food Service Program

The USDA’s Food and Nutrition Service (FNS) is responsible for increasing food security and reducing food insecurity in partnership with cooperating organizations by providing children and low-income people access to food, a healthy diet, and nutrition education. FNS administers 15 nutrition assistance programs, including the Summer Food Service Program (SFSP).

The SFSP is a federally-funded, state-administered program. The SFSP reimburses providers who serve healthy meals to children and teens in low-income areas at no charge primarily during the summer months when school is not in session. During the school year, many children receive free and reduced-price breakfast and lunch through the School Breakfast Program (SBP) and National School Lunch Program (NSLP). SFSP is designed to fill the nutrition gap by providing meals to children during the summer months.1 SFSP was first created as part of a larger pilot program in 1968. It became a separate program in 1975.2

State education agencies oversee the program in most States, while SFSP is locally run by approved sponsors, such as school districts, local government agencies, camps, or private, nonprofit organizations. Sponsors receive payments from USDA, through their State agencies, for the meals served. Open sites operate in low-income areas where at least half of the children come from families with incomes at or below 185 percent of the Federal poverty level, making them eligible for free and reduced-price school meals. Meals are served free to any child who visits an open site. Enrolled sites provide free meals to children enrolled in an activity program at the site where at least half of them are eligible for free and reduced-price meals. Camps also may participate in SFSP. They receive payments only for the meals served to children who are eligible for free and reduced-price meals.2

At most sites, children receive either one or two reimbursable meals each day. Camps and sites that primarily serve migrant children may be approved to serve up to three meals per day. Sites cannot claim meal reimbursements for meals that exceed the maximum number of approved

1 http://www.fns.usda.gov/sites/default/files/sfsp/SFSP-Fact-Sheet.pdf.

2 http://www.fns.usda.gov/sfsp/frequently-asked-questions-faqs http://www.fns.usda.gov/sites/default/files/sfsp/SFSP-Fact-Sheet.pdf meals. The cap on the number of meals served is based on the capacity of the site to prepare and distribute meals.3

For a meal to be eligible for reimbursement, all food components in the required minimum serving sizes must be offered. However, summer sites may use offer versus serve (OVS) meal service that allows meals to be fully reimbursed if all food components are made available, but the child has declined a certain number of items. A food component is one of the food groups that comprise a reimbursable meal. Sites must offer the food components that comprise a reimbursable meal in at least the minimum required amounts. A food item is a specific food offered within the food components comprising the reimbursable meal. For example, separate ½ cup servings of peaches and pears are two food items that comprise one component, the fruit and vegetable component. All food components that make up a reimbursable meal should be served to each child all at the same time.4 Please click the following link for more details on the SFSP meal pattern requirements for reimbursable breakfast, lunch/supper and snacks.

Participation in the Summer Food Service Program

In July 2015, more than 2.5 million children participated in SFSP.5 However, summer meals served through SFSP (not including the Seamless Summer Option) reached only a little over 11 percent of the 22 million children receiving free or reduced price lunches during the school year.6

FNS implemented four different enhancements7,8,9 to the traditional SFSP to investigate ways to increase program attendance: the Extending Length of Operation Incentive (2010-11), the Activity Incentive (2010-11), the Meal Delivery demonstration project (2011-12), and the Food Backpack demonstration project (2011-12). The enhancement projects produced mixed results with the extension of operation days resulting in the most consistent participation increase.

However, none of these pilots has been implemented at a national level.

The Summer Electronic Benefits Transfer for Children (SEBTC) project uses electronic cards to deliver Supplemental Nutrition Assistance Program (SNAP) or Special Supplemental Nutrition

3 http://www.fns.usda.gov/sites/default/files/SFSP09-2014os.pdf.

4 http://www.fns.usda.gov/sites/default/files/NutritionGuide.pdf.

5 The USDA FNS National Data Bank provides a single official repository to support the analysis and public release of FNS program information through the Food Programs Reporting System (FPRS), data from various FNS programs are extracted and imported into the NDB database.

6 http://www.fns.usda.gov/pd/child-nutrition-tables.

7 L. Elinson et al. (2012). Evaluation of the Summer Food Service Program Enhancement Demonstrations: 2011 Demonstration Evaluation Report. Prepared by Westat. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service. Project Officer: Chan Chanhatasilpa.

Available from: http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation.

8 L. Elinson et al. (2012). Evaluation of the Impact of the Summer Food Service Program Enhancement Demonstrations: 2012 Demonstration Evaluation Report. Prepared by Westat. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service. Project Officer: Chan Chanhatasilpa. Available from: http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation.

9 S. Baldi et al. (2014). Evaluation of the Impact of Wave 2 Incentives Demonstrations on Participation in the Summer Food Service Program: FY 2012. Prepared by Insight Policy Research, Inc. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service. Project Officer:

Chan Chanhatasilpa. Available from: http://www.fns.usda.gov/evaluation-impact-wave-2-incentives-demonstrations-participation-summer-food-service-program-sfsp-fy.

http://www.fns.usda.gov/sfsp/sfsp-meals-and-snacks http://www.fns.usda.gov/sites/default/files/SFSP09-2014os.pdf http://www.fns.usda.gov/sites/default/files/NutritionGuide.pdf http://www.fns.usda.gov/pd/child-nutrition-tables http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation http://www.fns.usda.gov/evaluation-summer-food-service-program-enhancement-demonstrations-2011-demonstration-evaluation http://www.fns.usda.gov/evaluation-impact-wave-2-incentives-demonstrations-participation-summer-food-service-program-sfsp-fy http://www.fns.usda.gov/evaluation-impact-wave-2-incentives-demonstrations-participation-summer-food-service-program-sfsp-fy

Program for Women, Infants, and Children (WIC) benefits to eligible children not adequately served by congregate feeding sites. The SEBTC evaluation assessed the impact of a $60 per child monthly benefit amount to be used on SNAP- or WIC-approved foods, depending upon the grantee’s EBT system. The $60 per child monthly benefit is comparable to the cost of free lunches plus breakfasts under the NSLP and SBP. Grantees provide the SEBTC benefit to households with children from pre‐kindergarten through 12th grade who were certified for free or reduced-priced school meals during the previous school year. Data from 2012 revealed that SEBTC significantly reduced Very Low Food Security among children.10 In addition, over 90% of those issued a SEBTC card used at least some portion of their benefits during the summer of 2013, suggesting that participation in SEBTC is feasible for the majority of those eligible.

However, SEBTC also has not been implemented at a national level.

A 2003 evaluation by the Economic Research Service, “Feeding Low-Income Children When School Is Out—The Summer Food Service Program,”11 was the first major study on the SFSP since 1986 and revealed key findings surrounding program participation along with meal quality and plate waste. States participating in the study indicated that recruiting new sponsors and encouraging the sponsors to increase their number of sites were key hindrances in increasing participation in SFSP. Interestingly, the majority of sites reported that they had the capacity to serve more children. However, staff at all levels most frequently cited lack of transportation as a barrier to children’s participation in SFSP. While the study provided valuable information based on feedback from State and local SFSP administrators, interviews were not conducted with participants.

Lastly, FNS currently is conducting a national evaluation entitled “Evaluation of the 2013 Summer Food Service Program Participant Characteristics” that assesses the characteristics of the SFSP at the state, sponsor, and site level in the summer of 2015. This study also includes survey data from a small sample of SFSP participants (n=30) and non-participants (n=25) in an effort to determine ways to increase participation in the SFSP. The final report for this study will be available in the summer of 2016.

Meal Quality of the Summer Food Service Program

While FNS regularly obtains information on the foods and nutrients provided by school meals through the School Nutrition Dietary Assessment12 reports, minimal data is available on the

10 A. Collins et al. (2013). Summer Electronic Benefits Transfer for Children (SEBTC) Demonstration: Evaluation Findings for the Full Implementation Year. Prepared by: Abt Associates, Mathematica Policy Research, and Maximus. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service. Project Officer: Hoke Wilson. Available from: http://www.fns.usda.gov/summer-electronic-benefits-transfer-children-sebtc-demonstration-evaluation-findings-full.

11 A. Gordon et al. (2003). Feeding Low-Income Children When School is Out—The Summer Food Service Program. Prepared for the U.S.

Department of Agriculture, Economic Research Service. Washington, DC. Available from: http://www.ers.usda.gov/publications/efan-electronic-publications-from-the-food-assistance-nutrition-research-program/efan03001.aspx.

12 M.K. Fox et al. (2012). School Nutrition Dietary Assessment Study IV Final Report. Prepared by: Mathematica Policy Research. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service. Project Officer: Fred Lesnett. Available from:

http://www.fns.usda.gov/school-nutrition-dietary-assessment-study-iv.

http://www.ers.usda.gov/publications/efan-electronic-publications-from-the-food-assistance-nutrition-research-program/efan03001.aspx http://www.ers.usda.gov/publications/efan-electronic-publications-from-the-food-assistance-nutrition-research-program/efan03001.aspx http://www.fns.usda.gov/school-nutrition-dietary-assessment-study-iv content and characteristics of SFSP meals. The previously referenced 2003 report by the Economic Research Service provided data on meal quality and plate waste of SFSP meals. Based on the study findings, the most frequent foods served for lunch were 1% chocolate milk, cooked or canned peaches, French fries, corn, carrots, sandwich bread and rolls. On average, lunches were high in total fat, saturated fat and sodium. Ultimately, 1/3 of the energy and nutrients of the lunches served were not consumed by the SFSP participants.

A case-study of SFSP meal quality was conducted from a one-week menu of breakfasts and lunches from a sponsor in Ohio during summer 2014.13 The case-study indicated that breakfasts and lunches were low in fiber; lunches were high in sodium, total fat and saturated fat while breakfasts were low in these nutrients; and, vegetables were only served twice over the course of the week.

III. STUDY RATIONALE

USDA is dedicated to ensuring that all of America’s children have access to safe, nutritious, and balanced meals. The SFSP Participant Characteristics and Meal Analysis of Quality (SFSP PC MAQ) will provide up-to-date and nationally representative data on participant characteristics and meal quality. This information will help in identifying strategies to increase participation in SFSP and to serve healthy meals that are well-received by program participants.

IV. SCOPE OF WORK

The contractor shall evaluate why children and their caregivers participate in SFSP and their satisfaction with SFSP, collect characteristics of SFSP participants and eligible, non-participating children and their families, and assess why eligible children and families do not participate in SFSP. In addition, the contractor shall describe the foodservice characteristics of SFSP sites, assess the quality characteristics and content of SFSP meals and snacks, and assess the facilitators and barriers to preparing and serving SFSP meals.

V. RESEARCH OBJECTIVES AND QUESTIONS

The evaluation will focus on the following six research objectives and related research questions.

At a minimum, the contractors shall address the research questions provided below. Contractors may propose additional research questions to address each objective.

Objective 1: Evaluate why children and their caregivers participate in SFSP and their satisfaction with SFSP.

13 L. Hopkins and C. Gunther (2015). A Historical Review of Changes in Nutrition Standards of USDA Child Meal Programs Relative to Research Findings on the Nutritional Adequacy of Program Meals and the Diet and Nutritional Health of Participants: Implications for Future Research and the Summer Food Service Program. Nutrients 7, 10145–10167.

1. How do children and their caregivers first learn of the summer program? What are the different types of information sent out to families? Do caregivers receive sufficient information in advance to help them determine if their child will attend SFSP? If not, what additional information is needed and what are the best ways and most convenient times to receive this information?

2. Who makes the decision for children to attend SFSP? What are the most important factors when deciding to participate in SFSP (e.g., hours of operation, site location, staff, transportation, site activities, meal service)? What deters children and their families from participating in SFSP? For children, does this vary by age?

3. Do children find the meals appealing and satisfying? If applicable, do caregivers believe the meals are appealing and satisfying to their children? What factors lead to children and caregivers finding the meals appealing or unappealing, and satisfying or not satisfying?

4. How frequently are participants attending SFSP sites? Does their participation fluctuate over the summer? Are participants attending the sites as often as desired? Why or why not? What are the most common reasons cited by participants and their caregivers on how often they attend, what could improve their attendance, what are the barriers and facilitators for their attendance?

5. What is participants’ and caregivers’ overall satisfaction with the program? Do participants and caregivers recommended SFSP to other families?

6. Do caregivers plan for their child to participate in SFSP next summer? What are the key reasons for deciding, or not deciding, to participate in SFSP next summer?

Objective 2: Collect characteristics of SFSP participating families and eligible, non-participating children and their families.

1. What is the current food security status of the families?

2. What are the demographics of the children and their families (e.g., age, race/ethnicity, income)?

3. Do the children and families participate in other federal assistance programs? If so, which ones? Do children and families participate in other meal programs (e.g., NSLP, SBP, Child and Adult Care Food Program (CACFP))? If so, which ones?

4. Do the children and families participate in other summer meal programs (i.e., SEBTC, Seamless Summer Option (SSO)? If so, which ones?

5. What are the typical child care arrangements during the summer? Do the families participate in other summer programs?

6. What are families’ available means of transportation to the SFSP site? Does the site provide transportation? If so, do the families have to pay for it?

7. What is the average distance to SFSP sites for children and their families? Are families aware of and/or attending the closest available SFSP site?

8. Do SFSP participating families and non-participating families differ by food security status, demographics, participation in other federal assistance programs, summer child care arrangements, and proximity to SFSP sites?

Objective 3: Assess why eligible children and families do not participate in SFSP using a subsample of non-participants.

1. Have eligible, non-participating children and families heard of the summer program? If not, what are the best ways for summer sites to provide them with this information? If so, what are their perceptions of the program?

2. What are the most common reasons cited by non-participants and their caregivers for why they do not attend SFSP programs, what could get them to attend (e.g., accessibility, activities, cost, etc.), and what are the barriers for their attendance?

3. Have eligible nonparticipants ever participated in SFSP? If so, why do they no longer participate? What could have facilitated retention in the program?

4. Do eligible, non-participating children live in an area eligible for SFSP? If not, what is the average distance for these children to reach an area eligible for SFSP?

Objective 4: Describe the foodservice characteristics of SFSP sites attended by study participants.

1. What type of meals do sites most often serve (e.g., lunch and snack, breakfast and lunch, etc.)?

2. Where do sites obtain food for meals and how is the food transported? What percent of sites offer vended meals?

3. What are the sites’ food storage capacities (e.g., refrigerator/freezer capacity, pantry, etc.)? What are the sites’ food preparation and/or cooking capacities (range, oven, microwave, etc.)? Is the site equipped to serve hot meals?

4. Who plans the menus (e.g., sponsor, vendor, site, etc.)? What influences a site’s menu planning practices (e.g., sponsor guidance, training, participant preferences and equipment availability, etc.)? Are cycle menus used, and if so, how often? How frequently is offer versus served (OVS) being utilized?

5. Where and when are the SFSP meals and snacks served? What are the characteristics of the eating facilities? How long are participants given to eat the meals?

6. What type of meal service do the sites utilize (e.g., family-style, plated, etc.)?

7. How frequently are local, seasonal and/or USDA foods offered? If offered, what are the most frequent types of these foods offered?

8. What percent of sites offer à la carte foods? Are those foods for purchase, where are they sold and by whose control? What are the most frequent à la carte foods offered?

9. What policies and procedures are used to: accommodate children with food allergies or other special dietary needs; adjust the serving sizes of foods by different age groups; and ensure food safety?

10. For school-sponsored SFSP sites (e.g., academic summer schools), how often do the sites continue to utilize their NSLP and SBP menus during the summer months? What are the key reasons for continuing to use NSLP and SBP menus?

11. What percent of sites have approval to prepare meals according to the CACFP meal standards?

Objective 5: Assess the characteristics and content of SFSP meals and snacks.

1. What specific food items are offered the most often? Least often? What are the most frequent food items offered among each of the SFSP meal component groups?

2. How many choices are participants offered for various menu elements, e.g., milk, grains, meat/meat alternates, fruits, vegetables, other foods (desserts, salty-snacks etc.)?

3. What are the most frequent types of fruits and vegetables offered? How frequently do meals include both a fruit and vegetable? How frequently are the following vegetable subgroups offered: dark-green vegetables; red and orange vegetables; starchy vegetables;

beans and peas; and other vegetables? What are the most frequent vegetables offered for from each of the subgroups? Which forms of fruits/vegetables are offered most often (fresh, canned, frozen, dried)? How frequently are fruit and vegetable juices served and how frequently are they 100% juice?

4. How frequently are whole grains, lean meats/meat alternates and low–fat/fat-free dairy products offered? What are the most frequent types of these foods?

5. How often are other foods that do not qualify as a SFSP meal component offered? What are the most frequent types of these other foods?

6. How frequently are foods high in solid fats, added sugars and/or sodium offered? What are the most frequent types of these foods?

7. How do the foods offered vary from day to day and from week to week?

8. Are foods served in a way that eases consumption by younger children (e.g., pre-sliced apples, pre-peeled oranges, etc.)? If so, how frequently does this occur and with what specific food items overall and by different age groups?

9. How do the meals and snacks adhere to the SFSP meal requirements?

10. How do the SFSP breakfasts and lunches compare to the meal patterns and nutritional standards for SBP and NSLP?

11. What is the distribution and average nutrient content overall and for each type of meal offered/served, including snacks, in a typical week?

12. What are the major food sources of calories and key nutrients in the meals and snacks offered, including at a minimum, total fat, saturated fat, cholesterol, sodium, vitamin C, vitamin A, iron and fiber?

Objective 6: Assess the facilitators and barriers to preparing and serving SFSP meals.

1. What do sponsors and sites believe constitutes a healthy meal? What are the greatest challenges in planning and preparing healthy summer meals? Do sponsors and sites feel that they have the tools and resources to prepare healthy and reimbursable meals that are well-liked by participants? If so, what are those tools and resources (e.g., Team Nutrition, MyPlate, USDA Healthy Meals Resource System, USDA Summer Meals Toolkit, and USDA What’s Cooking? Recipe Bowl, etc.)? If not, what tools and resources do sites need to prepare those types of meals? What limitations, if any, do sponsors and sites have in accessing tools and resources?

2. What are the reasons for deciding to prepare meals on-site versus serving vended meals?

3. Do sponsors and sites try to incorporate local foods as part of their menus? If so, what are the benefits and challenges when serving local foods?

4. Do sponsors and sites feel that the meals served are well-received by participants? Why or why not? Do more participants show up on days when certain meals are served?

5. For vended meal sites, do sponsors and sites collaborate with the vendor on meal planning and preparation?

6. Are participants’ likes and dislikes, as well as cultural and ethnic practices, considered when planning meals? If so, how?

VI. GENERAL BACKGROUND INFORMATION ON TECHNICAL APPROACH

Qualifications required by a winning contractor of this acquisition include high level of knowledge of SFSP operations and extensive skills in research design, sampling, data collection, statistical analyses, and writing comprehensive research reports. The successful contractor’s proposal shall provide sampling, recruitment, and data collection plans that clearly and explicitly document their understanding of, and remedies for, the challenges associated with collecting data during the short timeframe necessary for SFSP data collection. The contractor also must have data collectors and interviewers who are proficient at collecting information from a wide variety of sources (e.g., State agencies, SFSP sponsors, SFSP sites, SFSP participants) using a variety of data collection techniques (e.g., CATI/CAPI data collection, on-site observation, record abstraction, and focus groups, etc.). The successful contractor must have a demonstrated track record of successful project completion involving collecting and analyzing, interpreting, and writing expertly about data based on food service records (e.g., menus, production records, recipes, etc.).

The contractor’s proposal shall address the evaluation parameters described below:

Study Design

The study design shall be nationally representative and include data collection with SFSP sponsors, site operators, participants and their families. The study also shall include a sub-study of eligible but not participating children, and their respective families to collect information on why they do not participate in SFSP. In addition, the contractor shall assess factors that might encourage or facilitate eligible yet non-participating children to attend SFSP in the future. Plans for sampling, data collection and analysis shall be included in the proposal and the contractor shall pay special attention to the challenges associated with sampling and collecting data from a national sample of SFSP sites. In particular, the contractor must consider in their sampling and data collection plan the challenges associated with 1) the lack of administrative data on open site participants; 2) the short timeframe of some SFSP sites (e.g., one-week camp); and 3) reaching participants at rural sites, early- and late-opening SFSP sites, and small sites. In addition, a mixed-methods approach, including qualitative and quantitative research methods, shall be utilized in order to address the research questions.

Sampling Frame

A database that includes all meal service sites that participate in the SFSP, along with various characteristics of those SFSP sites and their participants, does not currently exist. The closest database available to FNS is the data from the Form FNS-418 Report of the Summer Food Service Program for Children. State agencies are required to report the information on this form annually, which includes the number of meals served by SFSP site type (e.g., school, government, residential camp sites, etc.), the number of sponsors, sites, and the average daily attendance during the month of July. The contractor shall coordinate with FNS before communicating with any States and locals. FNS may provide the contractor with State agency contacts, if available, who can provide the most current list of SFSP sites and the necessary information for the nationally representative sample of SFSP sites.14

Sampling

The primary sampling strategy for the study shall produce estimates generalizable to all SFSP sites. FNS requires a nationally representative sample of SFSP sites that will yield population estimates with a precision of ± 5 percent at the 95 percent level of confidence for the overall SFSP site population. The confidence intervals for key subgroups shall not exceed ± 10 percentage points or 10 percent of mean values, respectively. Contractors shall provide initial sample size estimates, expected margin of error and minimum detectable differences expected for the total population and the various subgroups in their proposal. Expected subgroups include:

• Site location (rural, urban, suburban);

• Type of site (e.g., open, closed-enrolled, restricted-open, camp, migrant, homeless, National Youth Sports Program (NYSP), SFA-sponsored, indoors, outdoors);

• Length of operation (daily, weekly, and monthly); and

• Participant proximity to site.

14 Please see section VII for more details on the requirements related to Government Furnished Information.

In addition, the contractor shall develop highly reliable estimates for the sub-study of children who are eligible to participate, but do not participate in SFSP. Selection of respondents for any qualitative interviews shall be conducted in a systematic way to ensure the results are as representative as possible of the SFSP population.

Data Collection

There are numerous methods that could be used to collect information from the participants and non-participants of SFSP as well as on the operations and quality of the SFSP meals. These methods include mail questionnaires, in-person interviews, telephone interviews, focus groups, direct observation, and menu analysis. In presenting a data collection plan, contractors shall determine from whom and how best to collect program information from the SFSP in a timely and accurate fashion. In addition, the contractor shall determine the most effective way to ensure representation from participants who attend specific sites, such as: open sites (and thus not officially enrolled in SFSP); rural sites; small sites; sites only open for a short time (e.g., a one-week camp); and early- and late-operating sites.

According to OMB standards, a non-response bias analysis shall be conducted and weights adjusted accordingly if the response rate is less than 80 percent for any population.

Analysis

The majority of analyses will be primarily descriptive in nature with the contractor providing descriptive tables, figures and appropriate explanatory text. Data shall allow inferences to be drawn to the universe of all SFSP sites in the U.S. Descriptive statistics shall be presented for each tabulated item for the total population and by relevant classification variables. When comparing differences between subgroups, the contractor shall perform statistical analyses to test the significant differences between the subgroups. The contractor’s proposal shall identify and justify any statistical tests that the contractor plans to utilize.

In addition to descriptive statistics, multivariate analyses shall be considered to describe the relationship between a given variable of interest (e.g., participation rate, participant satisfaction, meal quality, etc.) and several explanatory variables.

Study and Analysis Plan

Contractors shall include a study plan in their proposal which presents the details of how they will address the research questions and objectives discussed in the Performance Work Statement.

At a minimum, the study and analysis plan shall include:

• Research objectives and their related research questions, as well as identification of variables and their relationship to research questions;

• Instrument specifications, development, and pre-testing plans, including rationale for selection pre-test respondents;

• Sampling and weighting plan covering the recommended sample sizes and methods of selecting SFSP participants and sites;

• Recruitment and outreach plans;

• Data collector training and monitoring;

• Data collection strategies and modes;

• Description and rationale for analytic techniques, including the link between the proposed analysis and the corresponding research question and evaluation objective;

• Data preparation activities, including but not limited to:

o Data verification, cleaning, entry and processing;

o Decision rules for dealing with missing, incomplete and out-of-range data;

o Data transformation activities and data reduction plans;

o Database design; and o Creation of restricted and public-use data files.

• Sample table shells, graphs, and figures that will be incorporated in the final report;

• A detailed description of tasks and deliverables. This section must include the proposed scheduling and sequencing of tasks, the identification of any key non-deliverable dates, and the designated time for FNS review of deliverables15. This section must include a strong emphasis on the proposed plan for successfully recruiting participants and SFSP sites and collecting data during summer operating hours.

VII. GOVERNMENT FURNISHED INFORMATION

Lists of State administrators of SFSP, their contact information and other necessary data needed for sampling and data collection may be provided to the contractor if available from FNS and only with at least 20 working days (4 weeks) advance notice from the contractor. The contractor shall maintain the confidentiality of the data and shall utilize the data only for this contract.

VIII. STUDY TASKS AND DELIVERABLES

Prior to submission to FNS, all versions of deliverables, including the initial drafts, shall be in final form and have undergone a thorough quality review process that adheres to the contractor’s quality control plan. In addition, each deliverable shall be reviewed by a professional copy editor. The contractor shall submit a cover memo with each deliverable that lists the personnel, including the copy editor, that reviewed the deliverable and deemed the deliverable acceptable for submission to FNS. In addition, all deliverables shall be consistent with the requirements of the most recent version of the Government Printing Office Style Manual. When multiple versions are necessary, each subsequent version shall be reflective of FNS’s feedback on the

15 See Section IX for additional details on the requirements for FNS’s review of deliverables.

prior version and include a separate, track-changes version that documents the changes made in response to FNS feedback along with any additional changes made by the contractor since the prior version, and provides justification for any comments made by FNS that were not incorporated into the revised version.

Task 1 Prepare a Revised Study and Analysis Plan

The Study and Analysis Plan shall be based on the contractor’s final proposal16 and changes made during the post-award meeting.

At a minimum, the plan shall include all variables listed on under the “Study and Analysis Plan” section on pages 10 and 11 of this PWS. In addition, the plan shall include all planned table shells, figures and graphs to show how the results will be presented. The plan shall include the supporting documentation noting the scientific literature that supports the data collection approach, survey instrument questions, and analysis plans.

The contractor shall arrange for external review of the draft study plan by persons with subject matter expertise in both the program and the methods used. The contractor shall submit the notes from the external reviewers and a summary of the changes made to the draft plan based on the external reviewers’ comments with the submission of the draft plan to FNS.

Deliverable 1.1: Draft Study and Analysis Plan with External Reviewer Summary Memo

Deliverable 1.2: Revised Study and Analysis Plan

Deliverable 1.3: Final Study and Analysis Plan

Task 2 Develop and Pre-Test Data Collection Plan and Instruments

The contractor shall develop all data collection instruments and procedures to collect data required for the study as outlined in the finalized Study and Analysis Plan. All data collection instruments shall include the supporting documentation for any survey questions that are utilized from other research studies.

FNS requires a full cognitive pretest of all data collection materials with respondents of the same demographics (comparable population) as those who will be responding to the instruments and a memorandum summarizing the results, including details on what was changed based on the pretest. The pretest shall be conducted on the revised version, after FNS reviews the initial draft and informs the contractor that the instruments are ready for pretesting. The revised version shall incorporate FNS’s feedback. Pretesting shall follow OMB guidance. The respondents included

16 The final proposal shall incorporate changes made during negotiations, including questions and answers, and any amendments related to the procurement.

in the pretest shall not be included in the study. The Pre-test Memorandum shall include a question-by-question analysis of results from cognitive pretesting, paying particular attention to how the draft instrument was changed based on pretesting. The memo shall include information on the amount of burden, clarity of questions, feasibility of administration, and reliability and validity of the instruments. The Pre-Test Memorandum shall be provided with the revised data collection instruments.

A copy of the finalized data collection instruments (to include screenshots of all web-based or electronic instruments) shall be submitted to FNS and included in the OMB package.

All translations shall be validated based on the scientific literature. The contractor shall translate the instruments in languages that ensure adequate and representative responses from the study participants. The pretest memo shall include an explanation of the translation and validation methodology, including any issues encountered and their resolutions.

The contractor shall provide FNS representatives the opportunity to test all instruments in the format in which they will be administered. This opportunity shall be provided at a location in the Washington D.C. area and at a time as agreed upon by FNS. The contractor is responsible for ensuring that FNS is provided all the necessary items to be able to review and test the content and functionality of all the instruments.

Deliverable 2.1: Data Collection Instruments and Procedures

Deliverable 2.2: Revised Data Collection Instruments and Procedures

Deliverable 2.3: Pre-Test Memorandum

Deliverable 2.4: Final Version of Data Collection Instruments and Procedures

Task 3 Develop Recruitment Communication Materials

Because this study anticipates collecting data from multiple audiences, the contractor will need to develop different recruitment materials for FNS Regional Offices, State Child Nutrition Program Directors, SFSP sponsors and sites, SFSP children/caregivers, and eligible, non-participating children/caregivers. The contractor shall prepare a draft letter for FNS to send to the Regions and States informing them of the objectives of the study and data collection procedures.

SFSP sponsors will need to verify that the sampled SFSP sites’ contact information (telephone number and address) is valid. State Directors may be requested to provide guidance in the appropriate protocol for contacting facilities.

The recruitment materials also shall include a consumer-friendly, glossy fact sheet or brochure describing the study purpose, activities, and timeline.

The contractor cannot contact any program representatives until the communication materials have been approved by FNS and the Regions and States have been notified about the data collection plans by FNS. In addition, all recruitment materials must be included in the OMB package in their final form. The deliverable schedule shall reflect this timing.

Deliverable 3.1: Draft Recruitment Materials

Deliverable 3.2: Revised Recruitment Materials

Deliverable 3.3: Final Recruitment Materials

Task 4 Prepare OMB Clearance Package

When developing the project schedule, the contractor shall allow a minimum of 52 weeks for OMB’s final approval from the publication of the 60-day notice in the Federal Register to final OMB approval. The approval process may take less than this amount of time; the contractor is advised that the amount of time for clearance is related to the quality of the initial submission of the OMB package.

The contractor shall prepare and submit a draft and final version of the Federal Register 60 Day Notice for publication that meets all the requirements of the Office of the Federal Register Document Drafting Handbook. The contractor shall draft responses to be sent by FNS for all public comments received in response to the published Federal Register Notice. The summary of comments shall include the actions taken to address the comments. The summary of public comments shall be included in Supporting Statement Part A and the individual comments and responses to the public comments will be provided as appendices to the OMB package.

The contractor shall submit to FNS a package requesting OMB approval for all information collection activities, including copies of all recruiting materials and data collection instruments and a supporting statement as set forth in 5 CFR 1320, Controlling Paperwork Burdens on the Public, the USDA Information Collection Package Submission Guidelines and OMB guidance including, but not limited to, OMB Guidance on Agency Survey and Statistical Information Collections and Revisions to the Standards for the Classification of Federal Data on Race and Ethnicity.

Note that scripts are required to be prepared and included in the OMB package for all communications (either by telephone, mail, email, or internet) with all persons and organizations contacted for data collection purposes. If web-based data collection is included, screen shots of web-based versions of instruments shall be included in the information collection package. If utilized, versions of the data collection instruments translated into other languages, such as Spanish, must be included in the OMB package.

http://www.archives.gov/federal-register/write/handbook/ddh.pdf http://www.archives.gov/federal-register/write/handbook/ddh.pdf http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf http://www.whitehouse.gov/omb/fedreg_1997standards/ http://www.whitehouse.gov/omb/fedreg_1997standards/

The contractor shall provide a clearance package that provides a narrative addressing each of the supporting statements (Parts A and B) with appropriate appendices. The contractor shall provide all the information necessary for completing the 83-I form. Here is the link for the 83-I form:

http://www.whitehouse.gov/sites/default/files/omb/inforeg/83i-fill.pdf.

The OMB package shall provide an explicit, concise description of the direct links between the study objectives, research questions, variables, instrument items, data analysis plans and desired outcomes.

For purposes of calculating response burden, the time required in communications with respondents and separately for non-respondents contacted in the project must be included in the burden table. The OMB package shall be submitted in MS Word format. The burden table shall be submitted in MS Excel as an appendix unless otherwise instructed by USDA.

The Information Collection Request package will be reviewed by the FNS Office of Policy Support (OPS), the FNS Planning and Regulatory Affairs Office (PRAO), USDA’s Office of the Chief Information Officer (OCIO), and USDA’s National Agricultural Statistics Service (NASS) prior to being submitted to OMB for review and approval. Revisions to the package may be required after each level of review. The contractor shall update the OMB package after each set of required revisions and edits, as necessary. Preparation of a high-quality initial draft, followed by timely responses to required revisions, will greatly facilitate the OMB approval process. The contractor shall participate in a conference call presentation of the study to OMB and/or draft responses to questions from OMB, as requested.

The 60-day notice shall not be approved until it is published in the Federal Register. The OMB package shall not be approved by the COR until OMB provides final clearance. For reference, recently published FNS Federal Register Notices are available online at www.federalregister.gov. Recently cleared FNS OMB packages are available online at www.reginfo.gov.

Deliverable 4.1: Draft 60-day Notice

Deliverable 4.2: Final 60-day Notice

Deliverable 4.3: Draft OMB Package

Deliverable 4.4: Revised OMB Package

Deliverable 4.5: Final OMB package

Task 5 Assemble Sampling Frame/Draw a National Sample of SFSP Participants http://www.whitehouse.gov/sites/default/files/omb/inforeg/83i-fill.pdf http://www.federalregister.gov/ http://www.reginfo.gov/

The contractor shall, in accordance with the Study and Analysis Plan, assemble the sampling frame and draw a nationally representative sample of children and families participating in the Summer Food Service Program. The contractor also shall draw a subsample of eligible children and their families who do not participate in the SFSP.

Deliverable 5.1: Summary Report on Final Sample

Task 6 Recruit the Sample

Following OMB approval, the contractor shall begin recruiting the study sample. Progress on recruiting the sites and participants shall be described in the monthly progress reports. A brief recruitment status report shall be submitted to FNS on a bi-weekly basis during the recruitment process. The report will document the sampling process, problems encountered and their solutions, results of the recruitment process and characteristics of the final sample. A final Summary Report on recruitment will be compiled as described above in the bi-weekly reports.

Deliverable 6.1: Recruitment Status Report (Bi-Weekly)

Deliverable 6.2: Summary Report on Recruitment

Task 7 Prepare Training Packages and Data Collection Manuals

The contractor shall develop data collection training manuals for use by data collectors to accompany the data collection instruments. The training packages shall include, at a minimum, an overview of the study, data collection instruments, procedures for data collection and simulated data collection exercises. Data collection training manuals shall be submitted to FNS for review. The training manual shall address all data collection and communication activities.

Prior to conducting the data collector training, the training manuals must be approved by FNS and address any comments received during the OMB clearance process.

Deliverable 7.1: Draft Data Collection Training Manuals

Deliverable 7.2: Revised Data Collection Training Manuals

Deliverable 7.3: Final Data Collection Training Manuals

Task 8 Recruit and Train Data Collectors

The contractor shall recruit and train data collectors to ensure a high quality data collection process. The data collector training shall take place as shortly before the actual data collection as possible. FNS Headquarters and Regional staff may attend the training sessions to observe and to serve as resources for the training. All trainees must show proficiency in the data collection activities to which they are assigned. The contractor shall prepare a memorandum summarizing the data collector training and provide explicit proof that the data collectors can competently complete data collection activities (e.g., certification test).

Deliverable 8.1: Training Summary Memoranda

Task 9 Collect Data

Data collection shall proceed according to the plan set forth in the finalized Study and Analysis Plan. Weekly status reports shall be submitted to FNS for the duration of this task. These reports shall include the progress that has been made (including, but not limited to, response rates per respondent group, change in response rates since previous biweekly report, number of non-respondents, number of attempts to reach non-respondents and mode of attempts, item level non-response, etc…), problems encountered and attempted solutions. If problems are encountered that cannot be resolved by the contractor, FNS shall be notified immediately (within 24 hours) to resolve the problems. At the completion of the data collection activity, the contractor shall prepare a brief memorandum summarizing the data collection effort.

Deliverable 9.1: Weekly Data Collection Progress Reports

Deliverable 9.2: Summary Report on the Quality of Final Data

Task 10 Create Study Database and Analyze Study Data

As the study data is received, the contractor shall undertake procedures to build a database and create analytic files consistent with the procedures described in the approved Study and Analysis Plan. Quality control measures shall be implemented to ensure the integrity of the data. The contractor shall provide FNS with analytic tables and figures as agreed upon in the Study and Analysis Plan. The analytic table and figure deliverables also shall include the analytical themes based on the qualitative analyses.

If changes in the analytic framework have occurred since the approval Study and Analysis Plan, the contractor shall submit a Memorandum of Intent outlining these changes for FNS approval.

Deliverable 10.1: Draft Analytic Tables and Figures

Deliverable 10.2: Final Analytic Tables and Figures

Deliverable 10.3: Memorandum of Intent (If Necessary)

Task 11 Prepare Final Study Report

The contractor shall prepare a draft, revised and final report that details the results of analyses. A detailed outline and subsequent drafts of the final report shall be submitted to FNS for review and comment. At a minimum, this report shall contain the following:

• Executive summary of the major evaluation findings;

• Introduction and background to the study;

• Delineation of study objectives and research questions;

• Overview of methods;

• Presentation of all findings of the study;

• A discussion of the study limitations;

• Study conclusions; and

• Technical appendices necessary to fully document all analytic procedures used, data collection instruments, and a detailed discussion of the methodology employed, such that another researcher could replicate the methodology without further consultation with the contractor.

A copy of the draft, revised and final report, including executive summary and all tables, graphics, and appendices, shall be submitted to FNS in two forms: a microcomputer word processing…