J_19 Survey Methodology 23March2010.doc

Survey Methodology for RGFSC II

Contents

21. Introduction

22. Purpose .

23. Audience .

24. References

25. General Methodology

36. Sampling Plan

3a. Sampling Plan

3b. Population .

3c. Confidence Interval

47. Sample Size .

48. Random Sampling

59. Weighting

510. Questionnaire

611. Treatment of Results

6a. Data Cleansing

6b. Outliers

6c. Complaints .

6d. Missing Responses

6e. Seasonality

612. Calculation of Composite Ratings

713. The RGFSC II Survey Process

714. Monitoring and Control Plan

815. Conclusions

Executive Summary

· Composite Ratings will be determined by a random-sample survey of 2,600+ Marines at USMC mess halls over each six month rating period.

· The methodology is from a GAO document Using Statistical Sampling, which is a reference for GAO auditors to conduct surveys.

· The survey was developed per the DON TQL Office Survey Handbook.

· The lower limit of the 95% Confidence Interval of the results, rather than the mean, will determine the rating. This is to incentivize reducing the variation among USMC mess halls.

· The survey responses are weighted to incentivize contractor improvement efforts on small mess halls as well as large ones.

· A Monitoring and Control Plan was developed to ensure the survey remains rigorously executed, and to minimize the likelihood of contractor disputes.

1. Introduction

a. Performance Incentives and Award Fee criteria have been developed to focus contractor efforts toward accomplishing the four primary goals relating to the RGFSC II Contracts as detailed in the Incentive/Award Fee Plan. A portion of this plan includes determining Composite Ratings that represent the contractor’s performance across CONUS mess halls under contract over a given period in four specific areas.

b. Surveys of Marines dining in all CONUS mess halls will determine Composite ratings of Food Quality, Service Quality, Cleanliness and Speed of Service. These composite ratings are elements of the calculation of Incentive Fees, and thus directly impact the dollar value of incentives awarded.

c. A well designed survey that is rigorously executed and controlled will (a) accurately reflect the contractor’s performance, (b) provide confidence that investments by the contractor to improve performance will be rewarded via the incentives, and (c) minimize contract disputes.

2. Purpose. The purpose of this document is to describe the methodology to be used to determine the Composite Food Quality Rating, Composite Service Quality Rating, Composite Cleanliness Rating and Composite Speed of Service Rating for the RGFSC II.

3. Audience. This pre-decisional document is intended for internal I&L use only, and should not be distributed to potential service providers. It is anticipated that a version of the methodology will be made available for proposal development.

4. References

a. Using Statistical Sampling, GAO Program Evaluation and Methodology Division (GAO/PEMD-10.1.6), rev 1992

b. Survey Handbook, Archester Houston, Ph.D., Department of the Navy Total Quality Leadership Office (TQLO Pub No. 97-06)

5. General Methodology

a. The general method for arriving at the Composite Ratings is to take an appropriately sized random sample survey of Marines’ satisfaction across all CONUS mess halls, including samples of meals at all times of day, over the incentive period.

b. The survey will include questions covering the essential elements of each Composite Rating area (Food Quality, Service Quality, Cleanliness and Speed).

c. The survey results will be analyzed and a measure of the central tendency (accounting for the accuracy of the survey), will be converted to a Composite Rating for each rating area.

d. The parameters of the sampling methodology were chosen to drive the service provider toward desirable outcomes.

e. In statistical sampling, the criticality of randomness to the quality of results is well understood. A Monitoring and Control Plan will be used to ensure the survey parameters are followed. This will ensure the contractor’s investment and improvement results are reflected in the incentives received.

6. Sampling Plan

a. The Sampling Plan has been iteratively developed per reference (a) using the following steps:

(1) Formulate the sample objective

(2) Define the sample unit

(3) Define the population of interest

(4) Develop the sampling frame

(5) Determine the type of sampling

(6) Describe sample selection procedures

(7) Define the confidence level

(8) Determine an acceptable confidence interval and sample size

(9) Develop data collection and recording techniques

(10) Choose the analysis method

(11) Explain how missing data and outliers will be handled Each of these steps is detailed in Appendix I. A summary explanation of the major decisions follows.

b. Population. Each time a meal is served in a USMC mess hall, there is an opportunity to positively or negatively impact the Marine’s quality of life. Given unlimited resources, we could obtain an exact measure of the contractor’s performance by surveying each Marine in all mess halls at every meal. Since we don’t have these resources, we must use statistical sampling in an effort to represent this same group. Therefore, the recipient of each meal served from a USMC mess hall is the population.

c. Confidence Interval

(1) When using a statistical sample to represent a population, there will be some error in the estimate of the population’s responses to the survey. These errors are known and calculable given the methodology we are using. They are called Confidence Level and Confidence Interval. Confidence Level will be discussed with Sample Size below.

(2) The Confidence Interval is a range of values above and below the mean that is believed, with a preassigned degree of confidence, to include the value being estimated. In this survey, we are estimating the satisfaction of all recipients of meals served from USMC mess halls. We have set the Confidence Interval at 95%.

(3) Within the Confidence Interval, we cannot determine where the true population satisfaction lies. In other words, the population satisfaction level can be at the mean, the upper limit or lower limit of the Confidence Interval.

(4) The dollar values associated with the upper and lower limits of the Confidence Interval will depend on the variation in the population responses (of which the standard deviation is a measure); the larger the variation, the larger the spread of values. It is important to note that the variation in population responses cannot be controlled by the sampling methodology. It represents the variation in satisfaction in the population. It can, however be changed by the contractor’s actions to improve the level of quality and service.

7. Sample Size. The sample size depends on the desired Confidence Interval, the Confidence Level (sampling error that can be tolerated) and the resources available to commit to rigorous sampling. Since we have set the Confidence Interval at 95%, choosing the sample size is the result of trade-offs between the Confidence Level and the resources available to commit to the random sampling effort.

a. The Confidence Level is the amount of sampling error that can be tolerated but that will still permit the results to be useful. For this effort, we want to be fairly certain of the results in order to give the contractor reasonable assurance that successful improvement efforts will be rewarded. We have chosen 1% sampling error as being appropriate, and the Confidence Level is therefore 99%.

b. In other words, we sample such that we are 99% certain that the population satisfaction is within the Confidence Interval.

c. The survey results will be made available to the contractor monthly in order to provide guidance for improvements, both by locality and by rating area (Food Quality, Service Quality, Cleanliness and Speed).

d. With roughly 6 million meals served per incentive period, a 95% Confidence Interval and a Confidence Level of 99%, the sample size is approximately 2,600 completed random surveys per period (6 months).

8. Random Sampling. In order for these 2,600 surveys to represent the population to the accuracy specified, they must be truly random – that is, each recipient of a meal must have a known probability of getting surveyed.

a. To accomplish this, a monthly Survey Schedule will be constructed by a random draw from all meals expected to be served in all mess halls. This will include all meal times (breakfast, lunch, dinner, brunch and ‘midrats’) and every day a meal is served (weekends, holidays, etc.). This Survey Schedule will not be given to the contractor.

b. Surveyors will visit the mess hall on the assigned day, and conduct the surveys. They will randomly select Marines at the mess hall to interview.

c. As a practical matter, the surveyors will conduct a small, fixed number of surveys (4 are planned) on the assigned days. This will not affect the randomness to any appreciable degree.

9. Weighting. Weighting is a method used to ensure that all members of the population are properly represented in the sample. It can also be used to over-represent certain segments; in other words, have a segment’s responses carry more weight in the overall rating.

a. Incentives shape the contractor’s actions both toward desirable outcomes and away from undesirable outcomes. An undesirable outcome (from the Marine Corp’s viewpoint) would be for the contractor to focus his improvements on larger mess halls and minimize efforts in the smaller facilities where there will be relatively fewer samples taken. To counter this, a weighting method will be used.

b. The issue of how to weight is similar to that faced by the Founding Fathers – how to prevent less populous states from being under-represented in government. We applied their solution and constructed a weighting system where each mess hall is represented in the sample proportional to the number of meals served (House of Representatives), and an additional representation where each mess hall is weighted equally regardless of the number of meals served (Senate). The results are combined equally to give the final Composite Rating.

c. This is a mathematical treatment to the results of the sampling. It will not increase the sample size, nor change the precision, but will have the effect of adding an incentive to improvements in small USMC mess halls in addition to the larger ones.

10. Questionnaire

a. After brainstorming the essential elements of the Rating areas, the questions were drafted by a team composed of men and women of different races, and examined for the following:

(1) Proper wording for the sample (Marines)

(2) Ambiguity

(3) Overlapping response categories

(4) Leading questions

(5) More than one issue per question

(6) Redundant questions

(7) The following biases

(a) Gender

(b) Race

(c) Regional bias

(d) Rank

(e) MOS

b. Additional testing of the questionnaire was conducted at the mess halls at Henderson Hall and at Marine Corps Base, Quantico. There were no adverse comments regarding the surveys from the Marines tested, which included Male, Female, White, Hispanic and African-American Marines.

c. The survey is provided in Appendix II. It is simple and succinct, and is unbiased as far as is able to be determined.

11. Treatment of Results

a. Data Cleansing. Survey data will be programmatically examined for obvious patterns that will cause rejection of an individual survey, such as all “N/A” responses or peculiar patterns. These will be removed from the data and will not count toward the total sample required. The facility, meal, etc. will be tracked to ensure these do not impact randomness (if these are random events, the sample remains random). If a negative impact is detected, changes to the sample plan will be made to correct.

b. Outliers. The survey responses are categorical (Outstanding, Excellent, etc.) and not open-ended or numeric. Therefore all surveys that pass data cleansing will be used.

c. Complaints. There are avenues other than the survey for Marines to submit complaints, minimizing the likelihood that the surveys will be used for this purpose. The Interactive Customer Evaluation (ICE) system is available and Marines with complaints will be directed to use ICE to register their complaint.

d. Missing responses on an otherwise valid survey will be ignored.

e. Seasonality. There is no evidence that the time of year will impact the responses. However, the data set will be analyzed for seasonal fluctuations. If these occur, a simple weighting method will be developed to compensate.

12. Calculation of Composite Ratings

a. Per the contract, a single figure is entered for each Composite Rating.

b. The data used for each Composite Rating will correspond to that portion of the survey covering that area. All sub-questions will count equally (e.g., cleanliness of the service line, dining area and table count equally towards the Cleanliness Rating).

c. Rounding. The maximum value of a 0.1% change in a Composite Rating results in an Incentive difference of approximately $665. A difference of this size would unlikely influence a contractor’s behavior and would not be considered material. The Composite Ratings will be rounded to one-tenth of a percent, with 0.05% and higher being rounded up, and less than 0.05% rounded down.

d. The lower limit of the Confidence Interval, rather than the mean, will provide the Composite Rating for each area. The rationale for this is (a) the lower limit is as likely to represent the population satisfaction as is the mean, and (b) should the contractor’s efforts result in reduced variation among the mess halls (a desirable outcome), incentives are there to reward this, even if the mean moves very little.

e. Using the example in the Incentive/Award Fee Plan, and using the distribution obtained in the Survey Testing, the award range of the 95% Confidence Interval is expected to be the mean ± approximately $133,000, or the mean ± 9.4%. We assume that recouping some portion of this will serve as an incentive to reduce the variation (i.e., improve the lower performing mess halls), while at the same time it is not so far from the mean as to limit the number of proposals. Calculations are shown in Appendix IV.

f. Regardless of the sampling methodology, there will be sampling error. The choice of a 99% Confidence Level will simply have to be accepted by the contractor. In real terms, the government accepts a 0.5% chance that the true population satisfaction is below the lower limit, and the contractor receives more Incentive than is warranted. The contractor accepts a 0.5% chance that the true population satisfaction is above the upper limit, and will receive less Incentive than is warranted. Reducing these chances of error will increase the sample size beyond practicality.

13. The RGFSC II Survey Process. The RGFSC II steady-state survey process is mapped in Appendix III.

14. Monitoring and Control Plan. To ensure the integrity of the survey results, the following controls will be put in place by the controlling organization:

a. Weekly

(1) Receive and record survey data

(2) Check data for variance: correct meal, day and number of surveys

(3) Analyze variance for patterns & impact on randomness

(4) Audit X surveys for correct transcription

(5) Back up data and records

b. Monthly

(1) Create Sampling Schedule for the following month

(2) Check Sampling Schedule for randomness, correct if required

(3) Distribute Sampling Schedule by 15th of the month

(4) Calculate interim Composite Ratings

(5) Independently verify ratings calculation & record

(6) Meet with contractor & present verified interim Composite Ratings

(7) Randomly spot-check surveyors/data collection provider.

Record results & update training if required

c. Quarterly

(1) Check for seasonal fluctuations in responses. Record analysis.

If found, design and implement corrections. Notify contractor.

(2) Check quarterly training (initial and sustainment) is complete

(3) Conduct Quality Assurance audit of controlling organization

(4) Record QA audit results. Adjust processes and/or training as required

d. Semi-Annually

(1) Compile Composite Rating report

(2) Internally route Composite Rating Report

(3) Archive Rating Period data, records, reports, etc.

15. Conclusions. The methodology described here provides a well designed survey that, if rigorously executed and controlled, leaves the contractor but one avenue to improve his incentive award; that is to improve the quality and service at USMC mess halls.

Appendix I: The Sampling Plan

The Sampling Plan has been iteratively developed per reference (a) using the following steps, with details as indicated:

a. Formulate the sample objective: Survey Marines dining in all CONUS mess halls to determine Composite ratings of Food Quality, Service Quality, Cleanliness and Speed of Service.

b. Define the sample unit: The sampling unit is the individual Marine immediately following a meal at the mess hall (before he has departed).

c. Define the population of interest: Each time a meal is served in a USMC mess hall, there is an opportunity to positively or negatively impact the Marine’s quality of life. Given unlimited resources, we could obtain an exact measure of the contractor’s performance by surveying each Marine in all mess halls at every meal. Since we don’t have these resources, we must use statistical sampling in an effort to represent this same group. Therefore, the recipient of each meal served from a CONUS USMC mess hall is the population.

d. Develop the sampling frame: The sampling frame is the method of obtaining the items in the population. This will be accomplished by creating a numerical list of each meal served on each day in every mess hall over a period (month), and assigning each meal a start and end number corresponding to the number of meals served at the corresponding meal in the prior year. The start and end numbers will not overlap. For example, assume that in the previous year there were 1 million meals served in all mess halls in January, and the first and second meals on the list served 200 and 350 meals, respectively. The numbering system for January of the current year would assign numbers 1-200 for the first meal in the first mess hall on the list, 201-550 for the second meal, and so on to the last meal where the end number would be 1 million.

e. Determine the type of sampling: Statistical sampling using simple random sampling. This is the most straight-forward type, and requires that each sampling unit (Marine eating in a CONUS mess hall) have an equal chance of being surveyed during the survey period.

f. Describe sample selection procedures: Create a monthly sample frame, and computer-generate a random set of numbers (size as required by the sample size) between 1 and the maximum number in the frame. The meals and mess halls corresponding to the random numbers will be sampled in that month. Unless there is a specific mechanical anomaly with the draw, the first set of numbers will be used. Multiple draws for any other reason will not be done.

g. Define the confidence level: 99%

h. Determine an acceptable confidence interval and sample size: 95% interval, with a sample size of 2600 completed surveys.

i. Develop data collection and recording techniques: Trained surveyors will travel to the mess hall for the surveys at the designated meal, and randomly select a fixed number of Marines, individually request that they take the survey using scripted language, then deliver a paper survey, recover the completed survey and finally thank the respondent using scripted language. The surveyors will electronically record the survey responses without filtering in any way, and send the responses to the controlling organization.

j. Choose the analysis method: Analysis consists of checking the data for seasonal variation, and eliminating records where the response was all “N/A” or other peculiar patterns. The data will be weighted to ensure attention is paid to small as well as large mess halls, and from the weighted responses, a 95% Confidence Interval will be calculated for each of the four rating areas.

k. Explain how missing data and outliers will be handled: Missing responses on an otherwise valid survey will be ignored. The survey responses are categorical (Outstanding, Excellent, etc.) and not open-ended or numeric. Therefore all surveys that pass data cleansing will be used.

Appendix II: The Survey

a. Appearance

b. Taste

c. Proper Temperature

a. How well were you able to communicate with the staff?

b. How would you rate the professional appearance of the staff?

c. How responsive was the staff to your needs?

d. How respectful were the employees to you?

a. Entrance (from outside to past the cashier)

b. Service Line

c. Dining Area

d. Your Table

e. Heads (restrooms)

4. Based on this dining experience, and your meal selection, How would you rate the speed of service?

5. On your visits to this mess hall in the past 30 days, how satisfied are you with the variety of food options?

Appendix III: Steady-State Survey Process

Appendix IV: Lower Limit Calculations

Assumptions: Population response variance = Survey Test variance Calculations based on Example: Incentive Fee = (# of meals served during period) x (KTR Fee CLIN) x (Quality Incentive Weight) x (Composite Rating) Constants:

# of meals served in period = 6,158,722

KTR Fee CLIN = $0.36 per meal

Difference Calculations:

Food Quality (Weight = 30%)

Composite Food Quality Rating

Mean Rating = 8.9 out of 10.0 (translated from 4.56 on scale of 1 to 5)

Lower Bound Rating = 8.16 out of 10 (translated from 4.26 on a scale from 1 to 5)

Food Quality Incentive Fee (Lower Bound) = $ 542,755.85

Food Quality Incentive Fee (Mean) = $ 591,976.36

Food Quality Difference $ 49,220.51

Service Quality (Weight = 20%)

Mean Rating = 8.2 out of 10.0 (translated from 4.28 on scale of 1 to 5)

Lower Bound Rating = 7.41 out of 10 (translated from 3.96 on a scale from 1 to 5)

Service Quality Incentive Fee (Lower Bound) = $ 328,580.14

Service Quality Incentive Fee (Mean) = $ 363,610.95 Service Quality Difference $ 35,030.81

Cleanliness (Weight = 15%)

Mean Rating = 7.4 out of 10.0 (translated from 3.96 on scale of 1 to 5)

Lower Bound Rating = 6.66 out of 10 (translated from 3.66 on a scale from 1 to 5)

Cleanliness Incentive Fee (Lower Bound) = $ 221,492.28

Cleanliness Incentive Fee (Mean) = $ 246,102.53 Cleanliness Difference $ 24,610.25 Speed (Weight = 15%)

Speed Rating

Mean Rating = 6.7 out of 10.0 (translated from 3.68 on scale of 1 to 5)

Lower Bound Rating = 5.96 out of 10 (translated from 3.38 on a scale from 1 to 5)

Cleanliness Incentive Fee (Lower Bound) = $ 198,212.31

Cleanliness Incentive Fee (Mean) = $ 222,822.56

Cleanliness Difference $ 24,610.25

Total Incentive at the Mean $ 1,424,512.40

Total Incentive at the Lower Bound of 95% Confidence Interval $ 1,291,040.58

Total Difference

$ 133,471.82 Appendix III: Steady-State Survey Process

Appendix I: The Sampling Plan

Appendix I: The Sampling Plan

Appendix II: The Survey

Appendix IV: Lower Limit Calculations

Appendix II: The Sampling Plan

Appendix II: The Sampling Plan

Pre-Decisional Material v2

_1301399492.vsd Control Organization Monthly Schedule

Generate Monthly Survey List

Distrinute List to Surveyors

Conduct Surveys per Schedule

Transmit Responses to Control Organization

Analyze data / Conduct QA

Monthly Update to Contractor