J-6_SURVEY_METHODOLOGY.docx

J-6 Survey Methodology for RGFSC III West Coast Contents

Executive Summary

· Composite ratings are determined by a random sample survey of patrons at USMC mess halls over each six month rating period.

· Survey methodology is derived from Using Statistical Sampling, a Government Accountability Office document that serves as a federal reference for conducting surveys, and The Survey Handbook, a document from the Department of the Navy’s Total Quality Leadership Office.

· The lower limit of the 95% confidence interval of the estimation of patrons’ satisfaction in each area, rather than the mean, determines composite ratings. This incentivizes reducing variation among USMC mess halls.

· A monitoring and control plan is presented to ensure the survey is consistently executed.

1. Introduction.

a. Performance Incentives and Award Fee criteria have been developed to focus contractor efforts toward accomplishing the primary goals relating to the RGFSC III Contracts as detailed in the Incentive/Award Fee Plan. A portion of this plan includes determining Composite Ratings that represent the contractor’s performance across CONUS mess halls under contract over a given period in four specific areas.

b. Surveys of patrons dining in all CONUS mess halls will determine Composite ratings of Food Quality, Service Quality, Cleanliness and Speed of Service. These composite ratings are elements of the calculation of Incentive Fees, and thus directly impact the dollar value of incentives awarded.

c. A well designed survey that is rigorously executed and controlled will (a) accurately reflect the contractor’s performance, (b) provide confidence that investments by the contractor to improve performance will be rewarded via the incentives, and (c) minimize contract disputes.

2. Purpose.

The purpose of this document is to describe the methodology to be used to determine the Composite Food Quality Rating, Composite Service Quality Rating, Composite Cleanliness Rating and Composite Speed of Service Rating for the RGFSC III.

3. References.

a. Survey Handbook, Archester Houston, Ph.D., Department of the Navy Total Quality Leadership Office (TQLO Pub No. 97-06)

b. Using Statistical Sampling, GAO Program Evaluation and Methodology Division (GAO/PEMD – 10.1.6), rev 1992.

4. General Methodology.

a. The general method for arriving at the Composite Ratings is to conduct an Equal Probability of Selection Method (EPSEM) sample survey of patrons’ satisfaction across all mess halls covered under RGFC III, including samples of meals at all times of day, over the incentive period. This method has been developed in accordance with reference a.

b. The survey will include questions covering the essential elements of each Composite Rating area (Food Quality, Service Quality, Cleanliness and Speed of Service).

c. The survey results will be analyzed, and a measure of central tendency (accounting for the accuracy of the survey) will be converted to a Composite Rating for each rating area.

d. The procedures of the sampling methodology were chosen to drive the service provider toward desirable outcomes.

e. In EPSEM sampling, randomness is critical to the quality of results. A Monitoring and Control Plan will be used to ensure the survey procedures are followed. This will ensure the contractor’s investment and improvement results are reflected in the incentives received.

5. Sampling Plan.

a. The Sampling Plan has been developed per reference (b) using the following steps:

(1) Formulate the sample objective.

(2) Define the sample unit.

(3) Define the population of interest.

(4) Develop the sampling frame.

(5) Determine the type of sampling.

(6) Describe sample selection procedures.

(7) Determine an acceptable confidence interval.

(8) Define the margin of error.

(9) Defining required sample size.

(10) Develop data collection and recording techniques.

(11) Choose the analysis method.

(12) Explain how missing data and outliers will be handled.

(13) Explain how ordinal ratings are converted to numerical scores.

(14) Explain why question five of the survey does not contribute to composite survey scores.

Each of these steps is detailed in Appendix I. A summary explanation of the major decisions follows.

b. Population. Each time a meal is served in a USMC mess hall, there is an opportunity to positively or negatively impact the patron’s quality of life. Given unlimited resources, we could obtain an exact measure of contractor performance by surveying every patron in all mess halls at every meal. Since we don’t have these resources, we must use EPSEM sampling in an effort to represent this same group. The population of interest is all patrons receiving meals at mess halls covered under the RGFSC III contract.

c. Confidence Interval.

(1) The Confidence Interval is a range of values above and below the mean of the sample that, with a pre-assigned degree of confidence, includes the value being estimated for the population. In this survey, we are estimating the satisfaction of all recipients of meals served from USMC mess halls.

(2) Within the Confidence Interval, we cannot determine where the true population satisfaction lies. In other words, the population satisfaction level could lie anywhere within the Confidence Interval with equal probability, including the upper and lower limits of the interval.

(3) When using an EPSEM sample to represent a population, there is a possibility that the population parameter will lie outside of the estimated range. The probability of this error occurring is known, and can be adjusted during analysis. The probability of this error is commonly referred to as either the alpha level or the confidence level. These terms are two inverse ways of identifying this probability. In this survey, we will use a .05 alpha level, which corresponds to a 95% confidence level. This means that there is a 5% chance that the population’s mean satisfaction score will lie outside of the confidence interval generated by our sample (2.5% chance that the true parameter is lower, 2.5% chance that it is higher).

(4) The dollar values associated with the upper and lower limits of the Confidence Interval will depend on the variation in the population responses (of which the standard deviation is a measure); the larger the variation, the larger the spread of values. It is important to note that the variation in population responses cannot be controlled by the sampling methodology. It represents the variation in satisfaction in the population. It can, however be changed by the contractor’s actions to improve the level of quality and service.

6. Sample Size.

The sample size depends on the desired confidence level, the margin of error, the variation in responses and the resources available to commit to rigorous sampling. Since we have set the confidence level at 95%, choosing the sample size is the result of trade-offs between the margin of error, variation identified in previous surveys, and the resources available to commit to the random sampling effort.

a. The margin of error is the maximum distance between the sample mean and the population mean. For this effort, we want to be fairly certain of the results in order to give the contractor reasonable assurance that successful improvement efforts will be rewarded. We have chosen a .05 margin of error in the scores as being appropriate. A .05 difference in satisfaction score will result in a maximum overall incentive difference of approximately $26,910.24. Food Quality (the largest portion of the incentive fee) makes up $11,532.96 of this difference, as calculated in Appendix V. This difference is smaller for the composite rating areas that receive smaller portions of the pool. A difference of this size would unlikely influence a contractor’s behavior and would not be considered material.

b. This means that we can assume with 95% confidence that the true population satisfaction score lies within .05 points of the mean score from the sample. Based on this margin of error, confidence level, and an average standard deviation for all composite areas of .8655 from FY17 surveys, the sample size for the incentive period would be 1,152. The sample size will be recalculated at the beginning of each incentive period based on the standard deviation of the previous incentive period. This will account for any changes in variation due to contractor efforts. The calculation method is outlined in greater detail in Appendix I.

c. The survey results will be made available to the contractor monthly in order to provide guidance for improvements, both by locality and by rating area (Food Quality, Service Quality, Cleanliness and Speed of Service).

7. Random Sampling.

In order for these surveys to represent the population to the accuracy specified, they must be truly random – that is, each recipient of a meal must have an equal probability of getting surveyed, as specified by the EPSEM.

a. The EPSEM selected is a stratified random sampling method. This will ensure randomness and equal representation between large and small mess halls. In this method, the sample will be stratified by mess hall. Every mess hall will conduct the same number of surveys.

b. To accomplish this, a monthly survey schedule will be constructed for each mess hall by a random draw from all meals expected to be served in each mess hall. This includes all meal times (breakfast, lunch, dinner, brunch, and ‘midrats’), and all days a meal is served (weekends, holidays, etc.). There will be an opportunity for dates to be adjusted prior to the sample to account for any mess hall closures. The Survey Schedule will not be given to the contractor.

c. Surveyors will visit the mess hall, or satellite, on the assigned day and meal time to conduct the surveys. Patrons will be randomly selected using a systematic sampling method. Patrons will be randomly selected to respond to the surveys through a standardized process that will be briefed to all surveyors, and can be found in Appendix I.

d. As a practical matter, the surveyors will conduct a small, fixed number of surveys as determined by the random selection process on the assigned days. This will not affect the randomness to any appreciable degree.

8. Weighting.

Weighting is a method used to ensure that all members of the population have an equal impact on the sample. It is often used to over-represent otherwise small segments of a population so that their input is equal to that of larger segments.

a. The Marine Corps wants to ensure that performance levels at all mess halls is satisfactory, and that the contractor is focusing on service at small mess halls with the same intensity that they do larger mess halls.

b. The stratified random sampling method was chosen for this reason. Each mess hall contributes the same number of survey responses to the overall sample, which will ensure equal representation between small and large mess halls.

9. Questionnaire.

a. After brainstorming the essential elements of the Rating areas, the questions were drafted by a team composed of men and women of different races, and examined for the following:

(1) Proper wording for the sample

(2) Ambiguity

(3) Overlapping response categories

(4) Leading questions

(5) More than one issue per question

(6) Redundant questions

(7) Gender, Race, Regional, Rank and MOS biases

b. Additional testing of the questionnaire was conducted at the mess halls at Henderson Hall and at Marine Corps Base, Quantico. There were no adverse comments regarding the surveys from the patrons tested, which included Male, Female, White, Hispanic and African-American patrons.

c. The survey script is provided in Appendix II.

d. The survey is provided in Appendix III.

10. Treatment of Results.

a. Data Cleaning. Survey data will be programmatically examined for obvious patterns that will cause rejection of an individual survey, such as all “N/A” responses or other peculiar patterns. These will be removed from the data and will not count toward the total sample required. The meal periods will be tracked to ensure these do not impact randomness (if these are random events, the sample remains random). If a negative impact is detected, changes to the sample plan will be made to correct.

b. Outliers. The survey responses are ordinal (Outstanding, Excellent, etc.) on a five point Likert scale. Therefore all surveys that pass data cleansing will be used.

c. Complaints. There are avenues other than the survey for patrons to submit complaints, minimizing the likelihood that the surveys will be used for this purpose. The Interactive Customer Evaluation (ICE) system is available and patrons with complaints will be directed to use ICE to register their complaint. Complaint data gathered in ICE will not affect scores generated by this survey.

d. If a survey shows ‘N/A’ for some but not all responses, the ‘N/A’ responses will not be counted in the Composite Rating, but the otherwise valid responses will remain included in the data.

e. Seasonality. There is no historical evidence that the time of year impacts the survey responses.

11. Calculation of Composite Ratings.

a. Per the contract, a single figure is entered for each Composite Rating.

b. The data used for each Composite Rating will correspond to that portion of the survey covering that area. All sub-questions will count equally (e.g., cleanliness of the service line, dining area and table count equally towards the Cleanliness Rating).

c. Rounding. We will round scores to the nearest hundredth. A score with a decimal value of .005 and higher will round to .01, while a score lower than .005 will round to .00. The maximum value of a 0.01 change in a Composite Rating results in a maximum Incentive difference of approximately $5,382.05. The Food Quality rating area, which makes up the largest portion of the incentive pool, would account for $2,306.59 of this difference. The maximum difference for the other areas will be smaller because they make up smaller portions of the incentive pool. A difference of this size would unlikely influence a contractor’s behavior and would not be considered material.

d. The lower limit of the Confidence Interval, rather than the mean, will provide the Composite Rating for each area. The rationale for this is that (a) the lower limit is as likely to represent the population satisfaction as is the mean, and (b) should the contractor’s efforts result in reduced variation within the sample being surveyed but no change to the mean, the incentive fee will increase because the range of the confidence interval will narrow and the lower limit will rise.

e. Using the example in Appendix V, and using the distribution obtained in FY17, the award range of the 95% Confidence Interval is expected to be the mean ± approximately $26,910.24. We assume that recouping some portion of this will serve as an incentive to reduce the variation (i.e., improve the lower performing mess halls), while at the same time it is not so far from the mean as to limit the number of proposals.

12. The RGFSC III Survey Process.

The RGFSC III steady-state survey process is mapped in Appendix IV.

13. Monitoring and Control Plan.

To ensure the integrity of the survey results, the following controls will be put in place by the Program Management Office:

a. Weekly

(1) Receive and record survey data

(2) Check data for variance: correct meal, day, and number of surveys

(3) Analyze variance for patterns

(4) Audit surveys for correct transcription

(5) Back up records

b. Monthly

(1) Create Sampling Schedule for following month

(2) Check Sampling Schedule for randomness (automated process)

(3) Distribute Sampling Schedule by last day of the month

(4) Calculate interim Composite Ratings (automated process)

(5) Independently verify ratings calculation & record (automated process)

(6) Meet with contractor & present verified interim Composite Ratings

(7) Randomly spot-check surveyors/data collection provider

(8) Record results & update training if required

c. Quarterly

(1) Check quarterly training (initial and sustainment) is complete

(2) Conduct Quality Assurance audit of controlling organization

(3) Record QA audit results. Adjust processes and/or training as required

d. Semi-Annually

(1) Compile Composite Rating report

(2) Internally route Composite Rating Report

(3) Archive Rating Period data, records, reports, etc.

14. Conclusions.

The methodology described here provides a well-designed survey that, if rigorously executed and controlled, leaves the contractor but one avenue to improve his incentive award; that is to improve the quality and service at USMC mess halls.

Appendix I. Sampling Plan

1. Formulate the sample objective. The objective of the survey is to estimate patrons’ satisfaction in composite rating areas by sampling patrons dining in mess halls covered under RGFSC III during the incentive period.

2. Define the sample unit. The sample unit is the individual patron.

3. Define the population of interest. The population of interest is patrons eating in mess halls covered under the RGFSC III contract.

4. Develop the sampling frame. The sampling frame is developed at each mess hall.

5. Determine the type of sampling. A stratified random sample will be used. The sample will be stratified by mess hall. Each mess hall will randomly sample meal periods after accounting for any mess hall closures, and will then conduct a fixed number of surveys at those meal periods. The patrons surveyed during these periods will be selected in accordance with the sample selection procedures described below.

6. Describe sample selection procedures. Each mess hall will conduct the same number of surveys each month. Through the use of an IT solution, surveyors will enter any planned mess hall closures for the following month to create a complete sampling frame of meal periods. The IT solution will then randomly select meal periods from the sampling frame during which the surveyors will conduct surveys. The surveyors will have a set number of surveys that will be conducted during each meal period. The surveyors will randomly select patrons to complete surveys, until the set number has been reached.

7. Determine an acceptable confidence interval. A 95% confidence interval will be used to determine composite ratings.

8. Define the margin of error. The margin of error is the maximum distance between the sample mean and the population parameter. A .05 margin of error has been selected to balance resources and accuracy as outlined in 6a of the J-6.

9. Defining required sample size. The sample size will be calculated using the following equation:

σ= the standard deviation of sample responses zα/2= the critical z score based on the confidence level E= the margin of error n= the sample size

The sample size will be recalculated at the beginning of each incentive period based on the average standard deviation for all composite areas over the previous incentive period. This will account for any differences in variation produced by contractor efforts. The number of surveys to be conducted monthly by mess hall will be calculated the following way:

n= the number of surveys to be conducted over the incentive period h= the number of mess halls in the region m= the number of surveys to be conducted monthly at each mess hall

The 6 used in the denominator is representative of the 6 month incentive period.

10. Develop data collection and recording techniques. The surveyors will conduct surveys using the script outlined in Appendix II. Patrons will record their responses in the survey presented in Appendix III. The ordinal survey responses are in the form of a five point Likert scale. These responses will be converted into a numerical score such that a 5 indicates the highest level of contractor performance, and a 1 indicates the lowest level of contractor performance.

11. Choose the analysis method. A 95% confidence interval will be calculated for each composite rating area. The lower bound of the confidence interval will be used to represent the composite rating for the area. The lower bound has been chosen for the reasons outlined in section 11d of the J-6.

12. Explain how missing data and outliers will be handled. During data cleaning, any surveys that have all N/A responses or other peculiar patters will be removed from the data set and will not impact the composite rating. Any N/A response on an otherwise valid survey (for example, N/A on bathroom cleanliness because the patron did not use the bathroom during that meal period) will not impact the composite score for the area, but valid responses from that survey will still be counted.

13. Explain how ordinal ratings are converted to numerical scores. The ordinal ratings will be converted to numerical scores according to the traditional five point Likert scale:

14. Explain why question five of the survey does not contribute to composite survey scores. Question five of the survey is for the Government to monitor customer satisfaction in menu variety. Menu variety is chosen by the Government in the J-4, and is not controlled by the contractor. Question five will not impact the composite rating for the contractor because the contractor cannot control the area it is assessing.

Appendix II. Survey Script

(SURVEYOR’S INSTRUCTIONS ONLY DO NOT SHOW TO RESPONDENTS)

PLEASE READ ALL INSTRUCTIONS BEFORE SURVEY.

Surveyors should arrive in civilian clothing to reduce possibility of rank bias. This page provides instructions for the person providing the survey (surveyor) to the customer (respondent). Please do not show these instructions to the respondents.

1. Items Required:

a. LFS Apps generated surveys (developed in accordance with Customer Survey Processes)

b. Appendix II, Survey Script

c. Pens / Pencils

d. Something to write on (clipboard)

e. Folder for completed surveys

2. The below is the text that will be used when addressing the respondent with regards to taking the customer surveys.

Note: It is crucial that the surveyor follow the direction to include the simple script to ensure that there is no influence or perception of influence by the surveyor.

3. Let the respondents take the customer survey without surveyor’s comments or influence.

4. Get the survey back from respondents, and thank them for their time.

Survey Script:

“Good <morning, afternoon, evening> I’m <your name> a Quality Assurance Evaluator for the Marine Corps. My task is to monitor this contractor’s performance. Do you have a moment to take a very short survey that will help improve the service and food at this mess hall? The survey is completely voluntary and we won’t take any personal information.”

If yes, Give form and writing utensil to the respondent. “Thank you. Please mark your response in the boxes next to each question with an x.”

After the respondent completes the survey and returns it to the surveyor, “Thank you for your time.”

If no, “Thank you.”

Appendix III. The Survey

1. Based on this meal, how would you rate the meal’s …

a. Appearance

b. Taste

c. Proper temperature

2. Regarding the mess hall staff at this meal …

a. How well were you able to communicate with the staff?

b. How would you rate the professional appearance of the staff?

c. How responsive was the staff to your needs?

d. How respectful was the staff to you?

3. How would you rate the cleanliness (include the senses of sight, touch, and smell) of each of the following areas?

a. Entrance

b. Service line

c. Dining area

d. Your table

e. Heads (restrooms)

4. Based on this dining experience and your meal selection, how would you rate the speed of service?

a. Speed of service

5. On your visits to this mess hall in the past 30 days, how would you rate the variety of food options?

a. Variety of food options

Appendix IV. The Steady-State Survey Process

Monthly Update to Contractor

Generate Monthly Survey List

Distribute List to Surveyors

Conduct Surveys per Schedule

Analyze data / Conduct QA

Transmit Responses to Control Organization Control Organization Monthly Schedule

Appendix V. The Incentive/Award Fee Calculation Examples

The total Incentive/Award Fee Pool is determined by the following equation:

m= number of meals served during the award period u= meal unit price p= industry standard profit margin a= incentive/award fee pool

Given 6,788,485 meals served, with a meal unit price of $8.09 and a 7% industry standard profit margin, the total Incentive/Award Fee Pool is:

The Incentive fee makes up 70% of this pool and is based on Composite Ratings. The Food Quality rating counts for 30% of the total pool, the Service Quality rating counts for 20% of the total pool, the Cleanliness rating counts for 10% of the total pool, and the Speed of Service rating counts for 10% of the total pool. The scores are measured out of 5 total points, and the portion of the 5 points earned determines the portion of the allocated funds for each area that is earned. So, if the following scores were achieved:

The award fee breakdown would be calculated as follows:

As referenced in sections 6a & 11e of the J-6:

If a .05 difference in the rating of Food Quality occurred (maximum difference allowed by the margin of error selected):

The difference between the lower bound and the mean (a .05 difference in score) is $11,532.96. When maximum error is calculated for all areas the total incentive difference is $26,910.24.

For further information on the Incentive/Award fee process, reference the J-7.

image4.png image5.png image6.png image7.png image1.png image2.png image3.png