Draft Task Order Proposal Evaluation Method 23 Sep.doc

DRAFT

Attachment XX

DELPHI METHOD (Modified) 1.0 General Information

This document provides the information necessary to competitively select a contractor for a given TLPS Task Order/Delivery Order (TO/DO). The modified Delphi Method is a technique for quantifying subjective or qualitative data for analysis purposes. It offers a procedure for arriving at a consensus among a group of knowledgeable individuals who are most familiar with the requirement.

The use of the Delphi Form minimizes the risk of a rushed formulation of, or commitment to, a specific contractor’s solution. Evaluators are asked to rank each contractor’s capability to meet the requirements/standards/factors specified on the Delphi Form. The forms are then collected, and each member's opinion on each criterion is presented for the review of other participants. Evaluators are allowed to adjust their rankings at this point. Following the evaluation review/update, a team leader is identified (usually the PM or Requirements POC) and the group is brought together to challenge each other's logic/rationale. As a result of this activity, a consensus is obtained for each criterion ranking. This convergence of opinion, in quantitative terms, results in the ranking of a given alternative.

An explanation of the fields found within the Delphi Form follows:

EC

Evaluation Criteria is a descriptive name of the requirements/standards/factors against which each contractor is measured. This information is only a short synopsis or descriptive name.

CD

Criticality Designators indicate whether an EC is classified as Critical (C) or Non-critical (NC). CDs are associated with weighting factor (WF) values below.

Non-Critical (NC) – WF values between 1 and 5

Critical (C) – WF values between 6 and 10

WF

Weighting Factors are numerical values used to establish the EC’s relative importance to the Air Force. Values range from 0 to 10, with 10 representing the highest degree of importance.

The contractor will receive the EC and the CD as part of the solicitation, but not the WF, as this is part of the evaluation team’s assessment of each criterion’s relative importance and an important element in the evaluation process.

2.0 Developing Evaluation Criteria 2.1 Evaluation criteria establish a uniform baseline against which Government evaluators evaluate each offeror's proposal to determine the proposal's value to the Government. Evaluation criteria may be quantitative or qualitative as shown below:

Quantitative ‑‑ meets requirement of 1000 hrs MTBF

Qualitative ‑‑ maximizes the use of Non-Developmental Items

Quantitative criteria employ specific objective (usually numerical) measures. An example is "the proposed design will meet the 5000 hour Mean Time Between Failures (MTBF) requirement." Failure to meet the criteria would result in a deficiency. Here, the writer of the criteria wanted to ensure that any system acquired would meet reliability requirements. Proposals failing to meet the criteria would be deficient and would be unacceptable for award. Using a quantitative criterion, an offeror who proposes 6000 hours MTBF (for example) might receive a strength based on exceeding the minimum requirement in a way beneficial to the Air Force and be contractually binding.

Qualitative criteria, on the other hand, are subjective. Failure to meet the criteria may represent a higher assessment of risk, a weakness in the offeror's proposed approach, or could be a deficiency if it posed unacceptable risk or was so inadequate as to be incapable of evaluation.

Evaluation criteria may be requirements driven or demonstrating a level of capability. An example of a requirement driven criterion is one that references a specific PWS paragraph, specification paragraph, or other contract requirement minimum level of compliance. A level of capability criterion requires the offeror to demonstrate a level of capability, which demonstrates the offeror’s understanding of the requirement.

When developing evaluation criteria for a TO/DO, take into consideration the nature of your specific requirement and the best method for evaluating those factors or elements that are important to the success of your acquisition. Is your TO/DO strictly a technical analysis or study that all prime contractors can accomplish? Then perhaps cost will be your most important evaluation factor. If you are procuring a complex software system, then a superior technical solution from a contractor with excellent past performance might be your priority. But what about the cost to design such a system? PMs must work within their respective IPTs to determine the best strategy for acquiring goods and services at the best value to the Government. Three different strategies for evaluating criteria under the TLPS contract are:

a. Combine technical and past performance factors

The advantage to this method is if past performance is already weighted in the Delphi worksheet, it need not be weighted separately.

b. Combine technical, cost, and past performance factors This method uses the final score in making the best value determination. However, one pitfall is with scoring cost – it can reduce the importance of the technical criterion.

c. Technical, past performance, and cost factors all evaluated separately The advantage to this method is it gives more flexibility in making a best value determination. Each factor can be weighed on its own merits. Any of the three factors may be used independently, or in combination, omitting those that may not be applicable to a specific requirement. If multiple factors are used, the weighting order will need to be listed, such as Technical #1, Past Performance #2, and Cost #3. Some, or all, factors can also be equal in weight.

Include risks identified during the risk assessment when developing your evaluation criteria. Risk is a critical element of this process – the better you can define the risks, the easier it will be to determine if an offeror’s solution has adequately addressed that risk. Evaluation criteria should focus on the critical capabilities the contractor will need to prove in order to ensure successful completion of your task. Provide enough description so the contractor knows what to provide. If you want contractors to explain how they are going to accomplish all the tasks required in your PWS, then one of the EC elements should say, for example, “Describe your technical approach to satisfying the overall requirements of the PWS.” The more detail included in the EC elements, or sub-elements if necessary, the easier it will be for evaluators to assess the offerors’ responses.

2.2 After developing your evaluation criteria, think about how you will assess the contractor’s ability to successfully complete those tasks or services in your requirement. Some areas you might consider include: (tailor to your specific requirement):

a.

Technical Requirements

(1) Contractor experience/knowledge related to the specific weapon system or subsystem

(2) Contractor experience/knowledge related to specific technologies, products, processes

(3) Contractor capability to provide services or items specific to the requirement

(4) Realistic approach to provide services or products

(5) Facilities and/or equipment to perform services or produce items

(6) Comprehensive systems engineering approach b.

Past Performance

(1) Experience with, and performance record on, similar projects

(2) Cost and schedule performance on similar projects

2.3 Next, designate whether criteria/requirements are critical (C) or non-critical (NC). Assign weight factors (WF) based upon the priority of the specific criterion/requirement. Make sure your WF is highest on your most important criteria/requirements; contractor proposals will focus on the highest-weighted factor(s). Be careful to note how the weighting will affect the overall score – the combined weights of several non-critical requirements can outweigh the importance of a few critical factors. In addition, too many critical elements can wash out the affect of any one factor.

a.

For non-critical (NC) items assign values between 1 and 5 b.

For critical (C) items assign values between 6 and 10

2.4 Cost and past performance factors: determine if either will be used, both will be used, or neither will be used. If both are used, choose whether they should be evaluated separately or combined; rank importance of each factor if evaluated separately. If both factors are combined, be careful to note how the combined weighting will affect the overall score.

3.0 Evaluating Proposals 3.1 Score each proposal from 0 to 5 in association with each criterion, 5 indicating the highest compliance level. A suggested guide for scoring is as follows:

SCORE

PROPOSAL FINDING

0.0 Unacceptable – proposal did not address criteria

1.0 Acceptable with high performance risk

1.5 Acceptable with moderate performance risk

2.0 Acceptable with low performance risk

2.5 Some requirements exceeded with high performance risk

3.0 Some requirements exceeded with moderate performance risk

3.5 Some requirements exceeded with low performance risk

4.0 Exceptional with high performance risk

4.5 Exceptional with moderate performance risk

5.0 Exceptional with low performance risk 3.2 All criteria listed will be evaluated.

3.3 Evaluators cannot reduce a contractor’s score for information not in the evaluation criteria.

3.4.

Each individual evaluator will evaluate each proposal.

3.5 The evaluation team will reach consensus and turn in one composite score.

3.6 Determine each contractor's effective value for each criterion by multiplying the assigned ranking by the weight factor (WF).

3.7 Determine the overall effective value of each proposal by totaling each of the values.

3.8 The proposal that supports the most critical requirements and has the highest total effective value is selected as the highest technically acceptable offer.

3.9 The decision authority will then weigh the technical evaluation against the cost evaluation to make the best value determination.

4.0 Past performance, if evaluated, will not be limited to TLPS performance, but may include relevant, recent past performance from other government contracts and/or commercial efforts.

SAMPLE EVALUATION WORKSHEET

Rating Total Contractor B

Rating Total Contractor C Rating Total

SAMPLE REQUEST FOR PROPOSAL (RFP) DATA SHEET

Rating Total Contractor B Rating Total Contractor C Rating Total

Total Evaluated Price (TEP) (See Note 2)

Instructions to Offerors:

1.

Contractor is to limit technical proposal to XX pages, font no smaller than Times New Roman, 11 point. Past performance information is included in the total page count (if applicable).

2.

Total Evaluated Price (TEP) Calculations:

a.

FFP Labor CLINs:

Cost reimbursable no-fee (CRNF) CLINs will not be included in the TEP calculations.

No adjustments will be made to the proposed hours

Verification of the rates will be made to ensure the proposed rates are not greater than the NTE rates in the basic contract b.

Labor Hour (LH) CLINs:

Cost reimbursable no-fee (CRNF) CLINs will not be included in the TEP calculations

Upward adjustments may be made to the proposed hours to satisfy the criteria of cost realism. Consequently, and for evaluation purposes only, the TEP may be different than proposed price. Downward adjustments will not be made to the highest proposed hours since the criteria of affordability will be implemented.

Verification of the rates will be made to ensure the proposed rates are not greater than the Not-to-Exceed (NTE) rates in the basic contract.

3.

Definitions:

a.

Affordable/Reasonable:

A price must be both reasonable and affordable. A price is reasonable if the proposed labor is not considered insufficient or excessive to perform the required effort. A price is affordable if it is within the Government’s funding profile. In determining affordability, all proposed costs will be considered; labor, travel, ODCs, and materials.

b.

Cost Realism of TLPS:

Cost realism analysis is an evaluation to ascertain potential cost understatement. Cost realism in the TLPS is determined by examining the proposed labor expense for cost reimbursable orders (labor hours and labor rates) to assess whether the estimated labor costs are realistic for the work to be performed, reflect a clear understanding of the requirements, and are consistent with the unique methods of performance described in the offeror’s technical proposal. In accordance with FAR 15.404-1(d), cost realism analysis shall be performed on all cost-reimbursement contracts [orders] to determine the probable cost of performance for each offeror. For LH contracts [orders] a cost realism analysis will be performed on the labor hours only.

c.

Total Evaluated Price (TEP) Adjustments:

On TLPS, for evaluation purposes, the TEP prices may be adjusted during the government evaluation for LH and Cost Reimbursement orders. If the cost realism analysis reflects that there is a significant difference between the proposed line item cost estimates for the labor CLINs and the government’s evaluation assessment, then adjustments will be made. The TEP on Firm Fixed Priced Orders will be calculated without adjustment to the proposed prices. The TEP on LH and CR may be calculated using the adjusted prices.

4.

Each proposal will be scored from 0 to 5 in association with each EC element, using the following scale:

SCORE

PROPOSAL FINDING

0.0 Unacceptable – proposal did not address criteria

1.0 Acceptable with high performance risk

1.5

Acceptable with moderate performance risk

2.0

Acceptable with low performance risk

2.5

Some requirements exceeded with high performance risk

3.0

Some requirements exceeded with moderate performance risk

3.5

Some requirements exceeded with low performance risk

4.0

Exceptional with high performance risk

4.5

Exceptional with moderate performance risk

5.0

Exceptional with low performance risk

Criterion must be tailored for each task order. Provide enough description so the contractors will know what to provide. In other words, let them know what you expect to see from them. The sample worksheet is for example only!

Total of Contractor B scores: (25+50+50+50+25+25+25+50+50+25+30+30 = 435)

Product of Weight Factor (WF) x Rating for Contractor A on each criterion.

Example shown: 5 x 2 = 10

Criticality Designator: Critical (C) or Non-critical (NC)

Weight factor:

Non-critical (NC) range must be 1-5;

Critical (C) range must be 6-10

Evaluator Rating:

Score between 0 and 5 (See para 3.1)