Attachment__5_-_NCSA_Task_Order_0001_Award_Fee_Plan_0001.docx

NCSA TASK ORDER – AWARD FEE PLAN

Contract Number:

Contractor

Coordination:

Board Chairperson: _________________________________ COL Monie R. Ulis

ARNG-HRR

Division Chief

Approved:

Fee Determining Official: __________________________________ Ms. Melissa Cossentino

NGB-PARC

DPARC

Table of Contents

1.0 Introduction

2.0 Organization

3.0 Responsibilities

4.0 Award Fee Evaluation Requirements

5.0 Evaluation Process

6.0 Award Fee Evaluation Methodology

7.0 Award Fee Periods

8.0 Modifications to Award Fee Plan

9.0 Award Fee Payment

Annex A – Award Fee Organization

AWARD FEE PLAN

1.0 INTRODUCTION

1.1 The purpose of this document is to outline the overall strategy, define responsibilities, establish procedures, and identify criteria for determining Non-Commercial Sustaining Announcements (NCSA) Award Fees applicable to (Contractor) under this Task Order. The Task Order specifically relates to support for the ARNG-HRR recruiting efforts. This plan also describes contract performance measures and provides an explanation of data sources, methodology, and performance thresholds associated with applicable Award Fees. The performance measures are defined in Attachment 04, NCSA Sample Task Order TV & Radio Designated Market Area (DMA) Acceptable Quality Levels (AQLs). Comment by Rubinstein, Judith S Ms CIV USA: Attachment 4 shows the number of airings for each DMA required to qualify for an excellent, very good, good, satisfactory, or unsatisfactory rating.

Attachment 3, and Section 6 of this document, indicate that the award fees will be based not only on the numbers of airings, as shown in Attachment 4, but also on the dayparts during which the airings occur. This introductory paragraph should so indicate.

Additionally, paragraph 3.7 and some of its sub-paragraphs, and 1.3, 4.4, 6.2.3 and 6.2.4, all imply that the award fee will also be determined on the basis of the contractor’s compliance with the requirements of the PRS. So do the first sentence of paragraph 1.3 and the third sentence of paragraph 1.7.

- If it is true that award fee determination will be based on the basis of the contractor’s compliance with PRS requirements, as well as on the basis of the numbers of airings, and on the dayparts the airings take place, I recommend revising this introductory paragraph to so indicate.

If that is not true, the above-referenced paragraphs should be revised so that they do not state or imply information that is not true.

Note also that the PARC office recommends adding a third metric for award fee determination, the third metric being the number of advertisements that air on stations that the target demographic is likely to watch or listen to. If this recommendation is adopted, this award fee plan must be revised accordingly, as must Attachment 3. Comment by Sicely, Brent W Sr SFC MIL USA: Added a block on my Attachment 6 Comment by Rubinstein, Judith S Ms CIV USA: Does the sample Task Order PWS call for placement of TV spots only, or does it also call for placement of radio spots? (See my comments on Attachment 2, the sample Task Order PWS, about this.)

If the sample task order PWS is calling for placement of TV spots only, the title of Attachment 4 must be revised accordingly, and the reference to Attachment 4 here in Attachment 6 must also be revised accordingly.

(Note also that if the figures in Attachment 4 reflect the combined numbers of radio and TV adds, those numbers need to be revised to reflect the required numbers of TV ads only. The same is true for Attachment 3.)

If the sample task order PWS is calling for placement of both radio and TV spots, leave this reference to Attachment 4 unchanged (but see the comment concerning use of the term “AQL”, which immediately follows the immediately comment). Comment by Sicely, Brent W Sr SFC MIL USA: Revised to read TV only Comment by Rubinstein, Judith S Ms CIV USA: Is the intended word “Defined” or “

Designated”? PWS paragraph 5.9.1 defines “DMA” as “Designated Marketing Area”, not as “Defined Marketing Area”. Comment by Sicely, Brent W Sr SFC MIL USA: Changed on my Attachment 6 Comment by Rubinstein, Judith S Ms CIV USA: This text accurately reflects the title of Attachment 4. However, I recommend revising the title of Attachment 4, because most of the figures shown in Attachment 4 really are not AQLs. They are really the performance standards that must be met in order for the contractor’s performance to receive the applicable rating (Excellent, Very Good, Good, Satisfactory, or Unsatisfactory).

An “Acceptable Quality Level” is the quality level that must be achieved in order for performance to be considered acceptable.

Except for the figures in columns O – R of Attachment 4, which are the columns which show the numbers of airings needed for a “Satisfactory” performance rating, the figures in Attachment 4 really are not AQLs. The figures in columns C – N, which show the numbers of airings for “Excellent”, “Very Good” and “Good” performance ratings, do not show the quality levels that must be achieved in order for performance to be considered acceptable. The figures in those columns show the numbers of airings that must take place in order for performance to be considered BETTER THAN acceptable.

The figures in columns S – T are not AQLs either. They represent “Unsatisfactory” performance, that is, performance that is NOT acceptable.

I recommend changing the term “Acceptable Quality Levels (AQLs)” in the title of Attachment 4 to “Performance Standards (PSs)” (or something like that). With this change, the title of Attachment 4 would read, “NCSA Sample Task Order TV & Radio Defined Market Area (DMA) Performance Standards (PSs)” (or something like that).

1.2 This Task Order with (Contractor) contains provisions for an Award Fee program. The primary objectives of the Award Fee program are: to motivate (Contractor) to provide services that exceed customer requirements and focus (Contractor)'s efforts in DMAs, in accordance with the defined AQLs. Comment by Rubinstein, Judith S Ms CIV USA: I recommend changing the term “AQLs” to “performance standards” or “PSs” (or something like that). See the last comment on paragraph 1.1 of this document. Comment by Sicely, Brent W Sr SFC MIL USA: Noted and this will be determined by Neal Dedicatoria.

1.3 (Contractor)’s performance will be assessed using the methodology outlined in this Task Order, the Performance Work Statement (PWS) and Performance Requirements Summary. References to the applicable PWS sections and weighting for the individual performance measures are outlined in the PWS. This action will be subject to a Firm-Fixed Price Award (FFP/AF) program, which recognizes minimum acceptable performance and rewards (Contractor) for increased levels of performance. Attachment 04 contains the required services (thresholds) and levels of performance that will be evaluated by this Award Fee program. The specific process for determining (Contractor)’s performance and corresponding Award Fee is set forth in paragraphs of this plan. Comment by Rubinstein, Judith S Ms CIV USA: This sentence implies that the contractor’s performance will be evaluated, and that the award fees will be determined, on the basis of compliance with the requirements of the PWS and the PRS (including report submission, use of logos, use of the ROI tracking tool, placement of spots in different languages, providing usage agreements, and posting advertisements on SBAs’ websites), in addition to being evaluated, with consequent award fee determination, on the basis of the standards shown in Attachment 4. This is not totally consistent with the next-to-last sentence of this paragraph, which implies that the standards shown in Attachment 4 will form the sole basis of evaluation of contract performance and award fee determination.

THE AWARD FEE PLAN MUST BE CLEAR AS TO THE BASIS OF AWARD FEE DETERMINATION.

See also the comments on paragraphs 3.7.1 and 3.7.2. Comment by Sicely, Brent W Sr SFC MIL USA: This is an AQ question for Mr. Dedicatoria Comment by Rubinstein, Judith S Ms CIV USA: It appears that this sentence is supposed to read, “References to the applicable PWS sections and weighting for the individual performance measures are outlined in the PRS [not “PWS”].”

Even with such a revision, this is not a totally true statement.

The PRS does not state weightings for the individual performance measures. It states the performance standards applicable to each listed task or deliverable, that is, the basis on which the performance of each listed task will be considered acceptable or unacceptable.

This sentence should either be deleted, or else, revised so that it states something that is totally true. Comment by Sicely, Brent W Sr SFC MIL USA: AQ question Comment by Rubinstein, Judith S Ms CIV USA: See the second comment on paragraph 1.1 (TV only, or radio also(?)).

Also see the comment on the first sentence of this paragraph 1.3 (award fee determination solely on the basis of the standards in Attachment 4, or also on the basis of compliance with PRS requirements(?)). Comment by Sicely, Brent W Sr SFC MIL USA: Changed and AQ should read comment 2 Comment by Rubinstein, Judith S Ms CIV USA: It appears that a specific set of paragraph numbers is supposed to be referenced here. Possibly, this should read, “is set forth in paragraph 6.0 and its sub-paragraphs” of this plan.”

If the suggested text does not accurately state the intended meaning, revise it accordingly. Comment by Sicely, Brent W Sr SFC MIL USA: AQ question

1.4 The Award Fee pool set forth in paragraph 7.0, Table 1, of this plan is established for this Task Order. A portion of the total Award Fee pool will be available for award in each of the evaluation periods specified. The Award Fee for each evaluation period is in addition to the agreed to pricing in the firm fixed fee portions of the contract.

1.5 Award Fees earned by (Contractor) will be awarded by the Fee Determining Official (FDO). Unilateral changes to the metrics or the associated methodology may occur up to thirty (30) calendar days prior to the beginning of an evaluation period. These changes will be implemented by means of a unilateral modification as described in Section 8.0. Any changes to metrics or methodology within the current period will be accomplished through a bilateral agreement. Comment by Rubinstein, Judith S Ms CIV USA: The usual practice is to state that the contractor “shall” do something, and that the Government “will” do something. Comment by Sicely, Brent W Sr SFC MIL USA: Accepted change Comment by Rubinstein, Judith S Ms CIV USA: The last sentence of this paragraph 1.5 contradicts this second sentence of this paragraph. The last sentence of this paragraph states, “Any changes to the metrics or methodology…will be accomplished through a bilateral [emphasis added] agreement.”

If changes will be made unilaterally, they cannot be made through a bilateral agreement.

If changes are made through a bilateral agreement, these changes are, by definition, bilateral, not unilateral.

The discrepancy between the second sentence and the last sentence of this paragraph must be resolved.

Note that the intention is that the contract that will include this award fee plan will be a contract for a commercial item. Under contracts for commercial items, all changes are made by mutual agreement of the parties. See paragraph (c), Changes, of the clause 52.212-4, Contract Terms and Conditions—Commercial Items. Changes can not be made unilaterally under contracts for commercial items.

If the service is to be acquired as a commercial item, the second sentence of this paragraph must be revised so that it does not discuss unilateral changes.

If ARNG wants to reserve the right to make unilateral changes to the metrics or the associated methodology, the service cannot be acquired as a commercial item.

The comments on this paragraph also apply to paragraph 8.1. See also the comment on the last sentence of paragraph 8.4 Comment by Sicely, Brent W Sr SFC MIL USA: AQ Comment by Rubinstein, Judith S Ms CIV USA: See the immediately preceding comment.

1.6 The determination of award-fee amounts are unilateral decisions made solely at the discretion of the Government.

1.7 Determination of (Contractor)’s performance and Award Fee eligibility will be based on attainment of the objective performance measures outlined in Attachment 4 and further explained herein. The method of assessing performance will be limited to (Contractor)’s activities and functions in performance as defined in this contract. (Contractor) will not be held responsible for failure to achieve performance measurements outlined in the PWS for reasons directly attributable to the Government. Examples of these are identified in the Introduction/Purpose section of the Surveillance Plan. Potential disagreements in this area are subject to the Disputes provision of this Task Order. Comment by Rubinstein, Judith S Ms CIV USA: This sentence does not appear to be totally consistent with the third sentence of this paragraph. This sentence indicates that the contractor’s performance will be assessed, and the amount of the award fee determined, on the basis of the numbers of airings (although not on the basis of the numbers of airings occurring during the preferred dayparts).

The third sentence implies that the contractor’s performance will be assessed, and the amount of the award fee determined, also on the basis of the contractor’s compliance with PRS requirements.

See the first comment on paragraph 1.1 of this document. Comment by Sicely, Brent W Sr SFC MIL USA: AQ Comment by Rubinstein, Judith S Ms CIV USA: See immediately preceding comment. Comment by Rubinstein, Judith S Ms CIV USA: I saw no such examples in the 10 October 2014 QASP that is posted in PCF.

Either provide a QASP that states such examples, or else, delete or revise this sentence. Comment by Sicely, Brent W Sr SFC MIL USA: AQ

1.8 The Government will assure that (Contractor) is given the opportunity to understand the basis for the award amount as defined by paragraph 5.5. Also, the Government will assure fairness of evaluation, and prompt and consistent feedback. This process will form the basis for Award Fee disbursements based on the final FDO determination.

2.0 ORGANIZATION

2.1 The award fee organization consists of the Fee Determining Official (FDO), Award Fee Evaluation Board (AFEB) Chair, AFEB Advisory Members, and the Award Fee Recorder.

3.0 RESPONSIBILITIES

3.1 Fee Determining Official (FDO). The FDO will:

3.1.1 Appoint the AFEB Chairperson.

3.1.2 Approve the Award Fee Plan.

3.1.3 Approve changes to the Award Fee Plan, and/or associated Award Fee criteria or methodology of determining amount(s) of Award Fee.

3.1.4 Be a member of the Award Fee board and determine the Award Fee earned for performance delivered during the evaluation period. Comment by Rubinstein, Judith S Ms CIV USA: The sentence is more grammatically correct with the indicated change. Comment by Sicely, Brent W Sr SFC MIL USA: Accepted change

3.2 AFEB Chairperson. The AFEB Chairperson will:

3.2.1 Chair the AFEB.

3.2.2 Provide executive-level feedback to (Contractor) on a regular basis in accordance with the areas of evaluation and criteria in the Award Fee Plan.

3.2.3 Appoint an Award Fee Recorder.

3.2.4 Recommend Award Fee Plan changes to the FDO.

3.2.5 Present the AFEB's recommended rating to the FDO.

3.2.6 Ensures sufficient training of all AFEB evaluators.

3.3 AFEB Advisory Members. AFEB Advisory Members are selected as appropriate to provide customer input and subject matter expertise to other AFEB members.

3.4 AFEB Members. The AFEB Members will:

3.4.1 Participate in the AFEB meetings and briefings.

3.4.2 Review performance reports for submission to the Award Fee Recorder. (Ref Para 3.7.3)

3.4.3 Review evaluation reports and recommend the rating and Award Fee amount to the AFEB chair and the FDO. (Ref Para. 3.7.4)

3.4.4 Recommend changes to the Award Fee Plan.

3.4.5 Maintain written records of (Contractor)’s performance within their assigned evaluation areas.

3.5 Award Fee Recorder. The Award Fee Recorder will:

3.5.1 Manage all activities associated with the AFEB process.

3.5.2 Consolidate assessments and recommendations for presentation to the AFEB and FDO at both the mid-term and final award (semi-annual and annual) fee evaluation periods.

3.5.3 Notify AFEB members of when and what required reports/briefings are due.

3.5.4 Record and distribute minutes of the AFEB meetings.

3.5.5 Maintain and update the Award Fee Plan as approved by the FDO.

3.5.6 Maintain the official Award Fee files, to include internal procedures and other documentation having a bearing on the FDO's decision.

3.5.7 Provide Award Fee overview and training for new team members when training is needed.

3.5.8 Document and maintain AFEB activities in official files.

3.5.9 Perform other duties as assigned by the AFEB Chairperson, including notification and scheduling of any advisory personnel needed by the AFEB.

3.6 Contracting Officer. The Contracting Officer will:

3.6.1 Act as the liaison between the (Contractor) and Government personnel.

3.6.2 Prepare FDO Award Fee determination letter along with a description of (Contractor)'s performance, strengths, weaknesses, improvement areas, and expectations for future performance.

3.6.3 Execute changes to the contract (which may include changes to the Award Fee Plan, the methodology in which any Award Fee is determined and distributed, or changes to the Award Fee pool available for Award Fee consideration.)

3.7 Lead Performance Monitor (LPM) and Performance Monitor responsibilities: the LPM will be designated by the Contracting Officer who will then designate Performance Monitors to monitor and collect performance data, as required, to accomplish tasks necessary to support the requirements of this plan and the (Program) Sustainment Surveillance Plan. All Performance Monitors will: Comment by Rubinstein, Judith S Ms CIV USA: Are the statements about IPTs and IPT Leads in paragraph 3.7 and its sub-paragraph true?

Will there by IPTs with one or more assigned evaluation areas?

If yes, no problem.

If no, revise paragraph 3.7 and its sub-paragraphs, so that the text states exactly what is true.

Examples of things to think about:

Will one IPT member assess report submission, and another member assess whether the required numbers of airings took place and took place during the preferred dayparts, and another member assess whether or not the contractor met PRS requirements concerning report submission?

Will one IPT member assess the numbers of airings in one region, and another member the assess numbers of airings in another region?

If the answer to at least one of those questions is “Yes”, the text of paragraph 3.7 might be reasonably accurate, although some of its sub-paragraphs may require revision. (However, see the immediately following comment.)

If the answer to both questions is “No”, that strongly suggests that some revision to paragraph 3.7 and its sub-paragraphs is necessary.

See also comments on paragraphs 1.3, 1.7, 4.4 and 5.0 – 5.4 and 8.3 of this document. Comment by Sicely, Brent W Sr SFC MIL USA: AQ question Comment by Rubinstein, Judith S Ms CIV USA: I did not see any document in PCF with the title, “Sustainment Surveillance Plan”.

Is the “Sustainment Surveillance Plan” the same thing as the “Quality Assurance Surveillance Plan”, which is referenced in paragraph 3.7.4?

A. If yes, for consistency, and to avoid confusion, I recommend referring to the “Sustainment Surveillance Plan” as the “Quality Assurance Surveillance Plan”.

B. If they are two different documents, put a “Sustainment Surveillance Plan” in PCF, and be sure that it accurately describes the methodology for Sustainment Surveillance that will be used for monitoring the contractor’s performance.

If there is no such document as a “Sustainment Surveillance Plan” that is a different document than the “Quality Assurance Surveillance Plan” for this acquisition, delete the reference to the “Sustainment Surveillance Plan”. Comment by Sicely, Brent W Sr SFC MIL USA: Noted and AQ question

3.7.1 Be familiar with the contract requirements and this Award Fee Plan and Comment by Rubinstein, Judith S Ms CIV USA: These paragraphs do not totally accurately reflect the text of the PRS.

The PRS for the basic contract largely discusses report submission (5 of the 12 rows) and required use of logos (3 of the 12 rows). Only 1 row of the basic contract PRS discusses obtaining the required amount of airtime, and there are no rows which discuss obtaining the airtime during the preferred dayparts. (The sample task order PRS does have a row addressing obtaining airtime during the preferred dayparts. (I recommend deleting that row for the reasons stated in my comment on paragraph 6.3 of this document.))

If contractor fails to meet PRS requirements in areas other than obtaining airtime (for example, if report submission is not timely), does that mean that the contractor will not receive an award fee, even if the required numbers of airings occur?

Also, the PRS does not really show performance rating criteria. Most of the rows of the PRS require the contractor to do something 99% or more than 99% of the time, the implication being that if the contractor does these things the required percent of the time, performance is acceptable, and if the contractor does these things less than the required percent of the time, performance is not acceptable. The PRS does not provide a basis for determining whether the contractor’s performance can be considered “Excellent”, “Very Good” or “Good”.

Examples of other things to consider:

- 2nd row of PRS:

If the monthly status report (MSR) is submitted before the 10th of the month, is the contractor considered to have exceeded performance requirements? If so, by when must the contractor submit the MSR to qualify for an “Excellent”, “Very Good” or “Good” rating for that requirement of contract performance?

- 10th row of PWS If the contractor posts the relevant advertisements on each State SBA’s web pages in less than 45 days of contract award, is the contractor considered to have exceeded performance requirements? If so, by when must the contractor post the relevant advertisements on each State SBA’s website in order to qualify for an “Excellent”, “Very Good” or “Good” rating for that requirement of contract performance?

Note that paragraph 6.0 of this award fee plan indicates that award fees will be based solely on the numbers of airings, and on the dayparts during which the airings occur, not on report submission, or use of logos, or posting advertisements on SBAs’ websites, or any of the other requirements addressed in the PRS. The text in the sub-paragraphs of paragraph 3.7 is thus not totally consistent with the text of paragraph 6.0.

Note also that paragraph 3.7.6 implies that the contractor award fee determination will be based on the numbers of airings, and on the dayparts during which the airings occur, while paragraphs 3.7.4 and 3.7.8 could be understood to mean that award fee determination will also be based on contractor compliance with PRS requirements, such as report submission, logo use, posting advertisements on SBAs’ websites, etc.)

See also comments on paragraphs 1.3, 1.7, 4.1, 4.4, 6.2.3 and 6.2.4 of this document. Comment by Sicely, Brent W Sr SFC MIL USA: I think this gets into the weeds of the PRS the Program Requirement Summary, especially the performance rating criteria for their assigned evaluation area(s).

3.7.2 Conduct assessments according to requirements and the NCSA PRS, assuring evaluations are fair and accurate.

3.7.3 Maintain written records of (Contractor)'s performance, “performance report” in their assigned evaluation area (s) that details specific examples where (1) improvement is necessary or desired; (2) improvement has occurred; and (3) performance is below, meets or exceeds contract requirements.

3.7.4 Review and coordinate evaluation reports prepared by AFEB recorder, for each period based on the processes outlined in the NCSA Quality Assurance Surveillance Plan for their Award Fee areas of evaluation. Comment by Rubinstein, Judith S Ms CIV USA: See comment on paragraphs 3.7.1 and 3.7.2. Comment by Rubinstein, Judith S Ms CIV USA: A QASP must accurately reflect the important tasks to be performed under the contract to which it pertains.

Note that the QASP for this acquisition that is in PCF lists a number of requirements which are not stated in either the PWS or the PRS.

Examples:

Paragraphs 8, 9 and 10 of the “Performance Requirements” section of the QASP respectively require the contractor to:

- Prepare an Annual Communications Plan detailing the contractor’s proposal to maintain, improve, and execute the NCSA and PSA program

- Prepare an Annual Outreach Opportunities Plan which explores NCSA- and PSA-related ideas and concepts and presents them for ARNG-GSS consideration

- Provide support the NAB Radio Trade Show.

Paragraph 12 of the QASP requires the contractor to submit the Monthly Invoice Supporting Data Report to ARNG-GSS and to include the detailed expenses that comprise the total amount due to the contractor on the monthly invoice.

Neither the contract PWS and PRS nor the sample task order PWS and PRS addresses any of these requirements.

Also, paragraphs 3, 4 and 5 of the “Performance Requirements” section of the QASP pertain to tasks associated with the preparation of creative materials. Since the contract for this acquisition of advertising services will not include preparation of creative materials – the creative materials will be prepared under a separate contract with a different contractor – QASP paragraphs 3, 4 and 5 thus appear to be inapplicable to this requirement. QASP paragraphs 3, 4 and 5 respectively require the contractor to:

- Support tasks associated with the Monthly Creative Materials Report.

- Make necessary arrangements to ensure preparation of appropriate creative materials for the SBAs

- Perform quality assurance prior to distributing creative materials packages to the individual SBAs

The QASP for this acquisition must be greatly revised so that it accurately reflects the important tasks to be accomplished under this contract, and so that it accurately mirrors the PWS and the PRS. Comment by Sicely, Brent W Sr SFC MIL USA: AQ question

3.7.5 Provide briefings and/or required reports to the AFEB recorder in preparation for performance evaluations and Award Fee determination.

3.7.6 Understand and monitor (Contractor)’s performance levels and the performance requirements necessary to achieve each level of performance as defined in Section 6.0, Award Fee Evaluation Methodology. Comment by Rubinstein, Judith S Ms CIV USA: See comment on paragraphs 3.7.1 and 3.7.2.

3.7.7 Be prepared to brief the AFEB on their specific assigned evaluation area(s).

3.7.8 Recommend changes to the Award Fee Plan; e.g., Award Fee pool reallocations, and criteria for weighting of performance areas, through the Award Fee Recorder. Comment by Rubinstein, Judith S Ms CIV USA: See comment on paragraphs 3.7.1 and 3.7.2.

3.7.9 Maintain open, honest, and frequent communications between team members.

4.0 AWARD FEE EVALUATION REQUIREMENTS

4.1 The Award Fee determination periods for FFP/AF will be twice a year. The specific evaluation period dates are shown in paragraph 7.0, Table 1. Award Fee evaluation and determination will cover areas of emphasis which reflect the balanced approach desired in order to achieve the performance goals of the NCSA Award Fee Program. Comment by Rubinstein, Judith S Ms CIV USA: This sentence could be understood to mean that award fee determination will be based on criteria other than the numbers of airings and dayparts during which the airings take place.

If that is true, it may not be necessary to revise this sentence.

If award fee determination will be based solely on the numbers of airings and dayparts during which the airings take place, this sentence should be deleted or revised.

See also comments on paragraphs 1.3, 1.7, 4.4 and 5.0 – 5.4 and 8.3 of this document.

4.2 The Award Fee process, being mostly objective in nature, will allow continuous monitoring of progress toward the performance levels associated with Award Fees. Continuous evaluation of performance will serve to advise (Contractor) of trends indicating strengths and/or weaknesses that might result in increases or decreases of Award Fee during evaluation periods.

4.3 All Award Fee periods will include an evaluation of (Contractor)'s performance. The purpose of the performance evaluations is to jointly assess (Contractor)’s Performance Base Line (PBL) performance during the period, identify strengths and improvements that occurred during the period, and recommend an appropriate Award Fee to be paid to (Contractor), if justified. Comment by Rubinstein, Judith S Ms CIV USA: What does the term “PBL performance” mean in the context of this paragraph?

What is the difference between saying that the purpose of the performance evaluations is to assess the contractor’s performance, and saying that the purpose of the performance evaluations is to assess the contractor’s Performance Base Line (PBL) performance?

If the contractor’s performance and the contractor’s PBL performance are two different things, what is the definition of “PBL performance”?

4.4 For each evaluation period, the PML will prepare the appropriate performance and evaluation reports documenting (Contractor)’s performance and corresponding Award Fee rating recommendation in accordance with the performance assessment methodology set forth in the (Program) Surveillance Plan. Comment by Rubinstein, Judith S Ms CIV USA: See comments on paragraph(s):

- 3.7, concerning IPT functions

- 3.7.1 and 3.7.2 concerning whether Award Fee determination will be solely on the basis of numbers of airings and dayparts during which the airings take place, or also on the requirements in the PRS, for example, report submission, logo use, and placement of advertisements on SBAs’ websites

- 3.7 and 3.7.4 concerning the Surveillance Plan

Revise this paragraph as necessary to make it state only what it is true.

4.5 Performance assessment results and supporting rationale will be reviewed, integrated, and compiled by the AFEB Chair, who will then develop and forward overall recommended to the FDO for use in determining Award Fee(s).

4.6 (Contractor) will be provided the results of these recommendations along with supporting material at least 5 business days prior to the FDO convening an Award Fee board. (Contractor) shall be provided an opportunity to address any ratings to the Award Fee board. Should (Contractor) deem it necessary, they will be provided the opportunity to submit an appeal of the board results to the Ombudsman.

5.0 EVALUATION PROCESS Comment by Rubinstein, Judith S Ms CIV USA: Will the evaluation process take place as described in the sub-paragraphs of paragraph 5.0?

If yes, no problem.

If no, revise the sub-paragraphs so that they describe how the evaluation process will actually take place.

5.1 Fourteen (14) calendar days prior to the end of the evaluation period, the Award Fee Recorder will establish a schedule for each required evaluation process step and notify the LPM, and AFEB Members of the upcoming Award Fee process requirements.

5.2 LPM will submit a coordinated performance assessment of each area of emphasis to the Award Fee Recorder. The assessment will be prepared cooperatively by the Performance Monitor for the area and reflect a recommended rating and areas of improvement for each area of emphasis. Comment by Rubinstein, Judith S Ms CIV USA: See comment on paragraph 3.7.

Do the terms “area” and “area of emphasis” mean determining if the required numbers of airings took place in a specific region of the country, or do the terms mean the different requirements summarized in the PRS, such as report submission, logo use, and posting advertisements on SBAs websites?

Does they mean something other than either of those things? If so, what do “area” and “area of emphasis” mean?

5.3 The Award Fee Recorder will compile an evaluation report with recommended Award Fee for review by the LPM. An assessment briefing based on the evaluation report will be prepared for review with the Performance Monitors not later than twenty (20) days after the close of the evaluation periods.

5.4 Not later than twenty-five (25) days after the evaluation periods, the LPM will present the assessment briefing to the AFEB.

5.5 The Award Fee Recorder will record AFEB discussions and deliberations in detailed minutes. These minutes, when combined with the briefings, will provide a clear and concise record of the events leading to the FDO decision. The minutes and briefings will comprise the official file to provide the support necessary to substantiate the Award Fee decision. The Award Fee assessments and recommendations proposed to the FDO will be shared with (Contractor) at least five (5) business days prior to the presentation to the FDO.

5.6 Not more than thirty-five (35) days after the evaluation periods, the AFEB Chair will present the Government’s assessment of strengths, improvement areas, and recommended ratings to the FDO. (Contractor), at their discretion, shall be given an opportunity to present additional pertinent information concerning the recommended ratings to the AFEB.

5.7 The FDO makes the final determination and awards the appropriate fee no later than forty-five (45) days after the evaluation period. The FDO through the Contracting Officer (KO) will notify (Contractor) in writing of the Award Fee earned and will provide an evaluation of (Contractor)'s performance.

5.8 The FDO will authorize the Contracting Officer (CO) to award the determined fee. The KO will issue a unilateral contract modification for the payment of the Award Fee to (Contractor) within fifteen (15) working days after receiving authorization from the FDO. The goal is to have the Award Fee payment to (Contractor) sixty (60) days from the end of the period.

5.9 The AFEB Chairperson may discuss with (Contractor), at an executive level, the decision of the FDO. However, the determination is a unilateral decision made solely at the discretion of the Government.

5.10 Should (Contractor) disagree with the decision of the FDO, they will be afforded the opportunity to submit an appeal of the decision to the Ombudsman.

6.0 AWARD FEE EVALUATION METHODOLOGY

6.1 Objective assessments will be used to evaluate (Contractor)’s overall performance and corresponding Award Fee during each evaluation period. The Award Fee criteria have been structured to achieve the performance-based objectives listed in paragraph 1.1. The Award Fee Pool will be funded by one source. The Pool will be based on the percentages as specified in Attachment 3 for each specified DMA and DMA Daypart, multiplied by the service price for each state. The Contractor must meet the number of statewide spots or returns on investment (total number of television spots) for the applicable State for consideration for each award fee. For a description on how Award Fees are earned, refer to paragraph 6.4.3. Comment by Rubinstein, Judith S Ms CIV USA: (Minor issue) “[C]riteria” is plural. Comment by Rubinstein, Judith S Ms CIV USA: Will the contractor be evaluated solely on the number of television spots, or also on the number of radio spots?

See the second comment on paragraph 1.1 of this document, and the first comment on Attachment 2, the sample task order PWS. Comment by Rubinstein, Judith S Ms CIV USA: Will the award fees be based:

- Solely on the numbers airings, and the dayparts the airings take place, as indicated here in paragraph 6.1, OR

- Also on the contractor meeting the requirements shown in the PRS concerning such things as report submission, as implied by the sub-paragraphs of paragraph 3.7, and paragraphs 1.3, 1.7, 4.1, 4.4, 6.2.3 and 6.2.4?

(The reference to “Attachment 02[*] of the Sample Task Order PRS” implies the latter.)

This issue must be clarified.

* The intended reference appears to be to “Technical Exhibit 1”, which is the PRS for the sample task order, not to Attachment 2. The sample PRS does not have an Attachment 02.

Possibly, the intended wording is, “and in the PRS of Attachment 02, the Sample Task Order PWS”.

The Sample Task Order PWS is itself Attachment 02 to the solicitation.

6.2 Technical Performance. Technical performance will be evaluated objectively utilizing the performance Award Fee framework as stated in Attachment 04 and in Attachment 02 of the Task Order PRS.

6.2.1 Under the FFP/AF PBL criteria, (Contractor) will be required to perform to the minimum “Threshold” (spots) requirement for each State at the aggregate to be considered for an award fee. Comment by Rubinstein, Judith S Ms CIV USA: See comment on paragraph 4.3.

6.2.2 A determination of the Government ratings will be based on monthly metrics as stated in Attachment 04 for each DMA and AQL level attained. The DMA award fee ratings are as follows: Comment by Rubinstein, Judith S Ms CIV USA: Attachment 4 shows quarterly metrics, not monthly metrics.

Will each month’s performance assessment be based on the metrics for the applicable quarter shown in Attachment 4?

Also, the term “monthly metrics” appears to be inconsistent with paragraphs 4.1 and 7.1 of this document, which respectively read, “The Award Fee determination periods for FFP/AF will be twice a year.” and “All Award Fee periods will be six (6) months in duration.”

Is the intended meaning that performance will be verified monthly, but that the award fee will be determined bi-annually? If so, this document should clearly indicate that meaning, as should the PWS and the PRS for both the basic contract and the sample task order. Comment by Rubinstein, Judith S Ms CIV USA: I recommend changing the term “AQL” to “performance standard” or “PS” or a similar term. See the fourth comment on paragraph 1.1 of this document.

Award-Fee Adjectival Rating Comment by Rubinstein, Judith S Ms CIV USA: FAR 16.401(e)(3)(iv) requires that award fee plans use the adjectival rating and associated description and the award fee pool percentages shown in this table, and allows for supplementation (not replacement) of the adjectival rating description.

The problem is that this table does not appear to accurately describe the methodology by which the amount of the contractor’s award fee will be determined.

Paragraph 6.1 describes the methodology by which the amount of the contractor’s award fee will be determined, and that methodology does not appear to be consistent with the methodology shown in this table.

It does not appear that either the percentages in the second column or the standards in the third column of the table will be used in determining the amount of the contractor’s award fee. The standards shown in columns C & D of Attachment 3 and in Columns C - N of Attachment 4, and the award fee percentages shown in columns L and M of Attachment 3, will be used to determine the amounts of the contractor’s award fees.

Although the presence of this table in the award fee plan indicates that award fees will be determined on the basis specified in FAR 16.401(e)(3)(iv), since the award fees will apparently not be determined on the bases shown in this table, in fact, the methodology for determining award fees does not appear to comply with FAR 16.401(e)(3)(iv).

** AQ and/or the requiring activity must either:

- Explain why the methodology for computing the award fees that is delineated in paragraph 6 of this sample award fee plan and in Attachments 3 and 4 complies with the requirements of this table, OR

- Develop a different methodology for determining the amounts of award fees that does clearly comply with the requirements of this table. **

It may be difficult to accomplish the first alternative. Saying that the contractor "exceeded almost all [or "many" or "some"] of the significant award-fee criteria" does not appear to make sense in the context of the award fee determining methodology described in paragraph 6.1 of this document. There are only two award fee criteria: (1) The numbers of times the advertisements aired in the preferred (top 100) markets, and (2) The numbers of times the advertisements aired in the most preferred markets (top 50) during the preferred dayparts. Even if we add a third award fee criterion, the numbers of advertisements that aired on stations which the target demographic is likely to watch or to listen to, that would still be only three award fee criteria. With only two, or at the most, three, award fee criteria, it does not make sense to say that the contractor exceeded "almost all" or "many" or "some" of the award-fee criteria.

Possibly, an argument could be made that since each criterion has multiple components (top 100 (or top 50) markets, and required numbers of airings (or required percentages of airings during the preferred dayparts) for 3 different performance levels (Excellent, Very Good and Good) for which award fees may be earned, it is possible to distinguish between meeting “almost all”, “many” and “some” of the award fee criteria. Even so, AQ would still have to demonstrate that the methodology for determining the award fees for “Excellent”, “Very Good”, and “Good” performance meets the requirements in the second column of the table that that methodology allows the contractor to earn 91%-100%, 76%-90% and 51%-75%, respectively, of the award fee pool for meeting the applicable performance standard.

One could also argue that exceeding the standards in almost all of the top 100 (or top 50) DMAs counts as “Excellent” performance; exceeding the standards in many DMAs counts as “Very Good” performance; exceeding the standards in some DMAs counts as “Good” performance; and meeting the standards in all DMAs, but exceeding the standards in no DMAs (or only a very few DMAs) counts as “Satisfactory” performance. However, the fact that for each DMA, the determination as to whether the contractor’s performance in that specific DMA is “Excellent”, “Very Good”, “Good”, “Satisfactory”, or “Unsatisfactory” is made with respect to that DMA exclusively, and not with respect to all of the top 100 (or top 50) DMAs collectively, makes it difficult to use this argument.

If it is the case, as indicated in paragraph 3.7 and its sub-paragraphs, and in paragraphs 1.3, 1.7, 4.1, 4.4, 6.2, 6.2.3 and 6.2.4 that award fee will also be determined on the basis of compliance with PRS requirements, one might be able to argue that the contractor can exceed almost all, many, or some of the contract requirements.

The Daypart Award Fee ratings are as follows:

Except for the bottom row of this table, the texts of the third column indicate that the numbers in the middle column are the percentages of airings that must have been played during PT and LE in order for the contractor to earn the rating listed in the first column, not the percentage of the award fee pool available to be earned.

Description

Unsatisfactory Comment by Rubinstein, Judith S Ms CIV USA: The text in the rightmost column in this row does not appear to make sense.

If the contractor’s performance is “Unsatisfactory”, it does not make sense to say that the “Contractor has exceeded the Daypart Group plays[.]” It would make much more sense to say that the contractor “has failed to meet” or “did not meet” (or similar language) the required number of Daypart Group plays.

Also, if the contractor’s performance is “Unsatisfactory”, and the contractor is totally ineligible for any award fee, why is it necessary to describe the percentages of plays that must have taken place during the preferred dayparts in order for the contractor to qualify for any award fee?

Although columns S –V of Attachment 4 do list numbers of airings for “Unsatisfactory” performance, the numbers in those columns do not appear to make sense.

Unless I am misunderstanding something, any number of airings that is less than the numbers of airings shown in columns O – R, which are the numbers of airings required for “Satisfactory” performance, would constitute “Unsatisfactory” performance.

For example, Column R shows that 4453 is the number of airings needed for a “Satisfactory” rating for 4th quarter for the Mobile-Pensacola (Ft. Walton) DMA in Alabama.

One would therefore conclude that 4452 or fewer airings for the 4th quarter in that DMA is considered “Unsatisfactory” performance.

However, Column V shows that 4380 airings, not 4452 airings, in that DMA during the 4th quarter is considered “Unsatisfactory” performance.

Suppose there are 4434 airings for the 4th quarter in that DMA. Is that considered “Satisfactory” performance or “Unsatisfactory” performance?

Suppose there are 4379 airings in that DMA during the 4th quarter. What performance rating would the contractor receive?

I recommend that the numbers in columns S - V be revised to either:

“< [the number in the corresponding block in columns O – R]”

OR

“< [(the number in the corresponding block in columns O – R) minus 1]”