Attachment 0002 QASP - UD SEAM.pdf

QUALITY ASSURANCE

SURVEILLANCE PLAN

(QASP)

FOR

Chief Digital and Artificial Intelligence Office (CDAO)

AI TALENT 2.0

Part 1

PERFORMANCE

MANAGEMENT

OVERVIEW

The dual purpose of the Quality Assurance Surveillance Plan (QASP) is to outline how the Government will: (a) assess and document Contractor performance in accordance with the requirements set forth in the Performance Work Statement (PWS), and (b) assess contract success. Specifically, the QASP details an effective and systematic surveillance method for each listed service, using defined metrics to measure and document the Contractor’s performance.

Continual contract assessment focuses on ensuring the contract, as written, continues to meet the needs of the customer. This QASP will explain the following:

• What will be monitored

• How the monitoring will take place

• Who will conduct the monitoring

• How the monitoring efforts and results will be documented

This QASP does not detail how the contractor accomplishes the work. Rather, the QASP is created with the premise that the contractor is responsible for management and quality control actions to meet the requirements set forth in the Performance Work Statement (PWS). It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance. Therefore, the QASP should recognize that unforeseen and uncontrollable situations may occur.

The objective of AI Talent is to provide direction/synchronization of efforts to modernize the Defense Department with Artificial Intelligence.

1. PERFORMANCE MANAGEMENT (CONTRACT ASSESSMENT)

The COR will choose from the following methods: periodic inspection, random monitoring, and/or direct observation to verify contractor performance.

1.1. Metrics: Metrics will be established to measure contractor compliance to contractual requirements. These metrics will be based in part on the deliverable requirements identified in the PWS, contractor reporting requirements, contractor and government relationship, and overall government satisfaction of contractor work performance.

1.1.1. Market research will be used as a tool throughout the life of the contract to ensure the most current, efficient, and effective assessment methods and techniques of the commercial marketplace are applied to the performance of services under the contract. At a minimum, market research (including an informal survey of customers) will be conducted annually prior to exercising the option for an additional performance period. The results of the market research/survey of customers will be utilized in making the decision as to whether it is appropriate and in the Government’s best interests to exercise the option. These results will be documented in the KO’s Determination and Findings (D&F) supporting option exercise if that course of action is selected.

1.1.2. All performance assessment data will be reviewed. If performance improvement is necessary, the corrective action will be requested from the contractor.

1.1.3. There will be no positive incentive other than award of the option year(s), and any negative incentive will be accomplished via FAR 52.246-4 Inspection of Services – Fixed Price, or by entry of appropriate reports in CPARS.

1.1.4. The Surveillance Procedures will be used to measure the contractor’s performance on this acquisition. Procedures for reductions in price when services do not meet contract requirements will be accomplished via the Government’s rights under FAR 52.246-4 Inspection of Services-Fixed Price.

1.2. Performance Deficiency Resolution: If discrepancies are found during performance, the COR shall notify the Contractor’s Team Lead immediately upon discovery. Whenever possible, performance deficiencies should be resolved at the lowest practical level. The COR shall determine the type of deficiency notification to issue based on the severity of the deficiency. The contract file shall be documented to reflect the type of notification issued, the name of the individual contacted and date of notification.

1.3. Nonconforming Services:

1.3.1. Definition: A nonconformance is defined as any condition or characteristic in any supply or service furnished under this agreement that does not comply with a requirement of this contract. A nonconforming item is an item submitted to the United States Government (USG) for acceptance with a nonconformance.

CORs will classify nonconforming supplies or services as critical, major, or minor based on the impact of the proposed nonconformance.

1.3.2. Classifications of nonconformity:

1.3.3. Critical: A nonconforming item is classified as critical when the nonconformance impacts safety.

1.3.4. Major: A nonconforming item is classified as major when the nonconformance impacts a requirement to the degree that the usability, functionality, or reliability of the item is materially affected.

Major nonconformance includes those things that involve health, performance, weight, interchangeability, survivability, maintainability, and effective use or operation.

1.3.5. Minor: A nonconforming item is classified as Minor when the nonconformance does not involve any of the factors listed above.

Such nonconformance impacts a requirement but does not materially affect the usability, functionality, or reliability of the item.

1.4. Contractor Proposed Acceptance: A nonconforming item will only be for a specific number of units or for a specified period of time. A written Contractor proposal for acceptance of nonconforming items enables the USG to determine the impact on performance, operational readiness, logistics support or other affected areas. The use of a nonconforming item shall only be approved for the express benefit of the USG for a time defined until the nonconformance is corrected. Adequate consideration shall be required.

1.5. Inform Contractor’s Contract Manager: The COR must always contact the contractor’s manager and inform the manager of any deficiency that was observed. This may be accomplished in person or electronically, but any verbal communication must be supplemented by submission of a Customer Complaint Record (CCR) or Corrective Action Report (CAR). The contractor will annotate on the report provided by the COR the Corrective Action (CA) the contractor will take to correct the defect and prevent its recurrence, and will return the completed report to the COR.

1.6. Contracting Officer Notification: If at any time the COR identifies a

“nonconformance” IAW FAR 46.101, the COR shall document their findings and notify the KO in writing. Email is acceptable, but deficiency should be followed up in writing on designated form and submitted with monthly inspection reports. All documentation will be uploaded to DoD Surveillance and Performance Monitoring (SPM), located at https://wawf.eb.mil.

1.7. Performance Deficiency Notification: There are three types of notification: verbal notification, written notification, and CAR:

1.7.1. Verbal notification is primarily used for minor discrepancies or tasks that can be re-performed.

1.7.2. Written notification is used when verbal notification does not result in correction or whenever the COR determines that written documentation is necessary. The notification will contain the statement, “Notice to Contractor: Failure to correct may result in a Corrective Action Report (CAR).”

1.7.3. A CAR will be initiated by the COR when previous notifications failed to result in corrective actions or when severe deficiencies exist. The CAR will be forwarded to the KO through the COR for action.

1.8. Remedies for Non-Conformance: If inspections indicate unacceptable performance, the COR will process a discrepancy record of the deficiencies to the contractor for correction. The contractor shall be given an appropriate time frame as stated in the PWS, after notification, to correct the unacceptable performance. If deficiencies are not corrected within the required time frame, the COR will contact the contractor for resolution or status. If the contractor disagrees with the noted discrepancy and an agreement cannot be reached, the discrepancy record will be sent to the KO for decision and validation.

When re-performance is not possible or corrective action is not performed to acceptable levels, contract price may be reduced to reflect the reduced value of the services performed by contract or otherwise; perform the services and charge to the Contractor any cost incurred by the Government, that is directly related to the performance of such service or:

1.8.1. Decrease the invoiced amount in relation to performance

1.8.2. Cure Notice

1.8.3. Show Cause Notice

1.8.4. Terminate the contract

1.9. Price Reduction Computation (if applicable): The KO will determine the action to be taken against the contractor, such as reductions in fee/price in accordance with FAR 52.246-4, Inspection of Services – Fixed Price. The

KO’s disposition of the deficiency will be annotated on the form and the COR and contractor will be notified of action.

2. Performance Assessment:

2.1. This QASP is based on the premise that the contractor, not the Government, is responsible for the management and quality control actions needed to meet the standards of the contract.

2.1.1. A Services Summary (SS) will be included in the PWS. This document will summarize the performance objectives and performance thresholds required by the Government in the contractor’s performance. These performance objectives will directly relate to mission essential items. The performance thresholds briefly describe the minimum acceptable levels of service required for each objective and are critical to mission success.

2.1.2. The COR will perform QA procedures following this QASP.

2.2. Initial Performance Review

2.2.1. The initial evaluation of contractor performance shall take place within 30 days after the contractor assumes full performance responsibilities (i.e., after completion of transition/mobilization) to ensure the contractor has successfully started performance, completed transition, is fully operational, and is within the estimated cost, schedule, and performance parameters of the contract.

2.2.2. Initial Performance Review Criteria

2.2.2.1. Technical. Assess the contractor’s conformance to contract requirements, specifications and standards of good workmanship (e.g., commonly accepted technical, professional, environmental, safety, or health standards).

2.2.2.2. Schedule. Assess the timeliness of the contractor against contract requirements, task orders, milestones, delivery schedule, and administrative requirements (e.g., efforts that contribute to or effect the schedule variance).

2.2.2.3. Management/Performance. Assess the integration and

coordination of all activity needed to execute the contract, specifically the completeness and quality of problem identification, corrective action plans, the contractor’s history of reasonable and cooperative behavior, customer satisfaction and management of subcontracts including progress on small business subcontracting goals.

2.2.2.4. Staffing. Assess the contractor’s performance in selecting, retaining, supporting and replacing personnel, when necessary.

2.2.2.5. Special Interest Items. Assess any significant modification to the contract made since contract award.

2.2.3. Assessment Values:

2.2.3.1. Green—No issues.

2.2.3.2. Yellow—Issue(s) but contractor has an adequate mitigation or corrective action plan in place.

2.2.3.3. Red—Issue(s) with inadequate or no contractor’s mitigation or corrective action plan. Any “red” assessment shall include the Government proposed actions with respect to the failing contractor.

2.3. CPARS: (Mandatory if contract is over $1 million) CPARS will be updated documenting contractor performance. The Assessing Official (AO) will be the COR; and the Reviewing Official (RO) will be the COR Supervisor (CORSUP). The contractor also has input prior to finalization. For FA7014 contracts, include this statement: The CPARS Focal Point for this contract is Mr. Tony Rogers, anthony.w.rogers3.civ@mail.mil) The website for the CPARS is: https://www.cpars.gov/

2.4. Combating Trafficking in Persons (DFARS PGI 222.17): The contractor shall comply with FAR 52.222-50, Combating Trafficking in Persons (CTIP). The COR will monitor contractor’s performance regarding trafficking in persons such that non-compliance with CTIP is brought to the immediate attention of the KO. All DoD military and civilians must receive CTIP awareness training; it is highly recommended that DoD contractors receive CTIP awareness training. The military has zero (0) tolerance for human trafficking. CORs are to report all violations to their KO immediately and keep a log for annual review of all concerns. The KO shall convey the information to the DoD CTIP Law Enforcement Support Office of Under Secretary of Defense (OUSD) Personnel and Readiness (P&R) Defense Human Resources Activity (DHRA).

http://ctip.defense.gov. The Department of State’s Office is to Monitor and Combat Trafficking in Person at http://www.state.gov/j/tip.

mailto:anthony.w.rogers3.civ@mail.mil https://www.cpars.gov/ http://ctip.defense.gov/ http://www.state.gov/j/tip

3. Revisions: Revisions are the responsibility of the government team. The COR reviews the PWS and QASP periodically for any required changes and shall provide any proposed changes to the KO for review. Revisions to the QASP are the joint responsibility of the contracting office and requiring activity and are only accomplished with the approval of and by the KO. This is a living document and can be changed at any time, but the KO must accept changes.

Part 2

ASSESSMENT

MANAGEMENT

SERVICES SUMMARY (SS)

PERFORMANCE OBJECTIVES, PERFORMANCE THRESHOLDS,

METHODS OF SURVEILLANCE AND REMEDIES

Performance Objective

PWS

Paragraph Performance Threshold Method of

Surveillance Provide qualified personnel at Call Order start

6.3/6.9 Performance is:

- excellent when 100%

- very good when 95%

- satisfactory when 90%

- marginal when <90%

- unacceptable when

<80%

… of qualified personnel are on the job at the start of the call order, unless previously negotiated by the KO.

COR Inspection at Call Order start

Maintain stable workforce

6.9 Performance is acceptable

when turnover is less than 20%. The CPARS rating for Management will be negatively affected if the contractor exceeds the maximum 20% turnover rate on any call order.

Personnel turnover will be documented by the COR and annotated on the annual CPARS rating

Effectively replaces/substitutes key personnel

6.9 Performance is acceptable

when key personnel vacancies are filled with qualified personnel within 14 calendar days of vacancy, unless approved in writing or otherwise directed by the KO AND there is no mission impact due to vacancies or unqualified personnel.

Tracked by the COR at time of vacancy

Develop and submit all required deliverables

Section IV Performance is acceptable when 95% of deliverable requirements are met and received on time AND information is accurate/complete

See deliverable due dates

Maintain professional workplace

6.11 Performance is acceptable

when there are <2 incidents per year

Complaints or written reports to

COR/KO

* The procedures set forth in the Inspection of Services Clause will be used to remedy all deficiencies. The re-performance times will be based upon sensitivity of the issue.

NOTE: The timeframes for re-performance should be established by the MFT, for the Services Summary items when the PWS and QASP are written.

Types of Surveillance to select from

Random Sampling: Appropriate for frequently recurring tasks. Evaluate randomly selected samples of the lot to determine the acceptability of the entire lot.

Random Inspection Guide, Method of surveillance, Lot size, Sample size, Performance requirement, Sampling procedure, Inspection procedure 100 Percent Inspection: Appropriate for tasks that occur infrequently. Inspect and evaluate performance each time task is performed Periodic Surveillance: Evaluation of samples selected on other than 100% or statistically random basis. (52.246-4) i.e. monthly, quarterly, semi-annually etc.

Customer Complaint: Complaints must be validated.

0BCORRECTIVE ACTION REPORT (CAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACTOR

2. CONTRACT NUMBER

3. TYPE OF SERVICES

4. FUNCTIONAL AREA

5. SUSPENSE DATE

6. CONTROL NUMBER

7. DEFICIENCY MAJOR MINOR

FINDING:

FINDING IMPACT:

Please respond with a written corrective action plan that details the corrective action of the cited deficiency, the cause of the deficiency, and actions taken to prevent recurrence by Suspense Date in Block 5. If date was not entered in Block 5, the contractor is not required to provide a response.

2B8. QUALITY ASSURANCE PERSONNEL (QAP)

3BTYPED NAME AND GRADE

4BSIGNATURE AND DATE

5B9. ISSUING AUTHORITY

6BTYPED NAME AND GRADE

7BSIGNATURE AND DATE

10. QAP RESPONSE TO CONTRACTOR CORRECTIVE ACTION AND ACTION TAKEN TO PREVENT RECURRENCE

11. QAP DETERMINATION

ACCEPTED REJECTED

12. CLOSE DATE

SAMPLE

1BCORRECTIVE ACTION REPORT (CAR)

8BContinuation Sheet (Number to correspond with applicable Item Number on reverse)

Corrective Action Report (CAR) Instructions

Block 1. Enter Contractor Name.

Block 2. Enter Contract Number.

Block 3. Enter Contract Name or Type of Services.

Block 4. Enter Functional Area of the Contract.

Block 5. Enter the assigned suspense date given the contractor to provide a response to the CAR. A date must be entered for a Major CAR. A date is optional at the discretion of the COR initiating the CAR if the finding is Minor.

Block 6. All CARs will be tracked with a Control Number. The functional commander/functional director is the Issuing Authority for his/her contract COR generating CARs. By providing the control number to the COR originating the CAR, the Issuing Authority demonstrates concurrence with the finding. The control number will be made up of the first two letters of the name of the site or contract, the last two digits of the calendar year, and a three digit number starting with 001 and progressing upward throughout the calendar year (e.g., WP04-001).

Block 7.

1. Check the block that indicates whether the identified deficiency is assigned as a Major or Minor finding. If you have not identified “major” and “minor” findings in your PWS, then line through the major and minor blocks on the form. [Remove this bracket of info after reading.

Major/minor findings can be defined as below, and placed in the PWS.

Major findings are contract non-conformances which are considered critical or major. IAW FAR

46.101 a major nonconformance means a nonconformance, other than critical, that is likely to result in failure of the supplies, or to materially reduce the usability of the supplies or services for their intended purpose. A critical nonconformance means a nonconformance that is likely to result in a hazardous or unsafe condition for individuals using, maintaining, or depending upon the supplies or services; or is likely to prevent performance of a vital agency mission. If at any time QAE/P identifies a condition as having a significant adverse effect on the quality of the activity, such as those stated below, QAP shall document their findings and notify the CO immediately in writing. Email is acceptable.

A. Contractor failure to meet a Performance Threshold.

B. Failure to provide adequate corrective action to preclude reoccurrence of Government identified findings.

C. Failure to provide corrective action to deficiencies identified by the contractor within a prescribed suspense period.

D. Any failure to adhere to security and/or safety regulations that results in a security or safety incident.

Minor Finding. IAW FAR 46.101 a minor nonconformance means a nonconformance, that is not likely to materially reduce the usability of the supplies or services for their intended purpose, or is a departure from established standards having little bearing on the effective use or operation of the supplies or services. When COR identifies a minor finding, the COR shall document the findings, but is not required to notify the CO. However, if the same minor finding is repeatedly identified, it may be an indication that a major finding is occurring or has occurred because the Contractor has not taken proper steps to prevent recurrence. In this case, the QAE/P shall notify the CO in writing (email is acceptable).]

2. Finding: Clearly state the details of the finding followed by a reference to the stated contractual requirement. The reference must state the portion of the contract, part, section, paragraph and subparagraph and must make a complete brief quotation of the contract reference.

3. State the impact that the finding has or could have on the accomplishment of the mission that the contract provides.

4. When determining how long to give the contractor to respond to the identified finding, the normal is 10 working days. The suspense date should reflect this unless the finding requires a greater amount of time to come to solution. Major findings require a contractor response. Minor findings may require a contractor response at the discretion of the COR initiating the CAR and/or the Issuing Authority.

Blocks 8 and 9. Contract CORs initiating a CAR must sign in Block 8 and the Issuing Authority signs in Block 9. Each annotates the date at the time of signature.

Block 10. Upon review of the contractor’s reply, the originator of the CAR will enter comments regarding acceptance or rejection of the contractor’s response. This block may also contain any comments regarding follow-on inspections conducted or needed at a later date to validate that the finding has not reoccurred.

Block 11. The COR selects Accept or Reject after reviewing the contractor’s response.

Block 12. The COR enters a close date after advising the Issuing Authority that the contractor’s response is acceptable and the Issuing Authority concurs. The Issuing Authority will then forward the closed CAR to the contractor and report accordingly in the monthly Certificate Of Service (COS) or other applicable CO services report. This routing remains the same if the contractor response is unacceptable and the CAR remains open until an acceptable response is received except that the Issuing Authority forwards the open CAR back to the contractor for additional information.

Continuation Sheet. The Continuation Sheet is to be used to expand on any information that will not fit in the applicable numbered block.

CUSTOMER FEEDBACK RECORD

DATE/TIME OF EVENT

DETAILS OF EVENT

ORGANIZATION

BUILDING NUMBER

INDIVIDUAL

PHONE NUMBER

NATURE OF COMMENT

CONTRACT REFERENCE

VALIDATION

DATE/TIME CONTRACTOR INFORMED OF OBSERVATION

OTHER COMMENTS OR NOTES

RECEIVED BY (NAME & DATE)