SW_Code_Assessment_QASP_Attachment_0003.pdf

W56JSR-17-R-0004

Quality Assurance Surveillance Plan (QASP) Software Code Assessment

09 November 2016

Version 4.0

1 Purpose 2 Authority 3 Scope 4 Government Resources 5 Responsibilities 6 Methods of QA Surveillance 7 Documentation 8 Performance Ratings and Criteria 9 Surveillance Matrix Appendix A - Surveillance Matrix

Quality Assurance Surveillance Plan (QASP)

Software Code Assessment

1 Purpose The Quality Assurance Surveillance Plan (QASP) is a Government developed and applied document used to make sure the systematic quality assurance methods are used in the administration of this effort. The intent is to ensure that the contractor performs in accordance with the performance objectives and the Government receives the quality of services called for in this effort.

The scope of this effort is for Communications-Electronics Command (CECOM) Software Engineering Center (SEC) to provide, as a service for Army Customers, software source code assessment to identify vulnerabilities and to assist Customers in remediating and repairing software code. The goal is to improve the cybersecurity posture of mission systems in a manner that will counter the growing threat to software intensive systems that the Army relies on to accomplish its mission.

To accomplish this, the Government intends to employ a third party software code analysis tool which will aggregate the results of multiple individual code analysis tools.

The requirements to be performed will cover two basic areas. First is the analysis of the system source code to determine if it is in suitable condition to be processed. If not, it may require that the code be compiled to allow it to be processed. This also covers the operation of the third party software code analysis tool which will perform the analysis and produce a basic report of vulnerability findings. The second requirement is to review these findings and to provide a subsequent report making the findings actionable as it relates to the specific system and customer.

2 Authority The QASP provides for inspections and acceptance of the services and documentation called for in service contracts, to be executed by the Contracting Officer or a duly authorized representative.

3 Scope

a. The QASP is put in place to provide Government surveillance oversight of the contractor’s efforts to assure that they are timely, effective and are delivering the results specified in the contract.

b. The contractors performance on under the contract resulting from this solicitation #W56JSR-16-R-0001 will be evaluated by the Government as described in this QASP.

The first evaluation will cover the period ending twelve (12) months after date of contract award with successive evaluations being performed for each twelve-month period thereafter until the end of the contract Period of Performance. For each period, the Government will evaluate the contractor’s performance. The evaluation will encompass all work performed by the contractor at any time during the period but will not include cumulative information from prior reports.

4 Government Resources The following definitions for Government resources are applicable to this plan:

• Contracting Officer - A person duly appointed with the authority to enter into (Procuring Contracting Officer (PCO)) or administer (Administrative Contracting Officer (ACO)) contracts and make related determination and findings on behalf of the Government. The PCO is identified in the contract. The ACO will be designated upon award.

• Contracting Officers Representative (COR) - An individual appointed in writing by the PCO to act as his/her authorized representative to assist in administering the contract. The COR will be appointed in the resulting contract. The limitations of authority are contained in a written letter of appointment.

5 Responsibilities The following Government resources shall have responsibility for the implementation of this QASP:

• Contracting Officer (PCO): The PCO ensures performance of all necessary actions for effective contracting, ensures compliance with the terms of the contract, and safeguards the interests of the United States in the contractual relationship. It is the PCO who assures the Contractor receives impartial, fair and equitable treatment under the contract. The PCO is ultimately responsible for the final determination of the adequacy of the Contractor’s performance.

• Contracting Officers Representative (COR): The COR is responsible for technical administration of the contract and assures proper Government surveillance of the contractors technical performance in accordance with the COR appointment or delegation letter. The COR is not empowered to make any contractual commitments or to authorize any changes on the Governments behalf. Any changes that the contractor deems may affect the contract price, terms, or conditions shall be referred to the PCO for action.

6 Methods of QA Surveillance

a. QASP - The below listed methods of surveillance shall be used by the COR in the technical administration of this QASP.

b. Contractor Performance Assessment Report System (CPARS) - An evaluation using the CPARS format will be performed at the end of each year of performance. The Government will address the quality of product or service, schedule, cost control, business relations, and management for this procurement. As this information may affect future source selections throughout DoD, and the continuation of the contract, the annual Government assessment will be used appropriately as an additional performance oversight and communication tool with the QASP.

7 Documentation In addition to providing monthly reports to the PCO, the COR will maintain a complete Quality Assurance file. The file will contain copies of all reports, evaluations, recommendations, and any actions related to the Government’s performance of the quality assurance function. All such records will be maintained for the life of the contract. The COR shall forward these records to the PCO at termination or completion of the contract.

8 Performance Ratings and Criteria

a. The Government will evaluate the contractor’s overall performance of the base indefinite-delivery contract at least every 12 months throughout the entire period of performance on the contract up to the final report for all orders IAW AFARS 5142.1502.

An interim CPAR is also required upon significant change within the agency, provided that a minimum of six (6) months of performance has occurred, such as a change in program or project management responsibility and/or a transfer of contract to a different contracting activity. The PCO, utilizing input provided by the COR, will assign one of the following Overall Performance Ratings:

• Exceptional (Dark Blue) - Performance meets contractual requirements and exceeds many requirements to the Government's benefit.

• Very Good (Purple) - Performance meets contractual requirements and exceeds some to the Government's benefit.

• Satisfactory (Green) - Performance meets contractual requirements.

• Marginal (Yellow) - Performance does not meet some contractual requirements. The element being assessed reflects a serious problem for which the contractor has not yet implemented satisfactory corrective actions.

• Unsatisfactory (Red) - Performance does not meet contractual requirements and recovery is not likely in a timely manner. Contractor's corrective actions to date are ineffective.

b. Performance Evaluation Criteria: The contractor’s performance will be evaluated using the following criteria and standards:

Criteria: Task Performance

• Exceptional - Work product frequently exceeds AQLs defined in Surveillance Matrix.

• Very Good - Work product always meets and occasionally exceeds AQLs defined in

Surveillance Matrix.

• Satisfactory - Work product routinely meets AQLs defined in Surveillance Matrix.

• Marginal - Work product occasionally fails to meet AQLs defined in Surveillance

Matrix.

• Unsatisfactory - Work product fails to meet AQLs defined in Surveillance Matrix.

Criteria: Staffing

• Exceptional - Contractor provides highly qualified personnel. Contractor reassigns personnel to ensure proper coverage. Actual lapses in coverage occur very rarely, if ever, and are managed in accordance with (IAW) contract terms and conditions.

Contractor ensures staff training remains current.

• Very Good - Contractor provides a mix of qualified and some highly qualified personnel. Lapses in coverage occur infrequently and are managed IAW contract terms and conditions.

• Satisfactory - Contractor provides qualified personnel. Lapses in coverage may occasionally occur and are managed IAW contract terms and conditions.

• Marginal - Contractor provides marginally qualified personnel. Lapses in coverage occur more than occasionally

• Unsatisfactory - Contractor provides marginally qualified or unqualified personnel.

Lapses in coverage occur regularly.

Criteria: Timeliness

• Exceptional - Contractor routinely delivers ahead of deadlines, schedules, and responds immediately to Government requests.

• Very Good - Contractor routinely meets deadlines and schedules and occasionally delivers early and responds immediately to Government requests.

• Satisfactory - Contractor routinely meets deadlines, schedules, and responds quickly to Government requests.

• Marginal - Contractor occasionally misses deadlines, schedules or is slow to respond to Government requests or is occasionally non-responsive to Government requests.

• Unsatisfactory - Contractor frequently misses deadlines, schedules, or is slow to respond to Government requests or is non-responsive to Government requests.

Criteria: Customer Satisfaction

• Exceptional - Routinely exceeds customer expectations.

• Very Good - Routinely meets or occasionally exceeds customer expectations.

• Satisfactory - Meets customer expectations.

• Marginal - Contractor occasionally fails to meet customer expectations

• Unsatisfactory - Fails to meet customer expectations

Criteria: Problem Resolution

• Exceptional - Problems are non-existent or the contractor takes corrective action without Government involvement.

• Very Good - Problems occur infrequently and are generally resolved quickly with minimal Government involvement.

• Satisfactory - Problems are resolved quickly with minimal Government involvement.

• Marginal - Problems are generally resolved but take unusual Government effort to resolve or take an excessive amount of time to resolve

• Unsatisfactory - Problems are unresolved, repetitive, or take excessive Government effort to resolve.

Criteria: Communication

• Exceptional - Contractor takes a proactive approach such that communications are almost always clear, effective and timely.

• Very Good - Contractor routinely communicates with Government in an effective and timely manner and is frequently proactive in managing communications

• Satisfactory - Contractor routinely communicates with Government in an effective and timely manner.

• Marginal - Contractor occasionally fails to communicate with Government in an effective and timely manner

• Unsatisfactory - Contractor often fails to communicate with Government in an effective and timely manner.

Criteria: Cost Management (Cost Reimbursement efforts only)

• Exceptional Reductions in costs to the Government below estimated costs are noteworthy. Contractor provides detailed cost analysis and recommendations to Government for resolution of problems identified. Funds and resources are optimally used to provide the maximum benefit for the funds and resources available.

Documented savings are apparent.

• Very Good - Contractor routinely completes the effort within the originally agreed to estimated cost and occasionally causes reductions in costs below estimated costs.

Provides cost analysis and recommends to Government resolution of problems identified.

• Satisfactory - Contractor routinely completes the effort within the originally agreed to estimated cost. Contractor provides measures for controlling all costs at estimated costs. Funds and resources are generally used in a cost-effective manner. No major resource management problems are apparent.

• Marginal - Contractor occasionally fails to complete an effort within the originally agreed to estimated cost, i.e. cost overruns occasionally occur.

• Unsatisfactory - Contractor routinely fails to complete the effort within the originally agreed to estimated cost, i.e. cost overruns frequently occur.

9 Surveillance Matrix The Surveillance Matrix (Attachment 1) is the list of performance objectives and standards that shall be performed by the contractor. This matrix details the method of surveillance the COR will use to validate and inspect these performance elements.

Inspection of each element will be documented in the COR file.

Appendix A - Surveillance Matrix

Statements Standards/AQLs Inspections Ratings 3.2.1 Provide customer and engineering support for software data analytics capability.

3.2.1.1 The Contractor shall participate in technical meetings with customers to discuss the customer's software data analytics requirements.

Deliverables:

A003 Meeting Minutes

a) STD: Attend and participate in scheduled meetings when requested by the Government.

AQL: 85%, attend and participate in scheduled meetings when requested by the Government.

b) STD: Provide meeting minutes for meetings requiring attendance in a timely manner and in accordance with the CDRL.

AQL: 80%, submission of all meeting minutes within 2 business days;

100%, submission of all remaining meeting minutes within 1 business day.

What: Meeting Attendance How: Meeting minutes or visual inspection.

Frequency: Each meeting Who: Government Technical Lead Standard(s): a, b

3.2.1.2 The Contractor shall prepare customer source code and/or binaries for code analysis and submit into the automated tool suite.

a) STD: Notify the Government Technical Lead within 1 working day of any significant issues that will cause delays for the analysis.

AQL: 90%, submission of all notifications within 1 business day;

100%, submission of all remaining notifications within 2 business days.

What: Delay Notification How: Email Who: Government Technical lead Standard(s): a

3.2.1.3 The Contractor shall generate and deliver reports containing tailored,

a) STD: Reports of raw data shall be provided upon completion of each automated analysis.

What: Reports How: Review and analyze the report.

aggregated results of the analysis.

Deliverables:

D001 Aggregated Report of Potential Defects D006 Summary View of Potential Defects D007 Tailored Aggregated Report of Potential Defects D008 Tailored Summary View of Potential Defects

AQL: 80%, submission of all reports within 5 business days; 100%, submission of all remaining reports within 4 business days. There shall be no more than 20% rework required.

b) STD: Reports shall be in accordance with the CDRLs and Government regulations, policy and mandates.

AQL: 100% in accordance with standard.

Frequency: Upon receipt of each report.

3.2.1.4 The Contractor shall attend meetings with customers to discuss assessment results.

Deliverables:

A003 Meeting Minutes

a) STD: Attend and participate in scheduled meetings when requested by the Government.

AQL: 85%, attend and participate in scheduled meetings when requested by the Government.

b) STD: Provide meeting minutes for meetings requiring attendance in a timely manner and in accordance with the CDRL.

AQL: 80%, submission of all meeting minutes within 2 business days;

100%, submission of all remaining meeting minutes within 1 business day.

What: Meeting attendance How: Meeting minutes or visual inspection Frequency: Completion of each effort

3.2.1.5 The Contractor shall provide status updates for project milestones.

a) STD: Provide status updates of milestone completion in a timely manner.

AQL: 80%, submission of all milestone status updates within 1 business day; 100%, submission of all remaining milestone status updates within 3 business days.

What: Reporting tool How: Review and analyze the report.

Frequency: Completion of each effort

3.2.1.6 The Contractor shall create and update processes and procedures for consistent software code analysis operations.

Deliverables:

D005 Process and Procedures

a) STD: Upon the identification of a new software assurance capability, draft procedures provided by date agreed to by the Government.

AQL: 90%, submission of all processes and procedures within 6 business days; 100% submission of all remaining processes and procedures within 10 business days.

What: Policies & procedures How: Review and analyze the policies and procedures.

Frequency: As Required

3.2.1.7 The Contractor shall provide demonstrations/briefings of the software data analytics capability.

Deliverables:

A002 Demonstrations/Briefing Material

a) STD: Contractor shall be prepared to conduct demonstration in a timely manner.

AQL: 90%, all demonstrations conducted within 5 business days;

100%, conduct all remaining demonstrations within 7 business days.

b) STD: Contractor shall provide draft demonstration/briefing material, as requested by the Government.

AQL: 90%, submission of all demonstration/briefing material within 3 business days; 100%, submission of all remaining demonstration/briefing material within 5 business days.

What: Demonstrations/Briefings How: Review and approve the draft demonstration/briefing material.

Frequency: As Required

3.2.1.8 The Contractor shall provide training enabling the government to implement the software data analytics capability.

Deliverables:

C004 Training Events

a) STD: Training shall provide attendees with the information necessary for a working understanding of the topic.

AQL: 90%, submission of all attendee training material within 5 business days; 100%, submission of all remaining attendee training material within 7 business days.

What: Attendee Training Material How: Review and approve the attendee training material.

Frequency: As Required Who: Government Technical Lead Standard(s): a, b

What: Customer feedback How: Review feedback from customer

b) STD: No unresolved user complaints that impact compliance with the standards or performance.

AQL: No more than 5% unresolved user complaints.

Frequency: Upon receipt of feedback Who: Government project lead Standard(s): b

3.2.1.9 The Contractor shall analyze and provide actionable reports for each software assessment.

Deliverables:

D003 Final software assurance assessment report

a) STD: Categorization will be based on applicable standards such as the DISA Application and Security Development Guide STIG and the DoD Risk Management Framework.

AQL: 100%, findings correctly categorized in accordance with the standards.

b) STD: Delivery of report is required following completion of PWS 3.2.1.3.

AQL: 90%, submission of all actionable assessment reports within 30 calendar days; 100%, submission of all remaining actionable assessment reports within 45 calendar days.

What: Actionable reports How: Review and analyze the report.

Frequency: Completion of each effort