L01_QASP.pdf

80AFRC19R0022

Attachment L01

Research Facilities and Engineering Support Services (RF&ESS)

Quality Assurance Surveillance Plan (QASP) August 28, 2019

Attachment L-01

INTRODUCTION

A. PURPOSE OF THE SURVEILLANCE PLAN

B. SCOPE

C. ROLES AND RESPONSIBILITIES OF PARTICIPATING GOVERNMENT

OFFICIALS

D. IDENTIFICATION OF THE TYPES OF WORK TO BE PERFORMED

E. METHODOLOGIES TO BE USED TO MONITOR THE CONTRACTOR'S

PERFORMANCE

F. REPORTING AND ANALYSIS OF SURVEILLANCE RESULTS

G. PERFORMANCE REQUIREMENT SUMMARY

CONCLUSION

INTRODUCTION

This Surveillance Plan sets forth procedures and guidelines that NASA Neil A. Armstrong Flight

Research Center (AFRC) will use in evaluating the performance of the Contractor, ____________, Contract Number __________________. A copy of this plan will be furnished to the Contractor so that they will be aware of the methods that the Government will employ in evaluating performance and to address any concerns that the Contractor might have prior to initiating work. This plan, including procedures and forms; it may be revised unilaterally by the

Government at any time during the contract period of performance.

This QASP is Government developed surveillance tool prepared in accordance with Federal

Acquisition Regulation (FAR) 46.401 and NASA FAR Supplement (NFS) 1846.401. It is not part of the contract, per NFS 1846.401, but provided to the Contractor for informational purposes only.

A. OBJECTIVE OF THE SURVEILLANCE PLAN

This Surveillance Plan is Government developed and applied document used to verify that systematic quality methods are used in the administration of the standards and metrics included in this contract. This QASP defines the process the Government expects to follow to obtain data, evaluate the contractor, and determine if contract performance conforms to contract requirements. The goal is to balance the level of Government surveillance with perceived impacts and risks associated with performance hereunder. The intent is to ensure that the

Government receives the quality of services called for in the contract and that the Government only pays for the acceptable level of services specified.

The QASP is intended to accomplish the following:

1. Define the roles and responsibilities of participating Government officials;

2. Identify the types of work to be performed;

3. Describe the evaluation methods that will be employed by the Government in assessing the Contractor's performance;

4. Define how the performance evaluation results will be applied during the preparation of the semi-annual assessment of their performance.

5. Provide copies of the monitoring forms that will be used by the Government in documenting and evaluating the Contractor's performance.

B. SCOPE

This QASP identifies the program requirements, strategies, resources, review and control processes, surveillance activities, and metrics for continuous measurement of Contractor performance. This plan provides effective and systematic surveillance methods for evaluating the Contractor services, processes, and products provided under this contract. The Government may evaluate work at any time during the Contractor’s work performance.

The intent of the QASP is to ensure that the Contractor performs in accordance with acceptable quality levels and the Government receives the quality of services and products called for in the contract. This QASP does not detail how the Contractor accomplishes the work. Rather, the

QASP is based on the premise that the contractor, not the Government, is responsible for managing its quality controls and ensuring that performance meets the terms of the contract. The role of the Government is quality assurance to ensure contract standards are achieved.

The QASP is intended to be a “living” document from which resources and activities will evolve from one phase to another during the life of the contract, and will be updated as required and defined in this document.

This plan is applicable to any service or product provided, as well as all areas in which work is being performed by the RFESS Contractor(s). Throughout this QASP, the term RFESS

Contractor is used. In terms of this plan, it should be known that unless explicitly stated, this term is applicable to both the RFESS Contractor and any and all subcontractors.

The surveillance program shall be a collaborative and integrated effort that includes all areas of contract management, including the following:

a. Quality Assurance

b. Procurement/Subcontracting/Purchasing

c. Business Management

d. Property

e. Environmental

f. Export Control

g. Safety and Health

h. Security

C. ROLES AND RESPONSIBILITIES OF PARTICIPATING GOVERNMENT

OFFICIALS

The following Government officials will participate in assessing the quality of the Contractor’s performance. Their roles and responsibilities are described as follows:

1. Technical Monitors (TMs) assist the Contracting Officer’s Representative (COR) and act as the authorized representative for the technical administration of specific tasking under the contract. The TM provides detailed technical oversight of the Contractor’s performance and reports their findings to the COR. TM findings are to be complete and impartial in nature to help assure proper Government surveillance of the Contractor’s performance. While the TM may serve as a direct conduit to provide Government guidance and feedback to the Contractor on technical matters, he or she is not empowered to make any contractual commitments or to authorize any contractual changes on the

Government’s behalf. Any changes that the Contractor deems may affect contract price, terms, or conditions shall be referred to the Contracting Officer (CO) for action.

2. Performance Monitors (PMs) oversee and report on performance for multiple WBS areas.

3. Contracting Officer’s Representative (COR) is responsible for monitoring, assessing, recording and reporting on the day-to-day performance of the Contractor. The COR has the primary responsibility for collecting the TM’s assessment forms, summarizing the results and, with the CO, making an independent assessment of the Contractor's work performance. An assessment will be presented to the CO following each semi-annual evaluation period. It is important for the COR to establish and maintain a team oriented partnering line of communication with the Contractor’s site manager (or other appropriate personnel) to ensure that required work is accomplished in an efficient and proper manner. Meetings shall be held on a regular basis in order to address in a pro-active manner concerns and problems that may arise during work performance. The

COR certifies the contractor’s invoices for payment in accordance with AFRC procedures. An Alternate COR may be delegated these responsibilities during the absence of the COR.

4. Contracting Officer is responsible for evaluating overall contract performance to include such items as contract compliance, contract administration, and property control;

reviewing the COR's assessment of the Contractor's performance; and resolving all differences between the COR's interpretation and the Contractor’s interpretation. The CO may call upon the expertise of other Government individuals as required. The CO must approve all modifications to this plan prior to commencement of the change. All CO determinations regarding this surveillance plan are final.

D. IDENTIFICATION OF THE TYPES OF WORK TO BE PERFORMED

This contract specifies that the Contractor will provide labor and materials to perform development, test, engineering, analysis, specification, design, development, installation, integration, operation, maintenance, documentation, training, customer outreach, change management, configuration control, troubleshooting and related functions. These responsibilities branch across three primary functional areas:

DATR operations and engineering services, SIM support services, and

Center-Wide IT Services

This a performance-based hybrid contract with Firm Fixed Price and Time and Material Contract

Line Item Numbers (CLINs). The Firm Fixed Price includes the above areas and the Time and

Material includes: Travel, over time, and training.

E. METHODOLOGIES TO BE USED TO MONITOR THE CONTRACTOR'S

PERFORMANCE

1. Management/Program Reviews: Regular reviews will be held with Contractor management to assess overall progress in meeting contract requirements. Appropriate

Government and Contractor personnel will attend these meetings to effectively convey and communicate issues and concerns. Government surveillance will be conducted as needed by TMs, COR or CO. These assessments will be used as a management tool to access overall contract compliance. These assessments will normally take the form of:

Semi-annual assessment of Contractor’s Performance

Technical Information Meetings

Review of Deliverables

Documentation of problems, issues and concerns

Data Collection

Performance Requirement Metrics

Reporting

Contractor’s Adherence to their Management Plan

Project Documents

2. Inspection and In-depth Observation: The Government will conduct routine inspections and observations of the work being performed. Formal inspections shall be performed periodically to ensure contract compliance. Inspections will utilize the PWS requirements as its baseline plus other controlling documents, as applicable. Areas of concern and commendation will be highlighted and brought to the attention of the

Contractor’s management.

3. Customer Feedback: Customers will be queried both formally and informally to provide insight into overall performance. When appropriate, a formal customer survey will be used to provide feedback. TMs, COR, and CO will regularly talk to customers and end users to monitor the Contractor’s performance. Results will be shared with the

Contractor’s management.

F. REPORTING AND ANALYSIS OF SURVEILLANCE RESULTS

The Government will use the BITs database utilizing the attached matrix to identify and document the Contractor's performance under this contract. The BITs will document the evaluator’s assessment of the Contractor’s performance and the impact or consequences of what was or was not performed in accordance with the contract.

1. Utilizing input from the TMs, the COR will evaluate performance using the following definitions and percentile ranges of Contractor performance:

Rating Definitions Percentile Range

Exceptional Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

91-100

Very Good Performance meets contractual requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

71-90

Satisfactory Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the contractor appear or were satisfactory.

51-70

Marginal Performance does not meet some contractual requirements. The contractual

31-50 performance of the element or sub-element being evaluated reflects a serious problem for which the contractor has not yet identified corrective actions. The contractor’s proposed actions appear only marginally effective or were not fully implemented.

Unsatisfactory Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which the contractor’s corrective actions appear or were ineffective.

0-30

2. The CO and COR will assess the overall contract performance. Examples of areas in which the Government may evaluate and rate Contractor performance to include:

Quality of Product or Service - compliance with contract requirements, accuracy of reports, and technical excellence.

Schedules - compliance with milestones; reliable, responsive to technical direction;

and actions completed on time.

Customer Satisfaction – end user’s satisfaction level with Contractor’s service.

3. The COR must substantiate all tasks, which he or she judges to be indicative of

"poor/unsatisfactory" performance. Performance at the "very good" level is expected from the Contractor. The COR will forward copies of all completed assessment forms to the CO by the 10th business day following the semi-annual being evaluated. The CO will review the semi-annual performance evaluation report prepared by the COR and provide the feedback that report has been received and reviewed, and what if any actions will be taken by the 10th business day. When appropriate, the CO may investigate any substandard performance further to determine if all the facts and circumstances surrounding the event were considered in the COR's opinions outlined on the performance evaluation form. The CO will immediately discuss every event receiving a substandard rating with the Contractor to assure that corrective action is promptly initiated. The CO will forward a semi-annual assessment report to the Contractor. For any negative findings, the Contractor is required to indicate in writing, the corrective action taken, within 10 working days from receipt of the summarized semi-annual report. When excessive errors in a particular area(s) result in negative findings, the Contractor shall include a corrective action plan in their response to the Government evaluation, identifying specific action to be taken, responsible technical entity, and schedule for implementation. The Government will place increased emphasis on areas of concern in succeeding evaluations to assure improvement in Contractor performance.

4. Upon effective corrective action the COR will close the corrective action plan, and forward copies of all completed actions to the CO. Original assessment forms will be retained in the contract file.

G. PERFORMANCE REQUIREMENT SUMMARY

The Performance Requirement Summary (PRS) lists and defines the tasks that the PM, and or

COR determine are most important to monitor and evaluate, the acceptable quality and frequency of inspections. The Business Information Tracking System (BITS) is used for recording the inspections.

Performance

Work

Breakdown

Structure

Performance Standards Performance Requirement

WBS 110 High caliber products are provided to customers, the percentage of rework on requirements for in-house software development for

Government acceptance tests, at no fault of the

Government, and not due to requirement change or negotiated required by date change.

No less than 3.1% of rework of requirements

WBS 110 High caliber products are provided to customers, the percentage of rework on requirements for in-house software development operational deliverables

(Operational Systems &

Applications), at no fault of the Government, and not due to requirement change or negotiated required by date change.

No less than 1.1% of rework of requirements

WBS 110 All mission systems & applications are stable and always ready for mission support, at no fault of the

Government, all contractor in-house deliverables should cause zero impacts to real-time mission activities. A minor impact is defined as:

problem(s) that can be fixed after the mission and don't need to re-fly the flight card(s) affected or causes a delay less than 30 minutes. A major impact is defined as:

problem(s) that causes immediate Return-To-Base

(RTB) or worse.

No more than 3 minor impacts to mission

WBS 110 Adherence to AS9100 and

ANSI/ISO/ASQ

9001:current version standards by ensuring configuration management

(tracked per release) for all in-house software and systems development is achieved 100% of the time following AFOP-7150.2-

001,and/or AFOP-7120.1-

002 current rev.

Configuration Management is achieved 98% of the time

WBS 110 All documentation for all in-house software and systems development, following

AFOP-7150.2-001 and/or

AFOP-7120.1-002 current rev. is complete.

Ability to properly upgrade or replicate deliverables is achieved

98% of the time

WBS

130/140/150/160

To enable successful and efficient mission operations, the requirement of a preflight

SOMP to be completed each mission is expected to be on time 100% of the time. When an area cannot/does not complete the preflight requirement on time, at no fault of the Government, rating per mission will be assigned as follows:

0 Points - Preflight SOMP complete with No Delay

.25 Points - Delay with No

Impacts

.5 Points - Delay with

Schedule Impact and No

Mission Impact 1 Point -

Delay with Schedule Impact and Mission Impact

2 Points - Delay with

Mission Cancellation

Points will be added on the last day of each month, and a percentage will be developed in conjunction with the number of missions. For example: 30 missions occurred during the month.

Two missions had delays with No Impacts. The monthly points are deducted from the number of mission

(30 - .5 = 29.5). That number will be divided by the total number of missions (29.5/30

= (.5) divided by 30 missions equates to a percentage of

98.3%. The 12 month cumulative percentage will be assessed pursuant to the

Adjectival Rating Scale.

SOMP completed 97% of resulting in no delay or mission impact

WBS 130 To enable successful and efficient mission operations, the requirement is that

Mission Build Disk be completed with no errors.

When an area cannot/does not complete a Mission Build

Disk with no errors, at no fault of the Government, rating per build will be assigned as follows:

0 Points - Mission Build

Complete with no Errors

.25 Points - Error with No

Impacts

.5 Points - Error with

Schedule Impact & No

Mission Impact 1 Point -

Error with Schedule Impact

& Mission Impact

Points will be added on the last day of each month, and a percentage will be developed in conjunction with the number of builds For example: 30 builds occurred during the month. Two builds had Errors with No

Impacts. The monthly points are deducted from the number of builds (30 - .5 =

29.5). That number will be divided by the total number of builds (29.5/30 = (.5) divided by 30 builds equates to a percentage of 98.3%.

The 12 month cumulative percentage will be assessed pursuant to the Adjectival

Rating Scale.

Completed with no errors 97% of time

Conduct and complete missions with no operator errors. When an area cannot/does not complete a mission with no operator errors, at no fault of the

Government, rating per mission will be assigned as follows:

0 Points - Mission completed with No Operator Errors

.25 Points - Operator Error with No Impacts

.5 Points - Operator Error with Schedule Impact and

No Mission Impact 1 Point -

Operator Error with

Schedule Impact and

Mission Impact

2 Points - Delay with

Mission Cancellation

Points will be added on the last day of each month, and a percentage will be developed in conjunction with the number of missions. For example: 30 missions occurred during the month.

Two Operator Errors with

No Impacts. The monthly points are deducted from the number of mission (30 - .5 =

29.5). That number will be divided by the total number of missions (29.5/30 = (.5) divided by 30 missions equates to a percentage of

98.3%. The 12 month cumulative percentage will be assessed pursuant to the

Adjectival Rating Scale

Completed with no errors 97% of time.

To ensure accurate tracking and reimbursement of resources, through no fault of the Government, BITS records are to be signed within 2 business days of staffing end time, or actual end of mission, whichever is later. Each instance will be rated as a pass or fail. The cumulative percentage of

"pass" ratings will be used to determine the percentage rating for the reporting period. For example, of 100 reports, 98 were completed timely and received a "pass" rating. The cumulative percentage (98/100) is 98%.

Timeliness: BITS records are to be signed within 2 business days of staffing end time, or actual end of mission 97% of time

WBS

130/140/150/160

Equipment which requires calibration is delivered/ready for pickup by Cal Lab on or before the due date. Each instance will be rated as a pass or fail. The cumulative percentage of "pass" ratings will be used to determine the percentage rating for the reporting period. For example, of 100 requirements, 98 were completed timely and received a "pass" rating. The cumulative percentage

(98/100) is 98%.

All equipment that requires calibration is delivered/ready for pickup by Cal Lab on or before the due date 96% of time

Perform all preventative maintenance on or before the due date. The completed PM

SOMP is expected to be completed on time 100% of the time. Each PM due will be rated as a pass or fail. The cumulative percentage of

"pass" ratings will be used to determine the percentage rating for the reporting period. For example, of 100

PMs 98 were completed timely and received a "pass" rating. The cumulative percentage (98/100) is 98%.

All preventative maintenance is performed on or before the due date

93% of time.

WBS

130/140/150/160

Maintain all equipment in an available status for mission scheduling. The cumulative percentage of up-time for the reporting period will be used to determine the percentage rating for the performance period.

Equipment shall be available for mission scheduling 100% of the time

WBS 210/220 To ensure high caliber products are provided to customers, minimize the percentage of rework on deliverables, at no fault of the Government, and not due to requirement change or negotiated required by date change.

50% of drawings completed and approved by deadline specified by the design engineer

WBS 220 To ensure high caliber products are provided to customers, minimize the percentage of rework on deliverables, at no fault of the Government, and not due to requirement change or negotiated required by date change.

Less than 20% of products require rework

WBS 220 Trouble calls are properly address/rectified within agreed upon time, at no fault of the Government, and not due to requirement change or negotiated required by date change. Major impact - Stop in testing (More than 8 hours) and Minor impact -

Delay in testing (Less than 4 hours)

No more than 4 minor impacts

(delay in testing)

WBS 220 NASA is able to properly upgrade or replicate deliverables.

All documentation is achieved for deliverables 91% of the time

WBS 220 Adherence to current

AS9100 standards for all deliverables. Submit System

Test Report to Configuration

Control Board to close

Configuration Change.

All documentation is achieved for deliverables 91% of the time within

5 business days.

310/330/360

Rapid initial response to customer tickets.

Initial response to customer ticket shall be within 4 business hours of technician receipt at least 75% of the time

WBS 310/330 Completion of Center projects and programs, tracked projects (ITR).

Deliverables must be completed on or before a negotiated due date 75% of the time

WBS 310 System documentation

(including processes and infrastructure), and application information are accurate and revised as necessary. Measure will be by the number of days documentation is approved as completed. TM approves complete.

95% of documentation completed within 90 days

WBS 310 AFRC flight projects are not delayed, archive real time and on-board flight data is available.

Flight Data Archive System (FDAS) are made available to user by the

Government negotiated due date

80% of the time

WBS 330 System documentation

(including mapped processes and infrastructure), operating system/platform information, including application information, must be accurate and revised as necessary. Measurement will be by the number of documents completed within the allotted timeframe approved by the TM.

Documents must be completed within the allotted timeframe and approved by the TM 95% of the time

WBS 330 Completed Center projects and programs, products ITRs.

ITRs must be completed by the negotiated due date, and with minimal rework at least 75% of the time

WBS 330 Innovation, planning, installation, and testing, of all hardware and software upgrades and modifications, including operating systems releases and mobile devices.

Upgrade tasks and projects must be completed by the negotiated due date 75% of the time

WBS 330 System security/vulnerability patches, updates and changes to hardware or peripherals, to include mobile devices, shall be implemented within

NASA guidelines. This will be measured by the patching compliance as reported through ITSEC-EDW.

Patching compliance percentage above 90%, at least 75% of the time

WBS 330 Maintain and update SSPs

(CITC Data Center, Flight

Loads Lab, and

Lab & Engineering Seats), and ensure that the systems are configured and managed in accordance with the SSP's guidelines and controls.

At least 75% compliant with the

Center AFOP-2810.1-002 and meet the Continuous Monitoring requirements for the SSPs

WBS 330 Support and manage the center-wide backup system.

The restoration of a complete system shall be completed within 2 business days (48 hours) of customer request at least 80% of the time

WBS 330 Support and manage the

Engineering Lab Seats systems.

Maintain the government mandatory

30-day inventory, except systems that have an approved IT security waiver at least 80% of the time

WBS 340 Conduct semi-annual review of Knowledge Base (Tier 1 scripts); ensure responsible parties update the information by the review date.

More than 50% of scripts are reviewed each period for accuracy

WBS 340 Maintain staffing of Help

Desk during business hours, as identified in the PWS.

Exception to operation hours include Center-authorized administrative time.

Roll to voicemail < 20%

WBS 340 Service requests assigned directly to contractor personnel are acknowledged and evaluated within 30 minutes.

30 minute response occurs more than 75% of the time

WBS 340 Positive customer satisfaction results from

Service Desk survey.

Positive response received more than 85% of the time

WBS 340 Timely notification of service impact to Center users and distribution of unanticipated emergency outages. Measure will be the time to distribute notification of emergency outages from the time the Service Desk receives written notification.

90% of emergency outage notifications shall occur within 30 minutes

WBS 340 Timely notification of service impact to Center users and distribution of scheduled outages.

60% of scheduled outage notifications occur more than 8 business hours before outage.

WBS 350 Attend recommended project meetings as established in each project’s communication plan.

Complies with plan more than 75% of the time.

WBS 350 Address Audit and

Assessment findings with project points of contact and ensure submission of corresponding waiver or

POA&M.

Complies with plan more than 75% of the time.

WBS 360 Trouble calls will be addressed and evaluated and security systems will be serviced on business days, or unless otherwise agreed upon.

All trouble calls will be addressed and evaluated within 8 business hours at least 75% of the time

WBS 360 Complete all projects by the customer or TM requested due date or agreed upon date.

All security systems will be serviced during normal business hours, unless otherwise agreed upon.

Completion of project by due date at least 75% of the time

WBS 360 System security/vulnerability patches, updates and changes, will be implemented within NASA guidelines. This will be measured by the Patching compliance as reported through ITSEC-EDW.

Patching compliance percentage above 90%, at least 75% of the time

WBS 360 Maintain accurate continuity of operations, procedures, system documentation

(including processes and infrastructure), and application information.

At least 75% of documents completed within the allotted timeframe

WBS 360 Center projects & programs, products must be completed accurately. Deliverables must be completed with minimal rework. Rework is defined as not achieving customer identified requirements.

Products must be accurate 75% of the time

WBS 410 Deliverables must be received when promised, at no fault of the Government.

Present all task related deliverables to customers by the negotiated due dates.

Deliverables are presented to the customer by the negotiated due date more than 75% of the time

WBS

410/420/430

Provide customers with deliverables that meet or exceed customer expectations and quality products that require limited amount of re-work.

Less than 10% of re-negotiated due dates are missed.

WBS 420/430 Present all task related deliverables to customers.

Deliverables are presented to customer more than 75% of the time

WBS 510 Submit accurate deliverable reports (as identified in the

DRDs) to NASA in a timely manner. Accurate means only minor errors, requiring no extensive rework.

Accurate reports are delivered one day after due date.

WBS 510 Ensure the safety of its people, and protection of its assets, operate in a safe manner.

There shall be no more than 1 negligent OSHA violation or no more than 10 minor incidents

WBS 510 Purchase Request shall be accomplished effectively and efficiently. NASA will measure the percentage of

Purchase Requests completed on or before the required by or negotiated date.

85% of Purchase Requests shall be accomplished by required due date

WBS 510 Utilize small businesses to accomplish purchasing functions.

8% of purchase dollars shall be awarded to small businesses

CONCLUSION

The CO may unilaterally change this plan prior to the beginning of an evaluation period. In addition, the Contractor may make recommended changes to the plan no later than 30 days prior to the beginning of the new evaluation period. The Contractor will be notified of changes to the plan by the CO, in writing, before the start of the affected evaluation period.

CONCUR: ________________________________________ DATE: ________________

Contracting Officer’s Representative

APPROVE: _______________________________________ DATE: ________________

Contracting Officer