AIE QASP Task Order 1.pdf

CUI

Quality Assurance Surveillance Plan (QASP)

DRAFT

MONTH 2027

Prepared By:

Product Lead Accessions Information Environment

Controlled by: ASA(ALT) Controlled by: PEO EIS/IPPS-A/AIE CUI Category(ies): GENERAL PROCUREMENT AND ACQUISITION Limited Dissemination Control: FEDCON

POC:

AIE QASP - DRAFT CUI

April 29, 2024 ii

April 29, 2024 iii

THIS PAGE INTENTIONALLY LEFT BLANK 14

April 29, 2024 iv

Contents 18

1.0 Introduction ......................................................................................................... 1 19

1.1 Purpose ............................................................................................................... 1 20

1.2 Intent .................................................................................................................... 2 21

1.3 Authority.............................................................................................................. 2 22

2.0 Contract Description .......................................................................................... 2 23

2.1 Background ......................................................................................................... 2 24

2.2 Scope of Work .................................................................................................... 3 25

2.3 Type of Contract ................................................................................................. 3 26

3.0 Roles and Responsibilities ................................................................................ 3 27

3.1 Government ........................................................................................................ 3 28

3.1.1 Contracting Officer ................................................................................... 3 29

3.1.2 Project Lead and Associate PL ................................................................ 4 30

3.1.3 Performance Monitors .............................................................................. 5 31

3.2 Contractor ........................................................................................................... 5 32

3.2.1 Contract Manager ..................................................................................... 5 33

3.2.2 Program Manager .................................................................................... 6 34

3.2.3 Quality Manager ....................................................................................... 6 35

4.0 Contract Quality Requirements ......................................................................... 6 36

4.1 Contractor’s Quality Control Program .............................................................. 6 37

4.2 Government Quality Assurance ........................................................................ 6 38

5.0 Government Surveillance .................................................................................. 7 39

5.1 Types of Inspection ............................................................................................ 7 40

5.1.1 100% Inspection ....................................................................................... 7 41

5.1.2 Random Sampling .................................................................................... 7 42

5.1.3 Periodic Inspection ................................................................................... 7 43

5.1.4 Customer Feedback ................................................................................. 7 44

5.1.5 After Action Review (AAR) ....................................................................... 7 45

5.2 Performance Requirements Summary ............................................................. 8 46

5.3 Surveillance Schedule ..................................................................................... 11 47

5.4 Surveillance Instructions/Checklists .............................................................. 11 48

April 29, 2024 v

5.5 Surveillance Documentation ........................................................................... 12 49

6.0 Nonconformance Reports (NCRs) (Attachment 1) ........................................ 12 50

6.1 Level 1 (Minor) .................................................................................................. 12 51

6.2 Level 2 (Major) .................................................................................................. 12 52

6.3 Level 3 (Critical) ................................................................................................ 13 53

6.4 Documenting NCRs .......................................................................................... 13 54

6.5 Tracking and Resolving NCRs ........................................................................ 14 55

6.6 Corrective Action Report (CAR). (Attachment 2) ........................................... 14 56

6.7 Performance Assessment Report (PAR). (Attachment 3) ............................. 14 57

7.0 Deduct Plan ....................................................................................................... 15 58

8.0 Data Analysis .................................................................................................... 15 59

9.0 Acceptance of Services ................................................................................... 16 60

10.0 COR Status Report ........................................................................................... 16 61

11.0 Contractor Manpower Reporting Initiative ..................................................... 16 62

12.0 Contractor Performance Assessment Reporting System (CPARS) ............. 16 63

13.0 Combating Trafficking In Persons (CTIP) ....................................................... 17 64

14.0 COR/Quality Assurance Surveillance Files .................................................... 17 65

15.0 QASP Changes ................................................................................................. 17 66 Attachment 1 - Nonconformance Report Worksheet ............................................... 18 67 Attachment 2 - Corrective Action Report (CAR) ....................................................... 19 68 Attachment 3 - Performance Assessment Report (PAR) ......................................... 19 69 List of Tables 72 Table. 1 Evaluation Ratings Definitions ........................................................................... 8 73 Table. 2Performance Requirements Summary (Surveillance Matrix) .............................. 9 74

April 29, 2024 1

1.0 Introduction 77

The U.S. Army Training and Doctrine Command (TRADOC), the senior responsible 78 official (SRO) for Army Accessions, and senior Army Leadership desires a modern 79 solution to simplify and streamline the recruiting process for the entire Army Accessions 80 Enterprise, including U.S. Army Recruiting Command (USAREC); U.S. Army Cadet 81 Command (USACC); U.S, Army National Guard (ARNG); and the U.S. Army Center for 82 Initial Military Training (C-IMT). 83 The Accessions Information Environment (AIE) will answer TRADOC’s modernization 84 solution as a fully integrated enterprise system implementing new business practices 85 and modernizing recruiting capabilities that meet all Army manning requirements. For 86 the first time, this standardization and modernization enables the Army to leverage 87 minimally customized Commercial Off-The-Shelf (COTS) solution hosted in a 88 commercial Salesforce and Amazon Web Service (AWS) Cloud Impact Level (IL)4 for 89 talent acquisition Army-wide. In concert with systems modernization across the 90 enterprise, AIE will originate the Army record for management in systems downstream. 91 The AIE mission is to develop, configure, and deploy a cloud based AIE solution using 92 Agile methodologies for all operational activities to allow for Continuous Integration / 93 Continuous Delivery (CI/CD), supporting all integration and testing activities necessary 94 per the Government, and compliance with Continuous Risk Management Framework 95 (cRMF). 96

1.1 Purpose 97

The purpose of the Quality Assurance Surveillance Plan (QASP) is to describe the 98 systematic methods the Government will use to evaluate Contractor actions while 99 performing the requirements as laid out in the Performance Work Statement (PWS) for 100 AIE. The QASP provides a means for evaluating whether the Contractor is meeting the 101 performance standards and acceptable quality levels identified in the PWS and 102 identification of surveillance methods used to document the performance. 103 The QASP has been developed by AIE to provide direction to personnel performing task 104 order surveillance activities. Personnel performing surveillance of the contract terms 105 and conditions and PWS requirements will periodically review the QASP throughout the 106 life of the contract. The QASP establishes procedures on how this 107 assessment/inspection process will be conducted. It provides details for a continuous 108 oversight process as follows: 109

• What will be monitored 110

• How monitoring will take place 111

• Who will conduct the monitoring 112

• How monitoring efforts and results will be documented 113

The Contractor is responsible for implementing and delivering performance that meets 114 contract standards using its Quality Control Plan. The QASP provides the structure for 115 the Government's surveillance of the Contractor's performance to assure that it meets 116

April 29, 2024 2 contract standards. It is the Government's responsibility to be objective, fair and 117 consistent in evaluating Contractor performance. 118 The QASP is not part of the contract, nor is it intended to duplicate the Contractor's 120 quality control plan. This QASP is a living document. Flexibility in the QASP is required 121 to allow for an increase or decrease in the level of surveillance necessary based on 122 Contractor performance. The Government will provide a copy of the QASP to the 123 Contractor to facilitate open communication. In addition, the QASP should recognize 124 that unforeseen or uncontrollable circumstances might occur that are outside the control 125 of the Contractor. As a result, the QASP should ensure early identification and 126 resolution of performance issues to minimize impact on mission performance. 127

1.2 Intent 128

The intent of this QASP is to: 1) ensure Contractor performance meets or exceeds 129 contract terms, conditions, and specifications; 2) hold the Contractor accountable for 130 quality control; and 3) encourage the Contractor to take appropriate steps to control and 131 improve quality. Accordingly, the Government will perform surveillance on the contract 132 in accordance with this QASP; and reserves the right to monitor the contract in any 133 manner necessary, at any times necessary, and at all places necessary to ensure that 134 the rendered services conform to contract requirements. The Government also reserves 135 the right to perform quality assurance at the Subcontractor level and perform quality 136 assurance at the Contractor's place of business, if applicable. Resolution of non-137 conforming services discovered at the Subcontractor level will be addressed with the 138 prime Contractor. 139

1.3 Authority 140

Authority for issuance of this QASP is provided under Part 46 of the Federal Acquisition 141 Regulation, Inspection of Services clauses, which provides for inspection, acceptance 142 and documentation of the service called for in the contract or order. This acceptance is 143 to be executed by the Contracting Officer or a duly authorized representative. 144

2.0 Contract Description 145

2.1 Background 146

Program Executive Office Enterprise Information Systems (PEO EIS) is seeking the 147 development, delivery, and deployment of AIE to provide a Customer Relationship 148 Management (CRM) system that will support the Army’s global recruitment and 149 accessions operations with over 28,000 core users. Mobility, reliability, secure 150 connectivity, and data access are paramount AIE attributes. The enterprise workforce 151 must be able to perform tasks using any device, anytime, anywhere, while maintaining 152 secure connectivity with the AIE. 153 Development will consider (1) Agile execution combined with Development, Security, 154 and Operations (DevSecOps) for all activities / tasks to allow for Continuous Integration 155 / Continuous Delivery (CI/CD) and compliance with the Continuous Risk Management 156

April 29, 2024 3

Framework (cRMF), (2) task automation to reduce / eliminate manual effort, and (3) 157 innovation and creativity in problem-solving. 158 For this contract, Agile execution is defined as a group of software methods (e.g., 159 Scrum, Kanban, SAFe®) based on iterative and incremental development and 160 deployment of software capabilities for end users, where requirements and solutions 161 evolve through collaboration between cross-functional teams. These teams include the 162 Government and Contractor communities working across parallel environments where 163 capabilities are implemented, and production system capability support is ongoing. 164 Contractors will apply and perform Agile execution in varying degrees to all aspects of 165 the program life cycle across and within key activities. These activities include 166 requirements analysis, elicitation, elaboration, decomposition, architecture, design, 167 development, implementation, test, integration, training material development and 168 deployment, in addition to support functions such as program management (planning, 169 reporting), configuration management, change control, release management, quality 170 assurance, and cybersecurity. 171

2.2 Scope of Work 172

The Army requires the following services, and the Contractor shall deliver an AIE 173 solution as follows: (1) design, configure, build, and test, (2) integrate and test the AIE/ 174 solution with the AIE production baseline, (3) develop AIE training material, (4) support 175 AIE release and deployment activities, and (5) transition AIE functionality to production 176 operations support. 177 These services will be acquired across the following tasks: 178

• Task 1: Agile Program Management 179

• Task 2: Solution Configuration, Development, and Test 180

2.3 Type of Contract 181

This is a Hybrid contract including Firm Fixed Price (FFP), Cost Plus Fixed Fee (CPFF) 182 Term, CPFF Completion, and Cost Reimbursable (no fee) line items for Travel and 183 Other Direct Costs (ODCs). 184

3.0 Roles and Responsibilities 185

3.1 Government 186

3.1.1 Contracting Officer 187

The Contracting Officer (KO) shall ensure performance of all necessary actions for 189 effective contracting and compliance with the contract terms and shall safeguard the 190 interests of the United States in the contractual relationship. The KO shall also ensure 191 the Contractor receives impartial, fair, and equitable treatment under this contract. The 192 KO shall determine and document the final assessment of the Contractor's performance 193 in the Government’s past performance tracking system. 194

April 29, 2024 4

Assigned CO: Megan A. Lamiotte, Contracting Officer 195 Organization or Agency: U.S. Army Contracting Command – Aberdeen Proving Ground 196 Telephone: (520) 669-7869 197 Email: megan.a.lamiotte.civ@army.mil 198 Contracting Officer’s Representative 200 The Contracting Officer’s Representative (COR) will be identified by separate letter. The 201 COR monitors all technical aspects of the contract and assists in contract 202 administration. The COR is authorized to perform the following functions: assure that 203 the Contractor performs the technical requirements of the contract: perform inspections 204 necessary in connection with contract performance: maintain written and oral 205 communications with the Contractor concerning technical aspects of the contract: issue 206 written interpretations of technical requirements, including Government drawings, 207 designs, specifications: monitor Contractor's performance and notifies both the 208 Contracting Officer and Contractor of any deficiencies; coordinate availability of 209 Government furnished property, and provide site entry of Contractor personnel. A letter 210 of designation issued to the COR, a copy of which is sent to the Contractor, states the 211 responsibilities and limitations of the COR, especially with regard to changes in cost or 212 price, estimates or changes in delivery dates. The COR is not authorized to change any 213 of the terms and conditions of the resulting order. 214 Assigned COR: TBD Contract Officer Representative 215 Organization or Agency: USARMY AIE 216 Telephone: 217 Email: 218

3.1.2 Project Lead and Associate PL 219

The AIE Project Lead (PL) and AIE Deputy PL (APL) provide primary program 220 oversight. While the PL and APL may serve as a direct conduit to provide Government 221 guidance and feedback to the Contractor on project matters, the PL and APL are not 222 empowered to make any contractual commitments or any contract changes on behalf of 223 the Government. The PL and APL will recommend a COR to the KO, ensure the COR is 224 trained before performing any delegated duties, and will support COR performance 225 assessment activities. 226 Project Lead: TBD 227 Organization or Agency: USARMY AIE 228 Telephone: 229 Email: 230 Deputy Project Lead: TBD 231 Organization or Agency: 232 Telephone: 233 Email: 234 mailto:megan.a.lamiotte.civ@army.mil

April 29, 2024 5

3.1.3 Performance Monitors 235

3.1.3.1 Functional Monitor 236

The AIE Functional Monitor will provide detailed functional oversight of the Contractor’s 237 performance regarding business value and alignment with the AIE functional 238 requirements and report his or her findings to the COR in a timely, complete, and 239 impartial fashion to support the COR’s administration activities. Any changes that the 240 Contractor deems may affect contract, price, terms, or conditions shall be referred to the 241 Contracting Officer for action. The Functional Monitor is not empowered to provide any 242 guidance to the Contractor or make any contractual commitments or any contract 243 changes on behalf of the Government. 244 Functional Monitor: TBD 245 Organization or Agency: 246 Telephone: 247 Email: 248

3.1.3.2 Technical Monitors 249

AIE Technical Monitors will provide detailed technical oversight of the Contractor’s 250 performance and report their findings to the COR in a timely, complete, and impartial 251 fashion to support the COR’s administration activities. Any changes that the Contractor 252 deems may affect contract, price, terms, or conditions shall be referred to the 253 Contracting Officer for action. Technical monitors may include performance monitors, 254 inspectors, technical experts, or others who provide information that helps the COR 255 monitor Contractor performance. The Technical Monitors are not empowered to provide 256 any guidance to the Contractor or make any contractual commitments or any contract 257 changes on behalf of the Government. 258 Lead Technical Monitor: TBD 259 Organization or Agency: USARMY 260 Telephone: 261 Email: 262

3.2 Contractor 263

3.2.1 Contract Manager 264

The Contract Manager manages the full range of acquisition activities. This individual 265 will be the primary point of contact for the KO and any designated CORs. Additionally, 266 the Contract Manager will 1) perform work related to Department of Defense (DoD) 267 Contracts Management, such as FAR, Defense Federal Acquisition Regulation 268 Supplement (DFARS), DoD Instructions (DoDI) 5000 series regulations and Clinger-269 Cohen Act compliance; 2) track deadlines, deliverables, and other obligations; 3) plan, 270 direct, and manage projects/operations; 4) track and maintain a record of modifications; 271

4) record and solve contract problems/issues; 5) report on contract deliverables and 272 performance, and 6) ensure product quality. In addition, the Contract Manager will be 273 knowable of different contract types; and will coordinate and interface with the 274

April 29, 2024 6

Government contracting office on contract-related action items and management 275 activities. 276 Contract Manager: TBD 277 Telephone: 278 Email: 279

3.2.2 Program Manager 280

The Contractor will assign an individual to serve as a Contractor Program Manager 281 (PM) for this effort. This individual will be the single point of contact for the Government 282 COR. The PM will be responsible for the performance of the work. The name of this 283 person and an alternate, who will act for the Contractor when the PM is absent, will be 284 designated in writing to the Contracting Officer. The Contractor PM and alternate will 285 have full authority to act for the Contractor on all matters relating to daily operation of 286 this contract. 287 Program Manager: TBD 288 Telephone: 289 Email: 290

3.2.3 Quality Manager 291

The Quality Manager will maintain an effective quality control program to ensure 292 services are performed as agreed to in the contract. This will include the following: 293

• Updating and implementing procedures to identify, prevent, and ensure the non-294 recurrence of defective services, to include products and software 295

• Using programmatic and technical metrics as quality measures to assess 296 performance, progress, speed, cybersecurity, and quality of the product team, 297 leveraging automated tools to the maximum extent practicable 298

Quality Manager: TBD 299 Telephone: 300 Email: 301

4.0 Contract Quality Requirements 302

4.1 Contractor’s Quality Control Program 303

The Contractor’s quality control program (IAW FAR Part 46 and applicable 52.246 304 Clause(s)) is the means by which the Contractor assures that work complies with the 305 requirement of the contract. The Contractor shall ensure services are performed in 306 accordance with the PWS. The Contractor shall identify, prevent, and ensure non-307 recurrence of defective services. 308

4.2 Government Quality Assurance 309

The Government shall evaluate the Contractor’s performance under this contract in 310 accordance with this QASP. This plan is primarily focused on what the Government 311

April 29, 2024 7 must do to ensure that the Contractor has performed in accordance with the 312 performance standards. It defines how the performance standards will be applied, the 313 frequency of surveillance and minimum acceptable defect rate(s). 314

5.0 Government Surveillance 315

5.1 Types of Inspection 316

The following Methods of Surveillance will be utilized in monitoring the Contractors’ 317 performance: 318

5.1.1 100% Inspection 319 The Government will inspect 100% of each deliverable or service performed. This 320 surveillance method is appropriate where 100% inspection is required and is practical. 321

5.1.2 Random Sampling 322

The Government will randomly sample less than 100% of a designated deliverable or 323 performed service on a random basis. This surveillance method is appropriate for a 324 deliverable or performed service that occurs infrequently and where 100% inspection is 325 neither required nor practical. 326

5.1.3 Periodic Inspection 327

The Government will inspect less than 100% of a designated deliverable or performed 328 service, on a periodic basis. This surveillance method is appropriate for a deliverable or 329 performed service that occurs infrequently and where 100% inspection is neither 330 required nor practical. 331

5.1.4 Customer Feedback 332

The following procedures will be followed when a complaint is received regarding the 333 Contractor/Contractor Employees. 334

• The COR will investigate and determine the validity of the complaint 335

• If the complaint is determined to be invalid, the COR will inform the person who 336 submitted the complaint of the reason(s) as soon as practical 337

• For validated complaints that are true contractual non-conformances, the 338 procedures in paragraph 6 below will be followed 339

5.1.5 After Action Review (AAR) 340

The Government will perform random AAR’s at the Planning Interval (PI) and/or sprint 341 level up to 100% of PI’s completed on the Task Order. This surveillance method is 342 appropriate for a deliverable or performed service which will capture best practices, 343 bottlenecks, and future PI/sprint development processes for implementation. 344

April 29, 2024 8

5.2 Performance Requirements Summary 345

In evaluating the quality of the Contractor’s performance, the following performance 346 ratings under Table 1 may be used IAW FAR 42.1503, Table 42-1. 347

Table. 1 Evaluation Ratings Definitions 348

Rating Definition Note

Exceptional – Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective.

To justify an Exceptional rating, identify multiple significant events and state how they were of benefit to the Government. A singular benefit, however, could be of such magnitude that it alone constitutes an Exceptional rating. Also, there should not have been any significant weaknesses identified.

Very Good – Performance meets contractual requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the Contractor were effective.

To justify a Very Good rating, identify a significant event and state how it was a benefit to the Government. There should not be any significant weaknesses identified.

Satisfactory – Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the Contractor appear or were satisfactory.

To justify a Satisfactory rating, there should have been only minor problems, or major problems the Contractor recovered from without impact to the contract/order.

There should not have been significant weaknesses identified. A fundamental principle of assigning ratings is that Contractors will not be evaluated with a rating lower than Satisfactory solely for not performing beyond the requirements of the contract/order.

Marginal – Performance does not meet some contractual requirements. The contractual performance of the element or sub-element being evaluated reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor proposed actions appear only marginally effective or were not fully implemented.

To justify Marginal performance, identify a significant event in each category that the Contractor had trouble overcoming and state how it impacted the Government. A Marginal rating should be supported by referencing the management tool that notified the Contractor of the contractual deficiency (e.g., management, quality, safety, or environmental deficiency report or letter).

Unsatisfactory – Performance does not meet most contractual requirements and recovery is not likely in a timely manner.

The contractual performance of the element or sub-element contains a serious problem(s) for which Contractor corrective actions appear or were ineffective.

To justify an Unsatisfactory rating, identify multiple significant events in each category that the Contractor had trouble overcoming and state how it impacted the Government. A singular problem, however, could be of such serious magnitude that it alone constitutes an unsatisfactory rating. An Unsatisfactory rating should be supported by referencing the management tools used to notify the Contractor of the contractual deficiencies (e.g., management, quality, safety, or environmental deficiency reports, or letters).

The Performance Requirements Summary or Surveillance Matrix (Table 2) is the list of 349 performance objectives and standards that must be performed by the Contractor. This 350

April 29, 2024 9 matrix details the method of surveillance the COR will use to validate and inspect these 351 performance elements. Inspection of each element will be documented in the COR file. 352 Performance objectives define the desired outcomes. Performance Standards define 353 the level of service required under the contract to successfully meet the performance 354 objective. The inspection methodology defines how, when, and what will be assessed in 355 measuring performance. The Government performs surveillance, using this QASP, to 356 determine the quality of the Contractor's performance as it relates to the performance 357 element standards. The Program Requirements Summary will be used to form the 358 foundation of the COR's inspection checklist. 359 Table 2 below provides list of Agile measures that will be tracked by the AIE program. 360 The goal of these measures is tracking incremental, secure and rapid value-added 361 capabilities to users. 362

Table. 2Performance Requirements Summary (Surveillance Matrix) 364

Type Deliverable or Required

Services Rating Area

Description Performance Standard(s) Objective

Acceptable Quality Level

(AQL)

Threshold

Method of Surveillance

Salesforce out-of-box utilization rate

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics

Maximizing Salesforce out-of-box utilization improves rapid and incremental delivery to users, and downstream capability support cost.

Accurate, complete. and support tasks and functions as directed by the Government

Meets the following at a minimum of 90%.

100% Inspection

Tracked on Agile tool such as JIRA at Epic level – lowest level user stories or tasks mapped to epics should be labeled as either custom or out of box

% of each high-level capability completed

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics

We must track capability delivery at epic level to gage progress.

Accurate, complete. and timely support of tasks and functions as directed by the Government

This will be determined by velocity and CI/CD efficiency. But it MUST be tracked.

100% Inspection

• Tracked on Agile tool such as JIRA at Epic level – lowest level user stories or tasks mapped to epics should be labeled as either custom or out of box

April 29, 2024 10

Type Deliverable or Required

Services Rating Area

Description Performance Standard(s) Objective

Acceptable Quality Level

(AQL)

Threshold

Method of Surveillance

# of user stories ready for development

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics

Rapid and incremental delivery needs adequate decomposed user stories with acceptance criteria ready to be development and configured.

More prioritized user stories decomposed, and ready for development then can be put on next two Sprints

More prioritized user stories decomposed, and ready for development then can be put on next two Sprints

100% Inspection

Tracked on Agile tool such as JIRA

Velocity

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics

Amount of work the team completes during a given period.

Velocity improvement over time

At least 75% of previous iteration.

100% Inspection

Tracked on Agile tool such as JIRA

User acceptance rate

Materials, Reports, and Metrics

Rate of developed/configured functionality that is accepted by users

Maximize developed and configuration to meet user needs

At least 50% of developed and configured user stories/tasks accepted by users.

100% Inspection

Tracked on Agile tool such as JIRA

Escape Rate

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics number of defects for a deployment that were identified after the release

Reduce defects that are deployed to production

Less than 15%.

Contractor shall resolve all post-production technical defects without any cost to government.

100% Inspection

Tracked on Agile tool such as JIRA

Recidivism Rate

PWS Section 5.1

A003 Materials, Reports, and Metrics

Materials, Reports, and Metrics

Percent of work returned to the developer for various reasons within the CI/CD Workflow

Minimize returned work to developers Less than 50%

100% Inspection

Tracked on Agile tool such as JIRA

April 29, 2024 11

*Note, the Government notifies the Contractor that it intends to adopt automated testing 365 and will establish metric(s) for automated testing at a later date under the terms of 366 QASP paragraph 15.0. 367 In addition, the Government will inspect user stories (hereafter stories) at the end of 368 each Sprint to evaluate the “Definition of Done” achieved during the Sprint. The target 369 “Definition of Done” for each Sprint will be defined at the beginning of the Sprint up to 370 and including integration and interoperability criteria. The Government expects that 371 stories for interoperability between applications and interfaces will be explicitly identified 372 as part of the development process. 373 A CPARS rating area (RA) is assigned to each relevant metric. The rating areas are 374 Quality, Management, Schedule, Cost Control, Small Business Subcontracting, and 375 regulatory compliance. Each rating area rating will be provided to the COR as input for 376 monthly and annual reporting. 377 The COR, along with the AIE-A PdM, AIE APM, and designated Government Functional 378 and Technical Monitors, will inspect all tasks to ensure Contractor compliance with the 379 requirements at the conclusion of each Sprint or PI. Delivery will occur by a data pull 380 within the AIE agile development tools repository. Inspection results will be recorded 381 and reported. The COR may find the delivery satisfactory even though further work is 382 required, provided that the specific requirements of the sprint are met. 383 At the conclusion of each sprint, the COR, along with the AIE PdM, AIE APM, and 384 designated Government Functional and Technical Monitors, will review the completed 385 stories and related functionality. Incomplete or inadequate code and stories will be 386 noted in a mutually agreed-upon issue tracker, such as Trello or GitHub Issues, and 387 links to each issue shared with the COR. The Contractor may respond in that tracker as 388 appropriate, addressing the accuracy and validity of the defect as well as any planned 389 corrective action (if not already noted). The Contractor team will discuss and document 390 actions to prevent recurrence in their sprint retrospectives. 391 At the conclusion of the period of performance, a similar procedure will be followed to 392 document discrepancies and to assess overall performance. 393

5.3 Surveillance Schedule 394

A Surveillance Schedule will be created for each month of Contractor performance and 395 filed in the Procurement Integrated Enterprise Environment (PIEE), Surveillance and 396 Performance Monitoring (SPM), Surveillance Documents Section of the Contract File 397 using the Add Smart Form function. The schedule is “Controlled Unclassified 398 Information (CUI)” and is not releasable to anyone other than those authorized by the 399 Government. 400

5.4 Surveillance Instructions/Checklists 401

Surveillance Checklists will be completed when surveillances are conducted and filed in 402 the PIEE, SPM, Surveillance Documents Section of the Contract File using the Add 403 Smart Form function. Supplementary data, such as end of sprint reports directly 404 extracted from the development tools that provide a snapshot of real-time progress will 405 be captured when possible. The objective is to take advantage of any tools and 406

April 29, 2024 12 streamlined processes available that capture results suitably to obtain the most 407 accurate results in an administrative-efficient manner. 408

5.5 Surveillance Documentation 409

No later than the 15th of each month surveillance documentation for the preceding 410 month will be filed in the PIEE, SPM, Surveillance Documents Section. Surveillance 411 documentation includes a COR Status Report, the Surveillance Schedule, all completed 412 Surveillance Checklist(s) for the period the COR Status Report covers, 413 Nonconformance Reports, Performance Assessment Report, Correction Action Report, 414 if any, and Contractor Deliverables, if any. 415

6.0 Nonconformance Reports (NCRs) (Attachment 1) 416

Nonconformance occurs when a Contractor fails to meet Contract or Task Order terms 417 and conditions and/or specifications or the services are defective in some manner. All 418 instances of nonconformance will be documented via a Non-conformance Report 419 Worksheet, Attachment 1 and the Contracting Officer will notify the Contractor of the 420 nonconforming performance via Letter. Resolution of nonconformance discovered at the 421 Subcontractor level will be addressed with the prime Contractor. Nonconformance is 422 classified as either: Level 1 (Minor); Level 2 (Major); or Level 3 (Critical). NCRs may 423 contain multiple nonconformances, each characterized by a level and type. The level of 424 the NCR should normally correspond to the highest level of nonconformance listed in 425 the NCR. For example, if an NCR had one Level 3, one Level 2, and three Level 1 non-426 conformances, the NCR level would be Level 3. The exception might be when the 427 aggregate of multiple nonconformances considered in whole associate to a more 428 serious type of nonconformance defined in FAR 46.101 (e.g., if an NCR had three Level 429 2 nonconformances but the definition of critical nonconformance better describes the 430 impact/effect when considered in whole, the NCR level would be 3). 431

6.1 Level 1 (Minor) 432

Level 1 nonconformances are those that are ‘minor’ in nature such that they can be 433 promptly corrected by the Contractor, present no need for root cause analysis or 434 determination if other contract requirements are affected, present no need for further 435 preventive action, and do not materially reduce the usability of the supplies, services, 436 construction, or deliverables. While the Contractor must correct the nonconformance, 437 further actions and Government follow-up are not required, and no written response 438 from the Contractor is needed. 439

6.2 Level 2 (Major) 440

Level 2 nonconformance(s) are those that are ‘major’ in nature such that they are not 441 promptly correctable, and/or warrant root cause analysis and preventive action, and/or 442 need action by the Contractor to determine if other contract requirements are affected. 443 These nonconformances are serious enough to warrant not only correction of the 444 nonconformance, but also a written response from the Contractor in the form of a 445 Corrective Action Plan (CAP), and Government follow-up. The qualifying distinction 446

April 29, 2024 13 between a Level 1 and Level 2 nonconformance is the need for a CAP and Government 447 follow-up. An NCR identifying a Level 2 nonconformance may also be issued after 448 repetitive Level 1 instances of the same or similar nonconformance demonstrate the 449 Contractor is failing to institute effective corrective action and/or failing to implement 450 effective quality and/or internal control(s). Repetitive Level 1/minor nonconformance of 451 the same or similar requirement(s) increases the risk to the Government that continued, 452 repetitious, or cumulative failures may materially reduce the usability of the supplies, 453 services, construction, work, or deliverables for their intended purpose or is an indicator 454 that root cause analysis, and a CAP is required to fix the underlying conditions creating 455 the nonconformance. Further, serious individual Contractor employee misconduct 456 discovered by the Government and traceable to contract requirements should normally 457 result in a Level 2 nonconformance because the serious nature of misconduct 458 necessitates Contractor management conduct root cause analysis, take preventive 459 actions, and requires Government follow-up (characteristics of a Level 2 NCR). Further, 460 issue an NCR with a Level 3 nonconformance for serious Contractor employee 461 misconduct only when there is a trend demonstrating a breakdown in Contractor 462 management or processes allowing serious misconduct to repetitively occur, or the 463 misconduct poses a hazardous or unsafe condition for the personnel receiving the 464 contracted supply, service, or construction, and misconduct can be traced to 465 nonconformance of contract requirements. 466

6.3 Level 3 (Critical) 467

Level 3 nonconformance(s) are those that are ‘critical’ in nature such that they represent 468 a serious weakness in internal controls and/or have actual or potential significant 469 impact(s) on contract performance and/or the supported unit/Government operations. 470 For these nonconformance(s), the qualifying distinction between a Level 2 and Level 3 471 nonconformance is the seriousness of the issue prompting the need for higher level 472 management to address the nonconformance and resulting corrective action. Due to the 473 nature of a Level 3 nonconformance, these types of nonconformance are candidates for 474 contract remedies. Therefore, before issuance of an NCR with a Level 3 475 nonconformance, consideration should be made whether an NCR or contract remedy is 476 more appropriate. Issuing a Level 3 NCR in lieu of a contract remedy would normally be 477 considered the Government’s final attempt to work with the Contractor to resolve a 478 contractual nonconformance before contractual remedies are used. However, issuance 479 of a Level 3 NCR is not a requirement before using contractual remedies. An NCR 480 identifying a Level 3 nonconformance (or a contract remedy) may be issued after 481 repetitive Level 2 instances of the same or similar nonconformance demonstrate the 482 Contractor is failing to institute effective corrective action or failing to implement 483 effective quality/internal controls. Repetitive major nonconformances with the same or 484 similar requirement(s) increase the risk to the Government that continued, repetitious, or 485 cumulative failures may significantly impact the mission, or may result in 486 hazardous/unsafe conditions. 487

6.4 Documenting NCRs 488

Use/fill out the Nonconformance Report Worksheet, Attachment 1, when nonconforming 489 performance is identified. Documentation, as a minimum, will include: the Contract or 490

April 29, 2024 14

Task Order number; reference to the specific contract requirement; the specific 491 discrepancy to the requirement; where it was discovered; the date and time it was 492 discovered; and the Contractor representative who was notified. Also, include a 493 suspense date for Contractor response/corrective action for Level 2 and Level 3 NCRs. 494 The Contracting Officer must ensure appropriate level review and approval of the NCR 495 Worksheet and then addresses the NCR in Letter Format to the applicable level of 496 Contractor management based on the seriousness of the nonconformance. 497

6.5 Tracking and Resolving NCRs 498

All NCRs will be included in the COR’s Status Report. The Government considers Level 499 1 nonconformance closed as of the issuance of the Letter to the Contractor. However, 500 your attention to address Level 1 Minor nonconformance is still required. Repetitive 501 nonconformance of the same or similar requirement would be an indication the 502 Contractor failed to institute effective corrective action and/or failed to implement 503 effective quality control and may result in a Level 2 NCR requiring submission of a CAP 504 to the Government. The COR and KO shall track Level 2 and Level 3 NCRs to ensure 505 the Contractor identifies and takes appropriate corrective action to the root cause and 506 the specific nonconformance is corrected within the required timeline. The KO will make 507 a final written determination of the Government’s remedies if the non-conformance is 508 not corrected by the Contractor after receiving notice and a reasonable opportunity to 509 correct the work. 510

6.6 Corrective Action Report (CAR). (Attachment 2) 511

When the COR determines formal written communication is required, the COR will 512 prepare a Corrective Action Report (CAR) (Attachment 2). The CAR is forwarded to the 513 KO for dissemination to the Contractor. The Contractor shall acknowledge receipt of the 514 CAR in writing. The CAR will specify if the Contractor is required to prepare a corrective 515 action plan to document how the Contractor shall correct the unacceptable performance 516 and avoid a recurrence. The CAR will also state how long after receipt the Contractor 517 has to provide a corrective action plan to the KO and COR. The Government may 518 withhold payment for services until corrective action is taken. The Government will 519 review the Contractor's corrective action plan to determine acceptability. Upon 520 Government acceptance of the corrected services the Contractor will be permitted to 521 invoice for the service fee. Any CAR may become a part of the supporting 522 documentation for contract payment deductions, fixed fee deductions, CPARS, or other 523 actions deemed necessary by the KO. CAR may be required under a Nonconformance 524 report. 525

6.7 Performance Assessment Report (PAR). (Attachment 3) 526

The Government will document performance and a PAR may be utilized in support of 527 documented performance included in the COR Monthly Reports in PIEE. Any report 528 may become a part of the supporting documentation for fixed fee payments, or other 529 actions. Government acceptance of services permits the Contractor to invoice for the 530 service fee. 531

April 29, 2024 15

7.0 Deduct Plan 532

The COR shall not certify satisfactory performance until all service / task defects have 533 been corrected. When the defects in services / task conduct cannot be corrected by re-534 performance, the Government may: 535

• Require the Contractor to take necessary action to ensure that future 536 performance conforms to the task requirements; and, 537

• Reduce the contract price to reflect the reduced value of the services performed. 538 The COR will, in addition to providing documentation to the CO, maintain a 539 complete quality assurance file. The file will contain copies of all reports, 540 evaluations, recommendations, and any actions relating to the Government’s 541 performance of the quality assurance function, including originals of all 542 surveillance activity checklists. All such records will be retained for the life of the 543 contract. The COR shall forward these records to the CO at completion or 544 termination of the contract. 545

The Government fully supports the Scaled Agile Framework (SAFe) and SAFe’s 546 concept to “Fail Fast”; however, the Government needs to ensure the Contractor 547 delivers a quality product in a timely manner. The Government will implement a Deduct 548 Plan that will support “Fast Fail” and aligns with the Government’s responsibility to 549 ensure the Contractor delivers a quality product in a timely manner. The Deduct Plan is 550 based on the following types of failures: 551

• Contractor Fault: Contractor fails to complete a user story and meet Definition of 552 Done 553

If the Contractor fails to deliver a user story planned to be developed in a 554 PI because of Contractor’s Fault and the failure is a first occurrence for 555 this user story, then the cost to develop the user story a second time will 556 be paid for by the Government as part of the Contractor’s planned effort in 557 a future PI. 558

If the Contractor fails to deliver a user story planned to be developed in a 559 PI because of the Contractor’s Fault and the failure is a second or more 560 occurrence in a PI, then the cost to complete will be deducted from the 561 cost to complete the user story in a future PI by the Contractor. 562

• Government Fault: Government Roadblock or delay prevents user story 563 completion 564

If the Contractor fails to deliver a user story planned to be developed in a 565 PI because of a Government Roadblock or delay, no matter the 566 occurrence for this user story, then the cost to develop the user story a 567 second or additional occurrence, will be paid for by the Government as 568 part of the Contractor’s planned effort in a future PI. 569

8.0 Data Analysis 570

Data Analysis will be performed to identify trends in cost, schedule and/or performance 571 risks. The results of Data Analysis will be documented in COR Status Reports and may 572 be used to make adjustments to surveillance schedules, and increase or decrease 573

April 29, 2024 16 surveillance, if deemed appropriate. Negative trends noted using Government 574 data/observations could result in issuing the Contractor a Nonconformance Report. 575 Trends noted using Contractor data will not result in a Nonconformance Report unless 576 the Contractor is not addressing the negative trend. 577

9.0 Acceptance of Services 578

When all services have been deemed acceptable and there is documented objective 579 quality evidence to support acceptable performance, the COR will accept the services 580 provided and authorize payment to the Contractor. This is accomplished by approving 581 the Contractor’s Receiving Reports in PIEE, Wide Area Workflow (WAWF). 582

10.0 COR Status Report 583

COR Status Reports will be submitted on a monthly basis, no later than the 15th of each 584 month for the preceding frequency, to the Contracting Officer via PIEE, SPM, 585 Surveillance Documents Section using the Add Smart Form function. COR Status 586 Reports will provide a synopsis of the Contractors performance for the inclusive dates of 587 the report. The synopsis will contain a summary of: surveillances performed; Customer 588 Feedback, if any; Nonconformance Reports, if any; an analysis of the Contractors 589 performance; recommended adjustments to surveillance schedules; and any other 590 pertinent contract information. 591

11.0 Contractor Manpower Reporting Initiative 592

In accordance with Office of the Secretary of Defense Memorandum, Revised 593 Department of Defense Contractor Manpower Reporting Initiative, dated Oct 16, 2019, 594 the…