ATTACHMENT_10_AFMAN_63-119.xls
XLS spreadsheet 186 KB Posted
- Attached to
- T-38A/B and A-10 Automatic Dependent Surveillance-Broadcast (ADS-B) Federal contract opportunity
- Solicitation number
- FA822017R0001
About this file
AFMAN 63-119
View the file
Other files for this federal contract opportunity
Show all 26
On GovTribe
Work with this file on GovTribe
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Index
| AFMAN 63-119 OT&E Template Index |
| How to use 63-119 templates: In the appropriate block under service designation type G, Y, or R (blocks defaulted to red "R"). The block will turn that color to reflect the current status. Typing N/A or a Question Mark will turn the block white. The blocks are not case sensitive. Clicking on the template designation in the index will take you to that template. Before using, save the file to your hard drive. |
| Attachment 2: |
ACQUISITION STRATEGY AND SCHEDULE
Attachment 3 ANALYSIS OF ALTERNATIVES (AoA) Attachment 4
CAPABILITIES BASED REQUIREMENTS DOCUMENTS (CBRD)
Attachment 5
THREAT & INTELLIGENCE DOCUMENTS
Attachment 6
INTEGRATED TEST TEAM (ITT) STANDUP AND ITT CHARTER
Attachment 7
AIR FORCE CONCEPTS
Attachment 8
LIFE CYCLE SUSTAINMENT PLAN (LCSP)
Attachment 9
INFORMATION TECHNOLOGY (IT) AND NATIONAL SECURITY SYSTEMS (NSS)
Attachment 10
TEMP (MS B AND BEYOND)
Attachment 11
INTEGRATED PLANNING
Attachment 12
PROGRAM PROTECTION AND SECURITY
Attachment 13
CONTRACTOR TESTING
Attachment 14
DEVELOPMENTAL TEST AND EVALUATION (DT&E)
ATTACHMENT 15
SOFTWARE DEVELOPMENT AND MATURITY
Attachment 16
LIVE FIRE TEST AND EVALUATION (LFT&E)
Attachment 17
MODELING AND SIMULATION (M&S)
Attachment 18
CONFIGURATION MANAGEMANT PLAN (CMP)
Attachment 19
DEFICIENCY IDENTIFICATION AND RESOLUTION PROCESS
Attachment 20
PRODUCTION REPRESENTATIVE TEST ARTICLES
Attachment 21
SYSTEM PERFORMANCE
Attachment 22
OPERATIONAL TEST AND EVALUATION PLAN
Attachment 23
PROGRAMMATIC ENVIRONMENT, SAFETY, AND OCCUPATIONAL HEALTH EVALUATION (PESHE)
Attachment 24
OPERATIONAL TEST TEAM TRAINING
Attachment 25
SUPPORT EQUIPMENT (SE)
Attachment 26
SUFFICIENCY OF SPARES
Attachment 27
SUPPORT AGREEMENTS
Attachment 28
PACKAGING, HANDLING, AND TRANSPORTATION
Attachment 29
PERSONNEL
Attachment 30
CONTRACTOR SUPPORT
Attachment 31
TECHNICAL DATA
Attachment 32
TEST AND EVALUATION RESOURCES
Attachment 2:
ACQUISITION STRATEGY AND SCHEDULE
Attachment 3 ANALYSIS OF ALTERNATIVES (AoA) Attachment 4
CAPABILITIES BASED REQUIREMENTS DOCUMENTS (CBRD)
Attachment 6
INTEGRATED TEST TEAM (ITT) STANDUP AND ITT CHARTER
Attachment 7
AIR FORCE CONCEPTS
Attachment 8
LIFE CYCLE SUSTAINMENT PLAN (LCSP)
Attachment 9
INFORMATION TECHNOLOGY (IT) AND NATIONAL SECURITY SYSTEMS (NSS)
Attachment 10
TEMP (MS B AND BEYOND)
Attachment 11
INTEGRATED PLANNING
Attachment 12
PROGRAM PROTECTION AND SECURITY
Attachment 13
CONTRACTOR TESTING
Attachment 14
DEVELOPMENTAL TEST AND EVALUATION (DT&E)
Attachment 16
LIVE FIRE TEST AND EVALUATION (LFT&E)
Attachment 17
MODELING AND SIMULATION (M&S)
Attachment 18
CONFIGURATION MANAGEMANT PLAN (CMP)
Attachment 19
DEFICIENCY IDENTIFICATION AND RESOLUTION PROCESS
Attachment 20
PRODUCTION REPRESENTATIVE TEST ARTICLES
Attachment 21
SYSTEM PERFORMANCE
Attachment 22
OPERATIONAL TEST AND EVALUATION PLAN
Attachment 23
PROGRAMMATIC ENVIRONMENT, SAFETY, AND OCCUPATIONAL HEALTH EVALUATION (PESHE)
Attachment 24
OPERATIONAL TEST TEAM TRAINING
Attachment 25
SUPPORT EQUIPMENT (SE)
Attachment 26
SUFFICIENCY OF SPARES
Attachment 27
SUPPORT AGREEMENTS
Attachment 28
PACKAGING, HANDLING, AND TRANSPORTATION
Attachment 29
PERSONNEL
Attachment 30
CONTRACTOR SUPPORT
Attachment 31
TECHNICAL DATA
Attachment 5
THREAT & INTELLIGENCE DOCUMENTS
Attachment 32
TEST AND EVALUATION RESOURCES
ATTACHMENT 15
SOFTWARE DEVELOPMENT AND MATURITY
AFMAN 63-119
| Compiled From: AFMAN 63-119,V4.3, 6 Oct 2015 | |||||
| Attachment 2: | |||||
| ACQUISITION STRATEGY AND SCHEDULE | USAF | Responsible Party | Comments | ||
| A2.1. | Ensure early operational tester (AFOTEC or MAJCOM) involvement when developing the acquisition strategy to ensure the strategy for T&E provides needed support. | R | PM | ||
| A2.2. | Develop realistic, achievable, event-driven acquisition and test schedules and ensure they are harmonized throughout all program documents. Avoid success-oriented schedules. | R | PM | ||
| A2.3. | Congressional and PPBE schedule constraints are incorporated into the acquisition schedule. | R | PM | ||
| A2.4. | Ensure sufficient and timely RDT&E funding and procurement appropriations are programmed during each budget cycle to keep the program in technical balance. | R | PM, OTO | ||
| A2.5. | Schedule sufficient numbers of certification reviews over the program’s projected life cycle. Frequency of reviews should increase as the program nears the start of dedicated OT&E. | R | PM | ||
| A2.6. | Resolve open issues, particularly with requirements, sufficiently early to permit orderly planning and transition to dedicated OT&E. | R | PM | ||
| A2.7. | If an incremental strategy is used, a clear distinction must exist between each increment for determining what will be tested, produced and/or fielded. | R | PM, User | ||
| A2.7.1. | Operational capabilities are clearly assigned to specific increments. | R | PM | ||
| A2.7.2. | Provisions exist for developing and operationally testing subsequent increments after the initial increment is complete. | R | PM | ||
| A2.8. | Ensure contract(s) capture the content of the most recent CBRD or appropriate requirement document. | R | PM | ||
| A2.9. | A CDT has been identified (if an ACAT I or MAIS program), or a Test Manager (or CDT) for other than MDAP and MAIS programs. | R | PM | ||
| Attachment 3 | |||||
| ANALYSIS OF ALTERNATIVES (AoA) | USAF | Responsible Party | Comments | ||
| A3.1. | The AoA (if required) may require updating, re-validation, and approval at the appropriate level prior to each milestone. | R | User | ||
| A3.2. | All reasonable alternatives must be objectively described. The military value of the final alternatives must be clearly identified. | R | User | ||
| A3.2.1. | All relevant costs must be identified, preferably using objective engineering and business estimates derived from accepted Air Force cost analysis principles and processes. | R | PM | ||
| A3.2.2. | All assumptions and constraints must be explicitly identified and supported by the latest CBRD, AoA guidance documents, or reasonable basis determined by the AoA sponsoring agency. | R | User | ||
| A3.2.3. | Acceptable ranges of performance must be established using rigorous cost-benefit, trade-off, and sensitivity analyses to show decision makers when and where certain degradations in system cost or performance yield outcomes that no longer satisfy the mission need. | R | User | ||
| A3.3. | Measures of Effectiveness (MOE) and Measures of Suitability (MOS) must reflect operational utility and show how they were derived from the requirements documents. | R | OTO | ||
| A3.3.1. | MOEs and MOSs at the operational task level must be "testable" in order to develop DT&E and OT&E plans and concepts. MOEs must be developed as early as possible and agreed to between user and tester. | R | OTO | ||
| A3.3.2. | The AoA’s MOEs, MOSs, Measures of Performance (MOP), and other criteria must be linked to system performance thresholds stated in the latest threat and requirements documents and "track" throughout the program's development. | R | OTO | ||
| Attachment 4 | |||||
| CAPABILITY BASED REQUIREMENTS DOCUMENTS (CBRD) | USAF | Responsible Party | Comments | ||
| A4.1. | The appropriate CBRD (i.e., Initial Capability Document (ICD), Draft Capability Development Document (CDD), or Capability Production Document (CPD)), and CONOPS must be coordinated and approved at appropriate levels prior to each milestone, after major program changes, and early enough to develop the TEMP and OT&E test concept and OT&E plan. | R | User | ||
| A4.1.1. | AF Form 1067s issued for modification programs that introduce new capability must include a Table of Performance Parameters/Attributes (KPP, KSA, other or attributes) with minimum Threshold/Objective values similar to the format for a CDD/CPD. | R | User | ||
| A4.1.2. | AF Form 1067s issued for permanent sustainment modifications should identify CDD/CPD requirements the modification is intended to sustain. | R | User | ||
| A4.2. | The CBRD must be based on the JPG, Joint Vision, Air Force Vision, and long-range planning inputs from Joint and Air Force concepts. | R | User | ||
| A4.3. | The CBRD’s capabilities must accurately flow down through the AoA, acquisition strategy, and TEMP to the OT&E concept and OT&E plan. | R | User | ||
| A4.4. | The proposed system design must satisfy projected operational requirements in the CBRD and SPG. | R | PM | ||
| A4.5. | The system must provide the needed capabilities against the most current validated threat described in the system’s threat documents. | R | PM | ||
| A4.5.1. | Ensure Modeling and Simulation (M&S) requirements are identified early to enable programmed funding. | R | PM | ||
| A4.5.2. | Cyberspace threats, attack surfaces, and security requirements must be current. (See Figure 2.3) | R | User | ||
| A4.6. | Joint, multi-national, multi-departmental, or multi-service uses described in the CBRD must be addressed during the system's development. | R | PM | ||
| A4.7. | All thresholds and objectives must be stated in operational terms and defined in measurable, beneficial increments of capability. | R | User | ||
| A4.7.1 | Ensure measureable and testable criteria for how the system supports military operations, is entered and managed on the network and how effectively it exchanges information is specified with threshold and objective values IAW the Joint Capabilities Integration and Development System (JCIDS) Net-Ready KPP requirement. | R | User | ||
| A4.7.2 | Cyber resiliency must be addressed through the JCIDS Survivability KPP. | R | PM | ||
| A4.8. | CBRDs must be stated in such a manner that testable MOEs, MOSs, and MOPs are quantitatively measurable through analytically-based evaluation methods when possible. | R | User | ||
| A4.9. | All CTPs, KPPs, MOEs, MOSs, MOPs, threats, definitions, and other criteria must be consistent (harmonized) across the most current support documents (e.g., CBRD, system threat assessment, AoA, Air Force concepts, APB, TEMP). | R | User | ||
| A4.10. | If increments of operational capability are planned, the CPD must be updated to describe the next increment prior to development of the OT&E concept and OT&E plan. | R | User | ||
| A4.10.1 | Use Joint Requirements Oversight Council (JROC)-approved “IT Box” strategy for future increments if specified. | R | User | ||
| A4.11. | Changes must be finalized and open issues resolved early enough to ensure no adverse impacts on the successful completion of dedicated OT&E. | R | User | ||
| A4.12. | The CBRD must contain a complete audit trail documenting rationale for all requirements changes, including changes from the APB. | R | User | ||
| A4.13. | Only systems with requirements to “protect users in combat” according to USC 10 § 2366 must be listed as “covered systems.” | R | User | ||
| A4.14. | The CBRD must state the appropriate cybersecurity impact values (High, Moderate, and Low) for Confidentiality, Integrity and Availability as well as listing the appropriate security overlays as described in DoDI 8510.01, Risk Management Framework (RMF) for DoD IT. | R | User | ||
| A4.14.1. | Cybersecurity capabilities must also include the requirement to register the system in the Enterprise Information Technology Data Repository (EITDR), assignment of qualified personnel to RMF roles as well as an initial security control baseline. | R | User | ||
| Attachment 5 | |||||
| THREAT & INTELLIGENCE DOCUMENTS | USAF | Responsible Party | Comments | ||
| A5.1. | Threat assessment document(s) must remain valid and current with updates made prior to each milestone. | R | User | ||
| A5.1.1 | Address program impacts of testing against emerging threats which may not be stated in the current validated CBRD including cyberspace threats and impacts. | R | OTO, PM | ||
| A5.2. | The system threat assessment document must be approved by AF/A2. For ACAT I programs, the System Threat Assessment Report (STAR) must be validated by DIA. | R | User | ||
| A5.3. | The system’s threat assessment document(s) must be consistent with current DoD threat projections and accurately reflected in the CBRD and AoA. | R | User | ||
| A5.4. | Sufficient threat detail must be provided to support system R&D and the development of realistic operational mission scenarios in support of the ITC, OT&E plan, and schedules. | R | PM | ||
| A5.4.1. | All threats must be described in system-specific terms and include system-to-system interfaces. | R | PM | ||
| A5.4.2. | Threat shot doctrine and employment tactics must be described. | R | PM | ||
| A5.4.3. | The reactive threat and potential countermeasures must be described. | R | PM | ||
| A5.4.4. | Sources for projections and areas of uncertainty must be cited. | R | PM | ||
| A5.4.5 | Adversarial cyber capabilities and tactics must be understood and described. | R | PM | ||
| A5.5. | Life-cycle mission data plans (LMDPs) shall be established by the program office, or its predecessor organization, for each Intelligence Mission Data (IMD)-dependent acquisition program and effort beginning at MS A. | R | PM | ||
| Attachment 6 | |||||
| INTEGRATED TEST TEAM (ITT) STANDUP, ITT CHARTER | USAF | Responsible Party | Comments | ||
| A6.1. | New-start programs direct establishment of an ITT in the initial ADM as soon as possible after the MDD. (Every program requires an ITT regardless of how long the program has been in existence.) | R | PM | ||
| A6.2. | A current ITT Charter describes ITT activities, membership, goals, products, responsibilities, and operating procedures. | R | PM | ||
| A6.2.1. | A CDT is identified for MDAP and MAIS programs, or a Test Manager (or CDT) for all other programs. This person and the OTO representative co-chair the ITT. | R | PM | ||
| A6.2.2. | The charter covers the entire life cycle of the program. | R | PM | ||
| A6.2.3. | If the system comes under an overarching ITT of related systems, the ITT Charter includes provisions for managing multiple systems. | R | PM | ||
| A6.2.4. | All program stakeholders are represented (e.g., other Services, interoperable systems, and organizations supporting all types of T&E activities). | R | PM | ||
| A6.2.5. | The ITT has sufficient membership participation to be effective. | R | PM | ||
| A6.3. | The ITT directs formation of sub-groups to address specific tasks and responsibilities. | R | ITT | ||
| A6.4. | Research is completed to identify and nominate an LDTO to the PEO or decision review authority, in coordination with AFMC/A3 or AFSPC/A5, as appropriate. | R | ITT | ||
| Attachment 7 | |||||
| AIR FORCE CONCEPTS | USAF | Responsible Party | Comments | ||
| A7.1. | The Air Force concepts must describe expected system employment and operating concepts, strategies, methods, and tactics in concert with the latest CBRD. | R | User | ||
| A7.1.1. | Sufficient detail must permit early development of operationally realistic test scenarios and tactics for the OT&E test concept and test plans. | R | User | ||
| A7.2. | Operational effectiveness and suitability requirements, criteria, thresholds, objectives, and definitions in the CBRD must accurately flow down (be linked) to the Air Force concepts, which must in turn be linked to the OT&E test concept and OT&E plan. | R | User | ||
| A7.2.1. | Changes in the CBRD, system threat assessment document, AoA, logistics support concepts (LSC), and TEMP must be analyzed for potential impacts on Air Force concepts, which in turn affect T&E plans. | R | User | ||
| A7.2.2. | Changes in the Air Force concepts must be finalized and open issues resolved early enough to ensure no adverse impacts on the successful completion of DT&E, Integrated Testing, Cybersecurity T&E, and dedicated OT&E. | R | User | ||
| A7.3. | Air Force concepts must be available to support development of operationally relevant DT&E and OT&E scenarios. | R | User | ||
| Attachment 8 | |||||
| LIFE CYCLE SUSTAINMENT PLAN (LCSP) | USAF | Responsible Party | Comments | ||
| A8.1. | The LCSP must describe the optimal system maintenance strategies, concepts, and methods based on the CBRD’s requirements. | R | User | ||
| A8.1.1. | The system must use an acceptable inter-Service, organic, and/or contractor mix. | R | PM | ||
| A8.1.2. | The LCSP must identify potential high-risk and problem areas (such as long lead items, TOs, system reliability, support equipment, training). | R | User | ||
| A8.2. | Logistics and readiness criteria, thresholds, objectives, and definitions in the CBRD must accurately flow down (be linked) to the LCSP, which must in turn be linked to the MOEs and MOPs in the OT&E concept and plan. | R | User | ||
| A8.3. | LCSP strategies and plans must be sufficiently detailed to support early development of the OT&E concept and OT&E plan. | R | User | ||
| A8.4. | Realistic operational and suitability test scenarios that support the integrated test plan must be developed from the LCSP and other Air Force concepts. | R | OTO | ||
| A8.5. | The system must be supportable in dedicated OT&E using the LCSP's strategies and plans. | R | PM | ||
| A8.6. | The system's design must successfully address the quantitative and qualitative constraints identified in the LCSP. | R | PM | ||
| A8.7. | The Doctrine, organization, training, materiel, leadership and education, personnel, and facilities (DOTMLPF) elements must be sufficient to support the LCSP and maintenance plan during dedicated OT&E. | R | OTO | ||
| A8.8. | The Depot Source of Repair (DSOR) decision has determined the optimal maintenance posturing decisions needed to support warfighter operational requirements. | R | PM | ||
| A8.9. | Reliability and maintainability (R&M) growth plans are developed, coordinated, and documented in the Systems Engineering Plan (SEP) and TEMP. | R | PM | ||
| A8.10. | The LCSP integrates the acquisition and product support strategies throughout the system’s life cycle. The LCSP must support Milestone B and follow-on decisions. Note: Space systems are exempt from this requirement. | R | PM | ||
| Attachment 9 | |||||
| INFORMATION TECHNOLOGY (IT) AND NATIONAL SECURITY SYSTEMS (NSS) | USAF | Responsible Party | Comments | ||
| A9.1. | The NR-KPP defined in the CBRD, consists of testable characteristics, and contains performance measures required for the timely, accurate, and complete exchange and use of information. | R | User | ||
| A9.1.1. | Architecture products (e.g., OV-5, OV-6, and SV-1 to SV-7) are developed and available to the test community. | R | User | ||
| A9.1.2. | Key interface profiles are identified and complied with as applicable. | R | User | ||
| A9.1.3. | Cybersecurity capabilities are planned and designed into system specifications and configurations using the latest threat estimates. | R | PM | ||
| A9.1.3.1. | An AO and Information System Security Manager (ISSM) are formally assigned in writing. | R | PM | ||
| A9.1.3.2. | Impact values for Confidentiality, Integrity and Availability as well as a listing of the security overlays are in requirements documents and the TEMP. | R | User | ||
| A9.1.3.3. | The cybersecurity strategy, as an appendix to the Program Protection Plan (PPP), is complete and available to the T&E community. | R | PM | ||
| A9.1.3.4. | RMF is implemented and the T&E community invited to observe and participate in process activities. | R | PM, LDTO | ||
| A9.1.4. | The High Performance Team’s (HPT) architecture expert has ensured compliance with the DoD Information Enterprise Architecture, Version 1.1 and provided a compliance statement to the program office. | R | User | ||
| A9.2. | The Information Support Plan (ISP), Security Classification Guide (SCG), and all RMF –related documents (including a compiled list of system characteristics or qualities required for system registration, key security-related documents such as a risk assessment, privacy impact assessment, system interconnection agreements, contingency plan, security configurations, configuration management plan, and incident response plan) are complete and available to the T&E community as early as possible. | R | PM | ||
| A9.2.1. | ISP, SCG, RMF documentation, and PPP are consistent with the TEMP, strategy for T&E, and support T&E execution activities. | R | PM | ||
| A9.3. | System cybersecurity training for AOs, ISSMs, security systems administrators, and users is available and completed. | R | PM | ||
| A9.3.1. | Trained teams are used to conduct passive and active scans to reveal system/network vulnerabilities, verify system protection and detection capabilities, and complete a NetRA or equivalent as outlined in the TEMP and ISP. | R | LDTO | ||
| A9.3.2. | Trained and certified teams are used as opposition forces to conduct penetration testing for assessing the cybersecurity posture of the system/network as part of a vulnerability analysis assessment or equivalent as outlined in the TEMP and ISP. | R | LDTO | ||
| A9.4. | NetRA and other interoperability or net-ready certification activities are complete. | R | PM, LDTO, JITC | ||
| A9.4.1. | All developer/test passwords, password scripts, and accounts in use during system development are deleted prior to operational testing. | R | PM | ||
| A9.4.2. | Compliance with cybersecurity vulnerability alerts will not impact any other type of system certification or potentially invalidate test data. | R | PM, LDTO | ||
| A9.4.3. | Data passed to and from other interoperable systems must be compatible. | R | PM | ||
| A9.4.4. | JITC has provided an OTRR Interoperability Statement as required. | R | PM | ||
| A9.4.5. | The AO has obtained an Interim Authority to Test (IATT) or Authorization to Operate (ATO) memo (as appropriate) prior to test efforts. | R | PM | ||
| A9.5. | Systems and subsystems comply with the USAF Electromagnetic Compatibility Program and Radio Frequency Spectrum Management guidelines. | R | PM | ||
| A9.6. | Other systems and subsystems required to interoperate with the test articles (including external systems) are available. | R | PM, OTO | ||
| Attachment 10 | |||||
| TEMP | USAF | Responsible Party | Comments | ||
| A10.1. | The TEMP must be updated, coordinated, and approved at appropriate levels prior to each milestone and after major program changes. | R | PM | ||
| A10.1.1. | Open issues must be addressed and resolved before submission to HQ USAF. Changes required by OSD or other decision authorities must be incorporated as agreed. | R | PM | ||
| A10.1.2. | Coordination must be timely and efficiently planned to minimize chances of late rejection and negative impacts on dedicated OT&E. | R | PM | ||
| A10.2. | Level of detail must be appropriate for the stage of development, and “TBDs” eliminated as much as possible. MOEs, MOSs, CTPs, COIs, and Decision Support Questions (DSC) are included in the Developmental Evaluation Framework (DEF) Matrix. | R | PM | ||
| A10.3. | The TEMP must accurately reflect the most recent CBRD, system threat assessment documents, LSC, Air Force concepts, and AoA. | R | PM | ||
| A10.4. | The TEMP must clearly summarize relationships between: 1) the strategy for T&E, program schedule, and required resources; 2) CBRD parameters, COIs, CTPs, MOEs, MOSs, and 3) DEF, KPPs, CTPs, KSAs, Failure Modes, Effects and Criticality Analysis (FMECA), interoperability requirements, cyber-security requirements, reliability growth, maintainability attributes and developmental test objectives, other evaluation criteria, and decisions supported. | R | PM, OTO | ||
| A10.4.1. | The OT&E concept and plan must be executable in terms of structure, schedule, and resources. | R | PM | ||
| A10.4.2. | The requirements strategy (as reflected in the ICD and draft CDD) and acquisition strategy are fully supported (manpower, funding, test infrastructure, articles including M&S, and agencies). | R | PM | ||
| A10.4.3. | T&E test resource shortfalls or limitations potentially impacting dedicated OT&E must be identified. | R | PM | ||
| A10.4.4. | Describe the M&S assets needed for dedicated OT&E. | R | PM | ||
| A10.4.5. | Ensure the VV&A process and agency responsibilities are described for each M&S capability, to include expected products and approvals. | R | PM, OTO | ||
| A10.4.6. | If LFT&E is required, include the LFT&E strategy in the TEMP. | R | PM | ||
| A10.4.7. | Appropriate cyber test measures included to evaluate operational capability to protect, detect, react, and restore to sustain continuity of operation. | R | PM | ||
| A10.4.8. | Ensure Cost Capability Analysis has been completed. | R | PM | ||
| A10.5. | The TEMP must describe what DT&E, OT&E, or integrated test has done or will do to ensure the system has the potential to meet operational requirements in dedicated OT&E, including assessing schedule and product risks with requisite margins, and assessing mitigation plans of above. | R | PM | ||
| A10.5.1. | Show how all COIs and MOEs and MOSs will be addressed in dedicated OT&E. | R | OTO | ||
| A10.5.2. | Contractor-conducted vs Government-conducted DT and OT are clearly distinguished and mutually supportive. | R | ITT | ||
| A10.6. | Rationale and provision must be made for any planned OT&E deferred beyond dedicated OT&E into Follow-On Test and Evaluation (FOT&E) or follow-on increments. | R | PM | ||
| A10.7. | Links to required detailed information cited in the TEMP must be functional and the linked information complete. Sufficient detail is available for: | R | PM | ||
| A10.7.1. | Reliability growth curves and planning. | R | PM | ||
| A10.7.2. | STAT calculations and analyses. | R | PM, OTA | ||
| A10.7.3. | Allocation of reliability among key components. | R | PM | ||
| A10.7.4. | Anticipated development and test problem areas. | R | PM, LDTO | ||
| A10.7.5. | Resolution of past deficiencies. | R | PM | ||
| A10.7.6. | Cyber T&E strategy and resources and includes specified cyber content: architecture, operational environment, evaluation structure, ATO, time and resources, cooperative vulnerability and penetration assessment, and adversarial assessment. | R | LDTO, OTO | ||
| A10.7.6.1. | Cyber test (cooperative vulnerability and penetration assessment, and adversarial assessment) events for DT&E, OT&E, and Integrated Test. | R | LDTO, OTO | ||
| A10.7.6.2. | RMF planning is described. | R | PM | ||
| A10.8. | The requirements strategy (as reflected in the ICD and draft CDD) and acquisition strategy are fully supported (manpower, funding, test infrastructure, articles including M&S, and agencies). | R | PM | ||
| A10.9. | Ensure the TEMP includes applicable test scenarios, appropriate data collection (established T&E database), and performance evaluation over the life cycle of the system. | R | PM | ||
| Attachment 11 | |||||
| INTEGRATED TEST PLANNING | USAF | Responsible Party | Comments | ||
| A11.1. | Ensure integrated test planning starts as early as practical to make T&E schedules and resource expenditures more efficient and eliminate duplication of effort. | R | PM | ||
| A11.1.1. | A rigorous SEP identifies how T&E will be used to achieve program goals and technical results. | R | PM | ||
| A11.1.2. | A rigorous TEMP specifies how T&E will be planned and used to verify and validate program requirements are met to ensure the system is operationally effective and suitable. | R | PM | ||
| A11.1.3. | DT&E and OT&E plans and concepts are structured so that OT can capture and apply DT&E data to reduce OT&E timelines and requirements. | R | ITT | ||
| A11.1.4. | OAs are planned at strategic points in the development program. OAs and early user inputs influence system design and function. | R | PM, OTO | ||
| A11.1.5. | Other types of T&E (e.g., cybersecurity, LFT&E, contractor) are incorporated as much as practical in the integrated test design. | R | PM | ||
| A11.1.6 | Dedicated operational test and developmental test objectives are not compromised. | R | ITT | ||
| A11.1.7. | STAT process employed to ensure T&E is effective, efficient, and appropriate factors and conditions selected to produce the data required to characterize system capabilities. | R | PM | ||
| A11.1.8. | Ensure the Integrated Test Concept (ITC) and TEMP reflect the most current program direction. | R | PM | ||
| A11.2. | Definitions, formulas, and evaluation criteria used to determine operational effectiveness and suitability must be consistent between all individual test plans and T&E documents. | R | ITT | ||
| A11.3. | A common T&E database is used to archive all T&E data from all test organizations. | R | PM | ||
| A11.3.1. | Parameters and formats are agreed upon between all test teams. | R | ITT | ||
| A11.3.2. | Test item configurations are rigorously controlled. | R | PM | ||
| A11.4. | Integrated test matrices are addressed in the TEMP and depicts all T&E events and who will accomplish them. | R | ITT | ||
| A11.4.1. | Duplication and voids in testing are minimized. | R | ITT | ||
| A11.4.2. | A prudent number of backup resources (e.g., test assets, funds) are available to supplement all testing if planned integrated DT&E/OT&E data is unusable or unavailable | R | PM | ||
| ATTACHMENT 12 | |||||
| CYBER RESILIENCY | USAF | Responsible Party | Comments | ||
| A12.1. | Cyber resiliency goes beyond “cybersecurity” to include cyber detection and response. Ensure cybersecurity phases shown in Figure 2.3 are reviewed to ensure currency of the strategies for operational requirements, acquisition, T&E, and cybersecurity. Use Figure 2.3 for the rest of this template. | R | ITT | ||
| A12.1.1. | System's Cybersecurity Strategy, Security Plan (SP), SCG, and PPP are current. | R | PM | ||
| A12.1.2. | Cyber resiliency assessments are integrated into DT&E and OT&E. | R | ITT | ||
| A12.1.2.1. | OT plan addresses required DOT&E cybersecurity content: TEMP linkage, architecture, intelligence community-validated cyber threat, operational environment, evaluation structure, time and resources, cooperative vulnerability and penetration assessment, and adversarial assessment. Plan should also address cybersecurity software assurance considerations. | R | ITT | ||
| A12.1.3. | Six-step RMF process (1. Categorize System, 2. Select Security Controls, 3. Implement Security Controls, 4. Assess Security Controls, 5. Authorize System, 6. Monitor Security Controls) is followed. Establish an ITT sub-group to monitor and control, if necessary. | R | PM | ||
| A12.1.4. | Confidentiality, Integrity, and Availability ratings as well as a listing of security overlays are properly assigned. | R | PM | ||
| A12.1.5. | Applicable overlays are applied so that appropriate security controls are selected and updated. | R | PM | ||
| A12.1.6. | Cyber-attack surfaces, threats, etc. are properly characterized and updated. | R | PM | ||
| A12.1.7. | Cyber kill chain is correctly understood, analyzed, and updated. | R | PM | ||
| A12.1.8. | System's Functional Hazard Analysis is current for cooperative vulnerability, penetration assessment and adversarial team baseline, and determining safety and real-world operations considerations. | R | PM | ||
| A12.2. | Cyber test infrastructure (with appropriate architecture, level of realism, and security) and documentation is available and described in the TEMP. | R | PM | ||
| A12.2.1. | System owners agree on rules of engagement for all teams. | R | PM | ||
| A12.2.2. | Reciprocity agreements are in place between teams and other Services. | R | PM | ||
| A12.2.3. | Test plans with refined cyber T&E scenarios, operational capability requirements, potential test venues, mission threads, and simulated scenarios are developed and approved. | R | ITT | ||
| A12.2.4. | Funding is available to complete cooperative vulnerability, penetration assessment, and adversarial assessment test events. | R | PM | ||
| A12.2.5. | The IATT and ATO are available at the appropriate times. | R | PM | ||
| A12.2.6. | SAR is prepared, recommended corrective actions and system weaknesses are addressed and prepared for. | R | PM | ||
| A12.2.7. | All cyber testing planned to be conducted on a cyber range is identified and all events integrated with OT&E and assessment activities. | R | PM | ||
| A12.2.8. | All necessary linkages between the cyber range and operational networks are developed. | R | PM | ||
| A12.2.9. | Integration plan established for system operators, network defenders, and threat emulations on planned Cyber Range if conducting cyber range testing. | R | PM | ||
| A12.3. | Cooperative vulnerability, penetration assessment and adversarial teams are available and scheduled. | R | PM | ||
| A12.3.1. | Testability of cyberspace requirements are determined and additional clarified. | R | PM, OTO | ||
| A12.3.2. | Applicability of network defender participation in adversarial assessment team OT&E events is determined. | R | OTO | ||
| A12.3.3. | Limitations of generating operational effects during cybersecurity adversarial assessment OT&E events due to safety and real-world operations considerations are identified and documented. | R | OTO | ||
| A12.3.4. | Quantitative cyber resiliency factors, descriptors and tailored measures are identified. | R | OTO | ||
| A12.3.5. | The threat basis on which vulnerability/penetration testing scenarios will be built is identified and documented in the test concept and scenarios. | R | OTO | ||
| A12.3.6 | The Test Resource Plan (TRP) includes updated resources and costs associated with cooperative vulnerability, penetration assessment and adversarial test events. | R | OTO, PM | ||
| A12.4. | Identify security constraints and their impacts on dedicated OT&E. | R | PM, LDTO | ||
| A12.4.1. | Receipt of permissions and rules of engagement before DT&E cooperative vulnerability, penetration assessment and adversarial events. | R | PM, LDTO | ||
| A12.4.2. | Cyber anti-tamper testing is integrated into DT&E and OT&E to the extent warranted and permissible. | R | PM, LDTO | ||
| A12.5. | System OPSEC plan is current. | R | PM | ||
| A12.6. | If NSA certification is required for classified/controlled cryptographic items, the program’s security verification test approach must be included in the TEMP. | R | PM, ITT | ||
| ATTACHMENT 13 | |||||
| CONTRACTOR TESTING | USAF | Responsible Party | Comments | ||
| A13.1. | Ensure all system specifications and contractor requirements support the latest CBRD. | R | PM | ||
| A13.2. | Ensure comprehensive contractor test plans for development, qualification, and production acceptance testing are in place. | R | C | ||
| A13.2.1. | Requirements and specifications must flow down accurately and clearly from prime contractors to subcontractors. | R | C | ||
| A13.2.2. | Contractor test strategies and methods must determine if all aspects of the specification and the CBRD can be met. | R | PM, LDTO | ||
| A13.2.3. | Test events should be performed with operationally relevant components/elements and under operationally relevant conditions and scenarios as much as possible with exceptions agreed to by all stakeholders and/or limitations cited. | R | C | ||
| A13.2.4. | Sub-system and system pass/fail specification thresholds must be directly traceable to the most current CBRD. | R | PM | ||
| A13.2.5. | A realistic, attainable, event-driven test schedule must be proposed and funded. | R | C | ||
| A13.2.6. | Known risks are reasonably and appropriately managed. | R | C | ||
| A13.2.7. | All contractor test data must be available in the system’s common T&E data base | R | PM | ||
| A13.2.8. | Ensure contractor testing is included in the ITC and described in the TEMP. | R | PM | ||
| A13.2.9 | Ensure the contractor is capable to plan and conduct special and formal multi-segment and system of system testing, including test support, handling, calibration, and transportation. | R | PM | ||
| A13.3. | Contractor testing must demonstrate the system and/or components are meeting the CTPs at prescribed threshold levels and within defined time frames at each step in development. | R | C | ||
| A13.3.1. | Government systems engineering analysis should determine if test results support achievement of the spec and if the system is projected to meet operational requirements. | R | LDTO | ||
| A13.3.2. | Fault tree analysis must be performed on the operational system and its external and internal interfaces to identify potential operational contributors to mission failure. | R | C | ||
| A13.4. | Available government facilities are used in contractor testing wherever cost-effective, available, and feasible. | R | PM | ||
| A13.5. | A deficiency resolution system must be in place and accessible to all test organizations to identify, track, and resolve test failures. | R | C | ||
| A13.5.1 | The contractor’s DR process must be compatible with the Government’s DR process. | R | PM | ||
| A13.5.2. | All test failures and resultant system design changes must be documented and analyzed for effectiveness. Tests must be repeated as necessary to certify specification compliance. | R | C | ||
| A13.5.3 | Document all changes to specification threshold (pass/fail) values and rationale. | R | PM | ||
| A13.6. | Contractor T&E data and information must be available in the required formats for Government review for impacts on DT&E and dedicated OT&E. | R | C | ||
| A13.7. | Planned contractor testing must be completed according to the contract before government acceptance and dedicated OT&E. | R | C | ||
| A13.7.1. | Contractor testing deferred beyond government acceptance of the system is documented for final certification of system readiness. | R | PM | ||
| ATTACHMENT 14 | |||||
| GOVT DEVELOPMENTAL TEST AND EVALUATION (DT&E) | USAF | Responsible Party | Comments | ||
| A14.1. | CBRD requirements must be accurately reflected in Government DT&E plans and be demonstrated during contractor and government DT&E. | R | PM | ||
| A14.2. | When design-cost-performance trade-offs are made that may impact CBRD requirements, user concurrence must be obtained and documented where appropriate. | R | PM | ||
| A14.3. | The DT&E schedule and testing must be planned and executed to allow sufficient time to certify system OT&E readiness, start and complete dedicated OT&E before FRP or fielding. | R | PM | ||
| A14.3.1. | DT&E, with inputs from LDTO, must validate contractor testing is complete, or a plan exists to finish testing. | R | PM | ||
| A14.3.2. | Sufficient suitability testing must be conducted to permit credible predictions about system Reliability, Maintainability, and Availability (RM&A). | R | LDTO | ||
| A14.3.3. | All CTPs must demonstrate satisfactory performance, or be supported by reliability growth plans and/or curves that show threshold attainment. | R | PM | ||
| A14.4. | A government-run DR system must be in place in support of DT&E and OT&E for identifying, tracking, reporting, and resolving DRs. | R | PM | ||
| A14.4.1. | Correction of all CAT I deficiencies including cybersecurity vulnerabilities identified during DT blue team/red team events are implemented before start of dedicated OT&E. | R | PM | ||
| A14.5. | A formal process is in place to control and track system configuration during DT&E that will support dedicated OT&E | R | PM | ||
| A14.5.1. | The system design must be stabilized sufficiently early with no major changes implemented in the OT&E test articles. | R | PM | ||
| A14.6. | Sufficient operationally relevant DT&E must be accomplished, culminating in a "dress rehearsal" in the final phase, to determine if CBRD requirements can be met before dedicated OT&E. | R | LDTO | ||
| A14.6.1. | Cooperative vulnerability and penetration assessment, and adversarial assessment tests of cyber resiliency are complete. | R | PM, OTO | ||
| A14.6.2. | Sufficient testing must be accomplished with other systems to support end-to-end cybersecurity and interoperability certifications. | R | PM | ||
| A14.6.3. | Required levels of performance must be demonstrated in the intended operational environment based on the CBRD, Air Force concepts, strategies, and plans. | R | PM | ||
| A14.6.4. | Sufficient workarounds acceptable to the OTO are identified for CAT II vulnerabilities deficiencies. | R | PM | ||
| A14.6.5. | If there are interoperability requirements, DT&E must take place at the system-of-systems level. | R | PM | ||
| A14.7. | LFT&E results (if required) must be available before start of dedicated OT&E. | R | PM | ||
| A14.8. | Formal certifications may be required from the following sources (among others). Ensure clearances and certifications are available for use in dedicated OT&E. | R | PM | ||
| A14.8.1. | Non-nuclear Munitions Safety Board | R | PM | ||
| A14.8.2 | Directed Energy Weapons Safety Board | R | PM | ||
| A14.8.3 | Flight Safety Board | R | PM | ||
| A14.8.4 | Airworthiness, Spaceflight Worthiness | R | PM | ||
| A14.8.5 | Range Safety | R | PM | ||
| A14.8.6 | Nuclear Weapons Center | R | PM | ||
| A14.8.7 | Institutional Review Board for Protection of Human Subjects in Testing | R | PM | ||
| A14.8.8 | SEEK EAGLE certification completed for threshold systems as a minimum | R | PM | ||
| A14.8.9 | AF Spectrum Management Office | R | PM | ||
| A14.8.10 | Authorization to Operate (ATO) | R | PM | ||
| A14.9. | For integrated testing, minimize duplication and voids in testing and the excessive use of facilities. | R | OTO | ||
| A14.9.1. | DT&E data formats and parameters are compatible with other tests to maximize data availability in the common database and usability for OT&E. | R | OTO | ||
| A14.10. | An agreed-upon plan and rationale must exist (e.g., in the TEMP) for testing any areas or capabilities deferred past the start of dedicated OT&E. | R | PM | ||
| A14.10.1 | If there are any incomplete test areas, explain why and give impacts on dedicated OT&E with inputs from the OTO. | R | LDTO | ||
| A14.11. | Ensure sufficient interim DT&E results and evaluations are available to support certification of readiness for operational testing. | R | LDTO | ||
| ATTACHMENT 15 | |||||
| SOFTWARE DEVELOPMENT AND MATURITY | USAF | Responsible Party | Comments | ||
| A15.1. | System software functionality, performance, and maturity must be assessed throughout the systems engineering technical reviews from SRR through OTRR and developmentally tested at the full system level (suitable for that increment) prior to starting dedicated OT&E. | R | PM | ||
| A15.2. | Define software-related exit criteria for MS B. These criteria may be modified and/or criteria added/deleted in response to CBRD changes during system development. | R | PM | ||
| A15.3. | Develop and implement a "requirements traceability" metric to measure adherence of software products (to include architecture, design, and code) to the CBRD. | R | PM | ||
| A15.4. | Operational databases are complete and sufficient for operational test and contain actual operational data. | R | PM | ||
| A15.5. | System level integration testing of software and hardware-software-firmware interfaces must be monitored, documented, and completed. | R | PM | ||
| A15.6. | Effective software configuration management and control procedures are in place. | R | PM | ||
| A15.7. | Software manuals and documentation must be validated and up-to-date with the current software baseline in support of dedicated OT&E. | R | PM | ||
| A15.8. | Software and firmware configurations must be fully documented and frozen before starting dedicated OT&E. Changes must not be implemented during dedicated OT&E that would impact the configuration being fielded or produced. | R | PM | ||
| A15.8.1. | Incrementally deployed software releases address specific capabilities and testable performance requirements and must be assessed ready for test. Each release must undergo dedicated OT&E. Software builds or increments that are not deployed individually (release) must still support full deployment system OT&E. | R | PM | ||
| A15.9. | The software must be stable (i.e., operate error free for a reasonable length of time prior to dedicated OT&E). | R | PM | ||
| A15.10. | Facilities, tools, and manpower must be sufficiently representative to support the OT&E plan and schedule, and fielding of the software. | R | PM | ||
| A15.11. | Required Software Assurance (SwA) Defense Information Systems Agency (DISA) Security Technical Implementation Guide (STIG) and CNSSI 1253 controls and protection mechanisms are identified, implemented, and tested to prevent system compromise, maintain integrity and availability, and prevent unauthorized access to systems and data. | R | PM | ||
| A15.11.1. | Require the use of automated vulnerability analysis tools & techniques throughout the lifecycle. Determine appropriate remediation strategies for all identified SwA vulnerabilities. | R | PM | ||
| A15.12. | For critical software, employ independent SwA Verification & Validation (V&V) organizations through DT. | R | PM | ||
| A15.13. | Known software and firmware vulnerabilities, exploitability levels, and discrepancies affecting system performance or the dedicated OT&E must be properly documented and appropriate corrective action(s) taken. | R | PM | ||
| A15.13.1 | The software must be analyzed for safety critical functions and determined acceptable for operational use. | R | PM | ||
| A15.14. | Sufficient regression testing must be accomplished at the unit, integration, and system-of-systems level to ensure changes do not introduce operationally critical faults and/or result in additional defects. | R | PM | ||
| Attachment 16 | |||||
| LIVE FIRE TEST AND EVALUATION (LFT&E) | USAF | Responsible Party | Comments | ||
| A16.1. | Review the most current threats and operational scenarios in the CBRD, threat documents, Air Force concepts, and AoA to assess whether or not a "covered system”. | R | PM | ||
| A16.1.1. | Consult AF/TEP, users, and OSD/DOT&E (in that order) for concurrence with the determination of covered system status. | R | PM | ||
| A16.2. | If the system is a covered system, determine LFT&E scope and complete a cost-benefit analysis. | R | PM | ||
| A16.3. | If full-up LFT&E is determined to be cost-effective and practical, develop an LFT&E strategy, to include the level of funding, and submit to OSD/DOT&E for approval | R | PM | ||
| A16.3.1. | Describe the LFT&E strategy in the TEMP and submit individual plans for “full-up system level” LFT&E to OSD/DOT&E for approval. | R | PM | ||
| A16.3.2. | Fully integrate the LFT&E strategy and plans into the overall strategy for T&E, TEMP, and integrated test plans. | R | PM | ||
| A16.3.3. | Plan for and fund LFT&E to be completed before start of dedicated OT&E | R | PM | ||
| A16.4. | If full-up LFT&E is determined not to be cost-effective and practical, prepare an LFT&E waiver request and an alternate LFT&E plan for the decision review authority or PEO and OSD/DOT&E approval before MS B. | R | PM | ||
| A16.4.1. | Describe the alternate vulnerability/lethality strategy in the TEMP. | R | PM | ||
| A16.4.2. | Plan for and fund “alternate” LFT&E to be completed before start of dedicated OT&E. | R | PM | ||
| A16.5. | Deficiencies identified during LFT&E that are to be corrected must be tracked and retested prior to certification for dedicated OT&E. | R | PM | ||
| A16.6. | Fully comply with all system-specific congressional direction regarding LFT&E. | R | PM | ||
| A16.7. | With regard to threat systems for LFT&E: | R | |||
| A16.7.1. | Threat "shot doctrine" and employment tactics must reflect the contents in the CBRD, Air Force concepts, and threat documents. | R | PM | ||
| A16.7.2. | Threat systems and threat models are VV&A’ed before use in LFT&E. | R | PM | ||
| A16.7.3. | Identify limitations in the test threats and voids in covering the threat spectrum. Describe proposed fixes. | R | PM | ||
| A16.7.4. | Where limitations exist in test threat systems, obtain approval to fill gaps with M&S and alternative systems. | R | PM | ||
| A16.8. | Develop a data reduction and common database for using all validated threat test data throughout the integrated test plan. | R | PM | ||
| ATTACHMENT 17 | |||||
| MODELING AND SIMULATION (M&S) | USAF | Responsible Party | Comments | ||
| A17.1. | Ensure M&S requirements are identified in CBRDs to obtain funding and support for their development or reuse | R | User | ||
| A17.2. | Develop a Modeling and Simulation Support Plan (MSSP) that links M&S requirements to the capabilities being developed and tested throughout the program (from the AoA through the MS C decision). The MSSP can be part of existing program, engineering or technical plans. | R | PM | ||
| A17.2.1. | Identify as early as possible the M&S support requirements, to include funding, over the entire system life cycle. | R | PM | ||
| A17.2.2. | The MSSP must address continuing ownership and maintenance of M&S assets after system fielding. | R | PM | ||
| A17.2.3. | Identify M&S linkages with planned interfacing and interoperable systems | R | PM | ||
| A17.2.4. | Check for archived M&S tools (e.g., with Air Force Modeling and Simulation Resource Repository (AFMSRR)) before building new M&S resources. | R | PM | ||
| A17.2.5. | Ensure programs obtain data and models for M&S from the required authoritative sources when available and feasible. | ||||
| A17.3. | Ensure M&S assets, test tools, and analysis tools will be available and usable for T&E as required. Testers must receive adequate training as required. | R | PM | ||
| A17.4. | Ensure M&S V&V plan and comprehensive schedule supports the integrated test plan and the dedicated OT&E plan and schedule. | R | PM | ||
| A17.4.1. | Scenarios, test tools, and analysis tools required for DT&E must be adequately documented. | R | PM | ||
| A17.4.2. | The design engineering data must be reviewed. Physics models can be V&V'd, whereas operations analyses are subjectively V&V'd. Empirical test data should be used to establish model credibility. | R | PM | ||
| A17.4.3. | Any M&S used to support dedicated OT&E must be accredited. | R | OTO | ||
| A17.5. | If M&S will generate results used to support a fielding and/or FRP decision on an OSD T&E Oversight program, OSD/DOT&E must approve its use in dedicated OT&E. | R | OTO | ||
| ATTACHMENT 18 | |||||
| CONFIGURATION MANAGEMANT PLAN (CMP) | USAF | Responsible Party | Comments | ||
| A18.1. | Configuration management must be a key element of a rigorous systems engineering process. | R | PM | ||
| A18.1.1. | The systems engineering process must be used for all system components and support items (e.g., hardware, software, support equipment, spares, Government Furnished Equipment (GFE)). | R | PM | ||
| A18.2. | A configuration control mechanism must be used to ensure the orderly transition from one decision review to the next, and from development to production. | R | PM | ||
| A18.2.1. | The Government must have sufficient control or oversight over the configuration to ensure changes do not invalidate the results of dedicated OT&E | R | PM | ||
| A18.2.2. | The exact system configuration must be traceable throughout the program. | R | PM | ||
| A18.3. | If known deficiencies remain in test articles before start of dedicated OT&E, the SEP must describe strategies for managing the following areas: | R | PM | ||
| A18.3.1. | System form, fit, and function must not be adversely affected as a result of each deficiency correction. | R | PM | ||
| A18.3.2. | The impacts of fixing before versus after dedicated OT&E must be assessed. | R | PM | ||
| A18.3.3. | All changes are documented and under configuration control. | R | PM | ||
| A18.4. | The system configuration and configuration of interfacing systems must be stable and production representative before the start of dedicated OT&E. | R | PM | ||
| ATTACHMENT 19 | |||||
| DEFICIENCY IDENTIFICATION AND RESOLUTION PROCESS | USAF | Responsible Party | Comments | ||
| A19.1. | A contractor-operated DR process, if established, will augment the Joint Deficiency Reporting System (JDRS) process. | R | C | ||
| A19.2. | JDRS incorporated and open to all stakeholders for promptly identifying, reporting, tracking, and resolving system deficiencies. | R | PM | ||
| A19.3. | A MIPRB must ensure resolution of all DRs and list the impacts to dedicated OT&E. | R | PM | ||
| A19.4. | A Deficiency Review Board (DRB) will periodically review, validate, and prioritize all open DRs. | R | PM | ||
| A19.4.1. | DRs should be rank-ordered, and the most critical worked first or as agreed by the user(s), operational tester, and LDTO. | R | PM | ||
| A19.5. | Open DRs from DT&E must not preclude successful conduct of dedicated operational testing and the achievement of operational requirements. | R | PM | ||
| A19.5.1. | Dedicated operational test results will not be invalidated due to deferred DR resolution. | R | PM | ||
| A19.5.2. | The DR analysis process must be complete and coordinated with users and testers prior to the start of dedicated OT&E. | R | PM | ||
| A19.6. | Known DRs or capabilities deferred beyond the start of dedicated OT&E must be reviewed and prioritized by a T&E DRB and an impact analysis performed. | R | PM | ||
| A19.6.1. | Category I DRs must be fixed and closure verified according to an agreed upon plan. | R | PM | ||
| A19.6.2. | Category II DRs must be fixed and closure verified, or suitable work-arounds provided. | R | PM | ||
| A19.7. | For DRs that cannot be resolved prior to start of dedicated OT&E, a plan exists for testing deferred capabilities and fixes after dedicated OT&E is accomplished. | R | PM | ||
| A19.7.1. | The plan addresses how open DRs are tracked from increment to increment after OT&E is complete. | R | PM | ||
| A19.8. | A Joint Reliability and Maintainability Evaluation Team (JRMET) and a Test Data Scoring Board (TDSB) must be established to review all Reliability, Availability, and Maintainability (RAM) data. | R | ITT | ||
| A19.9. | Plan of Action and Milestones (POA&M) shows how cybersecurity DRs and vulnerabilities will be resolved. | R | PM | ||
| ATTACHMENT 20 | |||||
| PRODUCTION REPRESENTATIVE TEST ARTICLES | USAF | Responsible Party | Comments |
This is the start of the file's text. The full file is on GovTribe.
File details come from the government source that posted it. Updated .