13.__T-6A_ADSB_Out_Draft_TEMP.pdf

T-6A Automatic Dependent Surveillance-Broadcast (ADS-B) Out

Draft Test and Evaluation Master Plan (TEMP) Annex to

Joint Primary Aircraft Training System (JPATS) TEMP

May 01, 2016

Prepared by

JPATS Integrated Test Team (ITT)

Note: This TEMP is solely for the development and installation of the ADS-B (Out) capability in the T-

6A aircraft and Aircrew Training Devices (ATDs). This TEMP is for use in conjunction with the legacy

TEMP that remains in place covering overall program needs.

DISTRIBUTION STATEMENT D - Distribution authorized to the Department of Defense (DoD) and

U.S. DoD Contractors only in accordance with (IAW) AFI 61-204 AFGM2 and comply with all applicable contractual license requirements (proprietary information, administrative and operational use), 29 July 2012. Refer other requests for this document to the JPATS Program Office, AFLCMC/WLZJA, 1970 Monahan Way, WPAFB, OH 45433-7211.

i

This Page Intentionally Left Blank ii

SIGNATURE PAGE

TEST and EVALUATION MASTER PLAN

T-6A ADS-B (Out) Program

ACAT III

SUBMITTED BY:

01 May 16

SEAN POLETE, COMMANDER, USN

JPATS Deputy Program Manager

Training Aircraft Division

Date

CONCURRENCES:

6 May 16

6 May 16

LARRY W. HUDSON, DAF

Deputy Chief of Maintenance

AETC/A4M

iii

TODD D. ST LAURENT, CAPT, USN

Program Executive Officer, Tactical Aircraft Programs Naval

Undergraduate Flight Training Systems

(PMA-273)

DENNIS S. LLOYD, CDR, USN

Commander, Air Test & Evaluation Squadron Two-Zero

(VX-20)

Beverley A. Gable \\signed\\ 10 May 2016 BEVERLEY A. GABLE, Ph.D., GS-15 DAF

Chief, Center Test Authority (CTA)

AFLCMC/AQT

PEO APPROVAL:

KEVIN BUCKLEY, SES

Program Executive Officer for Mobility iv

CONTENTS

1 PART I – INTRODUCTION

1.1 PURPOSE

1.2 MISSION DESCRIPTION

1.3 SYSTEM DESCRIPTION

1.3.1 PROGRAM BACKGROUND

Key Interfaces

Key Capabilities

System Threat Assessment

2 PART II – TEST PROGRAM MANAGEMENT AND SCHEDULE

2.1 Test & EVALUATION (T&E) MANAGEMENT & ORGANIZATIONAL CONSTRUCT

JPATS Test & Evaluation Manager

JPATS Program Manager

JPATS Program Office

Program Manager, Naval Undergraduate Flight Training Systems (PMA-273)

Test and Evaluation Squadron TWO ZERO (VX-20)

Role of the Lead Operational Test Organization

Defense Contract Management Agency (DCMA)

Role of the Contractor

2.2 COMMON T&E DATABASE REQUIREMENTS

2.3 DEFICIENCY REPORT (DR)

2.4 TEMP UPDATES

2.5 INTEGRATED ADS-B (OUT) PROGRAM TEST SCHEDULE

3 PART III – TEST AND EVALUATION STRATEGY

3.1 T&E STRATEGY

Decision Support Key

3.2 DEVELOPMENTAL EVALUATION APPROACH

Developmental Evaluation Framework

Test Methodology

Modeling and Simulation

Test Limitations

3.3 DEVELOPMENTAL TEST APPROACH

v

Mission Oriented Approach

Developmental Test Events & Objectives

3.4 CERTIFICATION FOR OPERATIONAL TEST & EVALUATION

OT&E Entrance Criteria

OT&E Exit Criteria

3.5 OPERATIONAL EVALUATION APPROACH

Early Operational Assessment

Operational Assessment

Sufficiency of Operation Test Review

Operational Utility Evaluation

3.6 LIVE FIRE TEST AND EVALUATION (LFT&E) APPROACH

3.7 OTHER CERTIFICATIONS

Supplemental Type Certificate

Military Type Certificate

Frequency Certification

Cybersecurity Accreditation

3.8 FUTURE TEST & EVALUATION

3.9 RELIABILITY GROWTH

3.10 LOGISTICS

3.11 ATD OPERATIONAL EVALUATION

4 PART IV – RESOURCE SUMMARY

4.1 INTRODUCTION

4.2 TEST RESOURCE SUMMARY

Test Article

Test Sites and Instrumentation

Test Support Equipment

Threat Representation

Test Targets and Expendables

Operational Test Support

Models, Simulations and Test-Beds

Joint Mission Environment

Special Requirements

4.3 FEDERAL, STATE AND LOCAL REGULATIONS

vi

4.4 MANPOWER/PERSONNEL & TRAINING

4.5 TEST FUNDING SUMMARY

5 APPENDIX A - BIBLIOGRAPHY

6 APPENDIX B - ACRONYMS

7 APPENDIX C – POINTS OF CONTACT

Figures:

Figure 1: OVERVIEW OF PLATFORM INFORMATION TECHNOLOGY

Figure 2: INTEGRATED ADS-B (OUT) PROGRAM TEST SCHEDULE

Tables:

Table 1: FLIGHT TEST PERSONNEL

Table 2: DEVELOPMENTAL EVALUATION FRAMEWORK

1 PART I – INTRODUCTION

Purpose

This draft Test and Evaluation Master Plan (TEMP) supports the T-6A Automatic Dependent

Surveillance Broadcast Out [ADS-B (Out)], Acquisition Category (ACAT) III program. A final

TEMP will support Milestone B currently scheduled in June 2017. The Federal Aviation

Administration (FAA) is requiring ADS-B (Out) performance when operating within U.S.

national airspace after January 1, 2020. The purpose of this TEMP is to outline the test strategy, including Developmental Test and Evaluation (DT&E), Integrated Test and Evaluation (IT&E), and Operational Test and Evaluation (OT&E) to support fielding of the ADS-B (Out) system on the T-6A aircraft. ADS-B (Out) is a joint United States Air Force (USAF) and United States

Navy (USN) modification to the T-6A aircraft. This TEMP also covers the definition, design, installation, test, and certification of the ADS-B (Out) installation on the T-6A ATDs.

Mission Description

The USAF Air Education and Training Command (AETC) and the USN Chief of Naval Air

Training (CNATRA) are responsible for providing undergraduate flight training. The T-6A aircraft provides primary flight training and is the first aircraft student pilots, student naval flight officers, and student combat system officers will operate at the training commands. As part of this mission, the aircraft must operate in National Air Space controlled by the FAA. Figure 1 provides an overview picture of the T-6A platform information technology.

FIGURE 1: OVERVIEW OF PLATFORM INFORMATION TECHNOLOGY

System Description

The T-6A ADS-B (Out) system will broadcast short transponder messages from the aircraft at defined intervals containing current flight and aircraft-specific data whether operating on the ground or during flight. Data parameters include aircraft type, location, speed, heading, altitude, and aircraft length and width. The transmissions also include data on sensor type and accuracy used to generate each data point.

The T-6A ADS-B (Out) system design will be a non-integrated solution, which allows for a more cost effective and quicker effort. This non-integrated solution was presented to and approved by

AETC on 8 Jun 15.

Input data will be acquired from the ARINC 429 data bus or by hard wiring between data sources and the ADS-B (Out) controller, as appropriate. A separate, independent Global

Positioning System (GPS) unit and antenna will be installed to satisfy all GPS-sourced data required to assemble the transmitted aircraft position message. A small controller will be installed on each instrument panel for manual data entry, operational display and power control to the system.

The T-6A Ground-Based Training System (GBTS) will be modified to incorporate ADS-B (Out) in all T-6A ATDs. The pilot interface and the simulated message organization will be representative of the installed system, including operational procedures and displays. The installation will also include fault indications and simulated shutdown procedures. All ATD coursework materials, operation, maintenance and operational procedures documents will be generated or modified as required.

1.3.1 Program Background

The Federal Aviation Administration (FAA) has mandated that all aircraft operating in class A, B, C, and certain E airspace after January 1, 2020 have a transponder operating with ADS-B

(Out) capability IAW FAA Advisory Circular (AC) 20-165B, which provides guidance for the installation and airworthiness approval of ADS-B (Out) systems in various aircraft. The new

FAA rule is contained in Title 14 of the Code of Federal Regulations (CFR) part 91, paragraphs

91.225 and 91.227. Title 14 CFR 91.225 is the mandating rule while Title 14 CFR 91.227 contains the implementing rules. AC 90-114 change 1, dated December 21, 2012, facilitates operations using ADS-B (Out) technology in compliance with Title 14.

JPATS Operational Requirements Document III, 005-88-III, 1 April 2000 indicates that any Air

Vehicle shall be able to operate within the operational constraints of USN and USAF installations for undergraduate aviation training with a transponder which does not restrict operations within Federal Airways or the National Airspace System.

The Air Force (AF) Form 1067 modification proposal for USN and USAF T-6As has been coordinated through JPATS, AETC, CNATRA and Air Force Materiel Command. Vice Chair

Secretary of the Air Force signed the Air Force Requirements Oversight Council approval memorandum on 26 Feb 15 for the AF Form 1067.

This program will design, develop, integrate, test, certify, and install the ADS-B (Out) capability into 487 T-6A aircraft and 88 T-6A GBTS ATDs. A technical data package will be required for updating the associated technical data and to support the operations and maintenance of the

ADS-B (Out) solution in the air vehicles and the ATDs. In addition, aircrew and maintenance training will be required to operate and maintain the ADS-B (Out) technical solution.

An FAA Supplemental Type Certificate (STC) will be obtained and maintained by the ADS-B

(Out) Contractor IAW FAA AC 20-165B. The Government will provide the Contractor with an operational T-6A aircraft for use as a test bed. The Contractor will develop, implement, and maintain an airworthiness program for the ADS-B (Out) modification, documented in an

Airworthiness Plan, which successfully awards a Military Type Certificate (MTC) for the modification.

The ADS-B (Out) Contractor will follow the data management strategy as outlined in the

Statement of Work to facilitate updating maintenance/flight manuals, work cards, checklists, procedures and other printed or electronic documents needed to operate and maintain the aircraft and the ADS-B (Out) system in its new configuration.

The Contractor will install the ADS-B (Out) system into each T-6A aircraft at all USAF and

USN operational and test locations. The Contractor will coordinate installation and initial operation and maintenance with the appropriate Air Force and Navy organizations. The

Contractor will provide initial training including troubleshooting, operation and maintenance procedures for the newly installed ADS-B (Out) system.

The Contractor will upgrade all T-6A GBTS Operational Flight Trainers (OFTs), Instrument

Flight Trainers (IFTs), Unit Training Devices (UTDs), and Egress Procedure Trainers (partial upgrade) at Joint Base San Antonio (JBSA)-Randolph, Vance AFB, Columbus AFB, Laughlin

AFB, Sheppard AFB, and the Combat System Officer training unit at Pensacola (AF).

Additionally, the Contractor will upgrade all OFTs and UTDs at Naval Air Station (NAS)

Pensacola. The USN test asset, PT-5, located at NAS Pax River will ferry to NAS Pensacola for

ADS-B (Out) modification. The USAF test asset, PT-4, located at Beechcraft Defense

Corporation in Wichita, KS, will ferry to JBSA-Randolph for ADS-B (Out) modification.

Key Interfaces

The T-6A ADS-B (Out) system must acquire specific data items via hard wiring, ARINC 429 data bus or pilot input. A control head will be mounted on both the forward and aft instrument panels to accommodate pilot data inputs, data display and operational controls such as turning on and off, standby and ALT mode.

Key Capabilities

The ADS-B (Out) capability requires specific information be automatically broadcast in a message defined in Title 14 of the CFR §91.227 and TSO-C166b. Airworthiness compliance will be evaluated based upon the applicable intended function rule (e.g. 14 CFR §§ 23.1301, 25.1301,

27.130 and 29.1301) recognizing that the intended function is to meet the equipment requirements in Title 14 CFR §91.227.

The T-6A ADS-B (Out) program currently has one Key Performance Parameter, which is to satisfy the FAA mandate to add ADS-B (Out) capability onto the aircraft by 01 January 2020.

There are currently no Key System Attributes, Critical Technical Parameters, Critical

Operational Issues (COI), Measures of Effectiveness (MOE) or Measures of Suitability (MOS) defined for the ADS-B (Out) program. However, the Air Education and Training Command

Studies and Analysis Squadron Test & Evaluation (AETC/SAS-TE) test plan may develop COIs, MOEs & MOSs to support their OT&E efforts.

System Threat Assessment

The T-6A is not a combat aircraft; however, cyber threats will be taken into consideration using

DoD Instruction 8500.01, Cybersecurity, which establishes a DoD Cybersecurity program to protect and defend DoD information and information technology. Cybersecurity testing will be further defined during future ADS-B (Out) ITTs.

2 PART II – TEST PROGRAM MANAGEMENT AND

SCHEDULE

T&E Management & Organizational Construct

The JPATS Program Office (PO) has overall program management responsibility for the T-6A

ADS-B (Out) program. The JPATS PO will manage overall program planning, but delegate day-to-day test activities and execution authority to the local level.

JPATS T&E Manager

The JPATS T&E manager is the ADS-B (Out) T&E Working-Level Integrated Product Team

(WIPT) lead who will manage all test processes, including the following activities:

Test Management coordination with JPATS Branch Engineering, PMA 273, AETC/SAS, CNATRA and the Lead Developmental Test Organization (LDTO) as detailed in the

JPATS ITT Charter

Assist the Project Manager by monitoring test program funding

Assist with making arrangements for delivery of Government Furnished Equipment

(GFE) required for use during testing

Assist with making arrangements for access to Government facilities by the development

Contractor as required

JPATS Program Manager

Chair the Operational Test Readiness Review (OTRR)

Certify ADS-B (Out) readiness for OT&E

JPATS Program Office

Provide overall ADS-B (Out) program management, administration and funding

Prepare the TEMP for concurrence and final approval

Participate in all T&E planning and execution

Manage the contract for ADS-B (Out) definition, installation and documentation

Complete the OTRR checklist and communicate across the JPATS PO to assess readiness to enter OT&E

Co-lead the ADS-B (Out) ITTs

Program Manager, Naval Undergraduate Flight Training Systems (PMA-273)

Provide the USN portion of the funding for the T-6A ADS-B (Out) program

Provide coordination with the LDTO for all required T&E support

Provide support to the T&E WIPT on all T&E related aspects of the program

Participate in all System Engineering Technical Reviews

Monitor the Contractor’s design/installation progress

Review all FAA STC and USG MTC certification data, evaluate system specification requirements, and characterize system performance

Co-lead the ADS-B (Out) ITTs

Test and Evaluation Squadron TWO ZERO (VX-20)

VX-20 is the ADS-B (Out) LDTO for JPATS

Provide Government pilots as appropriate throughout flight testing

Co-chair for the Test Readiness Review and Flight Readiness Review

Write and manage Deficiency Reports (DRs)

Provide support for Contractor and Government ground and flight testing in accordance with Naval Air Systems Command Instruction 3960.4C, Change 1 as required

Provide T&E comments and guidance throughout ADS-B (Out) development

Provide a final report at the end of DT&E with-in 90 days from the completion of the last flight test

Chair Deficiency Report Boards (DRBs) for DT&E

Co-lead the ADS-B (Out) ITTs

Lead Operational Test Organization (AETC/SAS-TE)

AETC/SAS-TE is the lead OT&E organization

Conduct OT&E as appropriate (may include Early Operational Assessment [EOA]), Operational Assessment [OA], Sufficiency of Test Review [SOTR], or Operational

Utility Evaluation [OUE])

Develop COIs, MOEs, and MOSs as required

Coordinate with LDTO regarding test strategy and data collection in an effort to integrate

DT & OT as appropriate and practical

Assess if the ADS-B (Out) installation is operationally effective and suitable

Provide support to the T&E WIPT on all OT&E related aspects of the program

Provide a final report within 60 days from the conclusion of OT including a fielding recommendation

Chair Deficiency Report Boards (DRBs) for OT&E

Participate in ITT meetings

Defense Contract Management Agency (DCMA)

Monitor Contractor development testing

Coordinate and conduct Government production acceptance efforts with the Contractor

Monitor component testing, inspection and acceptance testing of the modified aircraft and

ATD systems

Act as the JPATS Program Office on-site agent

Review and comment on the Contractor’s Acceptance Test Procedures for the aircraft modification kits and for the installation of the kits into the T-6A operational fleet

Contractor

Integrate ADS-B (Out) into the T-6A aircraft and T-6A GBTS OFTs, IFTs, UTDs and egress trainers (partial upgrade)

Define, plan and execute activities and schedules required to obtain STC and MTC

Provide contractor test pilot(s) if necessary for CT/DT efforts and coordinate FAA test pilot participation for certification events

Provide any information/data required to conduct Technical Review Boards & Safety

Review Boards during the entire T&E effort

Provide project test reports as specified in the Statement of Work (SOW)

Participate in ITT meetings

Common T&E Database Requirements

The Contractor will set up, structure and format a common T&E database for storing test plans, sharing data as it becomes available during Contractor testing (CT), Developmental testing (DT), and Operational testing (OT), and tracking watch items, trouble reports, and other discrepancies.

The Contractor has overall responsibility for maintaining the common T&E database. This T&E database will be easily accessible to, shared with, and provided to the Government, and be secured from cyber threats. The Government will use this capability to set up file plans, load supporting documents, and add data such as Maintenance/Flight Manuals, checklists, work cards, and performance data as referenced IAW CDRL A012.

Deficiency Report (DR)

Contractor and government deficiencies and watch items will be uploaded into the Joint

Deficiency Reporting System using the format and priority ranking schemes described in TO 00-

35D-54 to provide for a common database across services and contractors. The Contractor and

LDTO will track deficiencies identified during the development, verification and validation, and certification testing from integrating the ADS-B (Out) into the T-6A avionics suite, and provide weekly Executive Summaries, which detail current DR status. Deficiency Review Boards (DRB) will be held as required for tracking DR priority and status (open or closed). DRB will be chaired by the LDTO during DT&E and AETC/SAS-TE during OT&E. The FAA will track DRs affecting STC certification. Category I and II DT&E DRs should be corrected and resolved before starting OT&E, and OT&E DRs should be corrected and resolved before Milestone (MS)

C. Category I DRs will be reported as they occur to the USG T-6A ADS-B (Out) Program

Manager. Some CAT I and CAT II DRs may have acceptable work arounds to permit fielding and correction as soon as possible or practical. Entry into OT and MS C is primarily based on

FAA STC, military MTC, and no open CAT I DRs that pose a safety of flight risk.

TEMP Updates

The ADS-B (Out) TEMP annex will be updated to support Milestone B with real time information as the program progresses and the Contractor and Government test agencies prepare test plans. The ITT will review any changes. There is currently no formal revision planned for a

MS C specific ADS-B (Out) TEMP since this program is ACAT III and a sub-system of the T-

6A aircraft. Updates may occur if all parties agree it is necessary to support future program milestones.

Integrated ADS-B (Out) Program Test Schedule

A copy of the ADS-B (Out) notional execution schedule is shown below in Figure 2. Contractors will prepare their own integrated master execution schedule as part of their proposal, which will be placed on contract at contract award. The Contractor schedule will include a line for the entire test effort including CT, DT, and OT.

FIGURE 2: INTEGRATED ADS-B (OUT) PROGRAM TEST SCHEDULE

Mod Kits

3 PART III – TEST AND EVALUATION STRATEGY

T&E Strategy

The ADS-B (Out) T&E strategy supports the system acquisition strategy as described in paragraph 1.3.1. Data obtained during Contractor and DT&E ground and flight testing will be used by the Contractor to obtain STC and MTC certifications. These certifications need to be in place prior to a MS C decision. Flight testing will focus on ADS-B (Out) capabilities IAW FAA

AC 20-165B and the USG T-6A ADS-B (Out) certification basis.

The ADS-B (Out) system will be tested using an IT&E philosophy. IT&E participants include

Contractor and Government DT&E and OT&E organizations from contract award to the end of certification. IT&E will be used primarily in the beginning of the test program during Contractor certification testing and DT&E. OT&E will use available data to the maximum extent possible to satisfy OT&E requirements. This arrangement will facilitate timely decision making throughout the program, reducing the development time and increasing the time available to field retrofits.

A test bed aircraft will be located at JBSA-Randolph, TX to ensure all aircraft maintenance needs are satisfied. Contractor personnel will have access to the test bed work area during business hours (Day shift: 0630-1430) and make arrangements for later shifts (Swing shift: 1400-

2200; Midnight shift: 2300-0700). The Contractor will complete all security procedures required to obtain access to the facility and the aircraft. The Government will provide the Contractor with access to a temporary workspace in a hangar that has desks and chairs. The Contractor will supply all test equipment, tools, materials and supplies required to define, install and test an operating ADS-B (Out) system on the test bed aircraft. Additional information is contained in the

Base Support Agreement.

Government pilots will be qualified in the T-6A according to FAA/USN/USAF organizational guidelines. Contractor pilots will be qualified IAW DCMA 8210.1(series) guidelines with flight authorization provided by a Government Flight Representative assigned to the contract. Either a

Government or Contractor pilot may act as pilot-in-command during all phases of ADS-B (Out) testing, including Contractor/FAA certification and Developmental testing. Operational testing will only use Government pilots. The Contractor will provide a qualified FAA test pilot when flying for FAA certification. Possible combinations of flight test personnel are listed in Table 1:

Pilot-In-Command 2nd Seat

Contractor Testing/FAA Cert LDTO Contractor or FAA Test Pilot

DT&E LDTO Contractor or LDTO

OT&E LDTO or 19 Air Force 19 Air Force

TABLE 1: FLIGHT TEST PERSONNEL

Decision Support Key

Key test events will support major acquisition decisions including:

Test plan reviews with the Contractor during both the Preliminary Design Review and Critical Design Review

Discuss test roles, responsibilities and resources during ITT meetings to maintain the overall program schedule

Collect test data to support issuing both a STC and MTC

Using DT&E and OT&E final reports to make a fielding decision for purchasing

T-6A production modification kits (i.e., MS C decision)

Developmental Evaluation Approach

VX-20 at Pax River, MD, has been designated the LDTO and will be responsible for

Government insight into the Contractor’s execution of testing, and oversight during the developmental phase of the ADS-B (Out) testing. The LDTO will provide oversight to ensure data compatibility, information sharing, “safe, effective, and efficient test execution”, and efficient testing between all test organizations. The PO is on track to provide an efficient and cost effective test program ensuring an operationally effective ADS-B (Out) system by January

1, 2020. VX-20 will work with the Contractor and JPATS PO to ensure timely delivery of final

DT&E reports to meet program milestones.

DT&E objectives supporting IT&E include verification and validation of design requirements, qualification of ADS-B (Out) component and system elements, and confirmation of the aircraft’s capability to meet all technical performance specifications. DT&E will ensure integration issues are resolved prior to starting OT&E, ADS-B (Out) fulfills mission requirements, and system performance has been evaluated against the specifications on contract.

The STC will be issued by the FAA’s Military Certification Office (MCO), which will verify compliance of the installed ADS-B (Out) system using Title 14 CFR 91.225 and FAA AC 20-

165B. MTC will be obtained by showing compliance with the USG provided T-6A ADS-B (Out) certification basis IAW MIL-HDBK-516C. All test flights will be flown under a Military

Experimental Flight Release (MEFR). An FAA issued Type Inspection Authorization (TIA) will be required prior to flying for FAA certification and issuance of the STC.

FAA testing may include Defense Contract Management Agency (DCMA) personnel and/or

USAF/USN participants witnessing laboratory, ground, and flight-testing. Since all flight testing will involve USAF/USN flight crews they will need to be incorporated into test planning and execution.

Tests conducted to support issuance of the STC will be reviewed by the FAA for both the sufficiency of the testing conducted and the adequacy of the results produced by testing. The

USG will review test results to determine if the data can be used to support MTC issuance.

Developmental Evaluation Framework

The Project Manager, in concert with members of the ITT, will closely monitor the development activities of the Contractor and will provide any needed direction during regularly scheduled

Program Management Reviews and ITT meetings. The Project Manager and representatives from VX-20 and AETC/SAS-TE may be present to witness or participate in Contractor DT&E and Certification Testing. Given this level of involvement with the development process, decisions concerning further DT&E and even OT&E may be made earlier than the norm for these kinds of activities. Table 2 lists functional evaluation areas, decisions supported, and system references, which augment the flight test program and Milestone C.

Functional Evaluation Area Decision Supported System Reference

Cyber-Security Requirements CDR DoDI 8500.01

Test Plans PDR, CDR CDRL 008

Risk Management Plan CDR CDRL A015

Test Readiness Review (TRR) CDR CDRL A014

Flight Readiness Review (FRR) Flight Testing CDRL A014

Safety Assessment Report Flight Testing CDRL A022

Spectrum Certification (Stage 3) Flight Testing DD 1494 Approved

Supplemental Type Cert (STC) Issued MS C FAA AC 20-165B

Military Type Certificate (MTC) Issued MS C MIL-HDBK-516C

TABLE 2: DEVELOPMENTAL EVALUATION FRAMEWORK

Test Methodology

The integrated ADS-B (Out) test effort will address the following methodologies:

Provide data to identify corrective actions

Measure system functionality

Assess system specification compliance

Identify system capabilities, limitations, and deficiencies

Assess system safety

Assess compatibility with T-6A aircraft

Support cybersecurity assessments

Support the FAA STC certification process

Support the USG T-6A ADS-B (Out) certification process

Verify achievement of required technical performance

Assesses the effectiveness and suitability of the ADS-B (Out) technical solution

Modeling and Simulation

No modeling and simulation is currently planned to support ADS-B (Out) testing.

Test Limitations

There are no test limitations anticipated on the ADS-B (Out) program.

Developmental Test Approach

Mission Oriented Approach

Integrated testing will be conducted to the maximum extent possible in real-world operational environments to evaluate the functionality and military utility of ADS-B (Out) on the aircraft. In addition, this approach will be used to evaluate the system performance in mission context when appropriate during DT to possibly influence the design, manage any risk and predict effectiveness and suitability. Any remaining operational testing will be accomplished during dedicated OT&E through AETC/SAS-TE.

Developmental Test Events & Objectives

Developmental testing will be performed in three primary categories: FAA STC, USG MTC, and system specification verification and validation.

FAA STC activities will be performed by the Contractor and validated by the FAA. The ITT will be involved throughout the FAA STC process, as necessary, to gather data in support of MTC and specification verification and validation.

USG MTC activities will be performed by the Contractor to evaluate ADS-B (Out) capability to meet MIL-HDBK-516C requirements.

Verification and validation testing will be conducted to ensure ADS-B (Out) performance complies with the contractual system specifications. Using the integrated test principle, verification and validation test data will be gathered during FAA STC and USG MTC certification, and dedicated specification verification and validation tests.

The Contractor is responsible for delivering the test plan and test procedures IAW the ADS-B contract, including an estimated number of ground and flight hours, required test articles, and list of test events.

The Contractor will use ADS-B (Out) System Requirements Document (SRD) dated 8 Feb 16 as guidance for developing FAA and DT&E test plans IAW CDRL 007, Requirements Traceability

& Verification Matrix, and CDRL 008, Test Plans. First draft of the test plans will be made available for Government review during PDR, and the final draft will be made available for

Government approval during CDR. Final test plans must be approved by the Government prior to starting flight-testing. Contractor participation in ITTs is encouraged for integrated test planning.

The following detailed objectives will be reviewed during DT&E:

Evaluate overall ADS-B (Out) system performance

Collect and verify data and system specification compliance supporting both FAA and military certification requirements and write DT&E reports

Evaluate structural adequacy and compatibility with the T-6A aircraft

Verify ADS-B (Out) has no negative impact on existing T-6A avionics

Evaluate transmitting capabilities and signal accuracy

Evaluate Human Factors for maintainers

Evaluate Human Factors for operators

Conduct an appropriately tailored cybersecurity assessment

Evaluate ADS-B (Out) Reliability, Availability and Maintainability

Evaluate airworthiness and safety of flight for any areas not covered by FAA certification

Evaluate training systems and support (pilots and maintainers)

Identify and mitigate system limitations; validate corrections for DRs identified during contractor or DT testing prior to starting OT&E

Determine if Logistics support is in place

Determine if System Safety parameters are in place

Integrate DT&E with OT&E whenever possible

Assess and demonstrate the system within the intended operationally relevant environment

Prepare for Operational Test Readiness Review

Certification for Operational Test & Evaluation

A tailored version of AFMAN 63-119 will be used to certify ADS-B (Out) readiness for OT&E.

AETC/SAS-TE requires both an Operational Readiness and System Safety release memo from the JPATS PO prior to starting OT&E. One memo will suffice as long as it contains the required information for both efforts. In addition, the following entry and exit criteria will be required for

OT&E.

OT&E Entrance Criteria

ADS-B (Out) system has a mature design and demonstrates stable performance

Ready to commence operational testing with an approved test plan

Required resources supporting OT&E are in place

Technical problems are resolved to reduce the chance of inadequate testing and unnecessary waste of test resources

ADS-B (Out) system is production and operationally representative

All critical issues and deficiency reports identified during contractor test and DT&E have been resolved

System capabilities have been demonstrated in an operationally realistic environment

All required certifications are in place and approved

Pilot and maintainer training have been completed

Updated maintenance manuals are in place

All pertinent contractor and government DT&E data has been provided to the OTA

OT&E Exit Criteria

ADS-B (Out) system has demonstrated acceptable performance

System effectiveness and suitability have been incorporated into the OT&E evaluation

The logistics support system has been established and tested

Adequate data is available to estimate ADS-B (Out) reliability

Operational Evaluation Approach

AETC/SAS-TE will plan and execute OT&E which will be a SOTR or OUE and may include an

EOA, and/or OA. SOTR is a method for reviewing all data which tracks and systematically reviews the design, operation, maintenance, system performance, documentation and development history. The SOTR may be used to evaluate the operational characteristics of relatively small modifications throughout the development cycle to avoid delay in introducing these modifications to the fleet. The OUE is a traditional approach for performing OT&E on small projects. Both methods would provide a fielding recommendation by AETC/SAS-TE.

Early Operational Assessment

The JPATS PO (AFLCMC/WLZ) will determine if an EOA needs to be performed on the ADS-

B (Out) program using inputs from members of the ITT. If conducted, the EOA would begin just after Preliminary Design Review (PDR) and would be a review of all program information to date. AETC/SAS-TE personnel would lead this effort with Contractor, PO and VX-20 personnel support as required. The technical information presented in an EOA would show potential design concept and avionics equipment shortcomings from an Operational perspective. No EOA is currently planned on the T-6A ADS-B (Out) program.

Operational Assessment

An OA would be conducted by AETC/SAS-TE in preparation for dedicated operational testing and MS C support. The OA is designed to be a progress report and not intended to determine overall effectiveness or suitability of the system. The OA provides early operational data and feedback from actual testing to developers, users, and decision makers. The OA also provides a progress report on the system’s readiness for operational testing, including an assessment of

ADS-B (Out) risk.

The JPATS Program Manager and AETC/SAS-TE will determine if a formal and extensive OA needs to be performed using inputs from members of the ITT. The OA would be conducted when an operating production unit is available shortly after Critical Design Review. AETC/SAS-TE personnel will have the opportunity to evaluate the production unit, with “hands on” access to the system cockpit displays and controls. Technical data concerning the system operation will be available, along with hands-on assessment of the performance of the system in a training environment. All available test data and reports would be thoroughly reviewed.

Sufficiency of Operational Test Review

Due to limited scope and complexity of the ADS-B (Out) system, integrated developmental and operational testing may provide adequate test data to support production and fielding decisions.

For ADS-B (Out), the lowest appropriate level of required operational testing may consist of a review of existing data rather than a separate, dedicated operational test event. A SOTR will only be accomplished when decided by AETC/SAS-TE and the T-6A PO. The SOTR would be used as the source of operational test information for supporting ADS-B (Out) production and fielding decisions.

Should AETC/SAS-TE perform a SOTR, it will be conducted near the end of ground and flight testing necessary to support FAA STC and USG MTC issuance. The SOTR will consider all available data to date concerning system development and may be supplemented with the most recent flight data; human factors test data and AETC pilot evaluations resulting from short test aircraft operations with an AETC pilot aboard. The SOTR output will be a final assessment of the system from the standpoint of operational viability. It is expected this assessment will be made prior to or approximately concurrent with the granting of the STC by the FAA and the

MTC.

Operational Utility Evaluation

AETC/SAS-TE may decide to pursue a traditional OUE construct which includes COIs, MOEs, MOSs and associated criteria assessing the operational effectiveness and suitability of the system being acquired. An OUE is an operational test conducted whenever a dedicated OT&E event is required, but the full scope and rigor of a formal Force Development Evaluation is not appropriate. This OUE would support a MS C fielding decision authorizing the production of

ADS-B (Out) modification kits and T-6A retrofits. OUE is at the discretion of AETC/SAS-TE and the T-6A PO.

Live Fire Test and Evaluation (LFT&E) Approach

There is no requirement for LFT&E on this program.

Other Certifications

The Contractor will perform all tasks necessary to demonstrate compliance with the FAA requirements specified in Title 14 CFR 91.225. The data acquired will support FAA STC and

USG MTC issuance. The Contractor will provide a Certification Plan which describes the ADS-

B (Out) installation and the specific testing needed to demonstrate compliance with functional and message content requirements for both airborne and ground operation. After the successful conclusion of the certification activities and issuance of both the STC and MTC, the Contractor will prepare a final summary report describing each activity completed during the design, development and test of the Aircraft with the ADS-B (Out) modification installed and operating in its final configuration.

Supplemental Type Certificate

The Contractor will develop and submit all appropriate data to the FAA to obtain, sustain, and retain ownership of the STC for the modification. The Contractor will obtain the STC from the

FAA prior to starting T-6A fleet modifications. A copy of this STC will be provided to the

JPATS Program Office.

Military Type Certificate

MTC verifies ADS-B (Out) installation satisfies the requirements of MIL-HDBK-516C. The

Contractor will submit the completed certification basis in order to obtain an MTC prior to modifying the fleet.

Frequency Certifications

The radio frequency assignment process and certification is very complex, time consuming, and requires approval by many different Government agencies, including FAA, FCC and various

DoD components. Frequency approval on T-6A ADS-B (Out) starts with the local Spectrum

Management Office (SMO). The local SMO acts as a shepherd for each request and passes each up through SMOs in both Air Force Material Command and Headquarter Air Force, then through

Interdepartment Radio Advisory Committee Frequency Assignment Subcommittee, which assigns frequencies on a national level. The frequency assignment then passes through National

Telecommunications & Information Agency and finally approved by the Joint Spectrum Center with inputs from FAA and FCC as required.

This process usually takes anywhere from six months to a year for final certification. There are also several stages of approval required for ADS-B (Out), including an approved temporary frequency authorization prior to starting ADS-B (Out) flight testing and Stage 3 approval prior to

MS C. Stage 4 approval is required prior to retrofitting aircraft in the field.

Cybersecurity Accreditation

The purpose of Cybersecurity is to identify, mitigate and eliminate computer and communication vulnerabilities within a weapon system. The T-6A aircraft will already have a system level

Cybersecurity Accreditation when ADS-B (Out) is installed on the aircraft. Per the ADS-B (Out)

SOW the Contractor will follow DoDI 8500.01 guidance for development of a Cybersecurity strategy.

Future Test & Evaluation

No further Test and Evaluation activity is anticipated after completing STC and MTC.

Reliability Growth

Reliability of the ADS-B (Out) system will be considered throughout system development, verification and validation, and certification test activities. Emphasis will be placed on reliability growth early in product development and integration to reduce the potential impact of equipment failure during flight test and operational use. A reliability growth assessment will be established to identify potential system risk effects of ADS-B (Out) on the T-6A. The data for reliability predictions will come from a variety of sources.

Logistics

The Contractor is providing two retrofit kits to support ADS-B (Out) test and evaluation. One kit will be installed on the T-6A test aircraft and the second kit will be used as a spare. Having a spare kit should keep the test effort on schedule in the event of premature component failure during contractor or developmental test periods. The Contractor is providing support equipment to test the ADS-B (Out) system to support flight testing. Consumable materials may be required to complete the retrofit on the test aircraft and will be provided by the USG through the normal

Contractor Operated and Maintained Base Supply request for supplies and services process used to support Time Compliance Technical Orders/Technical Directive efforts.

Product support elements such as training, supply support, support equipment and technical data will be evaluated throughout DT&E by VX-20 (limited assessment) and OT&E by AETC/SAS-

TE. The Contractor will develop test plans for DT&E and AETC/SAS-TE will develop test plans for OT&E. In addition, the adequacy of the logistics processes to support operational requirements will be verified during OT&E. Interim and final test reports will summarize evaluations of the product support elements.

ATD Operational Evaluation

AETC/SAS-TE personnel will perform an assessment on the first ATD fitted with the ADS-B

(Out) and will report all observed issues and deficiencies in the modified ATD to the Contractor and the JPATS Program Office. The Contractor will analyze this data and propose a course of action to eliminate these deficiencies. After acceptance of this action plan, the Contractor will incorporate the changes and document the improved system operation IAW the SOW.

AETC/SAS-TE will evaluate the final changes in ADS-B (Out) ATD performance.

4 PART IV – RESOURCE SUMMARY

Introduction

Testing is expected to occur solely at Randolph AFB, Texas, but may take advantage of other

DoD ranges, bases, facilities, and resources whenever deemed beneficial to the program.

Test Resource Summary

Test Article

The PO will coordinate with AETC to provide a production representative, operational T-6A aircraft based at JBSA -Randolph, Texas for the Contractor’s use for the duration of the program.

The provided T-6A aircraft will be available at contract award in the first quarter of Fiscal Year

‘17 through issue of the MTC. The Contractor test article will be the first article production kit.

Test Sites and Instrumentation

The PO will coordinate with AETC to procure the primary test site, JBSA-Randolph, TX. The

Contractor will be responsible for identifying, procuring and installing any special instrumentation required for conducting the ADS-B (Out) test program. The Contractor will remove any temporary modifications and special instrumentation items from the test aircraft upon receipt of the STC and MTC. If the ITT decides to use a secondary test site for DT&E then arrangements will need to be made through the JPATS Test manager and approved by the ADS-

B (Out) Program Manager and the JPATS Program Manager. A contract modification will be required for the Contractor to support DT&E at the secondary test site.

Test Support Equipment

The Contractor is responsible for providing test equipment for the ADS-B Out system to coincide with test and retrofit schedules. One approach could be to update to the current GFE, IFR-6000 transponder test set, with Option 3 software. Optionally, future test equipment may be different from current test equipment depending upon offeror’s proposals. The actual support equipment solution will not be determined until contract award. We anticipate no additional support equipment to execute this program.

Threat Representation

ADS-B (Out) will not require live threat representations.

Test Targets and Expendables

No test targets or expendables will be required for this test program.

Operational Test Support

AETC/SAS-TE will monitor system maturity throughout ADS-B (Out) design, installation and testing. AETC/SAS-TE will request pilot support from AETC/A3 and VX-20 to perform OT&E testing as necessary. No other external support is expected.

Models, Simulations and Test-Beds

No modeling and simulation is planned during this program. An operational T-6A aircraft will be provided for system development and testing.

Joint Mission Environment

This TEMP is a joint USN and USAF document identifying ADS-B (Out) test requirements from both services on the T-6A program. FAA AC 20-165B provides guidance for the installation and airworthiness approval of the ADS-B (Out) system in the T-6A aircraft.

Special Requirements

Stage 3 frequency certification approval is required prior to MS C, and stage 4 frequency certification approval is required prior to retrofitting the T-6A fleet.

Federal, State and Local Regulations

All T&E activities will comply with federal, state and local environmental regulations. The

National Environmental Policy Act documentation that addresses specific test activities and any known issues that may require mitigation to address significant environmental impacts will be submitted prior to the start of testing.

Manpower/Personnel & Training

The Contractor will provide subject matter experts to conduct on-site training for pilots and maintenance personnel. The Contractor will provide training for ADS-B (Out) transponder, GPS, and any other new avionics LRUs that aircrew and maintenance personnel will have to operate and maintain. At a minimum this training will consist of theory of operation and operational procedures for the newly installed ADS-B (Out) technical solution (to include new GPS operation and procedures) IAW CDRL A034 and RFP contract number FA8617-16-R-6207.

Training will be conducted no later than 30 days prior to the first aircraft being available for flight operations. It will consist of two days training, and will take place at a central location defined by AETC, most likely JBSA-Randolph, TX.

Test Funding Summary

Adequate funding has been estimated and programmed to support all required Contractor tests, FAA certification tests, DT&E and OT&E. Development funding covers Research, Developmental, Test & Evaluation and the initial two air vehicle and one ATD test kits.

Procurement funding covers retrofit kits, initial spares, and support equipment to install ADS-B

(Out) capability into 487 T-6As and 88 ATDs.

5 APPENDIX A - BIBLIOGRAPHY

1. Air Force Policy Directive 62-6, USAF Airworthiness, 11 June 2010

2. Air Force Instruction 62-601, USAF Airworthiness, 11 June 2010

3. MIL-HDBK-516C, Department of Defense Handbook, Airworthiness Criteria, 29 February, 2008

4. Air Force Instruction 99-103, Capabilities-Based Test and Evaluation, 16 October, 2013

5. Incorporating Test and Evaluation into Department of Defense Acquisition Contracts, October 24, 2011

6. DoDI 5000.05, Operation of the Defense Acquisition System, January 7, 2015

7. DoD Test and Evaluation Management Guide, Sixth Edition, December, 2012

8. Federal Aviation Administration Advisory Circular, AC 20-138C, May 8, 2012

9. Code of Federal Regulations (CFR) §91.225, General Operating and Flight Rules, May 7, 2015

10. Code of Federal Regulations (CFR) §91.227, Automatic Dependent Surveillance-

Broadcast, January 1, 2011

11. Federal Aviation Administration Technical Standard Order TSO-C254C, Universal

Access Transceiver (UAT) Automatic Dependent Surveillance-Broadcast (ADS-B)

Equipment Operating on Frequency of 978 MHz, December 2, 2009

12. Federal Aviation Administration Technical Standard Order TSO-C166b, Extended

Squitter Automatic Dependent Surveillance-Broadcast (ADS-B) Equipment Operating on

Frequency of 1090 MHz, December 2, 2009

13. Federal Aviation Administration Advisory Circular, AC 20-165B, Airworthiness

Approval of Automatic Dependent Surveillance – Broadcast (ADS-B) Out System, December 7, 2015

14. Federal Aviation Administration Advisory Circular, AC 20-151A, Approval of Traffic

Alert and Collision Avoidance Systems, Versions 7.0 & 7.1 and Associated Mode S

Transponder

15. Advisory Circular (AC) 90-114 change 1, Automatic Dependent Surveillance-Broadcast

(ADS-B) Operations, December 21, 2012

16. Project Test Plan Policy for Testing Air Vehicles, Air Vehicle Weapons, and Air

Vehicles Installed Systems, NAVAIRINST 3960.4C CH-1, April 9, 2013.

17. System Requirements Document, ADS-B (Out) Capability for the T-6A, dated 8

February 2016

6 APPENDIX B - ACRONYMS

AC Advisory Circular

ACAT Acquisition Category

ADS-B (Out) Automatic Dependent Surveillance – Broadcast Out

AETC Air Education and Training Command

AETC/SAS Air Education and Training Command/Studies and Analysis Squadron

AFLCMC Air Force Life Cycle Management Center

AFLCMC/WLZJ AFLCMC Designation for the JPATS Program

AFOTEC Air Force Operational Test and Evaluation Center

ATD Aircrew Training Device

CFR Code of Federal Regulations

CNATRA Chief of Naval Air Training

COI Critical Operational Issues

CT Contractor Testing

DCMA Defense Contract Management Agency

DoD Department of Defense

DR Deficiency Report

DT Developmental Testing

DT&E Developmental Test and Evaluation

EOA Early Operational Assessment

FAA Federal Aviation Administration

GBTS Ground-Based Training System

GFE Government Furnished Equipment

GPS Global Positioning System

IAW In Accordance With

IFT Instrument Flight Trainer

IOT&E Initial Operational Tests and Evaluation

IT&E Integrated Test and Evaluation

ITT Integrated Test Team

JBSA Joint Base San Antonio

JPATS Joint Primary Aircraft Training System

LDTO Lead Developmental Test Organization

MOE Measures of Effectiveness

MOS Measures of Suitability

MS Milestone

MTC Military Type Certificate

NAS Naval Air Station

OA Operational Assessment

OFT Operational Flight Trainer

OT Operational Testing

OT&E Operational Test and Evaluation

OTRR Operational Test Readiness Review

OUE Operational Utility Evaluation

PMA-273 Naval Undergraduate Flight Training Systems

PO Program Office

§ Section

SOTR Sufficiency of Test Review

SOW Statement of Work

STC Supplemental Type Certificate

T&E Test and Evaluation

TEMP Test and Evaluation Master Plan

TSO Technical Standard Order

USAF United States Air Force

USG United States Government

USN United States Navy

UTD Unit Training Device

VX-20 US Navy Test and Evaluation Squadron

WIPT Working-Level…