AFLCMC_Det_3_OI_99-103_08_Nov_2017.pdf

BY ORDER OF THE COMMANDER AFLCMC Det 3 OI 99-103

AIR FORCE LIFECYCLE MANAGEMENT CENTER (AFMC)

8 November 2017

Test and Evaluation

TEST AND EVALUATION PROCESS

COMPLIANCE WITH THIS PUBLICATION IS MANDATORY

ACCESSIBILITY: This publication is available digitally from the OPR.

RELEASABILITY: Distribution F – Further dissemination only as directed by AFLCMC Det 3

(8 November 2017).

OPR: AFLCMC Det 3 Chief of Test Certified by: AFLCMC Det 3/CC (Mr. Jason Pritchett) (Lt Col Christopher D. Gentile) Supersedes: ASC/WII Det 3 OI 99-103, 10 February 2012 Pages: 33

SUMMARY OF CHANGES

This instruction supersedes ASC/WII Det 3 OI 99-103, 10 February 2012. The document was revised to update the change in unit designation and coordination requirements resulting from the new Memorandum of Agreement (MOA) between AFLCMC Det 3 and 412 OG. It reflects the latest changes from the Air Force (AF), Major Command (MAJCOM) and Center Test Authority and Center Safety directives and instructions, as well as changes resulting from Department of Defense 5000 series and the Office of the Secretary of Defense, Acquisition, Technology and Logistics realignment. This OI is an extensive revision and should be fully reviewed.

1. INTRODUCTION

1.1. Authority and Purpose. AFPD 99-1 (reference 1), AFI 99-103 AFMCSUP (reference 2), AFI 91-202_AFMCSUP (reference 3) establish policies and processes for AF test and evaluation (T&E). The AFLCMC Det 3 has been designated Lead Developmental Test and Evaluation Organization (LDTO) responsibilities for MQ-1, MQ-9 and YQ-11 (reference 4).

This Operating Instruction (OI) establishes procedures for AFLCMC Det 3, as an acquisition and test authority, to implement and execute the AF’s T&E requirements.

1.2. Applicability.

1.2.1. This OI applies for air vehicles for which the AFLCMC/WII is System Program Manager (SPM). This includes government-owned, contractor-operated air vehicles.

AFLCMC Det 3 OI 99-103 8 November 2017

1.2.2. This OI applies for all other government-owned aircraft conducting T&E by AFLCMC/WII or its contractors (e.g., other SPM’s aircraft, Foreign Military Sales (FMS), etc.).

1.2.3. This OI applies if government personnel or equipment are involved.

1.3. Tailoring. The AFLCMC Det 3/CC can tailor or waive requirements in this OI on a case-by-case basis.

1.4. Supplements. No supplements to this OI are allowed.

1.5. Relationships with Other Instructions.

1.5.1. Modification Management and Airworthiness. Modification and airworthiness processes are integral to the test process; however, these processes are separate and distinct.

Procedures and documentation for Temporary 1 (T-1) and Permanent (P) modifications are covered in AFMCSup_AFI 62-601 (reference 5). Additional procedures and documentation for Temporary 2 (T-2) modifications are covered in the AFLCMC/WII Operating Instruction 21-126 (reference 6).

1.5.2. Flight Operations. AFLCMC Det 3 has established a MOA with the 412th Operations Group Flight Operations Authorities (FOA) for oversight of flight operations (reference 7). If a FOA MOA conflicts with this OI, the MOA will take precedence.

1.5.3. Test Director/Test Conductor (TD/TC) Qualification and Training. Defined in AFLCMC Det 3 OI 99-108 (reference 8).

2. ROLES

Throughout this OI, Detachment can be substituted for Squadron. Squadrons may delegate responsibilities to an OL. The primary AFLCMC Det 3 roles are the following:

2.1. AFLCMC/CC. Provides oversight of AFLCMC Det 3. Approves high safety risk tests IAW 412 OG Memorandum of Agreement (MOA).

2.2. AFLCMC/WII Senior Material Leader (SML). Approves medium safety risk tests.

Coordinates with AFLCMC/CC for approval of high risk test. Approve and coordinate medium risk (or higher) test sorties IAW 412 OG Memorandum of Agreement (MOA).

2.3. AFLCMC/WII Chief Engineer (CE). Ensures and maintains operational safety, suitability and effectiveness (OSS&E) for all AFLCMC/WII programs. Authors and maintains airworthiness process defined in AFMCSup_AFI 62-601.

2.4. AFLCMC/WII Lead Engineer (LE). The CE will assign an LE to each program.

Implements AFMCSup_AFI 62-601. Obtains the Flight Test Military Flight Release (MFR) IAW AFMCSup_AFI 62-601.

2.5. AFLCMC/WII Chief Developmental Tester (CDT). Provide T&E oversight of all Medium Altitude UAS related test activities IAW AF and MAJCOM and Center policies.

Evaluate all test requirements from Program Managers (PM), engineers and customers.

Participate in all phases of the acquisition cycle to include, among other duties, Integrated Test Teams (ITT), Test Integrated Product Teams (TIPT), system safety groups, Configuration Control Boards (CCB), product improvement working groups, Technical Review Boards (TRB), Safety Review Boards (SRB), test planning and reporting and aircraft certifications.

Assess the validity of all Det 3 test requirements and provide test execution options to PMs and ITT.

2.6. Squadron Commander (AFLCMC Det 3/CC). Oversees all aspects of T&E flight operations, test execution, safety and aircraft configuration control at the Squadron level.

Ensures test and safety plan approval, flight authorizations and test card approvals are conducted IAW the FOA MOA(s) and this OI.

2.7. Squadron Director of Operations (OL-Det 3/CC). Supervises all flying operations under tactical control (TACON) of the Squadron. Ensures that appropriately reviewed and approved test plans are effectively transitioned to flight operations. Ensures participating aircrew members are appropriately qualified. Provides aircrew training and other expertise to support the full spectrum of Squadron T&E requirements. Coordinates external assistance for flight crew support. Assures effective flight crew briefings and debriefings are conducted for all T&E missions IAW AFI 11-2FT Volume 3 paragraph 2.10 (reference 9). Coordinates with emergency support organizations for aircraft emergency and crash/rescue efforts unique to test operations and locations.

2.8. Squadron Chief Engineer. Ensures programs are executed IAW AF and AFLCMC Det 3 technical processes, in particular test, modifications and airworthiness. Verifies applicability of technical processes to programs. Ensures testing adequately meets test objectives relating to program requirements.

2.9. Squadron Chief of Safety. Supervises the safety of Squadron ground and flight operations. Ensures identification of mishap authorities and implementation of mishap procedures. Serves as Safety Review Board Chairman when delegated authority by

AFLCMC/SE.

2.10. Squadron Chief of Test. Responsible for all aspects of the test process and is the focal point for T&E in the Squadron and Operating Location. Acts as the primary interface between AFLCMC Det 3 and the Chief Developmental Tester. Assists with the development and approval of all test and safety plans and reports. Maintains a Squadron test schedule and priorities for all active and planned test projects. Provides T&E expertise and guidance on test-related issues. Executes group-level approved TD/TC training. Assigns Test Director/Test Conductor (TD/TC) for each T&E mission and ensures the TD/TCs are qualified and trained IAW AFLCMC Det 3 OI 99-108 (reference 8). Maintains Squadron’s TD/TC Letter of X’s and provides to the Squadron Commander on a quarterly basis. Ensures the test file is complete. Personnel authorized to perform Chief of Test duties will be appointed by the AFLCMC Det 3/CC.

2.11. Squadron Flight Test Engineer (FTE). A government representative that plans, provisions, executes and reports on a specific test project. Prepares and coordinates development, review and approval of test plans, safety plans, test cards, and reports. Supports conduct of test plan working groups, technical review boards, safety review boards, and test readiness reviews. Ensures proper test configuration control to meets test requirements.

Coordinates and schedules test assets and support resources with supporting agencies and test ranges. Tracks test execution, and produces quicklooks, watch items/deficiency reports and final test reports. Maintains the Project Test File.

2.12. Squadron Project Aircrew. Flight operations focal point for aircrew during a test project. Responsible for familiarity with test plan objectives, test specific limits, THAs and GMPs and adherence during flight test. Provides expertise on piloting, flight test techniques and operational aspects of test execution. Supports test plan, test card and technical order development and reviews. Participates in simulations and evaluation teams to support test card, Human Machine Interface (HMI) and flight/weapon manual development.

2.13. Squadron Project Engineer. Responsible for the acquisition life-cycle execution of their specific program. Communicates with PM on all relevant activities. Participates in ITTs and TIPTs and ensures all requisite coordination and documentation is accomplished for the project. Supports test plan, modification and technical order development and review. Works with LE to obtain airworthiness approvals. Coordinates unique test support assets, hardware, special frequencies, etc. with contractor, government and civil agencies to ensure effective and efficient T&E execution. Addresses issues, investigations, deficiencies found during testing.

2.14. Squadron Chief of Product/Quality Assurance (PA/QA). Ensures that maintenance and production concerns are addressed throughout all phases of T&E. Responsible for all modification/installation procedures and final PA/QA inspection for aircraft modifications IAW AFMCMC/WII Operating Instruction 21-126 (reference 6).

2.15. Test Director/Test Conductor (TD/TC). A qualified individual from the government, contractor or a participating test organization (PTO) with overall responsibility for the safe and efficient execution of specific test IAW an approved test package. TD/TC qualification and training is defined in AFLCMC Det 3 OI 99-108 (reference 8). Some tests may require both a TD and TC, depending on the complexity, risk level or special requirements of the test. The TC is subordinate to the TD; a TD may assume both roles.

2.16. Pilot in Command (PIC)/Aircraft Commander (AC). Responsible for overall safe operation of the test aircraft.

2.17. Participating Test Organization (PTO). Per AFI 99-103 a PTO is any test organization required to support an LDTO by providing specific T&E data or resources for a T&E program or activity. The PTO should designate a primary individual for all LDTO communication and coordination. Attends (or designates representative to attend) all TRBs and SRBs and ensures appropriate review of all test plans, test project safety review (TPSRs) and reports. Participates in ITTs, TIPTs and TRRs. Ensure all LDTO instructions, policies and procedures are followed.

Coordinates with Chief of Test on all PTO requirements. When requested, provides regular cost updates, schedule updates, test status, quick look reports and/or final reports to the LDTO.

3. PROCEDURES

3.1. Test Schedule. A flight test schedule will be maintained to properly plan activities and resources. The Chief of Test and/or Project Engineer will ensure the Program Manager is aware of actual or potential schedule issues. The schedule should account for review, coordination and approval with entities outside the Squadron. A test project schedule template is included in Attachment 6 to provide test planning events and expected times.

3.1.1. For Medium and High risk tests, time should be provisioned between the TRB/SRB and the intended start of testing to allow for review and approval of the test package by the coordination and approval authorities.

3.1.2. Flight operations waivers that require FOA, AFMC/A3V or AFMC/A3 approval should be provided to the waiver authority immediately after the conclusion of the SRB.

Pre-SRB coordination with the waiver authority is recommended.

3.2. Test Package Development, Review, Approval and Amendments. All T&E projects require a test package consisting of a test plan and safety plan. This includes T-2 modifications that install a system under test or flight test instrumentation. Technical and safety planning are an interdependent and iterative process. Technical plans should consider safety to personnel, aircraft, facilities and test equipment. Safety planning should be incorporated early in the project.

3.2.1. Test Plan Development. The Chief of Test, Project Engineer and FTE shall work with the test plan author in the development of the test plan. The Contractor or Government may write the test plan in any format provided its content answers the following questions:

What is the test article description?

What are the objectives?

What data is needed (both technical and safety-related)?

How will the data be collected?

How much data is needed?

How will the data be evaluated?

What criteria must be met to finish the test?

Additional guidance for preparing test plans may be found in the "Air Force Flight Test Center (AFFTC) Test Plan Preparation Guide” (reference 10), EDWARDSAFBI 99-101, 412 TW Test Plans (reference 11) and/or NAVAIRINST 3960.4 (reference 12). A Test Kickoff is a best practice that gains agreement on test requirements and approach prior to test plan development. A Test Kickoff guide is included in Attachment 7.

3.2.2. Technical Risk Factors. Technical risk is the combined effect of probability and severity that specific technical issues that could impede successful accomplishment of test objectives. The Technical Review Board (TRB) shall review test planning for technical risk factors, assess the level of risk for each factor and mitigate factors that can be controlled. The TRB shall describe Medium and High technical risk factors and associated mitigations in Section V, Review Synopsis of the Test Project Safety Review (TPSR).

Table 3.2.2, though not all inclusive, identifies some common technical risks and their associated technical risk levels.

Table 3.2.2 Technical Risk Factors & Descriptions.

Risk Factors Low Technical Risk Medium Technical Risk High Technical Risk

System-under-test maturity (hardware/software components)

Proven, mature

Less mature with significant progression/development (i.e., large step forward in construction/materials or design/logic to achieve the next level of capability).

New design and/or not previously tested/proven.

Confidence in test methodology

Common type of testing.

No adjustments to methodology anticipated.

Uncertainties exist in exact methodology required to meet objectives. Adjustments anticipated.

Most appropriate test methodology is unknown.

Large methodology changes will be required after testing begins.

Instrumentation adequacy

Mature/proven.

Parameters, sample rates, truth sources, resolution, etc. are all available, correct or known.

Less than mature/proven and/or parameters, sample rates, truth sources, resolution, etc. may not all be available, correct or known.

Immature/unproven and/or parameters, sample rates, truth sources, resolution, etc. are unknown or unavailable.

Contractor and AFLCMC Det 3’s experience with modification type

Experienced modifying this type of aircraft or system.

Little experience modifying this type of aircraft or system.

No experience modifying this type of aircraft or system.

Test Team experience and qualifications

High number of individuals qualified in and/or experienced with testing of this kind.

Little experience and/or qualifications among test team members with testing of this kind.

No experience or qualification among test team members with testing of this kind.

Resources or test range availability

Not a factor (no issues or concerns).

Resources/range availability is a concern.

Resources/range are known to be unavailable and for all or a portion of the testing.

Schedule adequacy

Number of sorties and schedule duration is adequate for meeting test objectives.

Number of planned sorties and/or schedule is aggressive and/or contingencies may preclude satisfying test objectives.

Number of planned sorties and/or test schedule is extremely aggressive and/or significant contingencies may undermine most or all of the testing.

Establishment and grasp of aircraft certification requirements

Certification plans have been established and number of uncertainties is minimal.

Aircraft certification requirements are understood, but there may be some additions that emerge later.

Aircraft certification requirements are not fully understood and the certification process may significantly impact/alter the test schedule.

3.2.3. Safety Plan Development. For every test plan there must be a safety plan. The test team shall identify specific test hazards, apply appropriate risk mitigation procedures and assign an overall level of safety risk in the safety plan. The AFLCMC Det 3 Test Project Safety Review (TPSR) Section IV, Section V, THAs, risk assessment matrix, and any other attachments comprise the safety plan. The test team shall use the safety plan template provided in Attachment 2.

3.2.3.1. Identify Test Unique Hazards. The test team will identify hazards that are unique to the series of tests. These “test unique” hazards are any conditions that have a potential of causing a mishap but are not associated with the standard operating procedures of the aircraft, test article, system under test or facility. In some cases, activities may elevate the risk associated with routine operational hazards, thus requiring additional safety planning such as a GMP or THA.

3.2.3.2. Eliminate or Control Hazards. The test team will eliminate or control hazards in the following order of precedence:

Design the test to eliminate the probability of the hazard occurring. This could include a decision to not perform the test or a redesign of the system if the risk is unacceptably high.

Change the test methodology to reduce the probability, severity, or exposure to the hazard (e.g., building up to the test condition).

Incorporate safety devices (e.g. spin chute, or additional power sources).

Provide caution and warning devices to detect an unsafe condition or trend.

Develop procedures and training when it is impractical to change the design or test methodology.

3.2.3.3. General Minimizing Procedures (GMPs). GMPs are stand-alone phrases/statements used to address items of test safety concern, such as test article restrictions, test build-up, critical parameter monitoring, weather, environmental criteria and flight test chase requirements. These procedures are beyond Standard Operating Procedures (SOPs) but do not address specific individual hazards.

3.2.3.4. Qualifications and Training. Piloting of Medium and High risk events by other than graduates of a TPS will be specifically documented in the safety plan for consideration by the test package approval authority.

3.2.3.5. Test Article Restrictions. The safety plan will identify any restrictions resulting from the safety planning. This may include any restrictions added at TRB/SRB or via coordination comments during the approval of the TPSR.

3.2.3.6. Special Considerations. Any notable activity or information that is important to safety planning but is not a THA, restriction or GMP may be a Special Consideration.

Examples include test sortie length limits, waiver dependencies, sortie complexity and identified but uncontrollable test hazards.

3.2.3.7. Test Hazard Analysis (THA). For a test unique hazard not controlled or eliminated by a GMP or test article restriction, a THA shall be written using Attachment 4 to control the hazard by directing mitigation and corrective actions. These THAs will be referenced in Section V paragraph 12 of the TPSR Form and are mandatory attachments. Each THA shall address a single hazard and shall be a stand-alone document. Test-unique hazards where risk cannot be minimized are Special Considerations.

3.2.3.8. Negligible Risk Safety Plan. A Negligible Risk (NR) Safety Plan is a shorter version of the TPSR form that simplifies Sections IV and V. An example is shown in Attachment 3. An NRR Safety Plan may be proposed by Chief of Test for tests that meet the following criteria:

All hazards have a risk of Negligible IAW the Test Risk Assessment Matrix.

Test activities are normal, routine, and operationally representative.

Test procedures do not involve the use of abnormal or emergency procedures, checklists or configurations.

For flight test, the system under test has no airworthiness impact, such that a failure or malfunction would cause the use of abnormal or emergency procedures to safely recover the aircraft.

3.2.4. Safety Risk Assessment. Risk is defined as a combination of mishap severity and probability of occurrence. Test teams use system safety techniques, prior experience, legacy system research and overall engineering judgment to assess risk. The residual risks determined by the test team are proposals for the SRB to affirm or adjust.

3.2.4.1. Determine Mishap Severity. The mishap severity category is a qualitative assessment of the worst credible mishap consequence that could occur with all mitigations in place. The assessment should incorporate engineering judgment and/or past experience with similar tests or systems with all minimizing procedures and corrective actions in place. Mishap severity categories are defined in Table 3.2.4.1.

(Note: Always use the latest financial thresholds in AFI 91-204.)

Table 3.2.4.1. Mishap Severity Definitions Description Mishap Definition

Catastrophic Loss of life, aircraft, or expensive and unique system. Loss > $2,000,000 (e.g., Aircraft Class A).

Critical Severe injury, lengthy hospital stay, permanent injury. Severe aircraft, equipment or property damage. Loss > $500,000 but < $2,000,000 (e.g., Aircraft Class B).

Marginal Minor injury, medical treatment, but no permanent injury. Minor damage. Loss > $50,000 but < $500,000 (e.g., Aircraft Class C).

Negligible Superficial injury, little or no first aid required. Very minor damage.

3.2.4.2. Determine Mishap Probability. The safety reviewers will subjectively assess the mishap probability with all mitigation in place. The mishap probability level should qualitatively and/or quantitatively measure the likelihood of the mishap occurring due to personnel error, environmental conditions, design inadequacies, procedural deficiencies, or system/subsystem component failure or malfunction. The assessment should incorporate engineering judgment and past experience with similar tests or systems with all minimizing procedures and corrective actions in place. Mishap probabilities are defined in Table 3.2.4.2.

Table 3.2.4.2 Hazard Probabilities Mishap Probability

Level Descriptive Quantitative

Frequent A Very likely to occur (e.g. test exceeds design limits or mishap occurred during similar testing, etc.)

> 10-1

Probable B Likely to occur (e.g. test at design limits or mishap almost occurred during similar testing)

< 10-1 but > 10-2

Occasional C Some likelihood to occur, but not expected < 10-2 but > 10-3

Remote D Unlikely to occur (e.g. test activity approaching design limits and done before w/no problems encountered)

< 10-3 *

Improbable E Rarely occurs (e.g. test activity within design limits and covered under normal operational procedures)

*Note: Centers may refine Remote and Improbable Quantitative probabilities to address values < 10-3 (e.g. defining Remote as < 10-3 but > 10-5 or Improbable as < 10-6)

3.2.4.3. Test Risk Assessment Matrix. Each hazard identified in a THA or Special Consideration will be assigned a risk level IAW Figure 3.2.4.3. Other organizations (FAA, Army, etc.) may have different risk assessment definitions. If AFLCMC Det 3 is the LDTO, this matrix takes precedence.

Figure 3.2.4.3. Test Risk Assessment Matrix

3.2.4.4. Overall Safety Risk. After all hazards have been identified and mitigations clearly defined, the test team will assess cumulative effects of the hazards and propose an overall safety risk level by using the Test Risk Assessment Matrix and the descriptions shown in Table 3.2.4.4. The overall safety risk level may be higher (but not lower) than the highest individual hazard.

3.2.4.4.1. A split risk assessment may be used in cases where safety risk is assessed differently for separate phases of a test project or for individual test events. For example, an overall HIGH risk may be assigned for a test program which includes flight envelope expansion, but a subset of that testing may be assessed as MEDIUM or even LOW. The test points in each risk category will be clearly identified in the TPSR risk assessment section. This allows lower risk testing to begin upon risk approval authority signature while higher risk approvals are obtained.

Table 3.2.4.4. Overall Safety Risk Level Assessment

Assessment Description and Implication HIGH RISK Tests or activities that present a significant risk to personnel, equipment, and/or property even after all precautionary measures have been taken.

MEDIUM RISK Tests or activities that present a greater risk to personnel, equipment, and/or property than normal operations even after all precautionary measures.

LOW RISK Test or activities that present no greater risk than normal operations. Routine supervision is appropriate NEGLIGIBLE Activities that are normal, routine and operationally representative

3.2.5. Technical and Safety Reviews. There are two reviews: Technical Review Board and Safety Review Board. The Chief of Test and SRB Chairman are responsible for determining readiness for TRB/SRB. Each Squadron is responsible for developing an appropriate review process to ensure the test package is ready for TRB/SRB. Typically, the TRB and SRB are conducted in a meeting format; TRB will precede the SRB and the two will remain conceptually independent. The Chief of Test shall obtain Chief Developmental Tester approval for an Electronic TRB/SRB in lieu of a meeting. Negligible risk test packages shall use an Electronic TRB/SRB.

3.2.5.1. Technical Review Board (TRB). The purpose of a TRB is to provide an independent assessment to determine technical adequacy and identify and mitigate technical risk factors.

3.2.5.1.1. Technical Adequacy. A test plan is technically adequate if its contents are appropriate, thorough and specific enough to answer the motivating question(s) for performing the test. At a minimum, objectives, methodology, instrumentation/data requirements, success criteria and evaluation criteria must be reviewed.

3.2.5.1.2. Technical Risk Factors. Reference paragraph 3.2.2.

3.2.5.1.3. TRB Chairmanship. The Chief Developmental Tester is the TRB chairman. The Chief Developmental Tester may transfer chairmanship on a case-by-case basis to a technically qualified individual who is senior in test experience (or has formal Test Pilot School training), with sufficient expertise in the test activity to be reviewed. If the Chief Developmental Tester is unavailable and there is no declaration of transfer, then the Chief of Test is chairman.

3.2.5.1.4. TRB Membership. The board members are the Squadron Chief Engineer/Project Engineer, LE, Chief of Test, project aircrew, and PA/QA representative. Any additional TPSR Section III signatories, such as the AFLCMC Det 3/CC, may opt to participate as a board member. The chairman may appoint additional individuals, such as project technical experts, to be board members.

3.2.5.1.5. TRB Attendance. All board members shall participate or designate an alternate. If a board member is absent, the chairman shall decide whether to postpone the meeting. Technical adequacy and risk factors shall not be finalized until every board member provides their technical assessment. Other attendees may be Squadron Chief of Safety, FTE, TD/TC, project aircrew, subject matter experts, PTO representatives, waiver authority staff and the Contractor.

3.2.5.1.6. TRB Conduct.

The FTE should provide a preliminary test plan to the attendees prior to the

TRB.

The chairman will lead the meeting.

The most technically qualified individual or FTE should brief.

The chairman will determine if briefing slides are required.

The briefing should include the program objective, test schedule, airworthiness readiness, test resources/range, test purpose, test objectives, test conduct/methodology, measures of performance, success criteria, evaluation criteria, operational limitations, technical go/no-go criteria, flight ops waiver requirements, limiting factors, test article configuration, results of previous tests, technical risk factors and test reporting.

The chairman will obtain input from the board members on technical adequacy of the test plan.

The chairman will obtain input from the board members on whether the technical risk factor list is complete and whether the mitigation applied and risk level assessed is appropriate for each factor.

The chairman will determine whether the test plan is technically adequate The chairman will assign the level of risk (low, medium or high) to each technical risk factor and determine whether the list of technical risk factors is complete.

Reference paragraph 3.2.2 for technical risk factors.

3.2.5.1.7. TRB Documentation. The FTE will publish minutes that include significant comments, concerns, and action items. The FTE will update the TPSR to include TRB action items, response summaries, affirmation of technical adequacy and assigned risk level. The TRB chairman will ensure the Program Manager is aware of any Medium or High technical risk factors because of their potential programmatic impacts. Determination that the test plan is technically adequate or test plan signature by the approval authority provides clearance to begin ground testing.

3.2.5.2. Safety Review Board (SRB). The purpose of an SRB is to ensure appropriate test hazards are identified, ensure the proposed risk control measures mitigate (minimize or eliminate) the hazards to an acceptable level, and to assess and recommend an appropriate residual risk level to the approval authority.

3.2.5.2.1. SRB Chairmanship. The AFLCMC/SET is the SRB chairman (reference 3). The AFLCMC/SET may delegate chairmanship to a project-independent and technically qualified individual who is senior in test experience (or has formal Test Pilot School training), with sufficient expertise in the test activity to be reviewed.

If the AFLCMC/SET is unavailable and there is no declaration of transfer, then the Squadron Chief of Safety or Chief of Test is chairman. The SRB Chairman must be a government employee.

3.2.5.2.2. SRB Membership. The board members are the Chief Developmental Tester, Squadron Chief Engineer/Project Engineer, LE, Chief of Test, Squadron Chief of Safety, project aircrew and PA/QA representative. The chairman may appoint additional individuals as board members, such as subject matter experts.

Any additional TPSR Section III signatories may opt to participate as a board member.

3.2.5.2.3. SRB Attendance. All board members shall participate or designate an alternate. If a board member is absent, the chairman shall decide whether to postpone the meeting. Safety risk level shall not be finalized until every board member provides their assessment. Other attendees may be TD/TC, FTE, project aircrew, flight operations, subject matter experts, PTO representatives, waiver authority staff and the Contractor.

3.2.5.2.4. SRB Conduct.

The SRB should be conducted prior to and separate from the test package approval briefing, if one is required.

The FTE should provide a preliminary TPSR to the attendees prior to the

SRB.

The chairman will lead the meeting.

The most qualified individual or FTE should brief.

The chairman will determine if briefing slides are required.

The briefing should cover the safety plan. Mandatory items include test aircrew procedures, test safety go/no-go items, flight ops waiver requirements, GMPs, Special Considerations, THAs and test team’s proposal for overall risk level.

The chairman will lead a general discussion of the test, identified hazards, and associated mitigations. The discussions should be candid and result in a general agreement by the board, although disagreements may occur.

Safety risk assessment should be IAW paragraph 3.2.4.

The chairman will poll each board member for recommended safety risk level for each hazard and the overall risk level.

The chairman will assess the safety risk level to be recommended to the approval authority.

3.2.5.2.5. SRB Documentation. The FTE will publish minutes that include significant comments, concerns, and action items. The FTE will update the TPSR to include SRB action items, response summaries and risk levels.

3.2.5.3. Electronic TRB/SRB. The Electronic TRB/SRB is a method to conduct the technical and safety reviews without a meeting. An Electronic TRB/SRB is appropriate when test activities are readily understood by reviewers, tend to be less complex and are lower in risk. Electronic TRB/SRB membership and attendees are the same as a

TRB/SRB.

3.2.5.3.1. Electronic TRB/SRB Conduct and Documentation. The FTE will send the test package to all board members and approval signatories electronically for parallel review. Board members will provide comments or questions. The FTE will staff the TPSR signature page separately for signatures. The FTE will update the test package to include Electronic TRB/SRB action items and response summaries.

3.2.6. Test Package Approval. The TPSR signature page is used to document approval of the test package and acceptance of safety risk. The TPSR is signed after TRB/SRB and all action items from Boards are complete.

3.2.6.1. Signatories. The TPSR preparer should complete and sign Section I. SRB members not listed in section III and any other notable SRB participants (such as PTO) should sign in Section II. Table 3.2.6.1 identifies the Section III coordination and approval authorities for all combinations of technical and safety risk levels.

Table 3.6.2.1. TPSR Section III Signatories

Safety Risk Level Section III Coordination Section III Approval

Authority

Negligible Squadron Chief of Safety

Chief Developmental Tester AFLCMC Det 3/CC

Low Squadron Chief of Safety

AFLCMC Det 3/CC AFLCMC/WII Chief Developmental Tester Lead Engineer

Medium

Squadron Chief of Safety

AFLCMC/WII

AFLCMC/WII Chief Developmental Tester

AFLCMC/WII Chief Engineer AFLCMC Det 3/CC

Flight Operations Authority (FOA) GP/CC

High

Squadron Chief of Safety

AFLCMC/CC

AFLCMC/WII Chief Developmental Tester AFLCMC/WII Chief Engineer

AFLCMC Det 3/CC

FOA GP/CC

AFLCMC/WII

FOA WG/CC

3.2.6.2. Routing. For test packages requiring AFLCMC Det 3/CC approval, the Chief of Test will facilitate test package routing for coordination and approval. For test packages requiring AFLCMC/WII or above approval, the Chief Developmental Tester will facilitate routing.

3.2.7. Test Package Expiration. The test package approval expires one year from final TPSR signature date unless otherwise specified by the TRB/SRB or test package approval authority. Re-approval will be obtained on a new TPSR signature page and can be accomplished with an Electronic TRB/SRB or TRB/SRB process. The original test package approval authority is approval level for reinstatement of a test package. A Test Readiness Review (TRR) should be re-accomplished after the test package is reinstated and before testing begins.

3.2.8. Test Package Amendments/Changes. It is common for project changes to arise after test package approval. Unexpected results, overly restrictive controls, hazards not previously identified or adequately controlled, configuration changes and changes in risk level all constitute reasonable grounds for changing test and safety planning. Test package amendments may require test card changes and re-approval—see paragraph 3.4.4. There are three types of test package amendments: out-of-scope, in-scope or administrative change. The Chief of Test will gain concurrence from the SRB Chairman when determining whether a proposed change is in-scope.

3.2.8.1. Out-of-Scope Amendment.

3.2.8.1.1. Definition/Scope. An out-of-scope amendment is any safety plan change or a major test plan change. Out-of-scope amendments are required for the following:

• Test plan changes that have actual or potential safety implications.

• Test or safety plan changes due to significant unexpected events to include damage to the test article or support equipment, exceeding a safety of test limit, or unfavorable departure from predicted results.

• Removal of overly restrictive controls.

• Proposed change in risk level.

• Test hazards not previously identified or adequately controlled.

• Substantive changes to test objectives, technical approach or test procedures.

• Test plan changes that fundamentally alter test objectives, methodology, test article configuration, data requirements or instrumentation requirements.

3.2.8.1.2. Approval. Out-of-scope amendments require a TRB/SRB (meeting or electronic) and a new test package approval IAW paragraph 3.2.6. The approval authority for a decrease in risk level will be based on the “original” risk level.

3.2.8.2. In-Scope Amendments.

3.2.8.2.1. Definition/Scope. An in-scope amendment is a minor test plan change that modifies test procedures but does not affect safety planning.

3.2.8.2.2. Approval. In-scope amendments may be documented in a memorandum and approved by the Chief of Test. The memo may be written by the government or contractor. The Chief of Test may determine in-scope amendments require higher approval to include: Chief Developmental Tester, SRB Chairman, Squadron Commander or original approval authority. The Chief of Test should provide the final memo to original TRB/SRB members and approval authority as an information copy.

3.2.8.3. Administrative Changes.

3.2.8.3.1. Definition/Scope. An administrative change clarifies information contained in the test package that does not affect test conduct or safety planning.

3.2.8.3.2. Approval. Changes are made directly to the test package and are approved by the Chief of Test.

3.3. Test Cards.

3.3.1. Test Card Development. All ground test and flight test events will be conducted from approved test cards or written test procedures. Test cards may be written by the contractor, AFLCMC Det 3 or a PTO. Any format is acceptable provided test cards contain sufficient information to adequately describe safe and efficient test point and test maneuver execution. Information applicable to the entire test mission may be included on cover cards rather than individual test cards. Test cards should be prepared early enough for sufficient review by the test aircrew. Whenever practical the test cards should follow a challenge-and-response format. The test cards should not replace or be redundant to the technical orders (TOs), flight manual (FM), Mission Operators Handbook (MOH), Aircrew Operating Handbook (AOH) or Pilot Operating Handbook (POH). Unusual situations, such as an aircraft’s first flight may dictate that all or part of a TO, FM, MOH, AOH or POH checklist be made a part of the flight test cards. Test cards may be tailored but should contain applicable information from the following:

Approval signature sheet Test plan test point number Risk level, if other than LOW, for each specific card Go/No-Go criteria Aircraft configuration (gear, flaps, store loads, software configuration, etc.)

Aircraft operating limit critical to the test point (airspeed, Mach number, angle of attack, angle of sideslip, load factor, center of gravity, etc.)

Test specific limits Test point initial conditions (airspeed, altitude, etc.)

Fuel Status Special test equipment/safety devices Challenge and response items, if required Flight test technique (FTT) required for test point Allowable data bands/parameter tolerances Expected or predicted results Individual test team member responsibilities for test point execution/data collection

3.3.2. Test Card Risk Levels. A risk level should be assigned for each test card. A test series may consist of cards at varying risk levels.

3.3.3. Test Card Approval. Test card packages shall be signed and submitted by the Test Director/Test Conductor, Project Aircrew and Chief of Test (or Project Engineer). Each card should be approved at its individual risk level. Test cards may only be approved concurrently with or after the TPSR has been approved. Test card approval authorities and delegation options are shown in Table 3.3.3.1.

Table 3.3.3.1. Test Card Approval Authorities Safety Risk

Level Approval Authority Coordination Delegation Option(s)

Negligible AFLCMC Det 3/CC AFLCMC OL-Det 3/CC AFLCMC OL-Det 3/CC

Chief of Test

Low AFLCMC Det 3/CC AFLCMC OL-Det 3/CC AFLCMC OL-Det 3/CC

Medium Flight Operations Authority

(FOA) GP/CC

AFLCMC Det 3/CC

AFLCMC OL-Det 3/CC None

High FOA GP/CC AFLCMC Det 3/CC

AFLCMC OL-Det 3/CC None

3.3.4. Test Card Expiration. Test cards expire concurrent with the test package expiration unless otherwise specified by the TRB/SRB or test package approval authority. Per the 412 OG MOA, this expiration is governed by EDWARDS OI 99-105 (reference 13).

3.3.5. Test Card Changes Due to Test Plan Amendments. The specific test cards affected by a test plan change must be reapproved by the original test card approval authority. Low risk test card re-approval for an in-scope test plan amendment may be delegated to the Chief of Test.

3.4. Test Readiness Review (TRR). A TRR is a review to ensure all necessary actions and documentation are complete and approvals are obtained prior to the start of a ground or flight test. A TRR is required for all test projects. The TRR must be conducted prior to the first flight of a test article as well as prior to a new test series. A test series is best defined as a sequence of tests which fall within the same relative time span and are conducted using the same test article configuration and same test documentation (e.g., test plan and safety plan).

3.4.1. TRR Attendees. The Chief of Test or FTE will lead the review. If available, original SRB members will be included in the TRR, otherwise minimum attendees should include Approval Authority, Chief of Test, TD/TC, project aircrew, project engineer, and appropriate contractor representatives.

3.4.2. TRR Conduct. TRRs for Low and Negligible risk tests may be conducted informally as walk-around coordination or with a formal meeting. TRRs for Medium and High risk tests will be a formal meeting. Attachment 5 provides a template checklist. The checklist will not be approved until all TRR action items are complete. The Squadron Commander is the approval authority for Test Readiness Review (TRR) regardless of the method of completion and may delegate it no lower than the Chief of Test.

3.4.2.1. Off-site testing typically incurs significant amount of effort and cost. Consider conducting a TRR or a pre-deployment briefing prior to traveling for off-site testing.

Squadrons can tailor their local procedures.

3.5. Test Conduct. In general, the scheduling, training, flight documentation, aircrew briefing, and mission reporting of all flight tests are accomplished IAW AF and AFMC instructions and are managed by the flight test unit’s flight operations. All test missions will have a designated

TD/TC.

3.5.1. Pre-Mission Planning. The TD/TC will decide the order and execution of the individual test points. The test sequence should be logical and follow a build-up approach using incremental steps. Criteria for advancement to subsequent test points should be well defined.

3.5.2. Mission Brief. Each test will have a mission brief prior to test which all participants attend. Each Squadron may develop their own procedures and checklist for the mission brief. The TD/TC and PIC/AC share mission briefing responsibilities. The following items should be covered:

Test Objectives/Requirements Test Crew Identification, Assignment and Duties Crew Seating and Communication Aircraft Configuration (software, weapons loading, maintenance issues, etc.)

Instrumentation (special procedures, circuit breakers, T-2s, etc.)

Test Card Procedures Specific Mission Procedures Flight and Test Limits Go/No-go Support Equipment (Ground Station, other aircraft, etc.)

Range Times TPSR (GMPs, THAs, Special Considerations, Test Article Restrictions, etc.)

Airworthiness or Military Flight Release restrictions Photo Requirements (Aspect, Sun, etc.)

Times (orbits, etc.)

Weather Constraints Emergency Procedures Actions and Key Words to Terminate Test Profile Crew Resource Management

3.5.3. Test Execution.

3.5.3.1. Safety. The PIC/AC is ultimately responsible for the safe operation of the test aircraft. The PIC/AC may decide to terminate the mission or test point for any reason.

Any crew member has the right to pause current testing and present safety concerns about a situation or development. Further testing will not be pursued unless consensus is unanimous amongst crew members.

3.5.3.2. Test Control. The Test Director/Test Conductor (TD/TC) orchestrates the test mission and has the final decision regarding test execution. Some tests may require both a TD and TC, depending on the complexity, risk level or special requirements of the test. The TC is subordinate to the TD; a TD may assume both roles.

3.5.3.2.1. Test Director. The test director is responsible for the efficient and safe conduct of a test mission IAW an approved test package. The test director:

3.5.3.2.1.1. Provides ground and flight test event information to the scheduling process.

3.5.3.2.1.2. Verifies that the test cards and/or procedures have been properly reviewed and approved.

3.5.3.2.1.3. Ensures key personnel attend both pre- and post-test briefings. The test director will ensure that test cards and safety packages are fully briefed and will ensure that test results are properly documented.

3.5.3.2.1.4. Supervises the Ground Control Station (GCS) and/or control room and validates qualifications of all test personnel in the control room.

3.5.3.2.1.5. Possesses the authority to terminate the test point or mission if the technical validity of the test is in question or safety is jeopardized.

3.5.3.2.1.6. Makes the final decision on the real-time selection of options during the test mission.

3.5.3.2.1.7. Maintains test discipline.

3.5.3.2.2. Test Conductor. The test conductor is responsible for real-time coordination of activities with the aircrew; paces the progression through the test cards, as agreed to in the mission pre- briefing; defers to the test director for decisions, as appropriate; and is the primary communicator to the aircraft/aircrew.

The test conductor:

3.5.3.2.2.1. Provides ground and flight test event information to the scheduling process.

3.5.3.2.2.2. Makes test-point terminate and go or no-go calls based on real-time engineering analyses of control room data. Terminates test points if the technical validity of the test is in question or safety is jeopardized.

3.5.3.2.2.3. Briefs test cards during mission briefing. Briefs test hazard analyses and general minimizing considerations for all applicable test points during mission briefing.

3.5.3.2.2.4. Coordinates control room and/or aircraft setup, when required.

3.5.3.2.2.5. Reports on test results via quicklook reports, watch items and deficiency reports.

3.5.3.3. Test Card Changes during Test Execution. During test execution, testing will be limited to the procedures defined in the approved set of test cards. Test cards may be deleted or reordered unless specific inclusion or order is dictated by the TRB or SRB. The Chief of Test may approve minor changes to test cards that do not impact safety. The test card approval authority must approve any significant changes. Test events not defined in the approved set of cards and/or not briefed at the mission brief are prohibited. Operations defined within an approved manual/instruction and performed by a trained and qualified operator are allowed if briefed at the mission brief.

3.5.3.4. Mishaps. Mishaps will be investigated in accordance IAW AFI 91-204 (reference 14). The owning unit AFLCMC Det 3/CC is responsible for notification and mishap investigation initiation.

3.5.4. Debrief. All test missions will be debriefed to ensure that test results are properly documented. Any safety concerns will be addressed. All test participants should be present for debrief unless excused by the TD/TC or PIC.

3.6. Test Completion or Termination.

3.6.1. Test Reporting. There are two types of reports that should be written during the course of a test program: a daily quick look report and a final report.

3.6.1.1. Quick Look Report (QLR). This report is a record of what happened during the sortie. At a minimum, it should note the sortie date and duration, the test conditions, what test points were accomplished, what problems occurred and any watch items.

Quick Looks will be sent to the test team, Squadron Commander, Squadron Chief Engineer/ Project Engineer, Chief of Test, Chief Developmental Tester and LE.

3.6.1.2. Final Report (FR). This is the official report that documents the test program.

Follow reporting guidelines defined in the test plan. Normally the Statement of Work (SOW) or the Contracts Data Requirements List (CDRL) will stipulate the timeline for putting out a final report, but in the absence of that guidance, final reports should be due within 60 days of the completion of the final test event. Additional guidance for writing test reports may be found in "The Author’s Guide to Writing AFFTC Technical Reports". Contractor format is acceptable provided it meets the intent of this document.

3.7. Lessons Learned Program. The test team should review appropriate historical lessons learned before commencing a new test project. Test team should document lessons learned regarding test planning, test execution, test reporting, and most importantly, test safety, throughout the test program. Lessons learned can be captured in emails, memos, hot wash meeting minutes, slides, etc. The Chief of Test is responsible for maintaining the unit’s lessons learned program and will provide copies to the Chief Developmental Tester. The Chief Developmental Tester will review and communicate lessons between the Squadrons.

3.8. Test File. The Chief of Test will maintain a file of all test documentation for each test project.