Bidders Library Operational Test and Evaluation - OTA Memo 5-31-2019.pdf

DEPARTMENT OF DEFENSE

AIR FORCE OPERATIONAL TEST AND EVALUATION CENTER

NAVY OPERATIONAL TEST AND EVALUATION FORCE

ARMY TEST AND EVALUATION COMMAND

MARINE CORPS OPERATIONAL TEST AND EVALUATION ACTIVITY

JOINT INTEROPERABILITY TEST COMMAND

31 May 2019

MEMORANDUM FOR DIRECTOR OPERATIONAL TEST AND EVALUATION

SUBJECT: Operational Test Agencies Six Core Test Principles

In order to meet the National Defense Strategy (NDS) objective to provide a more lethal, resilient, and rapidly adapting Joint Force, the Department of Defense (DoD) must Deliver Performance at the Speed of Relevance. As DoD and the Services prioritize speed of delivery, continuous adaptation, and frequent modular upgrades when providing capabilities to the warfighter, the operational test agencies (OTAs) need to adjust their focus of purpose from solely supporting the acquisition and warfighter communities, to that of “delivering combat capability at the speed of relevance”. To that end, this memo outlines six core test principles designed to ensure the future relevance, credibility and timeliness of operational testing (OT). Each Service OTA along with the Joint Interoperability Test Command (JITC) have agreed to these principles as a framework to achieve the NDS vision.

The six core test principles are concepts that have been used in the acquisition community for some time. However, the OTAs are putting a renewed focus and emphasis on each of them. By doing this, we will not only provide combat capability to the field at the “speed of relevance", but also do it at a more affordable cost. The six principles are: Early OT Involvement; Tailor to the Situation; Continuous and Cumulative Feedback; Streamline Processes and Products; Integrated and Combined Collection/Test; and Adaptive.

Early OT Involvement: The intent of this principle is for the OT teams to be engaged with a program from its very inception. The earlier an OT team becomes involved, the greater influence it can have on requirements definition, budgeting, contracting, and engineering to ensure the entire test community is part of a system’s development. By applying an operational perspective very early in a program, we can reduce changes later in the acquisition lifecycle and, in turn, reduce overall program cost. To be clear, early involvement is more than “observation” from an oversight prospective or “being seated at the table” as a symbol of collaboration; it means truly being part of the team to ensure relevant and credible information is provided to the decision makers when they need it. As an example, early OTA involvement means shaping each test event to simultaneously meet Contractor Test (CT), Developmental Test (DT) and OT objectives. In other words, to the maximum extent possible, design test events in an environment that can collect data once to answer the respective test objectives.

Tailor to the Situation: The intent of this principle is to provide the test teams the flexibility to adjust their tests as needed in order to field capabilities as rapidly as possible. Given that many programs today are rapid in nature and the fact the acquisition community is working hard to speed product delivery by exploring streamlined approaches and more agile techniques to procure products, our OT teams will need to adjust our methods in order to meet the unique needs of every program. We want each of our teams to know they have the flexibility to tailor their test planning, execution and reporting as needed in order to field capability as rapidly as possible. Furthermore, as each test team builds a test design for their program, they should determine and align the overarching purpose of the test with the warfighter (for example, “characterize” versus “demonstrate”). What decision is being supported? What information is needed to support that decision? It is essential, in the spirit of the OTAs providing warfighting capability, that the warfighter has direct input and rationale as to why OT is being conducted.

This information will be critical as we march toward the “speed of relevance”.

Continuous and Cumulative Feedback: The intent of this principle is to ensure that OT provides timely feedback regarding the problems it discovers throughout the life of a program, especially in the earlier stages. The current acquisition timeline calls for OT reports at specific milestones in the process.

With today’s fast paced acquisitions, these reports may be irrelevant before they’re even published. In order to provide relevant information, we will provide “Continuous and Cumulative Feedback" to the program office and all stakeholders regarding our findings. Depending on the nature of the program, we may still provide reports at specified milestones, but these reports will be a cumulative document, containing all the OT feedback leading up to that particular milestone or decision point. We want the developer and the program manager to know test results almost as fast as the OT teams. This approach changes the paradigm from OT being a “final exam” for a program manager, to OT being a partner in the system development and yet still delivering combat capability through timely independent assessments.

Streamline Processes and Products: Closely aligned with the principle of “Tailor to the Situation”, the intent of this principle is to remove the bureaucratic constraints of the current acquisition processes.

As a new program comes online, test teams should have the flexibility to modify existing procedures. To enable fielding at the “speed of relevance”, the test team must have the ability to streamline test processes and products to best meet the needs of their program. Along these lines, and tied to the previous principle, we want the teams to be able to create new products and processes, as needed, as they plan, execute and report on their systems.

Integrated and Combined Collection/Test: Closely coupled with “Early OT Involvement”, the goal of this principle is to merge the primary test stakeholders: the contractor/developer testers (CT), DT and OT, into one unified test team. As the program progresses through its acquisition lifecycle, we will no longer conduct sequential testing but instead will pursue synchronized collection and data sharing among all three test communities. We should remove the artificial CT, DT, and OT barriers and think in terms of utilizing all test events at any point in the program to achieve CT, DT and OT objectives in a collaborative fashion to the maximum extent possible. This “One Team” approach enhances communication across the entire program and encourages earlier and faster testing.

Adaptive: With today’s rapidly evolving technologies, the OTAs must be able to adapt to whatever program is presented to them. With the push within DoD for prototyping and its associated rapid fielding, the OTAs must not be encumbered by existing bureaucratic processes, and instead, must be allowed the freedom to change as the test proceeds in order to take advantage of learning during the test process.

They should not be constrained by a “checklist mentality” or rigid test designs that exceed the scope of the specific and evolving warfighter concerns. With this flexibility, our teams can meet the demands of fielding at the “speed of relevance” without compromising their independent analysis.

A fundamental premise for the successful implementation of these six principles is trust; trust among all stakeholders. The “speed of relevance” will be determined by the speed of trust--we must all trust each other to accomplish our assigned duties in order to deliver combat capability at the “speed of relevance”. The same way our warfighters, our sons and daughters do in the field, so must we do as we acquire the systems they will rely on to achieve victory.

Finally, we are fully aware that a dedicated focus on these principles will require a significant culture change among all stakeholders. Because we have conducted OT&E the same way for many decades, this will not be an easy undertaking. However, in order to meet the NDS challenge to provide more lethal combat forces, we must change the way we approach OT&E. These six principles are the first and most critical step to deliver combat capability at the “speed of relevance”.

Point of contact for this memo is Jeff Olinger, AFOTEC/CA, 505-846-6925 or jeffrey.olinger.2@us.af.mil.

MICHAEL T. BREWER P.A. SOHL

Major General, USAF Rear Admiral, USN Commander, AFOTEC Commander, OPTEVFOR

JOEL K. TYLER JUSTIN W. EGGSTAFF

Major General, USA Colonel, USMC Commander, ATEC Director, MCOTEA

SHAWN ROBERTS

CAPTAIN, USN

Commander, JITC