Bidders Library Operational Test and Evaluation - DoTE MEMO 6-3-2011.pdf

OFFICE OF THE SECRETARY OF DEFENSE

WASHINGTON, DC 20301

JUN - 3 2011

MEMORANDUM FOR: SEE DISTRIBUTION

SUBJECT: Test and Evaluation of Defense Programs

The Defense Acquisition Executive (DAE) recently chartered an independent team to assess expressed concerns that the Department's developmental and operational test communities' approach to testing drives undue requirements, excessive cost, and added schedule into programs and results in a state of tension between Program Offices and the Testing Community. Concurrently, the Director, Operational Test and Evaluation (DOT&E), and the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation conducted a systematic review of recent programs to address the question of whether testing increases costs and delays programs and, if so, by how much. The results of both efforts indicate that testing and test requirements by themselves do not generally cause major program delays. However, other problems were identified.

More specifically, the DAE assessment team "foundno significant evidence that the testing community typically drives unplanned requirements, cost or schedule into programs."

The DOT&E review examined 40 programs that experienced significant delays during the past few years andfound that 7 of those programs experienced some delays due to testing itself.

However, none of those programs was delayed solely by problems in testing and in no case was a testing problem a principal cause of delay. In 37 programs, there were delays caused by problems discovered during testing. Problems found by testing were shown to cause much longer delays than any delay caused by testing itself Furthermore, a DOT&E review of 76 recent programs found the cost of operational test and evaluation to be only about 1 percent of a program's total acquisition cost, exclusive of test infrastructure costs. We will continue our efforts to make testing both rigorous and efficient.

One area the DAE assessment identified for improvement is the relationship and interaction among the testing, requirements, and program management communities. In that regard, we callyour attention to four specific issues that we need your assistance in addressing:

the need for closer coordination and cooperation among the requirements, acquisition, and testing communities; the need for well-defined testable requirements; thealignment of acquisition strategies and test plans; and the need to manage the tension between the communities.

First, we need stronger mechanisms for rapid adaptation to emerging facts by the requirements, acquisition, and test communities and less resistance to change. These communities all share the goal of providing our forces in combat with equipment that worksas soon as possible; however, they often fail to work together effectively to achieve that goal when change is needed. Leadership at the Service level and throughout each community must empower andrequire their subordinates to work together to identify and resolve issues u beginningat the earliest stageof requirements development and extending throughout design, development, and testing. Both AT&L and DOT&E will work with the Services to identify and implement whatever policy changes may be needed to promotethis empowerment as part ofthe ongoing revision ofthe Department's acquisitionpolicies and practices; however, we believe that this is largelya matter of eitherempowering mid-level leaders to make necessary changes or more frequent senior leader involvement through vehicles like Configuration Steering Boards conducted during a program's development and test phases.

Second, we need the requirements process to produce well-defined, and therefore testable, requirements. Our results indicate the requirements process needs to be more agile and responsive to change as knowledge increases. From the outset, requirements development must be informed by technical feasibility and rigorous trade-off analysis. Defining requirements in ways that are clear and testable, including in a well-defined and comprehensive operational mission environment, should be achieved as early as possible. Operators, developers, program analysts, and testers should all participate in the development of requirements so that they are defined in ways that provide meaningful increments of operational capability, enable efficient program execution, and are testable. Interoperability needs in particular should be clearly defined. Key Performance Parameters should also be limited to those considered critical to assuring mission success in combat.

Third, we have found that acquisition strategies and test plans are often misaligned. A central element of all our acquisition strategies should be an executable plan to use developmental and operational testing together as a means to achieve and demonstrate success.

Programs often lack the budgetary and contract flexibility necessary to accommodate discovery and respond to problems discovered during program execution. The acquisition and test communities must work together to assure that this does not occur. Testers must work with developers to develop test plans that are both rigorous and efficient and that account for real-world constraints and expectations.

Fourth, although the independent review found that tensions between programs and the test community are to be expected and healthy for the most part, occasionally the tensions grow to the point of frustration and animosity. These tensions can be mitigated or eliminated through early and objective communication of concerns and issues and with the constructive involvement of senior leaders when necessary.

We are committed to working with you to resolve all of these issues and doing everything we can to efficiently and effectively equip our Warfighters with proven products that will ensure their success. We solicit your input regarding the issues we have identified and any recommendations you may have for improving the performance ofour acquisition and test activities.

Our points of contact are Mr. David Ahern, Deputy Assistant Secretary of Defense for Portfolio Systems Acquisition, at 703-693-3614 and Dr. Catherine Warner, Office of the Director for Operational Test and Evaluation, at 703-697-7247.

M MJ2-

Michael Gilmore

Director

Operational Test and Evaluation

JUN 0 3 2011 Ashton B. Carter

Under Secretary ofDefense for Acquisition, Technology and Logistics

DISTRIBUTION:

SECRETARIES OF THE MILITARY DEPARTMENTS

CHIEF OF STAFF, ARMY

COMMADANT OF THE MARINE CORPS

CHIEF OF NAVAL OPERATIONS

CHIEF OF STAFF, AIR FORCE

VICE CHAIRMAN OF THE JOINT CHIEFS OF STAFF

SERVICE ACQUISITION EXECUTIVES

ASSISTANT SECRETARY OF DEFENSE FOR RESEARCH AND ENGINEERING

DIRECTOR, COST ASSESSMENT AND PROGRAM EVALUATION

DIRECTOR, DEFENSE PROCUREMENT AND ACQUISITION POLICY

DIRECTORS OF THE DEFENSE AGENCIES (ATTN: ACQUISITION EXECUTIVES)

DEPUTY ASSISTANT SECRETARY OF DEFENSE FOR DEVELOPMENTAL TEST AND

EVALUATION

DEPUTY ASSISTANT SECRETARY OF DEFENSE FOR C3, SPACE AND SPECTRUM

DEPUTY ASSISTANT SECRETARY OF DEFENSE FOR PORTFOLIO SYSTEMS

ACQUISITION

DEPUTY ASSISTANT SECRETARY OF DEFENSE FOR SPACE AND INTELLIGENCE

DEPUTY ASSISTANT SECRETARY OF DEFENSE FOR SYSTEMS ENGINEERING

PRESIDENT, DEFENSE ACQUISITION UNIVERSITY