Bidders Library Operational Test and Evaluation - DoTE MEMO 01-06-2010.pdf

OFFICE OF THE SECRETARY OF DEFENSE

1700 DEFENSE PENTAGON

WASHINGTON, DC 20301·1700

JAN II mo

OPERATIONAL TEST

AND EVALUATION

MEMORANDUM FOR THE COMMANDER, U.S. ARMY TEST AND

EVALUATION COMMAND

COMMANDER, OPERATIONAL TEST AND

EVALUATION FORCE

COMMANDER, AIR FORCE OPERATIONAL TEST

AND EVALUATION CENTER

DIRECTOR, MARINE CORPS TEST AND

EVALUATION ACTIVITY

COMMANDER, JOINT INTEROPERABILITY TEST

COMMAND

SUBJECT: Reporting of Operational Test and Evaluation (OT&E) Results

The statutory responsibilities of the Director, Operational Test and Evaluation, include prescribing policies and procedures for the conduct of operational test and evaluation in the Department of Defense. Currently, DoD! 5000.02 (December 8,2008) specifies the following:

OT&E shall be used to determine the operational effectiveness and suitability of a system under realistic operational conditions, including joint combat operations;

determine if thresholds in the approved CPD [Capability Production Document] and critical operational issues have been satisfied; assess impacts to combat operations; and provide additional information on the system's operational capabilities.

I have observed differences in evaluating and reporting operational effectiveness and suitability among the operational test authorities that indicate the multiple purposes of OT&E contained in this definition may be causing confusion. Accordingly, the purpose ofthis memorandum is to clarify the policy for evaluating and reporting operational effectiveness and suitability.

The primary purpose of initial OT&E is contained in the first clause quoted above:

to determine operational effectiveness and suitability. The context of the determination is discussed clearly in the manual for the Joint Capabilities Integration and Development System (JCIDS). In that manual, operational effectiveness is defined as follows:

Measure of the overall ability of a system to accomplish a mission when used by representative personnel in the environment planned or expected for operational employment of the system considering organization, doctrine, tactics, supportability, survivability, vulnerability, and threat.

The Defense Acquisition Guide repeats this definition and expands on it, emphasizing "The evaluation of operational effectiveness is linked to mission accomplishment."

These documents treat operational suitability in a similar manner.

Thus, operational effectiveness and suitability must be evaluated and reported on the basis ofwhether a system can be used by Soldiers, Sailors, Ainnen, and Marines to accomplish a combat mission. The appropriate environment for that evaluation includes the system under test and all interrelated systems (that is, its planned or expected environment in tenns of weapons, sensors, command and control, and platfonns, as appropriate) needed to accomplish an end-to-end mission in combat. The data used for evaluation are appropriately called measures of effectiveness, because they measure the military effect (mission accomplishment) that comes from the use of the system in its expected environment. This statement of policy precludes measuring operational effectiveness and suitability solely on the basis of system-particular perfonnance parameters.

The second purpose of aT&E stated in 5000.02 is more closely linked to the development and acquisition process, which is "to detennine if thresholds in the approved CPD and critical operational issues have been satisfied." The Milestone Decision Authority needs this information in making a Milestone C procurement decision. The JCIDS manual specifies that "The CPD defines a single increment of the perfonnance attributes (key performance parameters, key system attributes, and other attributes) to support a MS C decision." The measures used are appropriately referred to in the context of "perfonnance" as in "key perfonnance parameters," or "measures of perfonnance." These "perfonnance attributes" are often what the program manager is required to deliver. But most important to this discussion, they are not the military effect or measure of operational effectiveness required for achieving the primary purpose of aT&E. Rather, they are, in general, system-particular performance parameters. As such they should be referred to as measures ofperformance (MOP) and not measures of effectiveness (MOE).

It is therefore unacceptable in evaluating and reporting operational effectiveness and suitability to parse requirements and narrow the definition of mission accomplishment so that MOP are confused with MOE. Basing evaluation solely on performance parameters can obscure the fundamental fact that the system being fielded is not operationally effective or suitable when used, in conjunction with interrelated systems, in combat by a typical military crew or unit. A narrow focus could also lead to erroneously evaluating as operationally ineffective a system that fails to meet certain performance parameters but nonetheless provides measurable improvement in mission accomplishment.

As has been the case since DOT&E was formed, my reports will continue to include a definite statement of the operational effectiveness and operational suitability of a system based upon its use to accomplish combat missions. All operational test plans from this date submitted for approval by DOT&E will contain explicit language acknowledging the requirement that operational test authorities also render evaluations on this same basis.

The Congress and the Secretary ofDefense have entrusted the operational test community with unique authority and responsibility to speak with clarity when rendering judgments of operational effectiveness and suitability. Working together, we must uphold this trust.

!1~~ ((Michael Gilmore

Director cc:

USSOCOM J8-0