MDA_Directive_8315.01_MS_VVA_w_CH1_Apr2011.pdf

MISSILE DEFENSE AGENCY

MODELING AND SIMULATION (M&S), VERIFICATION,

VALIDATION, AND ACCREDITATION (VV&A)

MDA DIRECTIVE 8315.01

January 21, 2009 Incorporating Change 1, April 6, 2011

Office of Primary Responsibility: Director for Engineering (MDA/DE)

DEPARTMENT OF DEFENSE

MISSILE DEFENSE AGENCY

7100 DEFENSE PENTAGON

WASHINGTON, DC 20301–7100

Change 1, April 6, 2011 2

DE 8315.01

SUBJECT: Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A)

References: Please See Enclosure 1.

1. REISSUANCE AND PURPOSE. This Directive reissues reference (a) to update the policy and responsibilities for implementing the Verification, Validation, and Accreditation (VV&A) of the Missile Defense Agency (MDA) Modeling and Simulation (M&S) and associated data at the System, Element, and Component levels. MDA uses system models to evaluate performance of the Ballistic Missile Defense System (BMDS) and its Elements; formulate evolution plans; guide investment strategies; develop system specifications; integrate and test the system; develop tactics, techniques, and procedures;

train operators; and verify and assess system performance. Therefore, the credibility of these M&S and the data underlying them are critical. Properly conducted VV&A establishes credibility and increases confidence in the M&S that provide a cornerstone for the Agency’s simulation-based development and test approach and enables the BMDS Operational Test Agency (OTA) independent M&S accreditation assessments for use in BMDS performance assessments.

2. APPLICABILITY AND SCOPE. This Directive governs the VV&A, and accreditation of the BMDS, its Element, and Component M&S necessary for performance evaluations; planning, acquiring, designing, developing, assessing, testing, training, and operating the BMDS as required by the Defense Acquisition Guidebook, of M&S used to simulate the BMDS, including its Element and Component M&S, necessary for performance evaluations; planning, acquiring, designing, developing, assessing, testing, training, and operating the BMDS as required by the Defense Acquisition Guidebook, which states that “All M&S used in test and evaluation must be accredited…” The VV&A Program shall develop a VV&A strategy that results in the accreditation of the models and simulations to represent the physical reality of the Operational BMDS. This strategy is to address individual M&S Verification and Validation (V&V), interim or partial accreditation of specific assessment and test events, BMDS Campaign, VV&A, and be documented in a BMDS Campaign M&S VV&A Plan. By integrating element data, algorithms and models into our system models, we assure the most accurate and current representations of our Elements and can capitalize on the wealth of Element subject matter expertise. Therefore, this Directive includes

MDA 8315.01, January 2009

Change 1, April 6, 2011 3 element data and M&S that are integrated into the system-level M&S. MDA must also have insight and confidence in Element and Component internal VV&A practices.

3. DEFINITIONS. Terms used in this Directive are defined in Enclosure 2.

4. POLICY

a. All Elements and Components of the BMDS shall fund and perform V&V of their Element/Component high fidelity reference M&S, their end-to-end Element M&S, and the representations or M&S of their Element/Component in the BMDS simulation and system-level events in accordance with references (b) and (f). High fidelity reference M&S are used to generate input data for system M&S, to predict Element performance for ground and flight tests, or to validate Element or system-level M&S.

b. All Elements and Components shall develop matrix of key data elements and parameters that include key sensitivities and areas of uncertainty relevant to predicting BMDS performance in order to drive an effective and efficient MDA Test Program, with the ultimate goal of gaining confidence in representation and prediction of BMDS performance.

c. All Elements shall execute a formal V&V program for their Element M&S and element data and representations used in system-level M&S. The Element and Component V&V programs shall be formally documented, annually coordinated through the Verification, Validation and Accreditation Working Group (VVAWG), and approved by the Modeling and Simulation Program Directorate (MDA/DES). The Element V&V program should update the assessment of models for every version generated, and shall produce a certification assessment for every event.

d. Element and Component offices shall appoint a technically qualified lead Government V&V Agent and provide that information to the Director for Engineering (MDA/DE) in writing. The agent’s primary responsibility shall be the coordination of the Element and Component office VV&A activities, and their responsibilities should be clearly stated in their individual performance objectives.

e. Element and Component offices and MDA/DE shall perform comprehensive M&S validation. This includes identification of data elements, key factors, and critical parameters based upon sensitivity analysis. This includes identification of key functions, Critical Engagement Conditions (CECs) and Empirical Measurement Events (EMEs) and associated rationale, operational range, data elements, and key test points based on uncertainty in M&S. Predictive and post-flight reconstructive modeling, in conjunction with flight and ground tests, or real-world events if available, will be used to anchor models and simulations used in MDA events, to include System-level Post-Flight

Change 1, April 6, 2011 4

Reconstruction and Post-Ground Reconstruction. One of the primary objectives of the Agency's test program is to anchor the system M&S. However, since Since real world data is limited, other validation techniques shall be used to ensure comprehensive M&S validation per reference (f).

f. Element and Component Offices will provide both predictive and post-analysis data collection, V&V results to DES for standardized reporting of integrated system-level M&S to MDA senior leadership.

e. g. Element, Component offices, and MDA/DE will execute a validation evaluation program across MDA, which will report status to senior leadership on a routine basis.

The use of validated models helps the Agency overcome testing limitations and extend capability assessments to a variety of engagement conditions that we cannot physically test due to cost or range safety constraints.

f. h. Operational M&S performance test parameters, acceptability criteria, and MDA BMDS systems engineering verification criteria shall be baselined and used for all MDA V&V efforts including, but not limited to, anchoring and benchmarking. These criteria identify accreditation parameters, their expected values, and the specified accuracy required for each measure. Additionally, M&S used in MDA events shall be evaluated on their use of MDA standard inputs and environments data, such as the common threat data and Earth Geopotential Model 98. Using these criteria and the event objectives developed by other MDA and warfighter organizations for the intended use, the event accreditation agents define specific event M&S performance parameters and acceptability criteria.

g. i. MDA/DE shall fund and perform integrated MDA VV&A at the system-level as required to support MDA builds; formally document MDA VV&A practices; formally document system-level MDA V&V plans and integrated reports plus accreditation plans and packages; and formally define, track, and report V&V requirements and status.

h. j. MDA/DE shall operate a problem reporting system to capture anomalies and exceptions to M&S discovered during the VV&A process. This system shall be used to provide feedback to M&S requirements, event execution leads, and potentially system specifications.

i. k. MDA/DE shall establish and lead the VVAWG to coordinate implementation of this policy, the VV&A processes, and event implementation details. All MDA-related VV&A organizations, to include Element/Component Program Office V&V Agents, shall participate in the VVAWG.

j. l. Element and Component offices shall fund and provide Subject Matter Experts (SMEs) to support major MDA events that require system-level VV&A. MDA/DES

Change 1, April 6, 2011 5 shall provide qualified technical teams (Federally Funded Research and Development Center, system software SMEs, and VV&A analysts) to work with Element and Component office SMEs to integrate all event VV&A activities and to produce appropriate system-level VV&A plans and reports. Analysis efforts shall be integrated into the VV&A process and shall use the key data and parameter matrices for evaluating predicted versus measured performance.

k. m. MDA Element and Component offices shall structure their requirements for deliverables, organizational conflict of interest provisions, and award fee plans in such a manner as to promote the effective transfer of necessary and critical data and M&S to the Government, sufficient to allow the Government to use the data and M&S for approved Government purposes as determined by the MDA.

(1) Proprietary M&S may result in restricted access to data and information necessary for the full and independent evaluation of the M&S. Proprietary M&S can have a valuable and an appropriate role in system development. However, access limitations associated with proprietary M&S may prevent or impede the openness necessary for detailed and unbiased evaluation of the M&S. Contract structures and deliverables, proprietary interests, and intellectual property rights cannot be an impediment to the objective of fully evaluating M&S. MDA encourages our industry partners' cooperation in achieving these policy goals through the use of appropriate solicitation requirements, contract structure, deliverables, organizational conflicts of interest provisions, and award fee plans. The Contracting Directorate (MDA/DAC) and Element/Component Program offices shall structure contracts through deliverables, organizational conflicts of interest provisions, and award fee plans to promote the transfer of necessary and critical M&S and associated data to the Government.

(2) MDA shall not acquire or use M&S data with limited or restricted rights (see the Defense Federal Acquisition Regulation Supplement (DFARS) Subparts 227.71 and 227.72) without prior coordination by the MDA Program Manager for M&S with the Director for Acquisition Management (MDA/DA), General Counsel (MDA/GC), and the approval of the MDA Director (MDA/D). Prior to seeking such approval, consider using independent third-parties, such as blind or shadow contractors, to enable a complete evaluation of the M&S data.

(3) Generally, contractors may not be required to provide the Government rights in data as a condition for being responsive to a solicitation or eligible for contract award.

See DFARS 227.7103-1 and 227.7203-1 for additional information.

Change 1, April 6, 2011 6

5. RESPONSIBILITIES

a. The MDA Director (MDA/D) is the approval authority for:

(1) M&S accreditation reports and letters.

(2) Acquisition or use of M&S data with limited or restricted rights.

b. The Director for Integration and Fielding (MDA/DF) and the Director for Test (MDA/DT) The Director for Test (DT) and the Deputy Director (DD) will work with MDA/DE and the Elements/Components to define a test program that anchors the system models and simulations across the expected operating spectrum and supports the Combatant Commands’ ability to develop concepts of operation and tactics, techniques, and procedures. This will be supported by Warfighter Operational Support (DTW), and Warfighter Strategic Integration (DDW).

c. The Director for Engineering (MDA/DE) will:

(1) Accredit system models and simulations for their intended uses as required to support BMDS Builds.

(2) Oversee the coordination of MDA event intended uses and test objectives among stakeholders, including MDA/DT and MDA/DF DD, the Director for International Affairs (MDA/DI), the Director for Agency Operations (MDA/DO), and the Director of the BMDS OTA.

(3) Work with MDA/DT, MDA/DF DD, and the Elements/Components to define a test program that anchors the system models and simulations across the expected operating spectrum through the use of test flight data.

(4) Conduct an annual review of the VV&A program.

(5) Lead the coordination among MDA/DT, MDA/DD, and the Element/ Component Program Offices, to jointly define a test program, employing data elements, key functions, and critical parameters defined at the Element and system-level that provides the appropriate validation and anchoring data for MDA M&S.

(6) Lead the V&V program that provides methods for Elements, Components, BMDS OTA, and M&S community to develop matrix matrices of key data elements and parameters. Matrices will identify key sensitivities and areas of uncertainty relevant for M&S to predict BMDS performance. Matrices will be used to identify gaps in M&S and to guide effective and efficient MDA Test Program.

Change 1, April 6, 2011 7

(7) Designate an Assistant Program Director for VV&A, responsible for monitoring and guiding all MDA M&S VV&A activities, who will establish quarterly management reviews and be responsible for monitoring and controlling funding issues related to the performance of VV&A.

(8) Chair Appoint a chair for the VVAWG. The VVAWG provides the forum to coordinate VV&A activities within MDA. It allows geographically distributed VV&A efforts through a formal periodic meeting process. The VVAWG will establish policies and processes for ensuring coordination among the various VV&A activities through the VVAWG meetings. Processes should include a set of common VV&A metrics for general use. Members include: MDA/DE, MDA/DT, MDA/DF DD, Element Program/Component Program Offices, M&S V&V agents, accreditation agents, BMDS Element and Component SMEs, the Director for Quality, Safety and Mission Assurance (MDA/QS), and the Director, BMDS OTA.

(9) Ensure close coordination between the VV&A community and other organizations, including M&S developers, Elements, Components, and users.

(10) Provide system M&S definitions to users, M&S developers, and VV&A agents. Element interaction specifications are particularly important.

(11) Conduct detailed review of key data elements and parameters necessary within system-level M&S to enable effective and thorough prediction of system-level performance of the BMDS.

(12) Provide MDA/DT with key factors, parameters, and test conditions necessary in ground and flight testing to support V&V of BMDS M&S.

(13) Conduct System Post-Flight Reconstructions as identified in the Integrated Master Test Plan (IMTP).

d. The Contracting Directorate (MDA/DAC) will assist the VVAWG to acquire M&S rights appropriate for full and independent evaluation of MDA M&S tools. Contract issues relating to proprietary M&S should be coordinated with MDA/GC prior to contract implementation.

e. M&S Developers will maintain M&S development documentation, conduct developmental V&V, and maintain configuration management controls over M&S development effort. Every release/change must be uniquely identified.

f. Responsible M&S User Organizations are those organizations that sponsor or conduct analyses, exercises, wargames, training, or operating the BMDS using M&S or M&S related data. MDA user organizations will:

Change 1, April 6, 2011 8

(1) Ensure that VV&A results (caveats, capabilities, and limitations) accompany event results.

(2) Identify M&S system configurations, pedigree of component models, software versions, threats, and scenarios for each intended use.

(3) Invite Element participation in planning, execution, and analysis for each intended use.

(4) Notify MDA/DE of any unscheduled use.

g. Element/Component Program Directors will:

(1) Conduct detailed review of key data elements and parameters necessary within Element/Component M&S to enable an effective and thorough prediction of system-level performance within the BMDS.

(2) In coordination with MDA/DE, identify and provide to MDA/DT with key factors, parameters, and test conditions necessary in ground and flight testing to support V&V of Element/Component M&S. functions, CECs, EMEs, and associated rationale, operational range, data elements, and key test points based on uncertainty in M&S.

(3) Verify and validate their Element/Component models using formally documented processes that are presented at the VVAWG and submitted to MDA/DE for approval annually.

(4) Fund and perform VV&A of respective Element/Component level M&S and V&V of Element/Component representations in system-level M&S, including anchoring to flight test data.

(5) Provide to MDA/DE the necessary V&V plans and reports, in accordance with MIL STD 3022, “Documentation of Verification, Validation, and Accreditation (VV&A) for Models and Simulations.” Also, provide the M&S data for system-level assessments and activities as required by the VV&A event milestones.

(6) Participate in the VVAWG.

h. The Director for Test (MDA/DT) will include system-level M&S requirements for validation data in the integrated master test plan IMTP and ensure that test data is available to MDA/DE by supporting data extraction and distribution as required.

Change 1, April 6, 2011 9

i. The Director for Quality, Safety & Mission Assurance (MDA/QS) will participate in MDA VV&A related activities.

j. V&V Agents will provide evidence of the fitness of the M&S for the intended use by ensuring that all the V&V tasks are properly executed and documented. V&V evaluations shall assess performance parameters and CECs, and when made appropriate by those assessments address impacts and root cause analysis. V&V evaluations shall also address scenario verification, including the threat/target implementation and the blue force laydown. Documentation includes plans, reports, and supporting evidence or data.

The V&V report content shall be delivered prior to the accreditation package (letter and report). Documentation V&V documentation shall be in accordance with MIL STD 3022.

k. BMDS Accreditation Agents will provide the accreditation assessment and recommendations. The BMDS Accreditation Agent works with the V&V team to ensure that all the necessary evidence regarding M&S fitness for the intended use is obtained;

collects and assesses the evidence; and, provides the results to the accreditation authority.

A final accreditation report shall be submitted 30 days after receipt of all V&V reports supporting the BMDS Build. MDA/DE shall provide accreditation assessment of all M&S used in support of MDA test and fielding objectives. In addition, MDA/DE shall provide V&V data sets necessary to support BMDS OTA independent analysis.

6. SECURITY. The VV&A process is geographically dispersed across a number of executing organizations. The classification of VV&A data and related team interactions and the methods for data distribution and transmittal shall conform to the BMDS security classification guides and relevant security practices and procedures.

7. REPORTING REQUIREMENTS

a. MDA/DE will develop a VV&A Campaign Plan for each BMDS Build.

b. For specified individual events within a build campaign, MDA/DE shall generate system-level MDA interim event V&V plans, which provide V&V planning guidance to Element/infrastructure program offices, oversee V&V activities, and report any issues requiring senior leadership assistance. Also, MDA/DE shall generate integrated MDA event V&V reports, which contain the Element/Component V&V reports, and provide briefings as required to senior leadership.

c. MDA/DE shall review and sign the event interim V&V plans and reports.

d. Building on Campaign event interim V&V reports, MDA/DE will develop a Campaign Accreditation Report. The accreditation authority, MDA/DE, shall review and

Change 1, April 6, 2011 10 sign the Campaign event accreditation letters. Results of BMDS OTA M&S accreditation assessments are will be reported to MDA leadership.

8. MANAGEMENT CONTROLS. The V&V agents and the accreditation agents shall provide monthly status and financial reports to MDA/DE.

9. EFFECTIVE DATE. This Directive is effective immediately.

Enclosures

1. References, continued

2. Definitions

Change 1, April 6, 2011 11

ENCLOSURE 1

REFERENCES, (continued)

(a) Missile Defense Agency Directive 8315.01 “Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A),” September 26, 2007 January 21, 2009 (hereby cancelled)

(b) Missile Defense Agency Directive 8315.02, “Modeling and Simulation,” January 13, 2009

(c) Missile Defense Agency Integrated Program Policy Implementation Guide, Version 2.0, 18 July 2005

(d) Ballistic Missile Defense System Security Classification Guide, 26 April 2004

(e) Defense Acquisition Guidebook, June 2006 (https://akss.dau.mil/dag/welcome.asp)

(f) Missile Defense Agency Verification and Validation Methodology Guide, May 2006

(g) DoD (DMSO) Verification, Validation, and Accreditation (VV&A) Recommended Practices Guide (RPG), http://vva.dmso.mil/, Build 2.5

(h) Operational Test Agency Accreditation Plan for the Ballistic Missile Defense System, 12 December 2005

(i) Operational Test Agency Team Operational Assessment Plan for the Ballistic Missile Defense System, Block 06, 26 June 2006

(j) Missile Defense Agency Assurance Provisions (MAP), Rev A, October 2006

(k) MIL STD 3022, Documentation of Verification, Validation, and Accreditation (VV&A) for Models and Simulations, January 2008

Change 1, April 6, 2011 12

ENCLOSURE 2

DEFINITIONS

Accreditation. The official certification that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose.

Accreditation Analysis. The analysis of the capability, correctness, accuracy, and usability of a model or simulation relative to a specific intended use and the risks associated with that specific use. Areas of emphasis for accreditation are the BMDS timeline/kill chain, threats, environment, sensors, weapons/interceptors, battle managers/command and control, and fire control and communications.

Agent. The individual, group, or organization designated by proper authority to perform a specific VV&A activity (verification, validation, accreditation) on a model, simulation, or simulation federation.

Anchoring. The comparison of selected data elements from a simulation/model to equivalent test data in order to validate the model. The test data and the model output are compared in detail and differences analyzed. [Differences may be analyzed as acceptable or requiring M&S modification.] Anchoring can be accomplished in three main ways:

statistically, technical judgment, or a combination of both.

Benchmarking. The comparison between a model's output and the outputs of other models or simulations, all of which represent the same input and environmental conditions. [Differences may be analyzed as acceptable or requiring M&S modification.] Benchmarking is often used when suitable anchoring data is not available; however, to be reliable, benchmarking used for validation requires accredited referent models.

Component. Subsystem, assembly, or subassembly of logically grouped hardware and software that performs interacting tasks to provide BMDS capability at a functional level.

Element. Executing organization for a complete, integrated set of components capable of autonomously providing BMDS capability.

System. A set of different elements so connected or related as to perform a unique function not performable by the elements alone. The most important and distinguishing characteristic of a system, therefore, is the relationship among the elements.

Validation. The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model. Because testing cannot be comprehensive or conclusive, other validation techniques should be employed to gain a full understanding of the simulation’s

Change 1, April 6, 2011 13 behavior. Among the other techniques employed are data validation, sensitivity analyses to evaluate input/output relationships, comparison with other models and simulations known (or assumed) to have validity in the operating range required (benchmarking), and the results of SME reviews of M&S outputs (face validation).

Verification. The process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications.

Verification of the correctness, consistency, and completeness of requirements is an essential V&V task.

Verification, Validation, and Accreditation (VV&A). Three interrelated but distinct processes that gather and evaluate evidence to determine the simulation’s intended use, the simulation’s capabilities, limitations, and performance relative to the real-world objects it simulates.