Bidders Library Operational Test and Evaluation - DISA Test Evaluation Process Guidebook.docx

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

TEST AND EVALUATION

PROCESS GUIDEBOOK

VERSION 1.2

Distribution C Distribution authorized to U.S. Government Agencies and their supporting contractors. This document describes procedures or guidelines for test and evaluation. Such information may be unclassified but is considered sensitive and its distribution is limited to entities that need it for Government purposes or to conduct official business for DOD. Other requests for this document must be referred to Commander, JITC, and ATTN: JTA, P.O. Box 12798, Fort Huachuca, Arizona 85670-2798.

OCTOBER 2020

UNCLASSIFIED//FOR OFFICIAL USE ONLY

JOINT INTEROPERABILITY TEST COMMAND

TEST AND EVALUATION PROCESS GUIDEBOOK

VERSION 1.2

OCTOBER 2020

Submitted by: Michael Koester Chief, Operational Test & Evaluation & Enterprise Services Division

Prepared Under the Direction of:

Akalnesh Mamo Joint Interoperability Test Command Fort Meade, Maryland

(This page intentionally left blank.)

DOCUMENT INFORMATION

DOCUMENT CHANGE HISTORY

FOREWORD

The Joint Interoperability Test Command (JITC) Test and Evaluation (T&E) Process Guidebook describes the T&E approach, activities, and tools necessary to establish a standard and repeatable T&E process to be applied to Defense Information Systems Agency (DISA) projects. The term “requirement” is used throughout this document to refer to use cases, features, epics, user stories, and product-backlog items for an Information Technology (IT) project. The term “project” is used throughout this document to mean any planned DISA IT undertaking as defined in DISA Instruction 610-225-2 “Acquisition Oversight and Management” dated 19 February 2015. The implementation and execution of this Process Guidebook, in coordination with the DISA Development and Business Center (DBC) offices, will standardize T&E for DISA projects.

A fundamental tool in ensuring the standardization of DISA projects is the JITC T&E Scorecard, which was designed to standardize the reporting of project status, from a T&E perspective. The JITC T&E Scorecard Areas of Evaluation (AoEs), and associated attributes, should also be used as the basis for a project’s analysis structure. The JITC T&E Scorecard is based on a combination of T&E lessons learned and the service management best practices found in the Department of Defense Enterprise Service Management Framework (DESMF).

The establishment of the Requirements and Analysis Office (RAO) and the Mission Partner Engagement Office (MPEO) within the DBC provides the opportunity for integrated and ongoing collaboration between these offices and JITC. JITC is positioned to work closely with these offices as part of mission partner engagement and requirements development to execute early T&E activities, resulting in project resource efficiencies. One output from JITC is Early Test Involvement (ETI), which provides an initial T&E Risk Assessment (RA) and the T&E Management Approach Recommendation (TEMAR), which are both described in this Guidebook.

The two key drivers that make this process an effective approach are: (1) dedicated ETI support during requirements definition and analysis activities and (2) alignment to the JITC T&E Scorecard AoEs throughout the project lifecycle. This T&E process ultimately provides leadership and other key stakeholders with the information required to make well informed fielding decisions.

The steps in the process from early test involvement during project initiation through the reporting of completed T&E include approaches for designing the Analysis Structure (AS) and T&E execution throughout the project lifecycle. This Guidebook identifies the project-related T&E phases and milestone where the T&E Scorecard and ETI products (contained in the appendices) are presented to project decision makers.

This T&E Process Guidebook, the JITC T&E Scorecard Guidebook, and supporting products, to include the T&E Scorecard Template, are located at: https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/ i iv

TABLE OF CONTENTS

Page

APPENDICES

APPENDIX A TEST AND EVALUATION SCORECARD A-1

APPENDIX B ANALYSIS STRUCTURE (EVALUATION FRAMEWORK AND

LIST OF FIGURES

Figure D-1. Project Risk Determination D-11

Figure F-1. Evaluation Strategy Process F-1

LIST OF TABLES

INTRODUCTION AND SCOPE

The Joint Interoperability Test Command (JITC) Process Guidebook establishes a standard process for the effective evaluation of new or enhanced Defense Information Systems Agency (DISA) Information Technology (IT) projects. The term “requirement” is used throughout this document to refer to use cases, features, epics, user stories, and product backlog items for an IT project.

The term “project” is used throughout this document to mean any DISA IT project as defined in DISA Instruction 610-225-2 “Acquisition Oversight and Management” dated 19 February 2015 and further identified in section 1.1, Applicability.

Historically, the concepts of early Test and Evaluation (T&E) involvement and T&E standardization have not been consistently implemented. The T&E Process (with a foundation in the JITC T&E Scorecard) provides the following value-added capabilities:

· Seamless integration of T&E into Project Management Plans, by establishing a positive working relationship between JITC and Program Managers (PM)s

· Early identification of potentially project-threatening difficulties (i.e., meeting timelines, defining requirements, and/or testing to requirements) through initial and follow-on risk assessment

· Support for the requirements-analysis process by providing a standardized framework for requirements analysis and definition

· Based on initial requirements analysis and risk assessment, recommendations of the best analysis structure and test strategy to the validate that the project is satisfying mission needs

· Assessment of whether a fielded project actually met the anticipated user needs, whether all of the project’s capabilities were used, or if a set of project’s capabilities was not used as intended

Consistent application of the T&E process described in this Guidebook will ultimately provide leadership and other key stakeholders with the information required to make well informed acquisition and fielding decisions.

This T&E Process Guidebook, the JITC T&E Scorecard Guidebook, and supporting products (including the T&E Scorecard Template) are located at: https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/.

T&E Scorecard Computer-Based Training (CBT) is available, at https://nit-jitc.nit.disa.mil/tools/lms/scorecard/.

Before preparing or updating the T&E Scorecard, testers should use this training to familiarize themselves with the concepts underlying DISA T&E and the Scorecard.

UNCLASSIFIED//FOR OFFICIAL USE ONLY

Applicability

This Guidebook applies to the T&E of DISA activities that are categorized by acquisition level as not being under the oversight of – and not requiring full reporting to – the DOD Director, Operational Test and Evaluation (DOT&E). This is defined in DISA Instruction 610-225-2 “Acquisition Oversight and Management” dated 08 June 2017. See Enclosure 5 of DOD Instruction (DODI 610-225-2) for the “Documentation or Information Requirements Matrix” for DISA activities. Use of this Guidebook is intended for the project-appointed test lead and test team members on DISA projects.

TEST AND EVALUATION

The “T” - Test is a structured activity performed under operationally representative conditions sufficient to allow for the collection of data needed to support specific evaluation objectives. For a more detailed definition see “Test” in the Defense Acquisition University (DAU) Glossary -https://www.dau.edu/glossary/Pages/Glossary.aspx

Testing done during system development and integration (Developmental Test) is considered Technical Verification for the purposes of this document. Technical Verification (TV) is usually conducted in a laboratory environment. TV includes acceptance testing (government and/or user), table top reviews, and other activities necessary to meet entrance criteria for operational testing. This is further detailed in Section 2.3.

Testing done with operationally representative users and conditions for the purposes of determining mission readiness (to include training, operations and maintenance -- typically referred to as Operational Test and Evaluation) falls under the categorization of Operational Validation (OV) for the purposes of this process. See Section 2.4 for more on OV.

The “E” - Evaluation is the process of logically assembling and analyzing data and comparing that data to expected performance to aid in decision-making. Evaluation may involve review and analysis of qualitative and quantitative data obtained from design reviews, hardware inspections, modeling and simulation, hardware and software testing, measures review, and operational equipment usage (DAU Glossary -https://www.dau.edu/glossary/Pages/Glossary.aspx.)

T&E SUPPORT TO THE DISA DEVELOPMENT AND BUSINESS CENTER

Establishment of the Requirement and Analysis Office (RAO), the Mission Partner Engagement Office (MPEO), and JITC within the DISA DBC provides the opportunity for early collaboration among these offices. This collaboration is the catalyst for implementation of T&E involvement activities across the lifecycle of a project. JITC is positioned to work closely with these offices as part of MPEO and requirements analysis to provide feedback in the form of a Requirements and Capabilities Gap Analysis, conduct an initial Risk Assessment (RA), and produce a T&E Management Approach Recommendation (TEMAR) during the Project Initiation Phase.

DISA also established the Infrastructure Development (ID) and Services Development (SD) Directorates within the DBC. JITC will leverage outputs from early collaboration and requirements analysis to provide continued T&E support to the development directorates throughout the project lifecycle.

The Component Acquisition Executive (CAE) is the Milestone Decision Authority (MDA) for DISA and chairs the Acquisition Review Board (ARB). The CAE is the designated entity at the organizational level with the ultimate responsibility for DISA acquisition programs and projects. The CAE retains the authority to delegate acquisition decisions down to the center level, taking cost, schedule, and performance reporting into account. The CAE generally reports to higher authority, including DISA senior leadership and, at times, Congressional oversight.

DISA’s tester of choice, JITC, is a non-Service Operational Test Agency for IT/National Security Systems. JITC is also the Department of Defense (DOD) Joint Interoperability Certifier, specializing in joint service communications and enterprise service interoperability. JITC provides risk-based test, evaluation, and certification services, tools, and environments to the Joint Warfighting IT capabilities that are interoperable and support mission needs.

Directorates within DISA, such as ID and SD, provide assured communications through the delivery of optimized cyber infrastructure solutions for DISA’s global partners. They also design, develop, integrate, and transition Business, Enterprise, and Command and Control systems, services and capabilities for DISA, DOD, other U.S. Government Agencies, and our Allies. Each DISA project falls within an appropriate directorate; a specific person within a directorate is assigned the duty of Project Manager (PM) for overall project management and status reporting.

The PM is responsible at the project level for balancing the many factors that influence project cost, schedule, and performance. The PM interprets the DODI 5000 Series regulations, and tailors its application to the project in accordance with the DISA IT Acquisition Guidebook. The PM’s role is to provide quality, affordable, supportable, and effective defense systems to the warfighter as quickly as possible. The Test Lead reports to the PM for T&E across the project lifecycle.

T&E PROCESS PHASES SUPPORT THE PROJECT LIFECYCLE

Once a decision to move forward with the acquisition is made by the Service Portfolio Council (SPC), a PM is assigned, and the project lifecycle begins. JITC is positioned to support DISA projects during the Project Initiation Phase, with the ETI effort and the T&E Scorecard. During test and evaluation, a project is technically verified and operationally validated to determine the extent requirements and user needs are met in support of an acquisition decision.

As figure 1 illustrates, a T&E phase has been devised for each phase of a project. The products and service offerings from JITC described in this Guidebook contribute directly to reporting and recording requirements for decisions and reviews.

Figure 1. Key Test and Evaluation Phases Integrated T&E is intended to result in resource efficiencies (saving time, money, people, and assets) and an enhanced data set for separate evaluations, with the goals: to have test teams collaborate along the project lifecycle; to plan events to produce credible qualitative and quantitative data useful to all evaluators; to address developmental, sustainment, and operational issues; and to satisfy multiple objectives without compromising the participating test organization's test objectives. Integrated T&E focuses the entire test effort on designing, developing, and producing a comprehensive plan that coordinates all test activities to support evaluation results for decision makers.

REQUIREMENTS ANALYSIS PHASE

The Project Initiation Phase is common to all DISA acquisition models described in the DISA IT Acquisition Guidebook. JITC supports project initiation via ETI, which primarily consists of Requirements Analysis. The JITC ETI Team provides T&E feedback during a project’s requirements definition to help mitigate risks to the project (and the supported mission). The intent of the Project Initiation Phase is for the JITC ETI Team to be involved at the earliest phase of the project up to the initial acquisition decision and to encourage the appropriate level of JITC participation throughout the project lifecycle.

The main deliverables created by the ETI Team from requirements analysis is an initial RA report and TEMAR, which includes the “Post Requirements Phase” T&E Scorecard (See Appendix A for details). Figure 2 shows the relationship of JITC products and support to a project’s reporting and approval requirements. The initial TEMAR will be delivered with (or incorporated into) the project requirements package.

The T&E Scorecard is used to brief the SPC and/or ARB as needed.

Figure 2. Requirements Analysis

REQUIREMENTS ANALYSIS AND FEEDBACK

During Project Initiation, requirements are identified, analyzed, translated, refined, and validated using various approaches to clearly convey the true operational need. Collaboration with all stakeholders (e.g., customers, mission partners, operations personnel, enterprise architects, and contracting officials) is critical during the shaping of requirements. JITC participation fosters the development of requirements that align with the T&E Scorecard AoEs.

The Requirements Definition process, described in figure 3, includes output from the Mission Partner Engagement Office (MPEO) and RAO Integrated Product Team (IPT) and results in a Requirements Package, which includes the acquisition strategy and project execution plan. The Requirements Package is a streamlined document that consolidates many of the DODI 5000-series requirements. The JITC ETI process is specifically designed to integrate with the Requirements Definition process and aligns with the processes described in the DISA IT Acquisition Guidebook; however, the processes described in this Guidebook can also be applied for projects following other acquisition guidance.

Figure 3. Requirements Definition Process

CONDUCT RISK ASSESSMENT

The initial RA (See Appendix D for the RA Template.) involves risk identification, likelihood of risk occurrence, and estimation of the mission impact for each of the JITC T&E Scorecard AoEs. The risk determination for each AoE is then aggregated to determine an overall risk level for the project.

As shown in the Risk Assessment Model (Figure 4), the initial and updated RAs for DISA projects are conducted based on the eight T&E Scorecard AoEs, employing the risk categories identified by DOT&E memorandum of 14 September 2010, “Guidelines for Operational Test and Evaluation of Information and Business Systems.”

Figure 4. Risk Assessment Model The outcome of the RA is an overall determination of risk for the program. This overall determination of risk then establishes the basis for the project’s TEMAR. The roles and responsibilities and levels of testing required are based on the outcome of the RA and described in the TEMAR. While numerous types of operational test events were historically conducted with varying objectives and rigor, it is imperative that a standard is established for the T&E Scorecard based evaluation of DISA projects. Note that DISA Acquisition Category I, Information Assurance) IA, II, and III programs should continue to follow DOT&E Guidance for formal operational testing. The T&E Scorecard will be used as the standard basis for their Analysis Structure, a gauge to determine readiness for operational testing, and a guide to focus the developmental test effort.

Based on the outcome of the risk assessment, the T&E lead (on behalf of the PM), will establish the project T&E roles and responsibilities by utilizing the following assessment criteria:

· High Risk: The PM is responsible for Technical Verification (TV); JITC will conduct OV in the form of Operational Assessment (OA). JITC must review and concur with TV/development testing (DT) plans and reports. JITC may also support, participate in, or lead TV events as requested by the PM.

· Medium Risk: The PM is responsible for TV; JITC will conduct OV in the form of a Field Assessment (FA). JITC must review and concur with TV/DT plans and reports. JITC may also support, participate in, or lead TV events as requested by the PM. Or, the PM is responsible for TV and the OV, in some form of acceptance testing; JITC must review and concur with TV/DT and OV/Operational Testing (OT) plans and reports. JITC may support, participate in, or lead test events as requested by the PM.

· Low Risk: The PM is responsible for all testing; JITC may provide feedback on plans and reports and/or support, participate in, or lead test events as requested by the PM.

The results of the initial T&E RA are used as the basis for development of the TEMAR. The initial TEMAR is updated as requirements are refined, and/or risks are mitigated during design. Any changes to the AoE level risk determinations or the aggregated risk level for the project will require an update to the TEMAR. T&E RA results will be used to determine the type and rigor of testing needed to address each AoE.

Each AoE is assessed to determine the level of risk that a TEMAR must adequately mitigate, and subsequently provide a well-defined level of T&E effort associated with testing. Each AoE must be accounted for and the risk measured based on “likelihood” and “impact.” The updated risk values will be used to determine the risk level for each AoE. The risk level for each AoE will be used to determine the T&E activities necessary for proper risk-based evaluation of the project.

PRODUCE INITIAL TEMAR

The initial TEMAR (See Appendix C for the TEMAR Template.) is a key part of the requirements package generated by the RAO-led IPT. The TEMAR describes the overall T&E objectives, the roles and responsibilities of key stakeholders, and the risk- based outcomes as shown in Figure 4. The TEMAR document will also describe the general test events, exercises, and environments required to evaluate the project, based on the T&E Scorecard AoEs.

The TEMAR advises the project on appropriate T&E activities to execute during the project lifecycle, including:

· Project-Specific T&E Strategy Development

· Overall T&E Tracking, to include the T&E Scorecard

· Technical Verification (Plan, Execute, Report)

· Operational Validation (Plan, Execute, Report) (when applicable)

· Approve T&E Strategy, Plans, and Reports

The elements contained in the project Requirements Package, along with the RA and TEMAR, inform the first key decision point (typically the SPC). At this point – depending on the level, resources required, and type of project – the Decision Authority may provide approval to proceed to an ARB and establish the project baseline.

The combination of the requirements definition, analysis, and design process outputs results in requirement and design packages that contain a full analysis of the project. JITC’s contribution is the initial TEMAR, which includes the results of the requirements analysis, the RA results, and the initial project T&E Scorecard. The initial TEMAR will be reviewed and updated following project design analysis.

DELIVER REQUIREMENTS ANALYSIS T&E SCORECARD (T&E SCORECARD 1)

The analysis conducted in the TEMAR is based on the eight T&E Scorecard AoEs. A convenient by-product of the initial TEMAR is the production of the “Post-Requirements Analysis Phase” T&E Scorecard – the first required T&E Scorecard for a project – based on the TEMAR results. This T&E Scorecard is provided to the customer in Section 3 of the TEMAR.

DESIGN ANALYSIS

The next step in the project lifecycle is the Design, Technology, Development, and Planning (DTDP) Phase, as identified in figure 5. After the project’s design is documented – and in anticipation of the next key decision point (typically a Chief Engineer’s Panel) – the T&E RA should be updated. Based on the results of the updated RA, an updated TEMAR and Design Analysis Phase T&E Scorecard will be generated to ensure T&E remains aligned to the chosen acquisition strategy.

Figure 5. Design Analysis Phase

UPDATED RISK ASSESSMENT

The initial RA is reviewed to determine if satisfactory progress has been made toward addressing risk items identified during the project’s initiation phase. The initial overall risk ascribed to the project should still determine the level of JITC involvement in the project, unless a higher risk is determined. The results of the updated RA are factored into the updated TEMAR and “post-Design Analysis” T&E Scorecard.

UPDATED TEMAR

The T&E team builds the Data Source Matrix (DSM) that presents data collection for every Measure of Performance (MOP) in the project’s Evaluation Framework (EF). The resulting analysis structure is used to design required test events. The DSM identifies those tests where the data will be collected to support the resolution of the MOPs. Well defined MOPs are essential in identifying which tests to conduct, and the DSM illustrates correlation. The products are the operating procedures for the project’s T&E.

Using the initial TEMAR as the foundation, an update is developed after DTDP that incorporates the full analysis structure (Appendix B) and test strategy. (See Appendix E for details.) This updated TEMAR will be presented for approval prior to production.

The completed design articles for a project are evaluated against the requirements to verify the outcomes of previous activities and identify issues before any commitment to building the system is made. The ultimate design review, if successful, therefore triggers the product build and a move into testing prior to approval for initial fielding. The updated design articles, RA, TEMAR, and post-Design Analysis T&E Scorecard are presented for approval prior to the next acquisition decision.

HIGH-LEVEL EVALUATION FRAMEWORK

A high-level Evaluation Framework (EF) is composed by the T&E team to lay the foundation for all T&E strategy and execution. The required testable areas that will validate the project requirements are identified, and the areas of evaluation are categorized into hierarchical groups:

· At the highest level are Critical Operational Issues (COIs). COIs are required mission focus items (sourced from Concept of Operations, capabilities documentations, etc.) which must be evaluated

· From each COI, mission needs are further defined by Measures of Effectiveness (MOEs) or Measures of Suitability (MOSs) that focus on mid- level or sub-activities of those COIs

· As depicted in figure 6, Test Measures (or MOPs) target low-level tasks and more in-depth data that satisfy the higher level, requirements-based COIs through support of the mid-level MOEs or MOSs

Figure 6. Analysis Structure

DELIVER DESIGN ANALYSIS T&E SCORECARD (T&E SCORECARD 2)

The results of the updated TEMAR and the EF are aligned with the eight T&E Scorecard AoEs. Data from these products should lead directly to the Design Analysis Phase T&E Scorecard, which is once again delivered as part of the TEMAR. This T&E Scorecard should be prepared to reflect the readiness of the project to proceed to testing and can be presented – along with the T&E Strategy – to the Chief Engineers Panel (CEP) who will consider the design review before allowing the project to go forward.

The project-specific T&E strategy provides a basis for planning, executing, analyzing, and reporting data from test events. It describes the processes, resources, risks, tools, schedules, organizations, platforms, and related elements for a project. For example, if 99-percent operational availability is required, related management requirements are able to monitor it, the technology is available to achieve it, and tactics, tools, and procedures can be applied to instruct operators on how to maximize availability. If ETI is used, the T&E Strategy is delivered as part of the updated TEMAR. See Appendix E for more on test planning, T&E Strategy, and reporting.

TECHNICAL VERIFICATION

TV Phase (illustrated in figure 7) is conducted during system development and integration (usually in a laboratory environment). TV includes traditional DT activities that are focused on verifying the technical attributes of the IT solution and includes early review and exercise of project monitoring and support processes. Such activities add acceptance testing (by the government and/or user), table-top reviews of processes and procedures, and other necessary activities undertaken to meet entrance criteria for OV. Data from TV activities should be captured and made available for analysis and used to optimize subsequent T&E activities.

Figure 7. Technical Verification TV is performance oriented. During TV, functional characteristics of the project are exercised and/or tested against Critical Technical Parameters (CTPs), Key Performance Perameters (KPPs), information-exchange requirements, and other validated system requirements to establish the degree to which the project satisfies the documented specifications of the system and the monitoring aspects of the system. TV also provides a preliminary look at the required support processes.

The T&E Team should ensure that each attribute of the eight T&E Scorecard AoEs has at least one test case that will exercise appropriate technical performance. Every function should have an associated performance parameter that establishes how well and/or how fast the function must perform to support the user’s need.

The purpose of TV is to identify vulnerabilities, which may be fed back to systems designers, developers, and engineers, so that mitigations can be implemented to improve resilience.

The T&E Team should consider providing a preliminary T&E analysis in support of the TV. This analysis should include discussion of the following:

· Critical missions and mission functions

· System components associated with the critical missions/functions

· Critical developmental software items

· Cybersecurity testing and cybersecurity assessment parameters

· Initial Evaluation Framework, with consideration of software assurance, security controls, anti-tamper, and supply chain risk management

· Follow-on Evaluation Framework laying the course for OV

At the completion of TV, the results are weighed against the fixed attributes on the T&E Scorecard, and the Test Lead completes the “Post-Technical Verification” T&E Scorecard to deliver to the project-responsible Program Management Office (PMO) or Program Executive Office (PEO) for presentation to decision makers at the Preliminary Network Operations (NetOps) Readiness Review Board (Pre-NRRB) and/or ARB as requested.

Operational Validation OV (highlighted in in figure 8) is typically accomplished in an operational or realistic operational environment with a representative sampling of users from mission partners and – if available – actual operations and support personnel. OV should be focused on the users’ ability to accomplish their assigned mission. While testing in an operational environment with representative users is indeed a requirement for formal Operational Testing, the concepts of OV should be incorporated in all test events whenever practical.

Figure 8. Operational Validation The following are the recognized OV events for DISA projects:

· Operational Assessment (OA): For projects that have been deemed “High Risk” in earlier analysis, an OA must be conducted by JITC in an operationally relevant environment, with testing executed by an operationally representative set of users that will leverage the project capabilities in support of real-world missions, tasks, and activities. This type of testing employs test scenarios, exercises, and simulations as needed to assess the key project capabilities.

· FA: If the Risk Assessment conducted on the project yields a “Medium” rating, an FA must be conducted by JITC in an operationally relevant environment. However, the testing does not have to be done by operationally representative users. Testers or other independent participants execute the test scenarios in an operationally relevant environment. This type of testing also employs test scenarios, exercises, and simulations as needed to assess the key project capabilities.

· Other approved acceptance testing: The testing may be conducted by the PM in any environment that provides the key project capabilities. The testing must employ operational scenarios executed by independent user representatives. This testing is conducted for projects determined to be low risk according to the approved updated TEMAR, or as part of an overall strategy that includes a follow-on FA or OA event.

During OV, the test events selected are oriented to user needs and should trace operational activities back to the business and capability level requirements.

Evaluator(s) validate the implementation plan of the project and ensure that the performance of capabilities meets the user’s needs.

“Metrics” are defined in the DAU Glossary as “Parameters or measures of quantitative assessment used for measurement, comparison or to track performance of production” (https://www.dau.edu/glossary/Pages/Glossary.aspx). “Measures” are defined by the National Institutes of Standards and Technology (NIST) as a “process of experimentally obtaining one or more quantity values that can reasonably be attributed to a quantity.” Metrics and measures are collected from TV and OV and analyzed to determine how well the defined criteria have been met. At a minimum, all key capabilities should be employed in scenarios to ensure all associated KPPs are met. Each capability must have at least one associated KPP. The final T&E Scorecard, or the “Post-Operational Validation” T&E Scorecard is prepared by the Test Lead and is required for presentation at the NRRB.

TEST STATUS MANAGEMENT

The T&E Process provides a means for continuous process improvement. T&E activities are monitored to capture T&E Scorecard usage trends across projects, collect lessons learned, and analyze results to support continuous improvement of the JITC T&E Scorecard, the T&E Scorecard Guidebook, and this Guidebook. The following areas, when properly applied, could promote positive change.

REPORTING T&E STATUS

Test teams will brief the test status using the JITC T&E Scorecard, on a schedule dictated by the project-specific T&E strategy. Briefings and/or presentation of the T&E Scorecard may also be periodically requested by stakeholders throughout the lifecycle process, to include presentation at review boards, such as the SPC, CEP, NRRB, and ARBs. A JITC-validated T&E Scorecard is required for all projects when briefing the NRRB. T&E reports and briefings should contain descriptions of any unresolved problems and their proposed resolution.

The analyzed test results are collected and presented to project management and other stakeholders as appropriate. At a minimum the test report should state:

· Whether the results of testing met the minimum threshold for expected results

· What gaps exist between the expected results and the achieved results

· What conditions precipitated the success or failure of the test

· What actions are necessary to rectify outstanding or unresolved issues

· The next step(s) in the T&E process

Each test result briefing utilizes the JITC T&E Scorecard. The T&E Scorecard reflects the current status of the project from a T&E perspective, provides an updated risk assessment based on analysis of trends, and describes any potential or developing areas of concern.

JITC will analyze the test data and report on the extent that the project’s requirements are satisfied. The report should describe areas of concern from the T&E perspective and clearly outline any deficiencies detected and any identified areas of risk. The JITC T&E Scorecard will be used to present this overview and brief the status of the project.

MONITORING THE T&E SCORECARD

The use of the JITC T&E Scorecard across all DISA projects will be tracked by the T&E Scorecard Team and will be maintained on the DOD Enterprise Portal Service DOD Enterprise Portal Service (DEPS) https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/. These records, along with trend analysis for T&E efforts and the capture of lessons learned, supports a continuous improvement plan for DISA T&E. The data is used to improve the Scorecard, training materials, and the T&E Process Guidebook. Lessons learned illuminate areas that require revision or explanation. This level of scorecard management helps maintain a consistent look, feel, and use of the T&E Scorecard and provides an up-to-date, standard reporting mechanism for testing. Maintaining metrics and measures provides previously undocumented invaluable information when evaluating the T&E process.

MAINTAINING THE T&E DASHBOARD

The Management and Information Decision Support (MIDS) system, hosted on the DEPS provides a means to update stakeholders on DISA’s T&E efforts. The JITC T&E Scorecard is an integral tool for reporting DISA’s T&E efforts. A near-term goal of the DISA T&E Process and T&E Scorecard is to integrate with the MIDS system to maintain an electronic T&E dashboard providing on-demand status for any DISA project. The dashboard would provide an electronic, up-to-date status for each project from the T&E perspective using the T&E Scorecard as the template. Individual T&E T&E Scorecards will be uploaded to the DEPS site https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/ until integration with MIDS is achieved.

INFORMING LEADERSHIP

T&E Scorecards will be included in all briefings to DISA governance boards; additional reporting on project status to DISA leadership should also be tied to the T&E Scorecard AoEs. T&E reporting based on the T&E Scorecard will provide DISA leadership with consistent information from requirements development through fielding of a new project.

SUMMARY

This T&E Process Guidebook, along with the T&E Scorecard Guidebook, provides additional value to the DISA T&E community. When used to their full potential, these tools assist T&E personnel in capturing and managing risks associated with fielding new, enhanced, or updated projects, and in recommending the best analysis structure and testing strategy to prove the project is satisfying requirements and is ready to operate at full capability. The ETI process and the T&E Scorecard are tools the DBC can use to bring T&E into the initial planning of a project’s lifecycle, improve early identification of risks, and manage the proper levels of test. The T&E Process Guidebook and T&E Scorecard Guidebook provide the following value-added capabilities:

· Establish a positive working relationship between JITC and PMs, leading to seamless integration of the T&E and Project Management Plans

· Help perform analyses of risk for projects, to include early identification of difficulty meeting timelines, defining requirements, and/or testing to requirements.

· Help the requirements-analysis process, by providing a framework for requirements analysis and definition.

· Assess whether a fielded project actually met the anticipated user needs, whether all of the project’s capabilities were used, or if a set of project’s capabilities was not used as intended.

APPENDIX A

TEST AND EVALUATION SCORECARD

The JITC T&E Scorecard provides a standard means for reporting project status from a T&E perspective. The eight T&E Scorecard Areas of Evaluation (AoEs), and associated attributes, should be used as the basis for a project’s analysis structure.

The JITC T&E Scorecard is based on a combination of T&E lessons learned and the service management best practices found in the DOD Enterprise Service Management Framework (DESMF), and other process guidance. The JITC T&E Scorecard Guidebook’s modules provide recommended measures and metrics for evaluation of each of the AoEs reported in the T&E Scorecard. Please refer to the JITC T&E Scorecard Guidebook for details on use of the T&E Scorecard. The T&E Scorecard AoEs are briefly described below:

· Capability: Addresses user-community needs and capability performance. In this area, requirements and associated Critical Technical Parameters (CTPs) Technical Performance Measurement (TPM), Technical Performance Parameters (TPP) and Key Performance Parameters (KPPs) are evaluated.

· Interoperability: Addresses interoperability based on the Net-Ready Key Performance Attribute (NR KPA) construct, i.e., supported military operations, effective end-to-end information exchanges, and the project’s ability to enter – and be managed on – the network. The project’s Joint Interoperability Certification status is also considered and reported under this AoE.

· Availability: Addresses the project’s ability to meet operational availability requirements, to include the processes to monitor and track availability. This includes the establishment and validation of operational reliability, availability, and maintainability metrics to include failover, redundancy and recovery activities.

· Capacity: Addresses the project’s capacity requirements in terms of concurrent users, transactions, and storage. The AoE also covers system resources, scaling processes, and demand forecasting.

· Transition: Addresses the project’s activities for transition planning and support, change management, configuration management, records management, transition/on-boarding, release, and deployment. This is accomplished through the review of the project’s plan and processes documentation.

· Service Operations: This AoE aligns with Department of Defense (DOD) Enterprise Service Management Framework (DESMF) service operations process areas to include event, incident and problem management, technical and application management, request fulfillment, and access management. It also addresses the Service Desk function and technical/application management requirements and capabilities.

· User Experience: Addresses usability, documentation, and user perception of value, training, and Section 508 compliance.

· Cyber: This AoE addresses and tracks the Integrated Cyber Test and Evaluation (ICT&E) processes used by Joint Interoperability Test Command (JITC) to determine the system’s overall ability to be found “Cyber Survivable and Cyber Resilient as deployed in a cyber-contested environment.” This process is predicated on the results from the required Developmental Testing Cybersecurity events and Operational Testing Cyber Survivability events outlined and defined in the DoD CIO Cyber Guidebook phases. The requirements under evaluation are the assigned Risk Management Framework (RMF) security controls and the Cyber Survivability Attributes as defined in the system’s Cyber Survivability Risk Category (CSRC). The system must survive based on the Adversary Threat Tier (ATT) defined in the CSRC and the system’s validated threat reference (i.e. Validated Online Lifecycle Threat [VOLT] document).

UNCLASSIFIED//FOR OFFICIAL USE ONLY

A-4 A sample JITC T&E Scorecard is depicted in figure A-1.

Figure A-1. Sample JITC T&E Scorecard

The preparation and presentation of the T&E Scorecard are required in the following four areas

1. Post-Requirements Analysis. This T&E Scorecard is developed from the results of the initial RA and it is included in the initial TEMAR, which will be discussed in greater detail in Appendix C. Prior to entering design, this T&E Scorecard will be included in the early Business Case Analysis (BCA), or “BCA-lite,” briefing to the SPC and the ARB at Milestone 1 in the project.

2. Post-Design, Technology, Development, and Planning. The T&E Scorecard is developed from the results of the updated RA and TEMAR, which will be discussed in greater detail in Appendix C. This T&E Scorecard should be used to brief the project to the CEP at Milestone 2 in the project.

3. Post-Technical Verification. The T&E Scorecard should be updated with the results of T&E conducted for technical verification, and should be used to brief a project’s status at a Pre-NRRB prior to operational testing, from a T&E perspective.

4. Post-Operational Validation (Final T&E Scorecard for NRRB). The T&E Scorecard should be updated with the results of T&E conducted for OV and should be used to brief a project’s status at the NRRB, from a T&E perspective.

The usable template of the T&E Scorecard (Figure A-1), as well as the current T&E Scorecard Guidebook, samples, and other information can be found on the T&E Scorecard Knowledge Management site on DISA DEPS: https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/

A-6

APPENDIX B

ANALYSIS STRUCTURE

(EVALUATION FRAMEWORK AND DATA SOURCE MATRIX)

Designing a comprehensive Analysis Structure ensures the fullest possible testing of requirements (mission, operational and functional) identified for a system. An analysis structure consists of two primary components:

1. EF – The EF presents the areas of evaluation and test measures that a test team will use to determine the level to which a project is operationally effective, suitable, interoperable, and secure.

2. DSM – The DSM provides the critical test planning details needed to execute the EF.

Within the EF, the areas of evaluation are categorized into two hierarchical groups: Critical Operational Issues (COIs) and Measures of Effectiveness (MOEs) or Measures of Suitability (MOSs). Below the areas of evaluation are the actual test measures, the Measures of Performance (MOPs) and Supplemental MOPs. The test measures can support either a Measure of Effectiveness (MOE) or Measure of Suitability (MOS). Figure B-1 shows the relationship between these measures and provides an analysis structure.

Figure B-1. Analysis Structure Developing the analysis structure begins with the formulation of COIs, stated as questions and typically derived from the mission objectives found in the…