JITC_Guide_to_Test_Documentation.pdf

JITC

GUIDE TO TEST

DOCUMENTATION

JUNE 2008

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

INTRODUCTION

The guide provides the details needed to implement JITC Instruction 210-85-01, Documentation of Test and Evaluation Activities. The guide and some resource information are on Hallways at Fort Huachuca and Indian Head in the JITC Guide to Test Documentation folder, and on the JITC Intranet Home Page under References, Policies and Procedures.

Applicability

The rules in the guide apply to all plans and reports written by JITC. All authors should assume the rules apply to their plan or report.

Exceptions In rare cases, JITC may best serve a customer by providing a product that deviates from these rules. In these cases, the author needs to get approval from division or portfolio management and the Policy Branch BEFORE the document is written.

Organization The guide first addresses the content needed in each section of a plan, report, and Interoperability Certification Evaluation Plan; then formatting instruction; and finally details in the Writer’s Reference.

Beyond the Guide The guide is intended to help you write clear, direct, effective test documentation. If you find that following any of the rules in the guide will make your plan or report repetitive, confusing, or incomplete, then STOP writing and call, e-mail, or visit the Policy Branch at 520-538-5127, gordon.douglas@disa.mil, Bldg 57305, room 254. We will work with you to find the best solution to your problem.

(This page intentionally left blank.)

i

TABLE OF CONTENTS

Page

TEST PLAN

Executive Summary System Functional Description Test Background Test Purpose Requirements or Required Capabilities Scope Limitations Methodology Example Results Tables Appendices

TEST REPORT

Executive Summary System Functional Description Test Background Test Purpose Scope Limitations Methodology Results and Analysis Conclusion(s) Recommendation(s) Appendices

INTEROPERABILITY CERTIFICATION EVALUATION PLAN

Before You Start Executive Summary System Functional Description Information Exchange Process Net-Ready Key Performance Parameter Requirements Test Procedures and Compliance Determination Methods Test Events ii

TABLE OF CONTENTS (continued)

Page

DOCUMENT ORGANIZATION AND FORMAT

Organization Format Cover – Format Instructions Signature Page – Format Instructions Intentionally Left Blank Page – Format Instructions Executive Summary – Format Instructions Table of Contents – Format Instructions Main Body – Format Instructions Tables – Format Instructions Figures – Format Instructions Acronyms Appendix – Format Instructions References Appendix – Format Instructions Points of Contact Appendix – Format Instructions

WRITER'S REFERENCE

General Guidance How To Reference Guides

TEST PLAN

Executive Summary Briefly summarize what the system does, what functions will be tested, and how.

Key Points Description Details

Mostly How Devote most of the space to what the system must do and how we will test it.

o Match functions to procedures.

o Mostly method of test.

For a General Audience

Write for high-level decision makers, not engineers or testers. Avoid technical and tester jargon.

o Help the Joint Interoperability Test Command Commander and other executives understand how the testing approach makes sense and that the test will be adequate for its purpose.

o Assume interest, not expertise.

Little Administrative Detail

Cover who, when, and where of the test very briefly. Executives need few details.

o Keep the focus on the test item, not the test or testers.

Only Critical Information

Delete everything not directly relevant.

Keep to one page or less.

o No room for boilerplate.

o No room for hype or program history.

Consistent Keep consistent with the body of the plan, particularly the purpose.

o Be careful not to modify when you summarize.

o Use the same terms.

Net-Ready Key Performance Parameter Testing

Identify any net-centric operations, standards conformance, or information assurance testing to be included in the test.

o Only identify actual testing, not document reviews.

o Don’t need to mention all elements.

Test Plan

System Functional Description Describe the important functions, missions, and uses of the system. Define what the users need from the system. Specifically address functions that use or provide network enterprise services and the exchange of information with other systems.

Key Points Description Details Identify the Users Tell Who Uses the System for What Purpose

Define the system’s role in supporting the warfighter or other system users.

o Include functions for users as well as operators.

o Explain how the system fits into the overall architecture.

(Critical or secondary?)

o If we are only testing some functions of a system, focus only on those functions.

Mission Perspective

Identify the missions that depend on the system. This will clarify for readers the potential impact of failures.

o Explain what capabilities are new or improved.

o Put technical failures in an operational context.

Avoid Cut and Paste

Our intent is to convey what the system should do, not to promote it.

o Avoid Program Manager or vendor hype.

o Avoid trade jargon and unsubstantiated capabilities.

Consistent with Other Sections

Clearly relate functions in this section to requirements and methodology in later sections.

o Each function should have related requirements.

o Avoid reader questions of “Why are they testing this?”

and “Why aren’t they testing that?”

o Methodology must address the requirements and relate back to functions.

o Use consistent organization of functions, requirements, and methodology.

Little Physical Detail

Include physical details only if they are relevant to how the system functions or how we will test it.

o More than a sentence or two is probably too much.

Interoperability and Net-Centric Functions

In all tests involving interoperability, explain the role of information exchange in fulfilling the functions of the system.

Describe any functions that provide or consume network enterprise services.

o Explain the role of information exchange in fulfilling the functions of the system.

o What functions depend on what exchanges?

o What functions of other systems depend on this interoperability?

o Identify potential net-centric attributes such as posting data or searching for data.

Test Background Explain why the test is needed, with supporting information only if it is directly relevant to what will happen in the test.

Key Points Description Details

Why Test Now

State the reason or reasons why we were asked to test the system; e.g., new capability, system upgrade, system is being used in a new way, new configuration, new environment.

o Identify the common sense reason that would make testing the logical thing to do.

o Not just why Joint Interoperability Test Command does testing.

o Not just that somebody asked us to.

o Not that the system needs to be certified.

Rationale for the Purpose

Provide the “why” for the “what” given in the Purpose.

o Give a logical reason for the purpose of the test.

o Don’t state what will be the purpose.

o Don’t describe this test. Test description belongs in

Scope and Methodology.

Only Relevant Background

If, and only if, previous testing will affect what or how we will test, then identify the relevant findings and show how they relate to the current test.

o Don’t need program history.

o Don’t need history of need for a function.

o Don’t need general history of testing program.

o If previous testing is mentioned, explain how it is relevant (some requirements already assessed).

Test Purpose Identify what the test will determine in one sentence.

Key Points Description Details

Primary Purpose

Identify the single most important purpose of the test. Address additional purposes in the Scope section.

o The primary focus of the test.

o What most of the testing is about.

What We Want to Find Out

Usually something like: “The test will determine the extent System A interoperates with other systems in a Joint Task Force network.”

Must be answered in Conclusion:

o Purpose: Determine if A and B are interoperable.

o Conclusion: A and B are (or are not) interoperable.

Short and Simple Stay clear and to the point. o Don’t use a paragraph of discussion or explanation.

o Don’t include lists, strings, or environments.

o Use the same terminology as in the Executive Summary.

Not Outcome Dependent

We want to determine something, not certify or validate.

o Success for us is getting the correct answer, not a pass for the system.

o Use unbiased terminology: our role as testers is to be objective. We only certify or validate when the facts warrant it.

Requirements or Required Capabilities Identify the requirements (or capabilities) the system must meet (or have).

Key Points Description Details

Mandatory Must either list requirements or summarize them here. Without clear requirements, it is impossible to understand the test.

o Cannot just refer to an appendix, standard, or other document.

o If no requirements, explain why.

o If we will evaluate, identify what the evaluation is based on.

Account for Every Requirement

Identify all requirements and indicate which, if any, will not be addressed during the test o Clearly address all identified requirements in the Methodology section. Don’t leave any loose ends.

o May include other Key Performance Parameters relating to interoperability besides the Net-Ready Key Performance Parameter.

Consistent with Functional Description

Organize requirements in a manner consistent with identified system functions and address all functions.

o Functions imply that there are requirements.

o Readers should be able to see what function would be affected if a requirement is not met.

Valid Sources For Assessments

Assessment requirements should be user validated.

o We cannot invent requirements, but we can use experience and engineering judgment.

o May use standards, requirements, capabilities, concept of operations, or other sources validated by users.

o Special cases may occur without user-validated requirements. Be very specific about conclusions from these events.

For Certifications Interoperability certification requirements must be Joint Staff (J6I) certified.

o Check the certification memo instruction for the specific details that will be needed there.

Requirements or Required Capabilities (continued) Net-Readiness Include requirements for the system to be net-ready. Identify the specific requirements that apply to the system. If formal requirements have not been identified, we will still assess the system net-ready status.

Key Points Description Details

Net-Ready Key Performance Parameter

Review all elements and show which of them or their parts are applicable. If not applicable, say why.

o Don’t just generically identify the Net-Ready Key Performance Parameter elements.

o Each system has unique requirements.

No Formal Net-Ready Key Performance Parameter

Explain that the system has no formal requirements for net-readiness, but the following elements will be used to characterize system net-readiness.

o Use requirements from available documentation.

o Relate to appropriate element; e.g., if an identified interface is a key interface, address as a Key Interface Profile.

Net-Centric Operations and

Warfare – Reference Model

Information Exchange in

Accordance with Integrated

Architecture Products

Key Interface Profiles

Assurance

Department of Defense

Information Technology

Standards Registry

Identify requirements related to operations in a net-centric environment.

o Identify all requirements for provision/consumption of enterprise services.

o Identify Communities of Interest and their applicable, shared-data requirements.

o Identify data/metadata and tagging requirements.

o Provide the status of Internet Protocol Version 6 compliance.

Summarize or list the Information Exchange requirements identified in the Integrated Architecture Products.

o Identify or summarize the system information exchange requirements. If any will not be addressed, explain the impact in the Scope section.

Identify any Key Interface Profiles for the system, the Key Interface Profile status, and any high-risk standards with those Key Interface Profiles.

o Put the Key Interface Profile declaration table in an appendix.

o List any high-risk Key Interface Profile standards not in the Technical View - 1 in this section.

o List any Key Interface Profile standards not in the Technical View - 1 in an appendix.

Summarize the applicable high-level Information Assurance requirements;

e.g., comply with Department of Defense Information Assurance Certification and Accreditation Process requirements.

o Generally too lengthy to list individual requirements in plan.

o If the system does not use the Department of Defense Information Technology Security Certification and Accreditation Process/Department of Defense Information Assurance Certification and Accreditation Process process, identify what other Information Assurance requirements apply.

List high-risk standards in the Requirements section. Include the Technical View - 1 as an appendix along with risk level and rationale.

o High-risk standards are those that are military unique, emerging technology, rarely implemented, or frequently violated.

o May need to use experience and engineering judgment if no Technical View - 1 or other source of standards is available.

Scope Outline what the testing will cover, emphasizing the extent of the test versus the total real-world requirements of the system. Include how we will evaluate applicable Net-Ready Key Performance Parameter elements.

Key Points Description Details

Test versus Real Environments

Explain how well the test environment and/or network represents the actual environment in which the system will operate.

o Are they the same, similar, or different in important ways?

o If different, explain why the differences are important.

o One realistic environment may not represent all real-world environments. If so, identify what is not represented.

Test versus Real Operation

Explain how well the system operation during the test represents the full range of potential system operations.

o Even if the environment is realistic, the performance demonstrated may not be. Performance with one or two users may not reflect performance with hundreds.

o Will we be able to fully and conclusively meet our test purpose?

Configuration Diagram, if Needed

Use a diagram to clarify relationships, connectivity, and information flow.

o May want to show test configuration and operational configuration or architecture.

o Especially important if configuration may impact performance.

Who, What, Where, When and Why They Matter

Briefly identify the organizations involved, what types of testing, where and when the testing will occur, and what these mean to the test.

o Indicate what the sites and organizations represent, such as a typical joint task force element at a deployed location.

o These details are important elements in showing test adequacy.

Net-Ready Key Performance Parameter

If there are applicable requirements for an element, identify our approach to address them. The approach typically will be either to review documentation (for the Department of Defense Information Assurance Certification and Accreditation Process or standards conformance certifications) or to conduct or observe testing.

o If there are no applicable requirements for an element, then you do not need to address it in this section.

o If you specify the approach to a requirement is to review documentation, then no additional information is needed in the Methodology section.

o For each high-risk standard, identify whether we will determine conformance through documentation or testing.

o If testing will be conducted using another plan, refer the reader to it and tell how the reader can obtain it.

Limitations Provide a short discussion of issues that will constrain what we can conclude from the test.

Key Points Description Details

Only Limitations on Conclusions

If the limitation will not affect the conclusion, omit it. However, no limitations means our conclusion is unequivocal.

o Not a place for detailing all our testing problems.

o Not just when or what we can’t test.

o Not a limitation if never in Purpose or Scope sections. If our purpose is to determine ability to support voice communications, it is not a limitation that we will not test video.

Always Include the Effect of Limitation

Explain the impact of each limitation on the conclusion.

o No value without a “so what?”

o Not just “so we can’t conclude anything about . . . .”

o For instance: Since video is a critical aspect of surveillance data, the system may not be able to support these key intelligence missions.

Always Include Risk to Users

Assess the risk to users: the likelihood of failure; the impact on the mission should the system fail or not be net-ready.

o Identify the risk to users, not to testers.

o How likely is there a problem in the untested area?

o How serious would a failure be to users?

o Is there risk to a particular group or mission?

Methodology Describe how the test item will be operated or exercised to determine if it meets requirements. Clearly state what the users/operators, system, and data collectors will do, and give the specific methodology for each Net-Ready Key Performance Parameter element to be tested.

Key Points Description Details

Primarily What the System Will Do; NOT What the Testers Will Do

Address how the system will be operated, then how data will be collected. Keep the focus on the system, not the testers.

o If most of the sentences start with “Joint Interoperability Test Command will . . . .” or “testers will . . . .” then the actual methodology is often missing.

o Avoid generic terminology such as scenarios, test scripts, test cases. Use specifics of what will occur during these events; e.g., users will direct aircraft, send images, modify purchase orders.

Most Important Part of the Plan

All other sections exist to support this.

The value a reader can get from a plan depends on this section.

o The primary purpose of the plan is to describe the test.

Do that here.

o Everything else provides the background to understand and evaluate the content of this section.

o If this section is sketchy, readers cannot assess the adequacy of the test.

Biggest Section

More specifics and detail belong here than anywhere else in the plan body.

“Users will perform their normal operational tasks” is not sufficient.

o Details here should provide a clear enough picture that only technical experts will need to read the appendices.

o If details in the appendix are only a few pages, put them in the body.

Consistent with Other Sections

Tie the system functions, requirements, and test procedures together to help the reader clearly see test strategy and thoroughness.

o Must have a clear track from function to requirement to methodology.

o Large System Functional Description and Requirements sections with a short Methodology section usually indicate inadequate methodology.

Test Procedures Must tell how the system will be tested and should include procedures for each requirement or type of requirement.

o Show how all requirements will be addressed.

o Cover conditions (no load, stressed, high and low bandwidths, new and experienced users, etc.), factors (file size, message type, operating mode, etc.), and sample sizes.

o Describe use of the system, not just using questionnaires; e.g., personnel will use the system under normal operational conditions for 3 weeks.

Net-Ready Key Performance Parameter

Think of this section as TEST methodology.

o Describe the specific testing we will do.

o If there is no actual testing of an element, there is no methodology for that element. Documentation review is addressed in Scope.

Example Results Tables Show how you will present results in tables. Use simulated content.

Key Points Description Details

First Step of Report

Use to begin the process of preparing the report.

o Include all requirements that will be addressed in tables.

o Should be able to fill in tables as data is collected and analyzed.

o May change tables in report if appropriate.

Layout and Content

Identify the organization of test conditions (environment, stress loads, etc.) we expect to be meaningful and clarify what data we intend to include.

o Layout shows what distinctions the reader will be able to make (large versus small files, images versus text).

o Content shows example data (attempts/successes, averages, maximums, percentages).

Simplicity and Clarity

Collocate criteria and results and other data that need to be compared.

o Include test criteria.

o The easiest comparisons are between adjacent columns.

o Use space to organize. Group to make trends and exceptions stand out.

o Avoid displays that hide variation in a sea of uniformity

(3571, 3571, 3571, 3517, 3571, 3571).

Omit if Trivial If no tables are necessary, omit. o Don’t create a table when a simple statement will do.

o Unnecessary to include statements like “We will present results in narrative.”

Net-Ready Key Performance Parameter

Probably will not need tables for results other than information exchange.

o Don’t need to show “status” (met/not met) tables without any results (actual numbers).

o Net-Centric Operations and Warfare Reference Model, Information Assurance, and Standards Conformance results normally not quantitative (more often just conforms or not).

Appendices Provide supporting information necessary to fully define the test and provide detail that would be needed to replicate it.

Key Points Description Details

Appendix Order Begin with Acronyms and end with References and Points of Contact.

Between Acronyms and References, present other appendices in descending order of importance.

o Acronym definitions first, so they are easy to find.

o Between Acronyms and References, place appendices such as test specifics, test configurations.

o Omit anything that does not directly contribute to understanding the test.

Detailed Procedures

If the test procedures are so complex that the plan body only summarizes them, describe the exact procedures in an appendix.

o The most important appendix contains the specifics of the test.

o Include criteria, data requirements, test conduct, and data collection not described in the body. Provide enough detail that another test team could replicate the test using only the test plan as guidance.

o Must track with the body. Put details in appendices. Don’t introduce unrelated test procedures.

Only as Technical as Necessary

May include more technical information than in the plan body, but keep as readable as possible, especially in procedures.

o Include the details needed by technical experts to understand the test.

o Provide specifics needed by users to implement.

o Provide enough information for the plan to stand alone.

Not Limited to Paper

Consider alternative media (electronic, Compact Disk, Digital Video Disk, etc.)

for very long items of interest only to specific customers.

o Value of plan not measured in pages.

o Use common sense to keep size reasonable.

Net-Ready Key Performance Parameter

Typical additional appendices might include Net-Centric Operations and Warfare Reference Model test details, Information Assurance details, Key Interface Profile declaration table and applicable standards, and Technical View - 1.

o Low-risk standards belong in an appendix, not in the body.

o Only include those appendices you need.

TEST REPORT

Executive Summary Summarize what the system did and did not do, operational impacts, and what conclusion can be drawn.

Key Points Description Details

Mostly What Was Found

Devote most of the space to the most important issues: what the system was able to do, not do, and the significance.

o Very brief functional sketch.

o Mostly results and meaning.

o Both the can and can’t results.

o NOT just the test plan’s Executive Summary with an added sentence or two.

and tester jargon.

o Help the Joint Interoperability Test Command Commander and other executives understand what the system can and can’t do and what that will mean for users.

o Assume interest, not expertise.

o Don’t need lots of detail (need to know rather than nice to know).

o Avoid unexplained bean counts (met 17 of 20 requirements).

o What the successes and failures are nearly always more important than how many.

Little Testing Detail

Cover who, when, where, and how only to the level they are important to the findings.

o If the test is complete and conclusive, very little detail is needed.

o Focus on the test item, not the test or testers.

o Add critical limitations if omitting might mislead.

Support the Conclusion

Include at least some data and logic that lead to the conclusion.

o Can’t just jump to a conclusion without anything to back it up.

o Conclusions should never be a surprise.

o Conclude what is, based on what was seen.

Only Critical Information

Delete everything not directly relevant to the findings, and keep to one page or less.

o More is not better.

o No room for hype or program history.

Consistent Keep consistent with the rest of the report, particularly the results and conclusion.

o Identical or very similar wording should be used here and in the sections in report body.

o Best to write this section last.

Net-Ready Key Performance Parameter

Include important findings in any of the Net-Ready Key Performance Parameter elements, such as critical Information Assurance vulnerabilities or standards conformance issues with potential operational impact.

o Don’t need to address every element if not applicable.

o Consider value to executive-level reader.

Test Report

Describe the important functions, missions, and uses of the system. Define what the users need from the system. Specifically address functions that use or provide network enterprise services and the exchange of information with other systems.

Key Points Description Details May Use this Section from the Plan

However, if testing identified new functions or results differ from the functions identified, new text is necessary.

o Users may have other needs.

o Check against results.

o Don’t let functions sound like results; e.g., the system provides seamless interoperable communications.

o Be sure to change verbs to past tense.

Identify the Users Tell Who Uses the System for What Purpose

Define the system’s role in supporting the warfighter or other system users.

o Include functions for users as well as operators, if different.

o Explain how the system fits into the overall architecture.

(Critical or secondary?)

o If we only tested some functions of a system, focus only on those functions.

potential impact of failures.

o Put technical failures in an operational context.

Our intent is to convey what the system should do, not to promote it.

o Avoid Program Manager or vendor hype.

o Avoid trade jargon and unsubstantiated capabilities.

Consistent with Results

Clearly relate functions in this section to results presented in the Results and Analysis section. Should see from results how well functions can be performed.

o Each function should have related requirements.

o Avoid reader questions of “What function requires this?”

and “Why didn’t they test that?”

o Use consistent organization of functions and results.

Little Physical Detail

Include physical details only if they are relevant to test results.

Interoperability and Net-Centric

In all tests involving interoperability, explain the role of information exchange in fulfilling the functions of the system.

Describe any functions that provide or consume network enterprise services.

o Explain the role of information exchange in fulfilling the functions of the system.

o What functions depend on what exchanges?

o What functions of other systems depend on this interoperability?

data or searching for data.

Test Background Explain why the test needed to be conducted, with supporting information directly relevant to what happened in the test.

Key Points Description Details

Use or Modify this Section from the Plan

May need to add relevant items or delete irrelevant ones.

o If customer needs changed after the plan was written.

o Delete items no longer of concern.

Why Test Now

State the reason or reasons why we were asked to test the system; e.g., new capability, system upgrade, system is being used in a new way, new configuration, or new environment.

o The common sense reason that made testing the logical thing to do.

o Not just why Joint Interoperability Test Command does testing.

o Not just that somebody asked us to.

o Not that the system needs to be certified.

Rationale for the Purpose

Provide the “why” for the “what” given in the Purpose.

o Give a logical reason for the purpose of the test.

o Don’t state what will be the purpose.

o Don’t describe this test. Test description belongs in

Scope and Methodology.

Only Relevant Background

If, and only if, previous testing had an impact on what was tested or found, indicate how the previous finding was relevant to the current test.

o Don’t need program history.

o Don’t need history of need for a function.

o Don’t need general history of testing program.

o If previous testing is mentioned, explain how it was relevant to current test and results.

Test Purpose Identify what the test was intended to determine in one sentence.

Key Points Description Details

Same as Purpose in the Plan

Should not change from the plan except in rare cases when extreme circumstances make the original purpose impossible.

o Don’t need to add additional purposes.

o Can report things beyond Purpose (if part of the testing goes beyond the Purpose, we can still report on it).

Primary Purpose Identify the single most important purpose of the test. Address additional purposes in the Scope section.

o The primary focus of the test.

o What most of the testing was about.

Short and Simple Stay clear and to the point. o Not a paragraph of discussion or explanation.

o Not a place for lists, strings, or environments.

o Use the same terminology as in the Executive Summary.

Answered in Conclusion

Conclusion must follow from the Purpose.

o Must be answered in Conclusion:

If the Purpose is “to determine if A and B are interoperable,” then the Conclusion must be “A and B are (or are not) interoperable.”

o Also should be consistent with the Executive Summary.

Not Outcome Dependent

We want to get information, not certify or validate a specific status.

o Success for us is getting the correct answer, not a pass for the system.

o Use unbiased terminology: our role as testers is to be objective. We only certify or validate when the facts warrant it.

Scope Outline what the test covered, emphasizing the extent of the test versus the total real-world requirements of the system. Include how we evaluated applicable Net-Ready Key Performance Parameter elements.

Key Points Description Details

Use or Modify Scope from the Plan

May need to add relevant things or delete irrelevant ones.

o If use or test environment changed from what was in the plan.

Test versus Real Environments

Explain how well the test environment and/or network represented the actual environment in which the system will be used.

o Are they the same, similar, or different in important ways?

o If different, why are the differences important?

o One realistic environment may not represent all real-world environments. If so, identify what was not represented.

Test versus Real Operation

Explain how well the system operation during the test represented the full range of potential system operations.

o Even if the environment was realistic, the performance demonstrated may not be. Performance with one or two users may not represent performance with hundreds.

o Were we able to fully and conclusively meet our test purpose?

Configuration Diagram, if Needed

Use a diagram to clarify relationships, connectivity, and information flow.

o May want to show test configuration and operational configuration or architecture.

o Especially important if configuration impacts performance.

Significance of Where and When

If locations and dates of testing were relevant to what was tested, explain how they are significant.

o Relevant location factors might be different missions, configurations, sizes.

o Relevant time factors might be high and low loads, periodic data roll-ups.

Net-Ready Key Performance Parameter

For all Net-Ready Key Performance Parameter elements, identify which applied and our approach to those that did.

o Since reports do not include a Requirements section, use this section to identify elements that do not apply.

o Don’t repeat things in the Methodology section if covered in Scope.

Limitations Briefly discuss issues that will constrain what we can conclude from the test.

Key Points Description Details

Use or Modify this Section from the Plan

May need to add or delete things as relevant.

o If significant deviations from the plan.

o If limitation no longer relevant.

Only Limitations on Conclusions

If the limitation does not affect the conclusion, omit it. However, no limitations means our conclusion is unequivocal.

o Not a place for detailing all our testing problems.

o Not just when or what we couldn’t test.

o Not a limitation if never in Purpose or Scope sections. If our purpose is to determine ability to support voice communications, it is not a limitation that we did not test video.

Always Include the Effect of Limitation

Explain the impact of each limitation on the conclusion.

o No value without a “so what?”

o Not just “so we can’t conclude anything about . . . .”

o For instance: Since video is a critical aspect of surveillance data, the system may not be able to support these key intelligence missions.

Always Include the Risk to Users

Include an assessment of the risk to users of failure: the likelihood of failure;

the impact on the mission should the system fail or not be net-ready.

o Identify risk to users, not to testers.

o How likely is there of a problem in the untested area?

o How serious would a failure be to users?

o Is there risk to a particular group or mission?

Methodology Briefly describe how we conducted the test and how we obtained the results.

Key Points Description Details

System Operation

Primarily what users did with the system.

o System is the focus, not test, testers, or data collectors.

o Describe use of system, not just using questionnaires; e.g., Personnel used the system under normal operational conditions for 3 weeks.

o Describe only significant deviations from the plan. (Include in Limitations section, if appropriate.)

Reduced from the Plan

Detail of plan not necessary. o This section is a support section, not the main focus as in the plan.

o But not “users operated the system.” Give the reader some idea about what the system was doing.

Only Relevant to Results

Provide just enough information for readers to understand how the results were obtained.

o Let the reader know if the test conditions are comprehensive or just a sample.

Details in an Appendix

Put test conduct and data collection details in an appendix.

Results and Analysis Summarize what happened during the test, including the factual and numeric outcomes relating to the requirements, and the operational meanings of the results.

Key Points Description Details

Good and Bad Report successful performance as well as problems.

o System value depends on both what it can and can’t do.

o Must include capabilities to put failures in context.

What Happened Include actual outcomes and numbers, not just “pass” or “met.”

o Don’t ask readers to “trust me.”

o Provide a clear, complete, performance picture.

o May need to report findings beyond planned measures

(system setup difficulties, network instability, alternative uses).

Address All Requirements

Operational as well as technical. o Systematically present results to cover all specified criteria.

o Explain any omissions.

o Present results next to criteria.

Explain Operational Meaning

Clarify operational significance of results.

o At a minimum, address all failures.

o Failure to meet a numeric goal not always significant.

o Don’t mix meaningful results with testing errors.

o Testing is to predict operational performance, not just report test bed outcomes.

Support Conclusions

Include analysis needed to draw the conclusion.

o Facts and discussion leading to conclusions belong here.

o No surprise conclusions.

Net-Ready Key Performance Parameter

Report results for all Net-Ready Key Performance Parameter elements, including Department of Defense Information Technology Standards Registry and other standards.

o Provide text and/or tables for all testing results (data).

o Provide status (met/not met, etc.) for compliance evaluations. If not compliant, identify failures and explain significance.

o Provide an overall summary table of the status of all elements.

Conclusion(s) Identify what we can conclude from the test results.

Key Points Description Details

First Address Purpose

The Conclusion statement must directly address the Purpose statement.

o Purpose: Determine if the system is effective.

o Conclusion: The system is effective.

THE Bottom Line

Not a discussion. o What you want the readers to remember.

o Don’t repeat the findings.

o Don’t dilute with minor points.

What IS True The system can or cannot interoperate with o Not what was seen (results).

o What can you conclude based on what you saw.

o Don’t need to say “based on . . .”

Other Conclusions

Only if other items are critically important.

o Separate items, not in one paragraph.

o Don’t need a conclusion for each Net-Ready Key

Performance Parameter element.

Recommendations When requested, provide recommendations based on test findings.

Key Points Description Details

Only if Requested

Not typically desired or required. o Not volunteered; should be customer requested.

Only Based on Results

Not just opinions or good ideas. o Must be objectively based on data: no surprises.

o Not just general opinions.

o Not self-serving, such as pay more money for Joint

Interoperability Test Command to do more testing.

Who Should Do What

Indicate who needs to take what actions.

o Clarify responsibility for what is recommended.

o Do not make recommendations to Joint Interoperability

Test Command.

Only in Areas of Joint Interoperability Test Command Expertise

Not cost or fielding (the system costs too much and should not be fielded).

o Not statements on technologies (e.g., Linux is better than Windows).

Appendices Provide supporting information necessary to describe the test and present the complete results.

Key Points Description Details

Appendix Order Begin with Acronyms and end with References and Points of Contact.

Between Acronyms and References, present other appendices in descending order of importance.

o Acronym definitions first, so they are easy to find.

o Between Acronyms and References, place appendices such as test specifics, test configurations, detailed test results.

o Omit anything that does not directly contribute to understanding the test and results.

Detailed Criteria, Procedures, Results and Analysis

If the main body only summarizes these items, present details in an appendix.

o The most important appendix contains the specifics of the test.

o Include criteria, data requirements, test conduct, and data collection not described in the body.

o Must track from the body. Put details in appendices. Don’t introduce unrelated test procedures.

o Arrange procedures, results, and analysis for ease of understanding.

Only as Technical as Necessary

May include more technical information than in the report body, but keep as readable as possible, especially in procedures.

o Include the details needed by technical experts to understand how we obtained our results.

o Provide specifics needed by users to implement.

o Readers should not have to consult other documents to understand.

Not Limited to Paper

Consider alternative media (electronic, Compact Disk, Digital Video Disk, etc.)

for very long items of interest only to specific customers.

o Value of report not measured in pages.

o Use common sense to keep size reasonable.

Net-Ready Key Performance Parameter

Typical additional appendices might include Net-Centric Operations and Warfare Reference Model test details and results, Information Assurance details and results, Key Interface Profile declaration table and applicable standards, and the Technical View - 1.

o Low-risk standards belong in an appendix, not in the body o Only include those appendices you need.

INTEROPERABILITY CERTIFICATION EVALUATION PLAN

Before You Start An Interoperability Certification Evaluation Plan (ICEP) is not needed for most interoperability testing.

The points below explain when it is needed and why.

Key Points Description Details

Can you use a test plan?

If you are able to write a test plan, you don’t need an ICEP. ICEPs describe the requirements and procedures when the details of implementation such as time, place, and resources are unknown.

o You always need a test plan.

o Don’t write any unnecessary documents.

o If interoperability testing details can be incorporated into another test plan, then an ICEP is probably not necessary.

Are there too many events for a single plan?

If you will conduct multiple test events, and each will have a separate test plan, you may use an ICEP to show what will be addressed in each test.

o The ICEP will identify all the requirements addressed in the testing program, and which requirements will be addressed in which event.

o The ICEP provides a means to show the overall test program and how the system will progress to final certification.

Will another organization conduct testing?

If another organization does the testing, the ICEP will provide the guidance they will need to design the test.

o The testers will need to know how to operate the system, what information to exchange, and the test conditions.

o The ICEP also needs to cover the data collection details, including what to collect and the amount of data required.

Do you have the information you need?

An ICEP requires a clear understanding of the system, its functions, and requirements. Such information must be included in the ICEP, or shortfalls clearly identified and corrected later.

o A generic “JITC does interoperability testing” document is not a useful ICEP. It does not give customers a product of value.

o If a customer or test organization requests an early version of an ICEP before all information is available, be sure to identify the document as an incomplete draft.

Net-Ready Key Performance Parameter

The ICEP must address all areas of the

NR-KPP.

o Identify the specific requirements that apply to the system in each area.

o Identify what will be needed to determine if each requirement is met.

ICEP

Executive Summary Briefly summarize what the system does, explain how it will be evaluated for net-readiness (or interoperability), and identify the activities that the test program will include.

Key Points Description Details

Mostly How Devote most of the space to what the system must do and the testing needed.

o Match functions to procedures.

o Mostly method of test.

o Not just vague “developmental and operational testing.”

and tester jargon.

o Help the Joint Interoperability Test Command Commander and other executives understand how the testing approach makes sense and that the test program will be adequate to determine interoperability and net readiness.

o Assume interest, not expertise.

Little Administrative Detail

Cover who, when, and where of testing very briefly, if known. Executives need few details.

o Keep the focus on the test item, not the test or testers.

Only Critical Information

Delete everything not directly relevant.

Keep to one page or less.

o No room for boilerplate.

o No room for hype or program history.

Net-Ready Key Performance Parameter Testing

Identify any net-centric operations, standards conformance, or information assurance testing to be included in the test program.

o Only identify actual testing, not document reviews.

o Don’t need to mention all elements if they are not going to be tested.

Describe the important functions, missions, and uses of the system. Define what the users need from the system. Specifically address functions that use or provide network enterprise services and the exchange of information with other systems.

Key Points Description Details Identify the Users Tell Who Uses the System for What Purpose

Define the system’s role in supporting the warfighter or other system users. If appropriate, break down by subsystems.

o Include functions for users as well as operators.

o Explain how the system fits into the overall architecture.

(Critical or secondary?)

potential impact of failures.

o Use to put technical failures in an operational context.

Our intent is to convey what the system should do, not to promote it.

o Avoid Program Manager or vendor hype.

o Avoid trade jargon and unsubstantiated capabilities.

Consistent with Other Sections

Clearly relate functions in this section to information exchange process, requirements, and test procedures and compliance determination in later sections.

o Each function should have related requirements.

o Avoid reader questions of “Why are they testing this?”

and “Why aren’t they testing that?”

o The Test Procedures and Compliance Determination

Method section must address the requirements and relate back to functions.

o Use consistent organization of functions, information exchange process, requirements, and test procedures and compliance determination.

Little Physical Detail

Include physical details only if they are relevant to how the system functions or to the testing.

Net-Centric

Describe any functions that provide or consume network enterprise services.

data, searching for data, or using enterprise services.

Information Exchange Process Describe the general flow of information into and out of the system to provide an explanatory transition from the missions and functions of the previous section to the details of the individual requirements listed in the next section.

Key Points Description Details

Primarily Functional, Not Technical

Explain the role of information exchange in fulfilling the functions of the system.

o What functions depend on what exchanges?

o What functions of other systems depend on this interoperability?

What is done with the information?

Identify information users and how they use it.

o By users of test system o By users of interfacing systems

Address all Interfaces

Usually the information over each interface is unique, so a discussion by interface is usually appropriate.

o Group interfaces if it will eliminate redundancy.

o Identify interfacing systems.

Net-Centric Processes

Identify any processes of posting, retrieving, pushing or pulling data, or using enterprise services.

o Indicate any applicable communities of interest you have identified.

Net-Ready Key Performance Parameter Requirements Identify the requirements the system must meet to be interoperable and net-ready. Identify the specific requirements, not just generic categories of net-readiness.

Key Points Description Details

Mandatory Must either list requirements or summarize them here. Without clear requirements, it is impossible to understand the test program.

o Cannot just refer to an appendix, standard, or other document.

o If you summarize, include sufficient detail to relate requirements to necessary testing.

Consistent with other Sections

Organize requirements in a manner consistent with identified system functions and information exchange process.

o Functions imply that there are requirements.

o Readers should be able to see what function would be affected if a requirement is not met.

Requirement Certification

Interoperability certification requirements must be Joint Staff (J6I) certified.

o Check the certification memo instruction for the specific details that will be needed there.

Net-Ready Key Performance Parameter

Review all elements and show which of them or their parts are applicable. If not applicable, say why.

o Don’t just generically identify the Net-Ready Key Performance Parameter elements.

o Each system has unique requirements.

No Formal Net-Ready Key Performance Parameter

Explain that the system has no formal requirements for net-readiness, but the following elements will be used to characterize system net-readiness.

o Use requirements from available documentation.

o Relate to appropriate element; e.g., if an identified interface is a member of the family of Key Interface Profiles, address as a Key Interface Profile.

Net-Centric Operations and

Warfare – Reference Model

Exchange

Key Interface Profiles

Identify requirements related to operations in a net-centric environment.

o Identify all requirements for provision/consumption of enterprise services.

o Identify Communities of Interest and their applicable, shared-data requirements.

o Identify data/metadata and tagging requirements.

o Provide the status of Internet Protocol Version 6 capability.

Summarize or list the requirements for Information Exchange identified in the Integrated Architecture Products.

o Identify or summarize the required exchanges across system interfaces.

Identify any Key Interface Profiles for the system, the Key Interface Profile status, and any high-risk standards with those Key Interface Profiles.

o Put the Key Interface Profile declaration table in an appendix.

o List any high-risk Key Interface Profile standards not in the Technical View - 1 in this section.

o List any Key Interface Profile standards not in the Technical View - 1 in an appendix.

Net-Ready Key Performance Parameter Requirements Identify the requirements the system must meet to be interoperable and net-ready. Identify the specific requirements, not just generic categories of net-readiness.

Assurance

Department of Defense

Information Technology

Standards Registry

Summarize the applicable high-level Information Assurance requirements;

e.g., comply with Department of Defense Information Assurance Certification and Accreditation Process requirements.

o Generally too lengthy to list individual requirements in plan.

o If the system does not use the Department of Defense Information Technology Security Certification and Accreditation Process/Department of Defense Information Assurance Certification and Accreditation Process, identify what other Information Assurance requirements apply.

List high-risk standards in the Requirements section. Include the Technical View - 1 as an appendix along with risk level and rationale.

o High-risk standards are those that are military unique, emerging technology, rarely implemented, or frequently violated.

o May need to use…