VA118-17-R-2324-009.docx

Software Assurance Standard Operating Procedure

1.0 TEST AND INDEPENDENT VERIFICATION AND VALIDATION

Test and Evaluation is the process by which a system or components are compared against requirements and specifications through testing. The results are evaluated to assess progress of design, performance, and supportability. Developmental test and evaluation is an engineering process used to reduce risk throughout the acquisition cycle. Operational test and evaluation is the actual or simulated employment, by typical users, of a system under realistic operational conditions.

Test and Independent Verification and Validation (T&IVV) should comprise a series of progressive system verification/qualification processes conducted sequentially within the system developer’s laboratory test-bed environment, the Government’s test-bed environment, and finally under full field conditions within an operational environment. Each test determines if the system, as developed, is sufficiently mature to continue to the next level of testing or deployed for use.

The T&IVV activities, for each Increment or product as applicable, should include vendor testing and risk-based independent Government testing in one or more Development Test Environments (DTE), then Operational Test and Evaluation (OT&E) at selected IOC sites within the components.

1.1 Software Code Quality Checking

High software quality helps to minimize sustainment costs, reduce run-time errors, and increase end-user satisfaction. Each software component being developed should be assessed relative to the overarching project architecture, and functional and technical requirements allocation to ensure the component design aligns with the overarching design. Additionally, the code should be analyzed against all federal/VA security standards, requirements, policies and industry best practices.

To support high-quality software products in the project environment, the government requires regular scanning of software code for latent software defects and security vulnerabilities and/or weaknesses, through a process known as Software Code Quality Checking (SCQC). The purpose of SCQC is to ensure delivery of secure, quality software code, clean code (e.g. free of fragments and dead code), benchmarked against recognized standards. The process focuses on the technical correctness of the code as well as freedom from security vulnerabilities. Software reliability and maintainability, along with a number of other factors, are measured throughout the entire system development life cycle (SDLC) through the use of SCQC tools.

The government defines SCQC as a scan of the source code, executables, and related artifacts, for example, documentation, to ensure that the system under development can continue with development, demonstration, and test, and can meet the stated performance, maintainability, and usability requirements within cost (program budget), schedule (program schedule), risk, and other system constraints. VA requires Cerner to provide evidence of having conducted a static code analysis, static security analysis and architectural analysis. The evidence should identify the processes and tools used, to include the use of external independent evaluators, together with the results of the scans. Cerner will ensure no new technical flaws or security vulnerabilities are introduced during the correction or enhancement of the code. Cerner will conduct SCQC throughout the updating, testing and sustainment phases of the project.

1.2 Software Reviews & Scans

SCQC encompasses the use of static code analysis, static security analysis, dynamic code analysis, dynamic security analysis and architecture analysis and is usually performed using automated tools.

· Static analysis is the analysis of computer software and related documentation that is performed without actually executing programs built from the software.

· Static security analysis is the analysis of computer software that is performed without actually executing programs to detect and report weaknesses that can lead to security vulnerabilities.

· Dynamic program analysis is the analysis of computer software and related documentation that is performed by executing programs built from that software on a real or virtual processor.

· Dynamic security analysis is the analysis of computer software that is performed by executing programs to detect and report weaknesses that can lead to security vulnerabilities.

· Architectural analysis may be supported by automated tools but are usually conducted by manual walk-through of documentation and visual inspection of the code.

All software developers should perform reviews and scans listed above to prevent/correct or mitigate defects and vulnerabilities to the greatest extent possible.

For development processes, Cerner should conduct static code and static security scans whenever code is altered. Dynamic code, dynamic security and architectural analyses should be conducted at the end of each sprint or more frequently when Cerner and the government determine it makes more sense to do so. Contractor should report SCQC results, weekly, using a defect removal efficiency matrix.

The Government may conduct its own independent informal SCQC inspection of the system under development at least weekly during the development of the system.

Cerner will make the following artifacts available:

· Source code and all design time libraries and licenses (static analysis)

· Executable code and libraries (dynamic analysis)

· Application configuration artifacts

· System Design Documents (SDD)

· System Sub-System Specification (SSS)

· System Sub-System Design Document (SSDD)

· System Security Authorization Agreement (SSAA)

· Interface Control Document (ICD)

· Database Design Document (DBDD)

· Test cases (dynamic analysis)

· Other artifacts proposed by the contractor Assessments of the contractor’s code should be reported in both the number and types of code quality and security vulnerabilities found as well as the technical debt or level of effort required to correct the defects.

1.2.1 Code Reviews

As a standard practice Cerner and the Government conduct formal code reviews together at least once each calendar month while code is being developed, tested and updated. These code reviews should include both technical controls as well as information assurance controls.

1.2.2 SCQC Tools

Contractor’s should be free to use its own tools and processes to meet the government’s standard for technically defect-free code that also contains minimal security vulnerabilities. However, software code quality should be assessed using the following tools or comparable tools, approved by the COR.

SONAR: SONAR is an open source platform used to measure and manage code quality. It covers the 7 axes of code quality: (1) architecture and design, (2) duplications, (3) unit tests, (4) complexity, (5) potential bugs, (6) coding rules and (7) comments. Multiple languages are supported through plug-ins. SONAR reports results based on the Software Improvement Group (SIG) Maintainability Model, The government uses SONAR to calculate the “technical debt” or cost to fix all defects reported by the tool. Technical debt is calculated based on the severity of the reported defects. Severities are Blocker, Critical, Major, Minor and Administrative. Cerner shall provide the government code that contains NO Blocker or Critical defects. The government should review the Major, Minor and Administrative defects and identify those defects that Cerner shall correct prior to acceptance of the code. SONAR incorporates a number of open source tools such as FxCop and FindBugs to identify defects. Using FindBugs to identify and correct defects when developing Java code, for example, should the number of defects and technical debt discovered by SONAR.

WebLayers: WebLayers is a licensed tool that provides static analysis of Java code. It checks artifacts included in the build for a piece of software. WebLayers runs in a JBoss server, and has a database which stores the results of scans. This document includes the queries used to create the reports from querying the database. Pending completion of these activities, the government should also scan Cerner’s code using webLayers and advise Cerner of the findings and potential corrective actions.

JArchitect: Visual JArchitect (JArchitect) is a licensed static analysis tool that is used for scanning Java-based applications. JArchitect utilizes a Graphical User Interface (GUI) and supports a large number of code metrics, allows for visualization of dependencies using directed graphs, and features a dependency matrix in order to provide troubleshooting assistance. The government may use JArchitect to identify defects such as circular dependencies, poor coding practices and the lack of best practices such as the need to refactor code. Contractor should deliver code that contains no circular dependencies, no components with a cyclomatic complexity greater than 78, and no more than 5% of the code requiring refactoring. The government should then identify other defect types reported by JArchitect and develop a corrective action plan in conjunction with the contractor. Upon acceptance of the corrective action plan, Contractor should fix the agreed-to types defects and ensure that no new defects within the same defect type are introduced.

NDepend: Visual NDepend (NDepend) is a licensed static analysis tool that is used scanning .NET-based applications. It is equivalent to JArchitect and the same criteria for JArchitect should apply to Cerner and the government.

Hewlett Packard (HP) Fortify: HP Fortify is a code validation program that scans source code for known security vulnerabilities. The government will provide Fortify licenses unless specifically addressed otherwise. Cerner should use Fortify to scan code and identify vulnerabilities. Vulnerabilities can be reported to VA using Common Weakness Enumeration (CWE) Identifiers using a Critical, High, Medium and Low rating.

CAST Management Studio: The CAST Management Studio is a licensed tool that allows scanning of applications to check for various metrics including security violations and quality of code. For any development in the Government’s Development and Test Center (DTC), the Government intends to issue licenses to Cerner to run CAST scans at the end of each sprint or sooner if it best supports the Contractor’s development activities. CAST aggregates defects using a set of “health” factors to categorize findings. These health factors include transferability, changeability, robustness, performance, security, programming practices, architectural design and documentation. The factors are scored from 1 (unacceptable) to 4 (exceptional). Cerner shall NOT deliver code to the Government that has a health factor below 2. CAST also identifies defects by weight using the same 1 to 4 scale. In addition, selected defects are identified for criticality. Cerner shall correct all critical defects prior to final delivery of the code to the Government.

1.3 Software Vulnerability

Cerner should strive to deliver code with zero high and zero critical vulnerabilities identified by Fortify scans or other SCQC tools. Cerner should provide risk assessment reports and a business case to address any medium or low level vulnerability.

1.4 Supply Chain Assurance

Cerner should implement a supply chain assurance process in accordance with NISTIR 7622, October 2012. The minimum criteria that Cerner shall meet is to assure through tools, techniques and processes that all third-party or open source software used in the SDLC by Cerner meets the government’s SCQC criteria.

1.5 Testing

Cerner should utilize Agile testing, a continuous process which takes place hand-in-hand with development and project management. As part of the Agile team, it is expected that Cerner will work closely with Government independent testing organizations and Government Independent Verification and Validation (IV&V) agents by providing metrics and eliciting early feedback from functional proponents, users, and other key stakeholders. Cerner should provide test resources to participate in daily scrum meetings Sprint planning.

As a standard practice, before acceptance of deliverable(s) from the Contractor, the intent of the Government is to advise, witness, and, at times, to participate in Contractor testing as to minimize risk of rework and limiting the amount of rework needed after deliverable(s) are received by the Government until it is evaluated as ready for full deployment.

1.5.1 Software Development Testing

Contractor should develop a Master Test Plan (MTP) which addresses the Contractor’s recommended strategy for testing the Capability to include, but not limited to: capability planned for development sprints within the Increment, integration test Sprints, performance test Sprints, Information Assurance (IA), scalability, software assurance testing, and interoperability testing. Contractor’s should provide demonstrations of functionality during development/integration sprints when a component of the Capability is ready for review by the scrum team and or validation by the Capability Owners (Functional user) that it meets acceptance criteria.

Cerner should develop a Test Plan for each planned Release of software to the Government to explain how Cerner should demonstrate to the Government that the deliverable(s) satisfy the Government’s requirements. Test Plans should address the following: Test Scenarios mapped to User Stories, Test Procedures, automated test scripts, and test data to be used in conducting tests. These items should be provided to the Government for their potential re-use in Government-led testing Sprints. Test Plans should be provided within five working days after approval to commence development and or integration. For example: At the end of every Sprint cycle Cerner will run Fortify and FindBugs and JArchitect will be run against the code monthly. Findings will be remediated by the end of two additional Sprint cycles. If Cerner is unable to remediate the findings within two Sprint cycles, development will stop until remediation is completed.

Cerner should develop Test Scenarios and Test Procedures that demonstrates business processes supported by the system and address full functionality for each user role. Test Scenarios should map to the integrated Business Requirements Document (iBRD), Requirements Traceability Matrix, and approved system design. Test Procedures should provide testability coverage for all new or changed interfaces and be sequenced logically and take into account dependencies, which should address functional requirements, database tables to be updated and checked, and expected output results.

In cases where the processing of “live” data files is necessary, Cerner should coordinate with the Government Capability managers for receipt of such files from the interface partner. No files containing Classified or Privacy Act Information shall be processed in the Contractor’s test environment. When re-validating software corrections resulting from previous test failures due to data issues, Cerner must use the original associated test file(s)/data that caused the problem as well as other data both conforming and not conforming to the approved format and retest the application as required. Any problem that cannot be resolved without impacting the schedule must communicated to the Project Manager and COR immediately for approval to fix or to move to the project backlog.

At the end of a development sprint or series of sprint(s) that delivers sufficient functionality of the Capability for deployment/release Cerner should conduct a Test Readiness Review (TRR) for the Government in order to present test findings and to turn the solution over to the project’s Configuration Manager for subsequent Government testing. The Capability must meet requirements of a Test Readiness Review (TRR) which requires removal of severity 1 & 2, and clusters of 3 or 4 defects. Table 1 describes the error severity levels and defect repair priorities.

Defect Severity Levels and Definitions

Severity Level 1 - Critical

The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system with no suitable workaround.

· Any defect that compromises patient safety or system security, (examples of system security defects include breach of confidentiality requirements of the Privacy Act, HIPAA or Federal Tax Information guidelines).

· Loss of system functionality critical to user operations with no suitable workaround (in other words, there is no way to achieve the expected results using the application).

· System crash or hang that prevents further testing or operation of the complete application or a section of the application.

· Any defect that causes corruption of data from a result of the system (as opposed to user error).

· Any defect in which inappropriate transmissions are consistently generated or appropriate transmissions of HL7 messages fail to be generated.

· Loss of functionality resulting in erroneous eligibility/enrollment determinations or communications not being sent.

· Loss of functionality resulting in erroneous eligibility/enrollment determinations or communications not being sent.

Severity Level 2 - High

The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system. There is no way to make the failed component(s) function. However, there are acceptable processing alternatives which will yield the desired result.

· A major defect in the functionality which does not result in corruption of data.

· A major defect in the functionality resulting in a failure of all or part of the application.

· The expected results can temporarily be achieved by alternate means. The customer indicates the work around is acceptable for the short term.

· Any defect resulting in non-conformance with to Section 508 standards.

· Any defect that results in inaccurate or missing requirements.

· Any defect that results in invalid authentication or authentication of an invalid end user.

Severity Level 3 - Major

The defect does not result in a failure, but causes the system to produce incorrect, incomplete, or inconsistent results, or the defect impairs the systems usability.

· Minor functionality is not working as intended and a workaround exists but is not suitable for long term use.

· The inability of a valid user to access the system consistent with granted privileges.

· Typographical or grammatical errors in the application, including installation guides, user guides, training manuals, design documents, etc.

· Any defect producing cryptic, incorrect or inappropriate error messages.

· Any defect that results from the use of non-standard data terminology in the application or documentation, as defined by the Department of Veterans Affairs.

· Cosmetic issues that are important to the integrity of the product, but do not result in data entry and or data quality problems.

Severity Level 4 - Minor/Exception

The defect does not cause a failure, does not impair usability, and the desired processing results are easily obtained by working around the defect.

· Minor loss of or defect in the functionality where a long term use exists.

· Low level cosmetic issues.

After Contractor testing the Government will typically conduct a "smoke" test of the application after successful install. Smoke Testing will include the following checks of the capability delivered in the Release: (1) Ability to Create, Read, Update, Delete data, (2) Checks on Interfaces, & (3) Checks on Roles and Privileges For each Release Cerner should document the results of their testing in a Test Report. The report should contain a detailed summary of test events, test findings, project backlog status, action items, and recommendations. All tool(s) and/or emulator(s) developed by Cerner specifically to test interfaces should always be delivered to the Government with the software for use in Government testing.

1.5.2 Test Support

After successful completion of the TRR, Cerner should provide technical support for subsequent Government testing. Assistance should be in the form of providing real-time availability of engineering staff to support trouble-shooting issues, explaining nuances of system design, assisting in the set- up of interfaces or emulation tools, performing quick fixes during a test, participating in meetings, performing detailed analysis and fault isolation. Cerner should also collaborate with the Government by allowing the Government testers to observe and report progress of the development/integration activities. This Quality Assurance process should include analysis on, but not limited to:

· Installation of the software in the development/integration environment

· Contractor Tools (IDEs, Build tools, Unit test tools, SOA messaging toolset) to ensure quality code

· Usage of Test Tools for automation by Cerner related to Progression, Regression, and Performance Tests

· Review of Contractor Test scripts for positive and negative testing

· Execution of system specific requested scripts

· Reporting for Test results and Defect Metrics

· Software deployment procedures

· Compliance with Measures of Effectiveness (MOE) and Measures of Performance (MOP) expected by Independent Tester All products must undergo field testing, including independent Operational Test and Evaluation before full deployment across the enterprise. Cerner should provide the artifacts listed in Table 2 to support fielding of a product at the initial test site.

For Operational Readiness Review, the submitted build is the final build targeted for IOC production deployment.

Software Package

Systems Management Guide

(2) System Design Document (SDD), (3) Data Definition

(4) Functional Flow Diagram,

(5) Hardware and Topological Architecture Diagram, (6) Nodes Connectivity Diagram

Systems Management Guide

Technical Reference Model

1.5.3 Test Metrics

This is the recommended minimum metrics for validating and forecasting workload, establishing the effectiveness of Contractor’s testing process and ensuring compliance of the product.

· Test Report showing the number of defects found during the Contractor’s testing and the severity and priority of those defects as defined by the government.

· Number of defects removed or repaired by Cerner prior to offering to the government and the number remaining by severity.

· A matrix identifying every requirement, the test procedure that shows where and how it was tested, the result of the test for that particular requirement, and any subsequent rework and regression testing involving the specific requirement.

· Test Report documenting the hardware configuration of each test event.

· Number of hours by resource category utilized for various test events.

· Defect categories shall be prepared and presented to the government for approval.

· Defect categories should show where the root cause of the defect originated (i.e., requirement, coding, integration, test procedure or execution).

Contractor should support Government tracking of software reliability by collecting and providing reliability data as a function of sprint test time by providing the following:

1. All instances of Priority levels of 1, 2, or Clusters of 3s or 4s defects (and associated defect number) found in a Test Sprint, with the goal of no more than 10 percent of any defect discovered in the sprint remain open at the end.

2. System availability measured by the percentage of time that the system performed as expected within the required timelines and accuracy of data measured.

1.5.4 Operational Readiness Review Support

Contractor should provide the following artifacts to support the Government’s Operational Readiness Review (ORR) prior to fielding:

(2) System Test Execution Log,

(3) System Test Defect Log, &

(4) Performance Test Results