Attachment 10h - SWFMEA RESULTS.xlsx

Observations

Users for these tests need to be both contractor provided and end users. The goal of each test should not be to show success as much as to identify failures or opportunities for growth. This will drive the system to a greater reliability.

General Failure Modes

Compilers out of spec with language standards Poorly Change Managed code base In the worst case, this could lead to a system abort for an out of spec compilation. This should be exceedingly rare. More likely, the result would be that the software runs in a previous configuration without updated corrections. In order to even reach this state, in most cases, you would need to override compiliation warnings and force the compiler to generate a poor build. IE - It is possible to generate a build of software with unfulfilled function pointers. If those pointers are ever acted on the software would abort. This is a more common occurence when porting a realtime OS to a new board. This should not happen on IBCS. n 1 4 4 No mitigation activity needed for build/compilation errors. Build configuration artifacts will need be generated and CM'd. Tests will need to be in place that confirm via build meta data that the appropriate configurations are in place. Even faults found at this level should not flow as far as becoming a fault as they will be corrected before ever leaving the build activity. It is relatively unlikely for build errors making it beyond the integration activities. This is a fringe case that only rarely occurs using established languages and compilers. It is much more likely that the developer misinterprets the language standard than there being a flaw with the software compilation. There is a higher chance of a build getting created with an incorrect set of configuration settings. This is also fairly well mitigated by process and the volume of testing that is done on the resulting product.

This is not representative of "compilation errors" although... if the compiler ignores these errors they would be relevant. Compilation errors must be corrected in order to achieve a functional executable. These will not be in the system.

Faulty Algorithm Algorithm Misdefined Wrong Algorithm Used Incorrect algorithm Specifications Data loss from numeric fidelity In general, this will result in IBCS mathematical operations driving to an incorrect value assessement. In less stressful cases this might result in degraded data fidelity. However, the algorithms driving track management and engagement operations do drive towards classifcation and are safety/mission critical. A failure in those cases could be catastrophic. That said, a poor algorithm should not cause a system abort. That would be logic error causing a bad pointer dereference or a divide by zero exception or potentially memory overruns.

NOTE- It's a misnmomer to tie "algorithms" to being strictly mathematical calculations. However, it common lingo this is how it is recognized. Under a more loose definition of "algorithm" (the steps necessary to perform a task), you could make an argument that every code activity is performing an algorithm. This is extreme for these purposes as it would tie all faults to being algorithm faults in additon any other fault type. y 3 3 9 Each IBCS Algorithm should be assessed for Mission Criticality.

Each Mission Critical Algorithm should have a Algorithm Design Document generated for it by the Systems team.

Systems will generate model objects to validate algorithm design prior to ADD generation.

The ADD will clearly define the algrothm logically and from a data fidelity perspective.

The ADD will provide a list of verification cases.

The ADD will be provided to Test and Software.

Test will verify the implemention against the verification tests.

An additonal option would be to allow the ADD development team auto generate code into the sofware product. This would eliminate any interpretation issues.

Corrupted Data Endian mismatch Incorrect Data Representation

Lacking Functionality Undefinded system spec The end result will be semi-indetereminate. However, by the point of formal delivery, the system will have been run and generally verified. Major issues will be accounted for prior to delivery. However, due to the volume of code generated and the overall test philosophy, not all control paths will be tested. There's a decent chance that some paths will lead into unintended behavior and/or have not been developed at all. y 2 3 6 There needs to be large volumes of full system integrated testing.

Individual functionalities need to be extensively tested to verify intended functionality.

End Users/Testers/Integrators/Developers/Designers all need to be included in the initial design.

A cultural paradigm needs to be put in place to choose to address needed design updates rather than point at a lack of requirement.

Teams must feel empowered to drive updates to design and to identify issues as design flaws.

A single modelling tool/tool family needs to be selected and all with access need to be trained on the tool Processes need to be vetted that ensure a natural flow of design activity into development activity Requirements need to be based on behaviors and contructed with enough detail as to stand alone

Under assessing failure conditions Not looking for failure conditions Errors/failures will occur. In most cases, software can handle the failure and keep the system performing nominal. Most software failures will be driven by faulty data. Not catching this will allow algorithms or other decision makers to work outside of their intended scope. Denominators not checked for 0 may allow mathematical faults. In the worst cases, not recognizing a fault will allow the system into System Abort situation or allow an engagment decision with erroneous data. Error detection is intended to protect you from the results of all other faults (in a manner of speaking) and as such the result of all those faults are pertinent for fault detection. y/n 2 4 8 Development and design activities need to be performed with the understanding that faults may occur - data will be provided to the best of capabilities but this is a radar driven system and radar data is typically noisy.

At some point in the future, IBCS will be tasked with interactin with new end items. There will be a need for flexibility in handling of data and interfaces with these.

All math functions should be checked for feasibility before being calculated. IE - check for 0 before divsions, handle the negative prior to handling square root, etc.

Unit tests should require that all potential faults be checked. Any that are omitted should be acounted for (by addition or my assessment). All fault cases should also be tested.

Under assessing failure conditions Not looking for failure conditions Errors/failures will occur. In most cases, software can handle the failure and keep the system performing nominal. Most software failures will be driven by faulty data. Not catching this will allow algorithms or other decision makers to work outside of their intended scope. Denominators not checked for 0 may allow mathematical faults. In the worst cases, not recognizing a fault will allow the system into System Abort situation or allow an engagment decision with erroneous data. Error detection is intended to protect you from the results of all other faults (in a manner of speaking) and as such the result of all those faults are pertinent for fault detection. y/n 4 4 16 Development and design activities need to be performed with the understanding that faults may occur - data will be provided to the best of capabilities but this is a radar driven system and radar data is typically noisy.

At some point in the future, IBCS will be tasked with interactin with new end items. There will be a need for flexibility in handling of data and interfaces with these.

All math functions should be checked for feasibility before being calculated. IE - check for 0 before divsions, handle the negative prior to handling square root, etc.

Unit tests should require that all potential faults be checked. Any that are omitted should be acounted for (by addition or my assessment). All fault cases should also be tested.

Poorly mapped memory Connection issues associated with interface devices Poorly design User Interface (I also map this issue to Faulty Interface) User Error Faulty Hardware The ablity for the system to store data, display data, or to receive commands is degraded or non-functional. The result of this could be a nonfunctional system or a system that cannot display accurate data or a loss of logging capability. y 2 3 6 Perform testing that results in tactical or beyond tactical memory and data storage activities.

Ensure that all logging activities have processes inplace (circular queues etc) to manage growing data sets.

Establish alerts monitoring for the connection and functionality of necessary IO devices.

Setup monitors and mitigation activities to ensure that drive space and memory are maintained.

Establish multiple mechanism for sharing Mission Critical information.

Maintain all required HW/SW updates.

Faulty Installation Corrupted Installation Disk NonCompatible Partial Release

Lack of ICD Poorly developed IDL Insufficient Network environment Conflicts in Interface version Faults in this category tend to show themselves in 3 ways: 1) Item A fails to establish a connection with Item B 2) Data between item A and item B mismatches (endianness/mismatched enumberations/using different fields/using different units/using different frequency) 3) Data is shared to a greater group than is necessary. This sets up a potential cyber issue and floods the network potentially causing throughput problems. y 4 3 12 Interface needs will be identified during design work with a well defined go-back plan worked into the development activity.

Working groups between external groups will be put together and collaborate on an agreed form and function of the interface activities.

ICDs will be developed and shared amongst all relevant parties. As ambiguities in the ICD language are identified revisions will be created to more explicitly clarify the utilization.

Interface revisions will be rolled into related parties concurrently with the goal being to eliminate opportunities for mismatched interface revisions

Success criteria for a given unit test needs to be reviewed and agreed on.

Testing needs to be performed by an independent party. --For a unit x, the units that rely on it are not guaranteed to be written by the same developer. so, deviations from the intended result will affect later results.

Logging needs to be sufficiently detailed to allow integrators to identify flaws in the developed items.

Faulty Memory Management Memory Leaks Poor Pointer Management Null Pointer Exceptions MisAligned Pointer

Poor Pointer Manangement Mismapped BSP Under-vetted driver code Generally, faults in this category are rarely seen but with catestrophic software consequences. Generally, this is only relevant in small batch hardware or when trying mix and match BSPs of related hardware (particularly older hardware). What happens here is not all of the capability of a board always gets mapped into the board support package being used... or that a function being implemented doesn't match the standards in the anticipated manner. This is not really pertinent with executables running on VMs however there is an analogue in that the hypervisor needs to be compatible with the underlying os. maybe 1 4 4 Use common well vetted hardware, firmware and software combinations.

Perform extensive testing utilizing the tactical physical and software computer environment.

Include all appropriate patches and firmware updates.

Utilize Virtual Machines wherever appropriate with accepted and vetted failover mechanism in the event of a failure to the VM. (this is available for VMWare) Where custom embedded hardware is required, perform only vendor vetted BSP updates and install all updates that are defined by the vendor.

FPGA/BSP/Bootloader/etc executables need to be tested at the same level as any mission critical component. Detailed analysis are often not feasible for these low level software items. So, analysis for these will need to be more of the form of success assessment - IE - if it starts the Bootloader was successful. After a large number of successful boots you can trust the bootloader. Analysis would, of course, have to be done on any drivers started up during that process.

Faulty User Instructions Incomplete/Incorrect User Instructions Missing user instructions

The SUM should be developed at the same time as the codebase. Mechanisms should be put in place to ensure the users are provided the most up to date instruction set.

User Instructions should be available from the UI and in an intuitive manner.

A hard copy of start up instructions will be provided at the console (for situations when the system is not up).

The user instructions will be available on the maintenance laptop along with the IETMs.

Invalid User Access Lack of User Restrictions Misdefined User Accesses Shared accounts Access is too tight Non Authorized Users with access Worst case in this situation is a non authorized user gaining accesss to the software (cyber issue) and acting on system capabilities that are intended for specific roles. During training activities personell should not be given access to to tactical interfaces. Only specific users should have authorization for defense planning or engagement operations (especially launch). y 2 4 8 Strict role definintions need to be defined and implemented within the system.

Integration testing should include attempts to access off limits capabilites.

Cyber Penetration testing should be performed by an independent team.

Mismatched Tools/OS Incongruent Versions Actual causes of these issue would normally be covered via other failure modes, however for COTS/NDI software the integrator does not always have that explicit view into the failure cause. So, it comes back to a generic knowledge of mismatched software components. The results of these failures can vary. For math or algorithm type libraries the api's may change resulting in mismatched funtion prototypes or skewed data results (swapping x,y,z coordinates for example). In some cases, these flaws can result in software crashes. The most commonly used highest profile COTS product is RTI DDS. Mismatches in versions of the IDL builds (not really COTS) would be catastrophic to the function of the IBCS system. A failure here could be not recoverable short of a rebuild and reinstallation of the software. y/n 3 3 9 Minimize the volume of COTS products running on the IBCS system. If it is not necessary it should not be on the machine.

Provide a set of machines to support daily activities. This would allow for the "wanted" tools without putting the "required" tools at risk. This could be for daily logs, email, etc.

Split the IBCS software amongts many virtual machines with redundant backup VMs prepared to immediately start. This won't keep failures from happening but will give you a fall back when the worst occurs.

Test, test, test.... Everything that runs on MC hardware needs extreme levels of test. Non-MC software can affect MC SW and HW when they reside together.

User Error User Mistakes End results are identical to those of a user working with faulty instructions. y/n 4 4 16 Safety and mission critical decision making should always require multiple levels of verification before being activite. (IE - no one button mission launch) The Users Manuals should be sufficiently fleshed out to instruct users how to revert back and recognize when a user mistake has been made.

The system should provide an alert when a user decision significantly differs from what is suggested by the system.

No user made decisions should be definitive and absolute. There should always be a mechansim to pull back from a decision.

System Abort (generic) Software "crash" In the case of the majority of software, a system abort would cause the specific software to fail. There may be a series of trickle down issues as dependant processes fail to receive commands or responses.

In the case of VMWare a potential System Abort could casue the entire system to become inoperable. y 1 4 4 Software will be primarily be run on Virtual Machines. While this will not preclude system aborts, it will allow an almost immediate restart of the system software (when configured in the correct manner).

Functional Requrement FMEA