AIAA-Perf-Anal-Final.pdf

Performance Analysis of an Autonomous Fault Management

System

Gordon B. Aaseng,1 Adam Sweet2, and John Ossenfort3

NASA Ames Research Center, Moffett Field, CA 94035

NASA has developed spacecraft autonomy technology for detecting, diagnosing, mitigating and resolving system failures with minimal assistance from ground support. These types of applications will be required for mission success on missions beyond Earth orbit when communications delays preclude immediate and continual mission support from the ground.

The emphasis for autonomy technology in the past has been on developing and proving the logic and algorithms needed to produce the information that will be needed for autonomous spacecraft operations. Relatively little effort has been made to determine the computational resource utilization that will be required for execution on flight computers with autonomy applications scaled sufficiently to achieve autonomous operations throughout a space mission.

Our group has developed a fault detection and diagnosis application, along with a failure impact determination reasoner, that covers a substantial subset of the Orion spacecraft systems. These autonomy applications have been executed on a flight quality PowerPC processor running with the VxWorks Operating System and the Core Flight System (cFS) platform and project independent reusable software framework. This paper describes a performance analysis of these autonomy applications using flight data from the Orion EFT-1 mission. The purpose of the analysis was to determine the performance of the application software at its current state of development, to identify opportunities for improving the performance, and to derive performance metrics that can be used to determine the processing requirements for deployable, full scale, autonomous spacecraft systems. The analyzed application is the Advanced Caution and Warning System (ACAWS) developed originally as a capability demonstration with the Ares 1-X test flight, followed by use with the Deep Space

Habitat technology demonstration project, and then adapted for the Orion EFT-1 mission, flown in December 2014. Following a successful flight following exercise of the EFT-1 mission running on a Linux PC, the software was refactored in C and C++ and executed within the cFS framework. Upon successful execution of the converted software running in the cFS framework on Linux, we transitioned to execution on a PowerPC 750 (PPC 750) SCS750®

Single Board Computer (SBC) developed by the Data Device Corporation (DDC) (formerly

Maxwell) with the VxWorks Operating System. When successfully executing the applications on the PPC and confirming that functional results matched the results obtained from the

Linux system, we were able to conduct a thorough performance analysis. The performance analysis that we have completed will provide much-needed information about the processing and data capacities that will be required for autonomous fault management applications. We discuss methods for deriving flight processing requirements for scaled up autonomous fault management applications, as well as methods for extending our methodology to a wider class of autonomy applications, such as planners, schedulers, procedure execution applications and decision reasoners that will comprise the suite of autonomy applications needed for safe and successful human flight into the Solar System.

1 Computer Scientist, Intelligent Systems Division, M/S/269, NASA Ames Research Center, AIAA Member 2 Computer Scientist, Intelligent Systems Division, M/S 269, NASA Ames Research Center, AIAA Non-Member 3 Computer Scientist, Intelligent Systems Division, M/S 269, NASA Ames Research Center, AIAA Non-Member

I. Nomenclature

ACAWS = Advanced Caution and Warning System

API = Application Programming Interface cFE = Core Flight Executive cFS = Core Flight System csv = Comma-separated Values

DDC = Data Device Corporation

EM-2 = Orion Exploration Mission #2

FIR = Failure Impacts Reasoner

FMEA = Failure Modes and Effects Analysis

GHz = Gigahertz

Hz = Hertz

ISP = Information Sharing Protocol

ISR = Interrupt Service Routine

MHz = Megahertz µsec = microsecond msec = millisecond

OSAL = Operating System Abstraction Layer

PPC = PowerPC

PSP = Processor Support Package

SBC = Single Board Computer

SBN = Software Bus Network

TEAMS = Testability Engineering Analysis and Maintenance System

TMR = Triple-Module Redundancy

VBA = Visual Basic for Applications

II. Objectives and Motivation for Performance Analysis

As spacecraft autonomy technology has been developed, challenging problems have been researched and resolved to assure that high quality, accurate and usable results can be produced by software under a wide variety of conditions and circumstances to relieve human experts of the need for continuous monitoring of spacecraft systems. As these autonomy technologies have matured, prospects for using and trusting autonomy technology are moving toward the realization of the promises such technologies have long offered.

A remaining challenge is to assure that intelligent, robust autonomy applications can be deployed on spacecraft with limited computing and data storage resources. Technology researchers have often used desktop environment computers that have become essentially unlimited to develop, test and validate autonomy application algorithms and architectures. Transitioning to flight computers requires significant revision of software to work on processors with slower clock speeds, different memory and register architectures, and significantly less programming language support. We have scaled an autonomous fault management application that performs fault detection and system-level fault diagnosis to cover a significant number of faults in multiple subsystems, and have conducted an analysis of processor and memory usage to first confirm that our applications can fit in a flight-like processing environment, and then to determine scalability metrics that can be used to size the flight processing requirements for highly autonomous spacecraft and to target areas for additional optimization research.

III. Technology Description

The fault management applications that are the subject of our performance analysis is a model-based detection and diagnosis system that we call the Advanced Caution and Warning System (ACAWS). ACAWS obtains data from the system, in our case the Orion Spacecraft, performs a set of tests on the telemetry that reduces numeric data to PASS, FAIL or UNKNOWN results, sends the test results to a Diagnostic Executive, and distributes the diagnostic results to consumers, who could be other software systems or human operators (McCann, 2013). In addition to diagnostics, ACAWS also performs failure impact assessment to identify equipment or functionality that has been affected by a system failure, such as the equipment that is lost when an electrical failure results in loss of power to parts of the system. The failure impact assessment technology, however, was not a part of our performance analysis. ACAWS in its current form was developed as a ground based flight-following technology demonstrator for Orion EFT-1 (Aaseng, 2015), but related elements have previously been used to diagnose faults for the Deep Space Habitat (DSH) (S.

Colombano, 2013) and the Ares I-X Ground Diagnostic Prototype (M. Schwabacher, 2010).

A. Software Architecture

The primary architectural components of ACAWS include:

- Fault Detector. Named acaws_fd, this application obtains data from the spacecraft, performs data cleaning, filtering and framing, determines spacecraft state information that is applicable to a test, and then performs a specified test on the data value. The output is a PASS, FAIL or UNKNOWN value for each test specified in a fault model.

- Diagnostic Executive. Named acaws_de, this application receives the PASS, FAIL and UNKNOWN test results from acaws_fd and passes the results to the Diagnostic Executive, TEAMS-RT®. TEAMS-RT® is a commercial software product, customized for execution with VxWorks on PowerPC processor and linked as a software library into the acaws_de application. acaws_de obtains results from TEAMS-RT® and distributes them as cFS messages to end users.

- Failure Impacts Reasoner (FIR). Also known as acaws_fir, this reasoner receives diagnostic results from acaws_de and identifies the system components and functions that are impacted by the diagnosed root cause failure.

This element is not a part of the performance assessment but is an important element of a fault management architecture, and we will discuss plans and approaches for assessing performance on this and similar types of reasoners. Since it needs to run only when a new failure has been detected, it can be scheduled at low priority as a background task.

acaws_fd and acaws_de are executed cyclically, normally at a 1 Hz rate. The execution rate is controlled by receipt of data from the telemetry system. ACAWS obtains telemetry data from the Mission Control Center’s telemetry distribution system, known as Information Sharing Protocol (ISP). ISP normally generates data at 1 Hz, although the rate can be increased or decreased when playing back recorded data, if needed. ISP was developed as a ground system running on Linux, and a separate application was developed for ACAWS using these libraries to act as a middleware, formatting the telemetry into cFS messages, and transmitting the messages to ACAWS on the PPC via another cFS application, Software Bus Network (SBN). This application, named isp_bridge, provides a sequence of messages each containing approximately 2000 data elements. ISP provides data only when a value or status changes, so the amount of data can vary considerably.

The software architecture for the analysis is shown in Figure 1.

B. Diagnostic Model

A diagnostic model of significant parts of the Orion spacecraft was used as the basis of our performance analysis.

The model was built with the Testability Engineering Analysis and Maintenance System (TEAMS®) tools provided by Qualtech Systems, Inc. (A. Mathur, 1998), consisting of components and their failure modes, and test points signifying the sensors and data that provide insight into the health of the system at various points. Tests, based on the sensor measurements from the spacecraft, are designed to detect one or more system faults. The diagnosis is based on these test results that are either PASS, FAIL or UNKNOWN. Any single FAIL test result implicates any failure mode which the test detects, while any PASS test result exonerates any failure mode that it detects. Any failure mode that is implicated and is not exonerated, is then diagnosed as a fault condition. Whenever possible, the model is developed to isolate fault conditions to a single, unambiguous failure mode. If sensor data is insufficient, or data is missing

(perhaps due to another failure), or for other causes, the diagnosis may be ambiguous. In this case, an ambiguity group with N possible failure modes could exist in which at least one failure mode is definitely present, but it is not possible to know which one of the N is actually present.

Our applications are designed to produce UNKNOWN results when there is any doubt about the data. By taking a conservative approach, we reduce the likelihood of either a false diagnosis or a false exoneration as much as possible.

The approach requires logic to validate any data before testing it. We test sensor data for off-scale values and exclude them from qualitative testing, and we also test selectively for non-changing data. If we normally expect to see sensors jittering from cycle to cycle, a sensor that stops changing can be excluded until it changes again. This ‘flatline’ condition could be used to indicate a fault, such as a failure of an analog to digital conditioning component, or it may be used only to suppress testing of a value from a flatlined sensor temporarily. The Fault Detector also suppresses evaluation of slow rate data when a value is not expected. The Orion spacecraft uses one-tenth Hertz (Hz) data for some sensors and data. When an updated data element arrives it is evaluated as either a PASS or FAIL result, but for the next 9 seconds, any test on the one-tenth Hz data is evaluated as UNKNOWN, so that if a failure occurs in between know that the valve is stuck closed, without knowing the specific physical cause. The failure modes per component, and the tests per failure mode, may vary considerably for other systems. The Orion model generally looks for primary component loss of function, without digging deeply into the exact root cause of a failure. Components such as valves or switches usually have a failed open and a failed closed failure mode, since the functional behavior and operational impacts are very different, but when different failure modes result in identical functional effects, we usually modeled a single general failure of the component.

C. Fault Impact Reasoner

The concept behind the Fault Impact Reasoner (FIR) is to leverage the existing work done to create the diagnostic model of the vehicle, but make a determination of the components impacted by a particular fault. In many ways, this can be visualized as the inverse of the diagnostic problem. Because the TEAMS® model is a fault-propagation model, wherein the faults are detected by sensing off-nominal conditions at certain test points (usually sensors on the vehicle) and flowing upstream to implicate the component failure modes that may have caused those off-nominal readings, the opposite can also be done. Once a failure mode is implicated, the path connecting it to all test points that might detect that failure give an idea of the components that also may be impacted by the failure. The term used for this backwards search of the model is “reverse propagation”. Of course, this means that the system must be modeled in such a way as to faithfully represent the connectivity of components; in some ways the model cannot be as simplistic as a “pure” fault-propagation model. A corollary “best practices” document for modeling had to be created during development in order to ensure that the model could be used in this way for both diagnostics and impacts.

For the FIR to operate, much of the work must be done up front in order to realize fast-time lookup of the impacts.

There are many more components along the impact path than there are failure modes along the failure path. The search space is also made more complex with the concept of “loss-of-function” versus “loss-of-redundancy”. Some components may have built-in redundancy that allows them to continue providing functionality through the use of a back-up string, even though they have been directly affected by the upstream failure. This is a case of “loss-of-redundancy” and it is important to alert crew or ground support that another independent failure may cause the component to lose complete functionality by taking out this back-up string. In the case of multiple failures then, all independent propagation paths affecting the same component must be searched in order to determine if it has lost full functionality or only lost redundancy. This has the effect of increasing the search space and eliminating the possibility for static lookup tables as a viable solution to the problem. Similar complexity is added when the system/vehicle changes modes of operation, where once viable propagation paths may be broken or re-connected. The current implementation handles this complexity by pre-computing a simplified version of the model in graph form that prunes all nodes not on the search path, and inserts additional nodes that can be turned on or off to enable or disable flow based on the state of the system. The resulting search operates on the simplified model and behaves similarly to a lookup table until it comes across certain decision points, at which time it briefly delays the lookup to resolve all other paths before continuing.

The FIR has been primarily executed on a Linux machine that receives inputs from the Diagnostic Executive.

Initial indications are that it requires substantially more processing under some conditions than other ACAWS applications, and that the processor resources can vary dramatically depending on the inputs. If there are no failures, or a failure with only a single impact, processing time is quite rapid. A failure with many impacts may require substantially more processing time. We have taken strides to minimize the processor resources required, but did not complete those improvements in time to include them in our performance analysis.

IV. Hardware and Operating System Environment

We conducted performance analysis on two PowerPC family processors, running the VxWorks Operating System and the Core Flight Executive (cFE) flight software middleware and identical application source code. Each of the architectural elements are described in the following paragraphs.

A. DDC SCS750® Processor

The processors used for the analysis are PowerPC family of processors from two vendors. The first is the SCS750®

Single Board Computer (SBC) from Data Device Corporation (DDC). The SCS750® processor is a space-proven and radiation tolerant SBC designed for high reliability and low power consumption. It operates in a cPCI backplane with a 6U form factor with a Triple-Modular Redundancy (TMR) PowerPC 750FX™ processor running at 400 MHz.

The board was designed, according to the manufacturer, for “… high performance computing for the most demanding space applications. Its design decisions have been driven by a guarantee of the highest reliability and performance. DDC has developed a comprehensive strategy to provide total dose, latch-up, and upset hardness for the

SCS750® SBC.”

The SCS750® features TMR in which three duplicate processors execute in lockstep. The output of each clock cycle is voted and the majority is output without delay. It has 256 Mb Synchronous Dynamic Random Access Memory

(SDRAM), with Reed-Solomon protected, Double-Device Data Correction volatile memory, and an additional 8 Mb

Electrically Erasable Programmable Read-Only Memory (EEPROM).

The SCS750® was procured by NASA in about 2014.

B. AiTech SP0-S Processor

The second processor is an SP0-S from AiTech Defense Systems, Inc., based on a PowerQUICC® III MPC8548E processor, developed by NXP Semiconductors (formerly Freescale), executing at 1 GHz with 1 Gb Double Data Rate, version 2 (DDR2) memory with Error-Correcting Code (ECC) in a 3U form factor cPCI bus.

The board manufacturer describes its radiation tolerant design features, stating that "Radiation Tolerant design with mitigation and/or immunity to SEE, TID and Latch-up. Radiation tested in UC Davis and IUCF to 100 krads (Si)

TID. The VxWorks BSP offers mitigation of Single Event Upsets; the resulting cross-sections are available upon request. The MPC8548E processor is manufactured using 90nm SOI technology and is immune to latch-up; other components in the standard configuration are latch-up immune to at least 65 MeV-cm2/mg for the SP0–S.”

The SP0-S was acquired by NASA in 2018, after using its predecessor board, SP0 for avionics development and evaluation work.

V. Software Test Environment

The analysis environment includes the VxWorks operating system, the cFE middleware layer and the application tasks that are the primary focus of the evaluation.

A. VxWorks

Software was executed on the SBCs using the VxWorks 6.9 Operating System from WindRiver Systems, Inc.

VxWorks is a real-time operating system used for many embedded devices and systems, including several successful space flight missions. It provides task management, signaling and communications, memory management and device management for a wide range of processors. Individual execution elements are known as tasks that are managed by

VxWorks. A task can be in one of 4 states, with valid transitions as shown in Figure 2.

The VxWorks task states are

Pended. The task is waiting for some event or condition to be satisfied before it can run.

Ready. All conditions required for execution have been met, but some other task at equal or higher priority is executing

Delayed. The task is asleep for some period of time

Suspended. The task is unavailable for execution, normally used for debugging or error management.

B. cFS

Core Flight System (cFS) and its foundation level Core Flight Executive (cFE) provide an architecture-neutral application middleware that provides for application portability from one processor and operating system to another with minimal application modification. Developed initially by Goddard Space Flight Center, it has been adapted for several projects to provide a common software environment that reduces the cost of software development by promoting software reuse. Using a Processor Support Package (PSP) and Operating System Abstraction Layer

(OSAL), it adapts commonly used operating system functions such as file management, input and output functions, signaling mechanisms, and message handling to a common Application Programming Interface (API) that is consistent across many different platforms. The cFE includes VxWorks tasks to perform time management, event services and the message-passing services that comprise a Software Bus, and cFS applications that support the ACAWS fault management applications. The list of tasks that execute regularly during ACAWS operations and are included in the performance analysis are shown in Table 2, below. These include cFE tasks, the cFS applications including the

ACAWS components and VxWorks tasks.

Figure 2 - Task state transition diagram shows the allowable transitions between states. The evente that can induce a state change are shown in the table in the lower section of the figure.

The cFS applications include:

- sbn. Software Bus Network is used for receiving and sending cFS messages between different processors over

Ethernet. It extends the cFS software bus, used for inter-task communications in cFS, across platforms. Applications can send and receive data without needing to differentiate between data coming from the same processor or a different one.

- sch. Scheduler provides periodic wakeup messages to applications. Receipt of a sch message can cause a task to change from a pended state to a ready state.

There are several VxWorks tasks that are always loaded and executing, and additional tasks provided by the cFS framework. These tasks can be configured and controlled by settings used to build the VxWorks kernel or the cFS framework. For the purpose of this analysis, these are considered fixed and always present.

Some of the additional VxWorks and cFS tasks that are running are described here. Only those that take appreciable amounts of time or affect performance are described. There are other tasks that run only sporadically or use miniscule execution time (less than 0.01% of CPU usage) that are not described.

- tExcTask. The VxWorks Exception Handler runs at a high priority, executing at 40 Hz (every 25 msec) and executing for about 21 microseconds (µsec) per iteration. With 40 iterations per second, the total task execution time of 850 µsec per second is not a significant use of time, but because the task is a high priority, it interrupts the ACAWS tasks many times during execution. The Exception Handler task initiates the cFS Scheduler task that is described above.

- tNet0. This is an asynchronous task that executes when there is Ethernet traffic either coming in or going out.

It often is associated with sbn, but also handles other Ethernet traffic. tNet0 execution times can range from a 20 µsec to about 1 msec. Periodically there is a flurry of execution cycles within a few msec, and extended periods with no activity. The tNet0 task is often in a Ready state but preempted by higher priority tasks throughout execution, such as when acaws_fd is running.

The SystemViewer data was first observed in the graphical displays that show task execution and significant events. SystemViewer has features for filtering by task, filtering by event, zooming in and out to look at various durations. An example is shown in Figure 3. The executing tasks are listed on the left, and pertinent task and event information is graphically shown at the right. The legend for commonly displayed information is shown below in

Figure 4.

Figure 3 - Example System Viewer plot shows tasks at the left, and execution states, durations and events in the plots to the right. The dotted magenta line indicates a task in a pended state, the cross-hatched line indicates a delayed task. Event markers indicate semaphores, message queues, timers and other events that can cause tasks to change execution state.

SystemViewer can zoom in to observe detailed execution and obtain fine-grain time measurements. A time tool can mark the beginning and end of a selection and provide the duration, start and end time to provide accurate measurements of selected sequences. The example in Figure 5 shows Interrupt1 causing the Exception Handler Task

(tExcTask) and sbn task to change from pended to ready, as indicated by the horizontal line changing from a dotted magenta to a green jagged line. The tExcTask, at a higher priority, runs to completion, and then sbn executes. Both the sbn and sch to change to the Ready state; sbn, at higher priority, runs until it requests a message queue and changes to a pended state. Then the sch task executes when it becomes the highest priority task in a ready state. It takes a semaphore and changes state to pended, awaiting a semaphore to be set by another task or ISR. It requests a message queue and changes to a pended state. The sch task changes from pended to running executes, and transitions to pended state and the execution returns to the ‘idle task’ waiting for the next event. 75.3 µsec has elapsed from the first interrupt that transitioned out of the idle task, until the sch task completes and returns to idle. This example is shown to provide an introduction to the kind of information that is available from the SystemViewer tool, not to do an analysis of the scheduler or sbn performance.

Figure 4 - The primary event and state indicators used in the System Viewer graphical depictions used in this report.

Figure 5 - An example of a detailed view covers about 200 µsec. Each tick mark in the scale just above the plot area is 10 µsec. Time is measured from the beginning of an ISR to completion of tasks whose state changed as a result of the interrupt processing.

VII. Performance Results

Our analysis was conducted over several months in which improvements to application software and expansion of the diagnostic model were completed in parallel with our analysis. Initially we had access to a single SCS750® PPC processor, and later added the SP0-S processor. The dynamic development and analysis environment reduced the statistical rigor that could be achieved in a constant hardware and software baseline, but it enabled quick comparisons of the effect of model, software and data changes. At this early phase of the analysis, the ability to characterize changes was considered more beneficial than a highly rigorous statistical benchmarking activity.

We will describe the results of our analysis in three different aspects. First, we describe a progression of performance on one processor, the SCS750®, as the software was changed, often as a result of prior measurements.

While not a formal code optimization process, the act of measuring software performance yielded information that was critical to making improvements. Second, we describe a series of measurements as we changed the amount of data to be processed by the ACAWS application. Our third analysis was a processor comparison of the SCS750® and the SP0-S at the maximum loading level that our software and Orion model could produce.

A. Software Improvement through Measurement

At the beginning of performance analysis on the SCS750® processor, the CPU was executing at over 100% capacity. When we began to assess the data, we quickly found substantial improvements to application software that reduced processor usage dramatically. These improvements included compiler settings as well as source code changes.

Once we were able to question why certain activities used as much resources as they did, it was very natural and relatively easy to make improvements in performance. We describe several discoveries made during our initial performance assessment and the improvements that were made as a result. The improvements described in the following paragraphs are summarized in Table 3.

A cursory review of acaws_fd development revealed that the build had been done with compiler debugging options turned on in parts of the software. The bulk of the ACAWS processing is performed in C++ libraries that are compiled separately from the C code that is compiled and linked with the cFS application load. These libraries were rebuilt with debugging compiler options turned off and the analysis was executed again, with notable improvements in performance.

We then undertook a series of code reviews to look for performance improvement opportunities. We found that copies of objects were made, when a reference to an existing object would provide the same result. Changing code from:

MonitorContainer::doEvaluations( const FdValueContainer valueContainer, const std::vector<bool> modes) to MonitorContainer::doEvaluations(

FdValueContainer const & valueContainer, std::vector<bool> const & modes) yielded an improvement of about 180 msec per execution cycle. Just two ‘ampersand’ characters in the source code made quite a difference!

We then reviewed the doEvaluations function, since it was a primary consumer of resources. We noted that it was retrieving a copy of an object, then making copies of its vectors of values and statuses. We added accessors to the container object to look up the value and retrieve a copy of the values directly, and the same for statuses. This was a more extensive change than the above, and gained an additional 40 msec improvement.

Continuing the review, we found additional cases where functions were passing arguments by value rather than by reference, and with accompanying changes to other derived classes, found that an additional 35 msec could be gained.

We next reviewed some internal data logging that had been added for functional analysis and debugging. While the logging provided valuable functionality during development and testing, it is not required operationally, and consumed quite a bit of processor time. We modified our process to provide an option to build with the logging or without it. Without the additional data logging we reduced an additional 100 msec, while retaining the ability to use logging for functional analysis and debugging when processor performance was not a high priority.

Another substantial improvement was realized by additional refactoring of the most-used data class objects. The original design had allowed for an option of using multiple threads within the Fault Detector, but by this time we realized that we would not need a multi-thread design within the Fault Detector. Without maintaining support for multi-threading, we found that a single data object could be created and used continually, rather than creating, copying and destroying objects every cycle. This required extensive rework of some functions but gained an additional 100 msec.

The final group of improvements were from several small changes made over several months. Once our development group started focusing on performance, finding improvements during routine functional testing and software maintenance became a normal part of development and gradually yielded improvements.

Table 3 – Significant improvements that were made during initial performance assessments on the SCS750® processor are summarized

Measurement

Average

Execution

Time (msec)

Description

1 1,155 Original measurement

2 650 Debugging compiler options removed

3 465 Changed copies of objects to references to objects

4 425 Added accessor functions to retrieve values

5 390 Changed additional functions to pass arguments by reference

6 290 Turned off internal data logging for functional analysis

7 190 Refactored value monitors

8 180 Final miscellaneous gains

The ACAWS Diagnostic Executive includes the COTS diagnostic reasoner, TEAMS-RT® and interface code that receives inputs from the Fault Detector, packages them for transmission to the TEAMS-RT® APIs, receives results from TEAMS-RT® and transmits results via cFS messages to off-processor applications including operator displays and data archives. Initial performance for the Diagnostic Executive was about 180 msec per cycle, and performance and code reviews similar to that described above for Fault Detector yielded similarly dramatic improvements. Final performance results for the Diagnostic Executive were reduced to about 35 msec per second. Most of the processing time was used by data preparation and distribution of diagnostic results. The actual diagnostic reasoning was found to use between 5 and 10 msec per second, and since this was such a small percentage of overall processing time, we did not conduct further assessments of the TEAMS-RT® diagnostic reasoner.

The remaining tasks, including network communications, timing control, and underlying cFS and VxWorks functionality use about 14 msec per second. Since these are much less than the ACAWS tasks, we did not focus considerable attention on these tasks. During initial assessments, however, we noted that task priorities in some cases were not as intended. The scheduler task was being run at a lower priority than some of the tasks to be scheduled, resulting in the scheduler not being able to run at the 40 Hz design rate. When we adjusted the priorities, all tasks executed in accordance with the planned timeline.

While it may be obvious, a clear observation from our initial assessment is that improvement requires measurement, and the converse, that measurement is a powerful driver of performance improvement.

B. Steady State Performance

Our final software and model configuration after performance improvement was analyzed to determine variability of task execution over time, best and worst case performance, timing stability, sequencing of tasks, and other performance characteristics. We also reviewed memory usage and variability to determine if any trends were apparent.

We conducted the most detailed analysis on the SCS750® processor and only later, compared selected results to the

SP0-S processor.

Processing times were measured by exporting performance data to a file in comma-separated values (csv) format.

The data for each measureable event included a timestamp, the task (also known as the context) associated with each event, the event name and event class, the CPU time used by the event, and event parameters.

In the raw data, execution times were associated with an event. When the event is a state change or an interrupt, the task that used the processor for the execution time associated with the event may not be the same as the context associated with the event. As higher priority events, including interrupts and their corresponding ISRs preempt execution of lower priority events, it was necessary to determine a task owner of each block of execution time. When interrupts preempted a task, we assigned a task parent so that when the ISR exited, we could correctly determine the task that was using the CPU at the point of interrupt and resumed execution on completion of the ISR. In some cases, interrupts were themselves interrupted by other interrupts, and when all the ISRs were completed, we were able to correctly assign time usage to the proper task. Once the time owner for each block of time was determined, we could then add the execution times for each task to accurately measure the total CPU time used by each. The task usage times extracted from the logged performance data established the primary basis for our CPU usage analysis.

In a hard-deadline real-time system, the processor usage of the worst-case cycle is critically important, because any cycle that exceeds the allowable processing time could result in loss of critical information and possibly a shutdown of software tasks. Designs can mitigate the impact of worst-case cycles by building in resilience to data loss, but we focused our attention in this part of the analysis on worst-case processor times rather than averages, since time from a lower loaded cycle cannot be recovered and used to make up for longer running cycles.

We directed our analysis at the largest utilizers of processor time, which continues to be the ACAWS Fault

Detector, followed by the ACAWS Diagnostic Executive in a distant second place, and the remaining cFS and

VxWorks functionality which uses relatively little processing.

We observed a notable processor utilization spike every 10 seconds in the ACAWS Fault Detector, and a similar, but smaller percentage difference in the ACAWS Diagnostic Executive task. The Orion data, like that of other real-time spacecraft, is organized in multiple rate groups; Orion data ranges from 40 Hz to 0.1 Hz. Because ACAWS operates at a 1 Hz rate, it takes a 1 Hz sample of any data with rates faster than 1 Hz, and ignores other data elements within that data cycle. For data at rates slower than 1 Hz, however, ACAWS only evaluates data in the expected cycles and passes an UNKNOWN result to the Diagnostic Executive on all other cycles. Orion transmits a considerable amount of data at 1/10th Hz, but distributed among 1 Hz minor cycles, so we would not expect the Orion data amount to be any higher at the boundary of a 1/10th Hz major frame. We continue to evaluate the cause of the processing spikes every 10 seconds; it could be due to unexpected bunching of the Orion data that we used, or there may be

ACAWS software characteristics that drive additional processing at an assumed 10-second boundary. Further analysis will be needed to determine the cause of the spikes and to make design improvements to flatten performance. We assume that a flatter, more even processing time per second is desirable for cyclic applications, and regular periodic spiking indicates that processing has not been optimally distributed among data cycles. If the spikes can be distributed among all cycles, we could anticipate a reduction of the maximum cycle time to one that is much closer to the average processor time. The maximum ACAWS Fault Detector processing time was found to be 178 msec per second, with an average of about 164 msec, over several data runs of about 30 seconds each. One sample is plotted in Figure 6 showing the overall ACASWS Fault Detector processing times, with a 1/10th Hz processing spike.

Figure 6 – Processing times for ACAWS Fault Detector over a 33-second sample show processor spikes of about

15 msec every 10 seconds.

The ACAWS Diagnostic Executive executes at a 1 Hz rate, driven by completion of the ACAWS Fault Detector.

When the Fault Detector completes processing, it puts its results into a cFS pipe, which in VxWorks is implemented as a message queue. When ACAWS Fault Detector sends its output message, the pipe causes ACAWS Diagnostic

Executive to change state to Ready, and when no other higher priority tasks are in a Ready state, the Diagnostic

Executive executes. The transition from Fault Detector to Diagnostic Executive is on the order of 600 µsec. Figure 7 shows a SystemViewer execution plot of a typical transition.

The Diagnostic Executive executes the cycle’s diagnostic reasoning, obtaining results from the TEAMS-RT® reasoner and then packaging results into cFS message structures. The Diagnostic Executive then waits for a message from the cFS Scheduler to transmit the message, assuring that diagnostic results messages are transmitted at precisely a 1 second rate. The message transmission requires approximately 1.2 msec to transmit a message containing the diagnostic results for the approximately 5400 failure modes in the model.

The Diagnostic Executive maximum processing time was 42 msec, with an average about 33.5 msec and a minimum of 28.8 msec in a typical 30 second data sample, as shown in Figure 8. The TEAMS-RT® diagnostic reasoning consumed about 5 to 10 msec. We did not fully characterize the cause of the variability. Analysis conducted by Qualtech Systems, Inc, who makes TEAMS-RT®, have determined that execution times can vary by the amount of data that changes from one cycle to the next. The Orion data would typically result in substantial number of tests changing between PASS and UNKNOWN, which could help to explain the variations. While a range of 14 msec may not be huge, it is a degree of uncertainty that warrants additional investigation. Time-critical software execution with peaks that vary significantly from the mean value may require extra margins of processor time, and make performance requirements verification more complex, so resolving unexplained dispersions early in development will be beneficial.

Figure 7 – A transition from ACAWS Fault Detector to Diagnostic Executive tasks shows the messages sent to the network and the Diagnostic Executive change the state of the receiving task from Pended to Ready, and task execution begins when no higher priority task is in the Ready state

Performance and stability of the underlying operating system and cFS middleware tasks were also reviewed to determine if jitter, drift or other variation could potentially impact execution performance. Overall processor utilization of the remaining tasks range from 12.7 msec to 15 msec, with an average of 13.3 msec. These included 3.7 msec for ISRs, 5.1 msec for cFS tasks including the Scheduler, SBN and time services tasks, and 4.4 msec for

VxWorks tasks including network drivers, exception handlers and data management.

As autonomy applications are considered for use in critical spacecraft control decisions, software robustness becomes a driving requirement, so our analysis has attempted to at least begin to look ahead to high-performance critical software application considerations. We noted a timing anomaly in the underlying cFS implementation. The cFS applications were configured to execute a Scheduler at 40 Hz (every 25 msec) based on a system clock interrupt at 5 msec. The Scheduler normally runs every 25 msec, with a maximum variation of 78 µsec. However, once per second, the Scheduler runs at 20 msec since the last Scheduler cycle, followed by a cycle that is 30 msec since the last

Scheduler cycle. These appear to be related to a resynch to the 1 second major frame that occurs every second, but apparently is not correctly implemented. We noted a similar, albeit different anomaly with the SP0-S processor; once per second, there was a single 35 msec cycle while all others were the expected 25 msec. While these anomalies did not significantly disrupt ACAWS processing, these sorts of timing anomalies could be potentially damaging. Finding and resolving such discrepancies early in development of applications will help to ease integration and verification of real-time systems.

Figure 8 – Processing times for the Diagnostic Executive show moderate variation from cycle to cycle.

C. Scalability Analysis

The initial analysis focused on performance of a model of constant size and data throughput. We conducted additional analysis to characterize the factors that affect performance as model size, data throughput and other factors change.

We assessed both the Fault Detector and the Diagnostic Executive processing time when varied by the number of tests for which a test algorithm was defined and for which data was supplied, using the Orion model with a constant size of 5372 tests defined in the model. The ACAWS Fault Detector reads additional data files at initialization that define what telemetry elements to use, and what to look for in the telemetry, in order to create a PASS, FAIL or

UNKNOWN result to send to the Diagnostic Executive. In the Fault Detector scalability analysis, we varied the number of tests defined in the Fault Detector initialization data files, as shown in Figure 9. The analysis confirmed that the processing time for the Fault Detector is a linear function of the number of tests defined

. With a minimum of two tests, the Fault Detector task used approximately 3.5 msec, indicating a task overhead of about 3 msec (it would not initialize and execute if no tests are defined), and scaled linearly. As currently implemented, all tests are simple tests that compare a single value to another value. If more complex tests are developed, we would expect more variation in the processing time per test, but we would still expect the total time to be the sum of the individual tests. Our initial analysis indicates that there are no interactions between tests that would result in exponential growth. For the SCS750® processor, processing times ranged from 45 to 53 µsec per second per test over several samples of varying sizes.

The Diagnostic Executive times also varied with the number of tests defined, but the variation is more complex.

Figure 10 shows the Diagnostic Executive performance times as the number of tests defined for the Fault Detector changed. The reasoner itself is processing a constant size model, and processing times for the TEAMS-RT® model are expected to depend the number of failure modes and tests in the model. The TEAMS-RT® reasoner is optimized to use recent data history, so cycles in which there are large amounts of change would take more processing time than cycles with little change from prior cycles. Qualtech Systems, Inc. has performed considerable optimization and performance analysis of the reasoner and we did not duplicate that analysis in our studies. The Diagnostic Executive task includes processing to package data from the Fault Detector task to send to the TEAMS-RT® reasoner, as well as to receive and process the results from the reasoner. The variation in Diagnostic Executive times appears to depend on the number of tests with an UNKOWN value. There is a minimal processing time when all values are UNKNOWN, which for the SCS750® processor, is about 11 msec per cycle. Processing time increases when more tests have values, varying from about 6.9 µsec per test to 12 µsec per test. This variation could be due to data changing from known to unknown, or it could indicate that it takes more time to process known values than unknown.

Figure 9 – Fault Detector scalability analysis shows a linear scaling based on the number of tests defined in the

Fault Detector test specification data. Using a model containing 5372 total tests, the analysis varied the number of tests actually mapped to telemetry from a minimum of 2 to 3500.

Figure 10 – The Diagnostic Executive execution times varied with the number of tests defined in the Fault

Detector initialization data. The Diagnostic Executive processes a constant size data vector, so the variation is only in the value of the tests. With few tests defined, the test results contains mostly UNKNOWN results, and as more tests are defined the values have more PASS or FAIL results and fewer UNKNOWN values.

D. Processor Comparisons

We repeated some of the analysis on a newer processor, the SP0-S processor built by AiTech Corporation. We repeated the largest configuration in which 3500 tests were defined and recorded the performance. We measured the

Fault Detector, Diagnostic Executive, and all other tasks on both processors. The SP0-S processor was considerably faster in all measurements. The SP0-S is a newer board, running at 1 GHz, while the SCS750® executed at 400 MHz.

The SCS750® can be configured to run at up to 800 MHz, but we used the default configuration of 400 MHz for our analysis. The SCS750® is designed with Triple-Module Redundancy (TMR) and several other features to assure high reliability and radiation tolerance; the SP0-S uses component level radiation tolerant design and mitigation strategies to achieve immunity from latchup and mitigation of single-event upset. We did not attempt to benchmark processors or account for the hardware features that could contribute to differences in performance. Our primary purpose was solely to gather preliminary information on the potential range of processing capacity available to autonomy applications. Requirements for power usage, mass, reliability, radiation tolerance and many other factors drive the selection of processors for spacecraft avionics design. The performance analysis that we have conducted could be used as one of many inputs into the specification of processors that support autonomy applications. We assume there are many processor choices available, and we can make no conclusions about where the two processors on which we measured our application’s performance fall on the performance scale of all viable processors.

There were notable differences in the task-based comparisons between the two processors that were not fully characterized. The Fault Detector on the SCS750® used 2.24 times as much processor time as the SP0-S, but the

Diagnostic Executive on the SCS750® used only 1.36 times as much processor time as the SP0-S, while the remaining

ISR, VxWorks and cFS tasks used 4.7 times as much as the SP0-S. We have not yet attempted to account for the variation in scalability factors between the two processors. Use of memory, caches, network access times and other factors could be involved. It appears that numerous factors in addition to CPU processing speeds have an effect on overall processing times. Overall, the performance differences are in line with the difference in the CPU clock speed differences.

Table 4 – Average processing times on the two processors used in the analysis, with execution times in msec

SP0-S SCS750®

Fault Detector 73 164

Diagnostic Executive 26 36

VxWorks and cFS Tasks 3 13

Processor Clock Speed 1 GHz 400 MHz

VIII. Performance Requirements and Path Forward

A primary objective of the analysis was to project processor requirement…