APCS SIR Sect L Att L.7 Amend 6.docx

Federal Aviation Administration

U.S. Department of Transportation

APC System PDP Stability Assessment Requirements

September 01, 2022

Focal Point Air Traffic Organization (ATO) Air Traffic Control (ATC) Communications Services Voice Switching and Recording Team, AJW-315

800 Independence Avenue SW Washington, DC 20591

i

Table of Contents

Introduction and Overview This document describes the requirements for conduct of the Stability Assessment to be conducted on the Air-to-Ground Protocol Converter (APC) System Previously Developed Product (PDP). The PDP Stability Assessment will occur during APC System source selection activities. The PDP Stability Assessment will be conducted by the Offeror at the William J Hughes Technical Center in Atlantic City, NJ.

PDP Loading Tool Requirements Functional Loading Tool Requirements The loading tool must contain all required software/hardware to perform all defined functions when connected via a networked interface.

The loading tool must contain all required software/hardware to perform all defined functions when connected via a legacy interface.

The loading tool must meet the maximum system sizing requirements listed in Table 3-6 while supporting traffic loads specified in Table 3-7 of Attachment J.1.1, APC System Technical Specification.

The loading tool must simulate the management and call traffic activity[footnoteRef:2] of APCs up to the sizing requirements listed in Table 3-6 of Attachment J.1.1, APC System Technical Specification. [2: APC management activity is defined as any network and/or data processing load placed on the APCS that does not pertain to placing and receiving calls. An example of this would be periodic status poll requests/responses to/from a management system. APC call traffic activity is defined as any network and/or data processing load placed on the APC System by APCs that supports call setup and audio transmission. Examples of this would include all SIP and RTP traffic to/from APCs.]

The loading tool must meet the PBM Call Distribution requirements listed in Table 3-7 of Attachment J.1.1, APC System Technical Specification.

The loading tool must simulate 100% Peak Busy Minute (PBM) call traffic load for up to maximum APC gateways allowed per Table 3-6 of Attachment J.1.1, APC System Technical Specification.

The loading tool must initiate and receive simultaneous A/G calls.

The loading tool must be compliant with ED-137 as implemented in accordance with Attachment J.1.1, APC System Technical Specification.

The loading tool must support simultaneous calls with, both SIP and RTP generation.

The loading tool must send and receive tones.

The loading tool or associated test equipment must measure audio quality.

The loading tool or associated test equipment must provide a minimum Mean Opinion Score - Listening Quality Objective (MOS-LQO) score for G.711 voice coding, in accordance with ITU-T P.862 and P.862.1 for all voice communications.

The Loading Tool Computer Human Interface (CHI) Functions

1. The loading tool must provide a computer Human Interface (CHI) that allows users access to features of the loading tool.

The CHI display must display in the English language.

The CHI must permit users to perform the following operations at a minimum:

1. Obtain health status of loading tool components;

2. Obtain connectivity status of APCS components under test by the loading tool;

3. Display, print and save to file all reports in format compatible with the Microsoft Office suite of products; and

4. Dynamically display the percentage of APCS load rate achieved and maintained.

Loading Tool Test Reporting

1. The loading tool must provide results of APCS loading during the APCS Stability Assessment that include but not limited to:

1. Start and stop time call traffic loading was active;

2. Total time duration call traffic loading was active;

3. Number of call traffic actions scheduled/attempted;

4. Number of call traffic actions that were blocked/failed to complete;

5. Number of call traffic actions successfully completed;

6. Average PBM load rate percentage achieved over the total time duration call traffic loading was active;

7. Average PBM load rate percentage achieved over the last 60 minutes;

8. Average PBM load rate percentage achieved over the last 12 hours; and

9. Average PBM load rate percentage achieved over the last 24 hours.

The loading tool must display and generate the following logs/reports:

1. Event Log Report containing at a minimum: time stamped entries for when loading tests are started and stopped.

2. Error Log Report containing at a minimum: time stamped entries for loading test failures; connection failures to APCS components under test; and loading tool component failures.

3. Call Traffic Loading Test Report containing at a minimum the metrics and call statistics specified in Section “a” above.

4. All logs and reports must use English as the default language.

The loading tool must report connection status to the APCS components under test.

The loading tool must store test results and error logs.

The loading tool must store all events and errors generated at the time they occur.

PDP Stability Assessment Documentation The following sections identify the activity and assessment documentation.

Stability Assessment Scripts/Procedures

1. The Offeror must develop scripts and/or test procedures for the Stability Assessment.

The Offeror must, if using scripts, present the scripts to the FAA 5 days prior to the Stability Assessment.

The Offeror must, if using test procedures, present the test procedures to the FAA 5 calendar days prior to the Stability Assessment.

Stability Assessment Hardware and Software Configuration Indices

1. The Offeror must provide a Software Configuration Index (SCI) and completed SCI Template that addresses the requirements defined in FAA CDRL/DID APC-SW004, Software Configuration Index (SCI), excluding sections 1 (Scope) and 2 (Reference Documents) and any subsections therein prior to the Stability Assessment. The SCI identifies the configuration of all the software products.

1. The Offeror must provide an electronic copy of the complete software configuration under test for the PDP. The information must include, at a minimum, software filename, software version number, date, and time each file was created.

2. The Offeror must provide an electronic copy of the complete software configuration under test for the traffic load generator. The information must include, at a minimum, software filename, software version number, date, and time each file was created.

The Offeror must provide a Hardware Configuration Index prior to the Stability Assessment. The Hardware Configuration Index identifies the configuration of all the hardware products.

1. The Offeror must provide an electronic copy of the complete hardware configuration under test for the PDP. The information must include, at a minimum, the name, the location, and the serial number of the component at the LRU level.

2. The Offeror must provide an electronic copy of the complete hardware configuration under test for the traffic load generator. The information must include, at a minimum, the name, the location, and the serial number of the component at the LRU level.

Stability Assessment Reports

1. The Offeror must, for each activity conducted, document the results of the activity.

The Offeror must present all Stability Assessment Report(s), including the Offeror Activity Logs defined in Section 3.4, to the FAA in an approved electronic format (e.g., Microsoft Windows/Office Suite or products, etc.).

Offeror Activity Logs

1. The Offeror must maintain an activity log for the duration of the stability assessment.

1. The Offeror must record all changes to the units-under-test configuration in the activity log.

1. The Offeror must record all maintenance actions to the units-under-test in the activity log.

1. The Offeror must sign the activity log at the end of the activity.

1. The Offeror must sign the activity results at the end of the activity phase.

1. The Offeror must, for each activity conducted, document daily activity information.

PDP Stability Assessment Conduct The following sections identify the activity and stability assessment activities.

General Requirements The following sections identify the general requirements for conduct of all stability assessment activities.

1. The Offeror must conduct the stability assessment in accordance with the scripts and/or test procedures provided to the FAA to verify that the solution meets the applicable requirements.

The Offeror must receive approval from the FAA for deviation from the scripts/procedures provided to the FAA.

The Offeror must note all deviations from the scripts/procedures provided to the FAA in the activity log.

The Offeror must redline all deviations from the scripts/procedures provided to the FAA on the stability assessment procedure document.

The Offeror must provide strict configuration control of the System Under Test.

The Offeror must receive FAA approval to make any changes to the installed baseline.

The Offeror must receive FAA approval to make any changes to the installed configuration.

The Offeror must document all changes in a change log.

The Offeror must inform the FAA of any deviations from, or waivers of, test requirements.

Test Briefings

1. The Offeror must provide evaluation briefings to the FAA.

The Offeror must, during the evaluation period, provide daily pre-run evaluation briefings.

The Offeror must, during the evaluation period, provide daily post-run evaluation briefings.

The Offeror must, during the pre-evaluation briefings, explain the test items to be conducted and the expected results.

The Offeror must, during the post-evaluation briefings, provide a synopsis of the day’s evaluation results.

The Offeror must, during the post-evaluation briefings, provide a synopsis of the day’s significant events.

Execution of Stability Assessment

1. The Offeror must, during Stability Assessment, verify system stability over 5 days of continuous system operations under 100% Peak Busy Minute (PBM) traffic load conditions as defined in Attachment J.1.1, APC System Technical Specification, as applicable.

The Offeror must, during Stability Assessment, verify system stability over 5 days of continuous system operations under call distribution percentages defined in Table 3-7 of Attachment J.1.1, APC System Technical Specification, as applicable.

The Offeror must, during Stability Assessment, ensure the system is running under continuous operational load for a continuous 5-day period.

The Offeror must, during Stability Assessment, ensure that at least 1 shift per day the Stability scripts/procedures include operational demonstration of complex call functions and features (including freestyle scripted scenarios, workstation operations, and management system operations).

The Offeror must, during Stability Assessment, exercise operator actions on the system in a manner that represents an operational environment (e.g., air-to-ground call functions, configuration, and maintenance operations.)

The Offeror must, during Stability Assessment, conduct the evaluation using a configuration that represents the maximum system size as specified in Attachment J.1.1, APC System Technical Specification.

1. The APC System configuration must include real system components as agreed to by the FAA.

2. The APC System configuration may include simulated system components as agreed to by the FAA.

The Offeror must, during Stability Assessment, measure and record the voice quality MOS-LQO score at least once per day.

The Offeror must, during Stability Assessment, document any deterioration of audio quality encountered during the evaluation.

The Offeror must, during Stability Assessment, document all errors that do not result in a system fault (e.g., LAN errors, Virtual Machine (VM) errors, process errors, memory errors, operating system time- outs, etc.).

The Offeror must, during Stability Assessment, document all errors that result in a system fault.

The Offeror must, during Stability Assessment, document all errors that result in a system failure.

The Offeror must categorize simple software faults as failures after 5 occurrences.

Entrance Criteria for Stability Assessment The Offeror must meet the following entrance criteria prior to the start of the Stability Assessment:

1. The APC system baseline under test has been identified;

All Stability scripts/procedures have been provided to the FAA;

Accreditation results have been provided to the FAA for all applicable test tools that are required for use in the conduct of the Stability Assessment;

The System Under Test does not have known deficiencies that affect the functions to be verified by the Stability Assessment; and, The Offeror must have certified the operation of the traffic load generator. The traffic load generator must be shown to meet the loading requirements above.

Failure Classification The Offeror must classify each system failure as Critical or Simple, and repeatable or non- repeatable as defined below:

1. A Critical failure causes the system to be unavailable to any one APC or the external interface for more than 10 seconds (including critical failures identified in Table 3-9 and 3.4.1.3.c of Attachment J.1.1, APC System Technical Specification). Unavailable means that the system does not respond to attempted inputs from that interface.

1. Table 3-9 APC System Critical Failure Parameters - The system does not suffer a cumulative loss of real resources in same time period (e.g., within the 30 minute MTTR window) for the APC system sizing per number of real A/G resources.

2. APCMS 3.4.1.3.c - The APCMS must base a Critical Failure on the loss of any of the following functions: login capability, configuration capability, maintenance functions, security functions, status monitoring and control functions (e.g., real-time status), database validation capability, or database transition from off-line to online capability.

A Simple failure is any other software failures that are not critical failures.

Failures Not Counted Against the System Under Test The Offeror must exclude any of the following from failure counts:

1. Failures that occur due to a known documented problem that has been dispositioned by the Offeror and the FAA;

Repeat occurrences of an initial failure that has been documented to the FAA and requires no operational work-around or has a suitable operational work-around;

CHI display issues (e.g., spelling errors, improper color indications on workstations, and improper sorting of reports);

Failures written in error;

Failures attributable to operator error;

Failures resulting from attempts to use unimplemented features scheduled for future releases;

Failures of any Government Furnished Equipment (GFE) unless the failures directly resulted from the System Under Test; and, Failures of test equipment unless the failures directly resulted from the System Under Test.

Verified hardware failures will not be counted toward the allowable thresholds.

1. A hardware fault shall be determined to be software if it cannot be demonstrated to be a hardware failure or due to an out-of-tolerance power condition unless the power out-of-tolerance is from APC system itself.

2. Intermittent hardware faults will be demonstrated to be hardware and corrected, or they will be logged as software faults.

Secondary failures as a result of simple common failure will be counted as a single failure.

Repeatable failures will only be counted if the failure is caused by the system itself or if the exact sequence that caused the failure is unavoidable, with no operational work-around.

Success Criteria The Offeror must measure, and report results in accordance with the success criteria listed below. All errors/failures/faults will be evaluated for criticality.

1. System did not experience any Critical Failures The number of Simple Failures did not exceed the specified number per LRU category (as defined in 1. through 5 below):

1. APC: 3 Failures

2. Workstations: 2 Failures

3. Management System: 1 Failure

4. System Power: 1 Failure

5. Network Components: 1 Failure The FAA reserves the right to further define the LRU categories based on the system design;

The total number of Simple Failures was less than or equal to 5 No restarts were required to regain functionality Exit Criteria Stability Assessment The Offeror must meet the following exit criteria by the end of the Stability Assessment:

1. The Offeror must have completed the Stability Assessment within the allocated time.

The Offeror must verify to the FAA that the software configuration was unchanged for the duration of the Stability Assessment period.

The Offeror must verify to the FAA that the traffic loading tool performed as required for the duration of the Stability Assessment period.

The Offeror must have completed all scripts/procedures The Offeror must have documented all failures The Offeror must have documented all discrepancies The Offeror must have documented all anomalies The Offeror must provide a Test Report for the Stability Assessment

Glossary Baseline - The approved, recorded configuration of one or more configuration items, that thereafter serves as the basis for further development, and that is changed only through change control procedures. For the purposes of this document, the term baseline is comprehensive, includes all phases of the program (including the development phase of the program), and is therefore not restricted to formal FCA/PCA approved baselines.

Database - a collection of data fundamental to the operation of a system or enterprise. Database usually connotes a systematized collection of data that can be immediately accessed and manipulated by a system for a specific purpose. Data Bank describes any collection of data that may or may not be interrelated or immediately accessible by a system.

Failure - The event, or inoperable state, in which any item or part of an item (hardware or software) does not, or would not, perform as previously specified due to loss of function or malfunction. A failure may be produced when a fault is encountered.

Fault - an abnormal operation of hardware or software. A fault, if it occurs, may cause a failure.

Procedures - Step-by-step instructions that a stability operator will perform to execute loading via the loading tool.

Scripts -Software instructions in a human-readable format that are compiled and/or processed by the loading tool. Scripts define what actions the loading tool is to perform. Consist of step by step cycles indicating actions to be performed.

Software - All stored programs and configuration files used by or executed by a processor or controller, regardless of implementation technique, and all associated adaptation data (e.g., configuration files, registry settings, database entries, etc.).

Unavailable - unavailable means the system does not respond to attempted inputs from that interface.

Acronyms

APCMS

CDRL

CHI

APC Management System Contract Data requirements List Computer-Human Interface

image1.jpg