Attach_2_RFIMS-SE-PLN-14-TEMP_v_1.2_20170419.pdf

Radio Frequency Interference Monitoring System (RFIMS)

Test and Evaluation Master Plan

Version 1.2

April 2017

U.S. Department of Commerce (DOC) National Oceanic and Atmospheric Administration (NOAA) National Environmental Satellite, Data, and Information Service (NESDIS)

RFIMS Test and Evaluation Master Plan RFIMS-SE-PLN-14 ii

TEST AND EVALUATION MASTER PLAN

FOR

RADIO FREQUENCY INTERFERENCE

MONITORING SYSTEM (RFIMS)

Prepared By: RFIMS Systems Engineering Team, Office of Satellite Ground Services (OSGS)

Approved By: ____________________________ Steven Grippando RFIMS Project Manager, OSGS Systems Engineering Deputy Division Chief iii

CHANGE RECORD

DOCUMENT TITLE: RFIMS TEST AND EVALUATION MASTER PLAN

VERSION DATE PAGES AFFECTED DESCRIPTION

0.3 October 13, 2016 All

First release of document for RFIMS peer review.

0.3B October 14, 2016 All Incorporated comments from RFIMS team.

0.3C October 25, 2016 All

Incorporated feedback from ITS and Aerospace. Added details from Test Data Approach; ITS Test Range Proposal.

1.0 November 14, 2016 All Baseline version accepted.

1.1 February 15, 2017 All

Updates to address vendor feedback on draft RFP and results of Tiger Team analysis of the sensitivity/detection threshold established previously.

1.2 April 19, 2017 All

Updated metrics for detection, classification, and identification based on RFIMS Tiger Team recommendations. Completed minor edits for refinement.

The document version number identifies whether the document is a working copy, final, revision, or update, defined as follows:

• Working copy or Draft: a document not yet finalized or ready for distribution; sometimes called a draft. Use 0.1A, 0.1B, etc. for unpublished documents.

• Final: the first definitive edition of the document. The final is always identified as Version 1.0.

• Revision: an edition with minor changes from the previous edition, defined as changes affecting less than one-third of the pages in the document. The version numbers for revisions 1.1 through 1.9, 2.1 through 2.9, and so forth. After nine revisions, any other changes to the document are considered an update. A revision in draft, i.e. before being re-baselined, should be numbered as 1.1A, 1.1B, etc.

• Update: an edition with major changes from the previous edition, defined as changes affecting more than one-third of the pages in the document. The version number for an update is always a whole number (Version 2.0, 3.0, 4.0, and so forth).

iv

Table of Contents

1. Introduction

1.1. Purpose and Scope

1.2. Definitions

2. Applicable and Reference Documents

3. RFIMS System Description

3.1. RFIMS System Background

3.2. RFIMS Requirements Flow Down

4. Test Program Summary

4.1. Test Approach and Objectives

4.2. Test Program Schedule

4.3. Roles and Responsibilities

4.3.1. RFIMS Project Systems Engineering

4.3.2. NTIA ITS

4.3.3. Aerospace Corporation

4.3.4. Vendors

4.3.5. Wireless Carriers

4.4. Test Activities

4.4.1. Environmental Survey Test

4.4.2. Site Survey

4.4.3. Test Range Development

4.4.4. Stress Test

4.4.5. Initial Concept Factory Test

4.4.6. Proof of Concept Demonstration

4.4.7. Prototype Factory Demonstration

4.4.8. Prototype Site Test

4.4.9. First Article Factory Testing

4.4.10. First Article Site Testing

4.4.11. End-to-End Test

4.4.12. Production Testing

5. Reporting

5.1. Requirements Verification Matrix

5.2. Anomaly Reporting and Resolution

v

5.3. Waivers and Deviations

6. Resource Summary

6.1. Testbed/ Test Range

6.2. Test Data

6.3. Government Furnished Items

7. Appendix A: Acronym List

Table of Figures

FIGURE 1: NOAA DOWNLINK FREQUENCIES IN THE 1675-1710 MHZ BAND

FIGURE 2: NOTIONAL HIGH-LEVEL RFIMS ARCHITECTURE

FIGURE 3: NOTIONAL BLOCK DIAGRAM OF RFIMS

FIGURE 4: TESTING THROUGHOUT THE RFIMS PROJECT PHASES

FIGURE 5: NOTIONAL RFIMS SUMMARY TEST SCHEDULE

1. Introduction

1.1. Purpose and Scope

The purpose of the Radio Frequency Interference Monitoring System (RFIMS) Test and Evaluation Master Plan (TEMP) is to document the approach and mechanisms for planning, execution and management of testing on the RFIMS Project to ensure the system is technically well-defined and meets all functional and performance requirements and operational needs. The goal of this document is to provide a plan that describes all the major test activities and how they will be managed.

This document clearly identifies the RFIMS test objectives and outlines the approaches to meet these objectives. It addresses the roles and responsibilities across the project, the requirements flow down, the test process, schedule of testing milestones, test reporting, and a summary of test resources. Detailed test plans and reports are described in contractor-developed documents and other project documents.

Additionally, this document reviews the initial concept definition evaluation, expanding the scope of the document more broadly than a standard system TEMP that solely focuses on verifying and validating system requirements. Because the RFIMS Project involves the research and development of new technology, the problem space requires studies and tests to be conducted to frame potential solutions.

The initial analysis and testing helps to shape the Concept of Operations, Statement of Objectives and Requirement Specifications. This testing is conducted using a structured test approach with test plans and test reports. Therefore, we have included the concept definition testing in the scope of this document.

Testing activities related to the RFIMS Small Business Innovative Research (SBIR) project are out of scope for this document.

1.2. Definitions

For the purposes of this document, the following definitions are used for test and evaluation activities:

Concept definition evaluation involves the process of understanding the problem space and study of the potential solution of the new technology. Testing is conducted to understand the existing environment, stressing situations and potential technology limitations.

The Demonstration process provides the opportunity to show the capabilities of a system that is still in development. While the system might not yet meet the requirements, it provides confidence in the emergent technology and software algorithms of the system and can be used as a checkpoint for continuation of further system development.

The Verification process confirms that system requirements are met and that all system elements perform their intended functions. The purpose of verification is to provide evidence that no error, defect, or fault has been introduced into the implemented system elements or the integrated system. It is used to confirm that the system has been made “right” according to the system requirements and selected methods, techniques, standards, or rules.

The Verification methods include inspection, analysis, demonstration, and test.

The Validation process is to provide objective evidence to show that a system in use will meet stakeholder needs in the expected operational environment. Validation is applied to the system at the appropriate phase in the system life cycle to prove the “right” system has been built.

2. Applicable and Reference Documents This TEMP has been developed in compliance with the following applicable documents.

Document Number Document RFIMS-SE-PLN-13 RFIMS Systems Engineering Plan RFIMS-SE-PLN-16 RFIMS Mission Assurance Plan

The following reference documents were used to draft this plan. They contain additional clarifying information and details.

Document Number Document RFIMS-PM-DOC-09 RFIMS Concept of Operations (CONOPS) RFIMS-PM-DOC-03 RFIMS Statement of Objectives (SOO) RFIMS-SE-DOC-17 RFIMS Requirements Document RFIMS-SE-PLN-15 RFIMS Test Data Plan RFIMS-PM-PLN-05 RFIMS Acquisition Plan RFIMS-PM-PLN-01 RFIMS Project Management Plan (PMP) OSGS 5820.01A OSGS Systems Engineering Process Document OSGS 5820.02A OSGS Change and Configuration Management Process Document OSGS 5820.03A OSGS Requirements Management Process OSGS 5820.04A OSGS Risk, Issue, and Opportunity Management Process Document ITS Test Range Approach Document ITS Technical Reports Aerospace Technical Reports U.S. Code of Federal Regulations, Title 47 DA-14-1023 FCC and NTIA: Coordination Procedures in the 1695-1710 MHz and

1755-1780 MHz Bands, Public Notice; 18 July, 2014, https://apps.fcc.gov/edocs_public/attachmatch/DA-14-1023A1.pdf

DOC NOAA, Transition Plan for the 1695-1710 MHz Band, 12 August, 2014 https://www.ntia.doc.gov/files/ntia/publications/doc_noaa_web-ready_1jul14_final_rev7_admin_chng.pdf

3. RFIMS System Description

3.1. RFIMS System Background

The National Oceanic and Atmospheric Administration (NOAA) manages and operates the Nation's operational environmental satellites through its National Environmental Satellite, Data, and Information Service (NESDIS) office. NESDIS provides timely access to global environmental data from satellites to promote, protect and enhance the Nation's economy, security, environment and quality of life. NESDIS operates a variety of satellite constellations in the L-Band, S-Band, and X-Band frequencies from multiple Federal earth stations across the United States. These frequencies are used to command satellites as well as to receive state of health and mission data from the satellites.

In March 2014, the FCC updated Title 47 Code of Federal Regulations (47 CFR) to reflect the 1695-1710 megahertz (MHz) band is occupied by both meteorological satellites and fixed mobile communications and wireless communications. It states that if protected Federal operations receive harmful interference from wireless carriers sharing the third Advanced Wireless Service (AWS-3) frequency band, 1695-1710 https://apps.fcc.gov/edocs_public/attachmatch/DA-14-1023A1.pdf https://www.ntia.doc.gov/files/ntia/publications/doc_noaa_web-ready_1jul14_final_rev7_admin_chng.pdf https://www.ntia.doc.gov/files/ntia/publications/doc_noaa_web-ready_1jul14_final_rev7_admin_chng.pdf

MHz, the AWS-3 licensee must modify its operations and/or technical parameters as necessary to eliminate the interference. The Federal Communications Commission (FCC) and the National Telecommunications and Information Administration (NTIA) publish coordination procedures for the 1695-1710 MHz band jointly via public notices. Public Notice 14-1023 provides guidance for how the licensees will coordinate with Federal operators, both formally and informally, to avoid interference.

This spectrum sharing agreement represents the first time the Federal Government will share the same frequencies with non-Federal users. The licensee, or wireless carrier, may use the 1695-1710 MHz band for uplink communications only from the user equipment (UE) to its network base station on a non-interference basis only. If UE uplinks are active while a NOAA earth station is receiving a satellite downlink signal from its meteorological satellites, the UE signal(s) may interfere with or degrade the NOAA downlink signal.

The Department of Commerce (DOC) Transition Plan lists each of the 17 NOAA Federal earth stations requiring protection, and indicates the transition timeline for spectrum sharing is 39 months after Auction 97 ended in January 2015.

NOAA operates both polar orbiting and geostationary satellites. The polar orbiting satellites, which include Polar-orbiting Operational Environmental Satellites (POES) and Meteorological Operational (MetOp) satellites, are in highly inclined low earth orbits (LEO), or “polar” orbits. Given 8-14 minutes of visibility per pass, this equates to a total of about 200-250 minutes of contact time on average, per day. Federal earth stations near the North Pole, such as Barrow, AK, may have as much as 950 minutes of visibility per day since they have more daily satellite passes. The downlink frequencies used by POES and MetOp are in the shared 1695-1710 MHz band.

The Geostationary Operational Environmental Satellite (GOES) is in a geostationary earth orbit (GEO). The GOES downlink frequencies are adjacent to the shared band in the 1675-1695 MHz band, but are still protected from interference from AWS-3 commercial wireless carriers.

The NOAA downlink frequencies for POES, MetOp, GOES, and future GOES-R satellites are shown below in Figure 1.

Figure 1: NOAA Downlink Frequencies in the 1675-1710 MHz Band

The purpose of the RFIMS is to support the sharing of spectrum between Federal incumbent users and commercial wireless Long-Term Evolution (LTE) licensees. NOAA will use the RFIMS to monitor the frequency environment and ensure there is no harmful interference to received satellite communications.

The RFIMS must monitor the integrity of the signal of interest at the receiver, as well as the overall spectrum environment at the receive location, to detect the presence of interfering signals. To do this, the system must perform the four basic monitoring functions: detect, classify, and identify the source of the interference, as well as, notify government operators and wireless carriers of said interference.

Monitoring will continue in order for the government operators to verify that the wireless carriers have mitigated the interference.

Detailed descriptions of the functional capabilities are as follows:

• Detect - The system should detect, in real-time, “interference events” in which the interference level lies at or above -161.4 decibel-watt (dBW)/180-kilohertz (kHz) interference power spectral density (IPSD) during NOAA’s earth station downlink reception. NOAA desires a monitoring system capable of adjusting the interference detection threshold between -161.4 dBW/ 180 kHz IPSD (i.e., highest sensitivity) and a less sensitive IPSD.Classify - The system should classify the types of RF interference it detects at -161.4 dBW/180 kHz interference power spectral density or higher. Where “classify” is the discrimination between 1695 – 1710 MHz LTE UEs or other co-channel RF transmissions and all other radio frequency interference (RFI) such as out-of-band and spurious emissions. NOAA desires a monitoring system capable of adjusting the interference classification threshold between -161.4 dBW/ 180 kHz IPSD (i.e., highest sensitivity) and a less sensitive IPSD.

• Identify - The system should determine if the RFI at -161.4 dBW/180 kHz interference power spectral density or higher is resulting from 1695 – 1710 MHz LTE UE uplink wireless transmission or other source of interference. The system should then identify the wireless carrier responsible for operating the tower(s)/sector(s) that are communicating with the UE(s) causing the interference in real time. NOAA desires a monitoring system capable of adjusting the interference identification threshold between -161.4 dBW/ 180 kHz IPSD (i.e., highest sensitivity) and a less sensitive IPSD.

• Notify - The system should notify NOAA operators, and potentially the wireless carriers, that wireless carriers are creating interference to NOAA.

The RFIMS subsystem at a NOAA earth station would forward information in real-time to a central node which will likely be co-located at one of the Federal earth stations. The central node would be responsible for monitoring the data collected at each of the 17 locations, and reporting status and interference information to wireless carriers and NOAA offices responsible for enforcing the spectrum sharing agreements. Figure 2 depicts a notional (not suggested) high-level RFIMS architecture (vendor developed solutions may vary).

Figure 2: Notional High-Level RFIMS Architecture

The RFIMS may consist of a monitoring system located remotely at each of the identified 17 NOAA Federal earth stations. Each system is responsible for monitoring the signal environment and detecting interference to the NOAA satellite downlink signals at an earth station. The system may include receive equipment connected directly to the NOAA downlink feed, as well as a receiver connected to an external monitoring antenna or antenna array. A notional (not suggested) block diagram of the system as integrated into a federal earth station system is shown in Figure 3 (vendor developed solutions may vary).

Figure 3: Notional Block Diagram of RFIMS

3.2. RFIMS Requirements Flow Down

The RFIMS Project Team developed their high-level requirements (Level 1 and Level 2) based on input from the NOAA Radio Frequency Management Division Spectrum Sharing charter, NESDIS requirements documents, and the RFIMS Statement of Objectives (SOO). Additionally, the Team used analyses from Aerospace and the National Telecommunications and Information Administration (NTIA) Institute for

Telecommunications Sciences (ITS) Non-Recurring Engineering (NRE) activities to guide the technical specifications for what the system could feasibly achieve given the cost and schedule constraints. Once on contract, the RFIMS Vendors will flow down lower-level requirements (Level 3) for designing and developing the system and interface. These requirements must trace back to the requirements sources.

The RFIMS Project will confirm the requirements traceability matrix (RTM) linking the Vendor requirements.

The Vendor will be responsible for verification of Level 3 requirements. The RFIMS Project Team will retain authority to approve or disapprove the status of each requirement verified during testing.

4. Test Program Summary

4.1. Test Approach and Objectives

The RFIMS test and evaluation program ensures that the system procured will satisfy the stakeholder requirements through a series of phased, incremental test activities leading to a system-level end-to-end test to show operations readiness. In order to achieve this, a test and evaluation approach has been developed that spans the entire project lifecycle. Early test activities are performed prior to preparation of project documents to develop system requirements and a valid concept of operations. This initial concept definition phase of the project entails analysis of the problem space through Aerospace and NTIA/ITS NRE studies. Additionally, these studies will involve creation of test data and a testbed/ test range.

Once the contract is awarded, the RFIMS Project will manage the Vendor(s) during three phases: Proof of Concept, Prototype, and Production & Deployment. During the Proof of Concept phase, the focus of the development is on detecting interference. Tradeoff analysis and technology assessments will guide design and development plans. At each subsequent phase of the project, the system complexity and technology plans will mature. During each of these phases, unit and component development integration and test will be conducted. The culminating product will then undergo rigorous verification and validation at the operational site prior to deployment. The additional systems will undergo production testing prior to operations at new sites. This phased approach is shown in Figure 4.

Figure 4: Testing throughout the RFIMS Project Phases

Concept Definition

• RFIMS Project, ITS, Aerospace:

• Evaluate RF Interference problem through environmental tests and site surveys

• Input to specifications and CONOPS based on simulations and analysis

Proof of Concept

• RFIMS Project, ITS, Aerospace, Vendors:

• Technology assessments; tradeoff analyses

• Demonstration of initial concepts at factory

Prototype

• RFIMS Project, ITS, Aerospace, Vendors, Wireless Carriers:

• Prototype development and test at factory and ITS Testbed using test data

• Interface testing with Wireless Carriers

Production & Deployment

• RFIMS Project, ITS, Aerospace, Vendors, Wireless Carriers:

• First article testing at factory and site

• End-to-End testing

A summary of planned test activities is provided in Table 1. Further details on the objectives, success criteria, participants, oversight, and test locations are described in Section 4.4.

Table 1: Summary of RFIMS Test Activities

Test Activity Objective Participants Location Environmental Survey Evaluate the radio frequency (RF) environment to form a baseline of the current spectrum use pre-LTE activity in AWS-3 band

ITS, Aerospace, RFIMS Project

NOAA Ground Stations, ITS site (Table Mountain, Boulder, CO)

Site Survey Evaluate the existing satellite receivers; examination of receiver characteristics, noise levels, link margins, interference, etc.

ITS, Aerospace, RFIMS Project

NOAA Ground Stations

Test Range Development

Develop and maintain a testbed/ test range which emulates a NOAA field station

ITS, Aerospace, RFIMS Project

ITS site (Table Mountain, Boulder, CO)

Stress Test Evaluate the stressing cases of RF interference for input to the specifications and CONOPS

ITS, Aerospace, RFIMS Project

ITS Test Range and/or NOAA Ground Stations

Initial Concept Factory Test

Test of the component integration and software functionality of the Vendors’ initial concepts

Vendors, RFIMS Project, ITS, Aerospace

Vendor Factory

Proof of Concept Demonstration

Demonstration of initial concept to Government; show capability for Detection and best effort for Classify, Identify and Notify functions

Vendors, RFIMS Project, ITS, Aerospace

Vendor Factory

Prototype Factory Test Test of the component integration and software functionality of the Vendors’ prototypes

Vendors, RFIMS Project, ITS, Aerospace

Vendor Factory

Prototype Site Test Test of the prototypes on the ITS test range; interface testing with the Wireless Carriers

Vendors, ITS, Aerospace, Wireless Carriers, RFIMS Project

ITS site (Table Mountain, Boulder, CO)

First Article Factory Test

Verify the function and performance of the first article

Vendor, RFIMS Project, ITS, Aerospace

Vendor Factory

First Article Site Test Verify the function and performance of the first article as installed at an operational site

Vendor, RFIMS Project, ITS, Aerospace, Wireless Carriers

NOAA Ground Station

End-to-End Test Verify the end-to-end system functional performance of all four key functions and interfaces.

Validate user operations.

Vendor, RFIMS Project, Wireless Carriers, ITS, Aerospace

NOAA Ground Station

Production Testing Verify subsequent production of systems. Test full-integrated system with all deployed sites.

Vendor, RFIMS Project, Wireless Carriers, ITS, Aerospace

NOAA Ground Stations

4.2. Test Program Schedule

The RFIMS test program schedule takes into consideration the acquisition schedule, Government mandated dates for sharing the spectrum and estimated development needs from the Vendors. In addition, the RFIMS test schedule is influenced by the readiness of the ITS testbed /test range and the external interface with the Wireless Carriers. A notional (not recommended) schedule is shown in Figure 5 that reflects the sequencing of test activities and the testing completed to-date. The timing and duration of future testing is dependent on proposed Vendor schedules and planned development activities.

CY 2016

Q1 Q2 Q3 Q4

CY 2017

Q1 Q2 Q3 Q4

CY 2018

Q1 Q2 Q3 Q4

CY 2019

Q1 Q2 Q3 Q4

CY 2020

Q1 Q2 Q3 Q4

Co nc ep t D ef in iti on

Sy st em D ev el op m en t

Figure 5: Notional RFIMS Summary Test Schedule

4.3. Roles and Responsibilities

4.3.1. RFIMS Project Systems Engineering

The RFIMS Project is responsible to OSGS for ensuring the performance and readiness of the delivered RFIMS for detecting, classifying, and identifying the source of interference and notifying Government operators and the Wireless Carriers. The project will manage the Vendor contracts and coordinate with external organizations to achieve project objectives. The RFIMS Project is responsible to:

• Propose test activities (objectives, execution)

• Review and approve individual test plans

• Review results from site surveys and evaluation studies

• Prepare Government sites for test activities

• Witness test activities and writing observation reports

• Review test reports

• Manage Interface test activities

• Collect and review and test artifacts

• Track anomalies and discrepancies

• Document test results in the Requirements Verification Matrix (RVM)

Stress Testing

Environmental & Site Surveys

OMB 39-month Transition

Develop Test bed/ test range

Contract Award

Initial Concept Factory Testing

Proof of Concept Demonstration

Production Tests

ETE Tests

Prototype Site Test

Prototype

1st Article

1st Article Site Test

4.3.2. NTIA ITS

Under a NRE task for RFIMS, NTIA ITS provides independent analysis and evaluation of the interference problem. Their role on the project is to:

• Propose test activities during Concept Definition Phase

• Propose recommendations for technical specifications based on test evaluations

• Create Monte Carlo simulations of interference problem

• Model receiver performance to characterize expected system baseline performance

• Conduct bench tests to measure interference susceptibility of POES receivers in the presence of characteristic LTE waveforms

• Take and measure spectrum measurements at NOAA Ground Stations

• Model aggregate interference to assess impact of temporal sharing on RFIMS detection, classification, and identification

• Conduct site surveys and test evaluations

• Generate test reports

• Develop and maintain a satellite earth station testbed/ test range

• Witness Vendors’ test activities and write observation reports

• Review Vendors’ test reports

4.3.3. Aerospace Corporation

Under a NRE task for RFIMS, the Aerospace Corporation provides independent analysis and evaluation of the interference problem. Their role on the project is to:

• Propose test activities during Concept Definition Phase

• Propose recommendations for technical specifications and Concept of Operations based on test evaluations

• Conduct site surveys, environmental studies and stress tests

• Generate test reports

• Create development environment to create test data

• Witness Vendors’ test activities and write observation reports

• Review Vendors’ test reports

• Assess and evaluate Vendors’ solutions

4.3.4. Vendors

Following the Concept Definition phase and Source Selection, one or more Vendors will conduct the Proof of Concept phase, the Prototype phase and the Production & Deployment phase of the RFIMS Project.

The Vendors will provide all management and engineering services, personnel, materials and equipment not otherwise provided by the Government, necessary to design, develop, integrate, and transition the RFIMS. They will be responsible to:

• Develop lower-level requirements that are testable

• Develop a project-specific verification and validation plan

• Create demonstration plans and test plans

• Conduct component testing at the factory for initial concepts, prototypes and first articles

• Demonstrate capabilities to Government team

• Support interface development and test activities with the Wireless Carriers

• Use ITS Test Range for test demonstration of prototypes

• Conduct site testing

• Write reports on demonstrations and tests

4.3.5. Wireless Carriers

The involvement of the Wireless Carriers is critical to ensure the developed RFIMS interfaces properly with their systems. They are responsible to:

• Support interface development and testing with the Vendors

• Perform the test activities during end-to-end system tests

• Confirm in a final written report that the RFIMS interface is suitable for interference mitigation coordination efforts

4.4. Test Activities

4.4.1. Environmental Survey Test

Test Description and Objective During the Concept Definition phase, the RFIMS Project issued NRE studies to NTIA ITS and Aerospace that included work for environmental survey testing. This testing was conducted at NOAA Ground Stations and at the ITS testbed/ test range site in Table Mountain, Boulder, CO. The spectrum activity for the AWS-3 band was measured and analyzed. These measurements and surveys provide both NOAA and Wireless Carriers an RF environmental baseline characteristics independent of actual satellite receivers. These measurements are sufficiently documented, so they can be repeated at any time in the future.

These test activities provide data results on AWS-3 band to provide a baseline of the existing environment before the existence of uplink LTE signals as observed from the external sensors and NOAA Ground Stations. This data provides an input to the technical performance specifications, such as limits and sensitivities, and the CONOPS.

Teams and Responsibilities These tests were performed by ITS and Aerospace Corporation Teams based on the direction of the RFIMS Project. ITS and Aerospace wrote the test plans, executed the tests, and issued reports. The RFIMS Systems Engineering Team reviewed the plans and reports and witnessed the test activities.

Results The ITS team performed environmental survey tests at Wallops, the NOAA Satellite Operations Facility (NSOF), Miami, and Table Mountain and their final report was submitted to the RFIMS Team. Aerospace conducted tests at Wallops and Miami and submitted their final report to RFIMS Team. As needed, additional environmental surveys will continue through 2017.

4.4.2. Site Survey

Test Description and Objective During the Concept Definition phase, the RFIMS Project issued NRE studies to NTIA ITS and Aerospace that included work for Site Surveys. This testing was conducted at the NOAA Ground Stations on the satellite downlink receivers.

These tests examined factors such as receiver filter characteristics, receive antenna characteristics, potential harmful interference, noise levels, and link margins. It is important to note that the results from each site differed from other sites. The RFIMS Systems Engineering Team, with the help of ITS and Aerospace, identified which sites are necessary to survey. The locations were chosen based on worst-case population density sites and quantity/size of the on-site antennas.

The objective of these test activities is to establish the RF characteristics of NOAA satellite ground stations at AWS-3 band before the existence of uplink LTE signals as observed by the actual satellite downlink receivers. These measurements and surveys provided NOAA with baseline characteristics of its satellite receivers. Moreover, it helped the RFIMS Project Team to understand the operational environment of these satellite receivers. The RFIMS Team will use this data as an input to technical performance specifications, such as limits and sensitivities, and the CONOPS.

Teams and Responsibilities These tests were performed by ITS and Aerospace based on the direction of the RFIMS Project. ITS and Aerospace wrote the test plans, executed the tests, and issued a final report. The RFIMS Systems Engineering Team reviewed the plans and reports and witnessed the test activities.

Results The results from these tests can be shared with Vendors and wireless carriers to inform them about the state of NOAA operations. The ITS and Aerospace teams performed environmental survey tests at Wallops, NSOF, and Miami and their final reports were submitted to the RFIMS team. As needed, additional environmental surveys will continue through 2017.

4.4.3. Test Range Development

Test Description and Objective During the Concept Definition phase, the RFIMS Project issued NRE studies to NTIA ITS, which include work for development of a testbed / test range. The objective of this test location is to provide an operational environment to study potential AWS-3 Band interference to NOAA satellite earth stations to the extent possible. The testbed/ test range site at DOC Table Mountain Facility provides an opportunity to characterize NOAA satellite downlink reception in the presence of simulated AWS-3 Band interference within a controlled environment and without interrupting the reception of meteorological signals at NOAA operational earth stations. The testbed/ test range will consist of system hardware to assess interference to emulated GOES and POES satellite receive systems. Initial test activities will include:

• Earth Station Receiver Modeling – collection of experimental data to validate link budget assumptions and validate interference protection criteria (IPC)

• RFIMS Evaluation Support – performing baseline scans of the RF environment and LTE user equipment emulation effort

• UE Aggregate Modeling – acquire UE aggregate data to validate model

Moreover, the testbed/ test range will be used for development, verification and validation activities for the RFIMS Project during the Prototype phase of the engagement. Vendors will use the test range to evaluate and demonstrate capabilities of their prototype concepts.

Teams and Responsibilities The ITS team will build and maintain the test bed test range under the direction of the RFIMS Project Team. ITS will support testing at the test range by the Vendors. The ITS team will also conduct their own testbed events while the RFIMS Systems Engineering Team witnesses the activities. Aerospace contribute to evaluation and recommendation for testbed/ test range activities.

4.4.4. Stress Test

Test Description and Objective The objective of stress testing is to determine the vulnerabilities of NOAA Ground Stations and characterize the behavior of Ground Station receivers in the presence of AWS-3 interference (actual or simulated). This will include determining the interference level at which the downlink signal is lost or out of lock. The testing will help the RFIMS Team understand the impact if, and when, UEs are operating inside the protection zone. These characterizations and test activities will likely be conducted at the ITS testbed, with potential testing at NOAA Ground Stations. The RFIMS Team can use this characterization to tailor specifications and refine the CONOPS.

Potential stress testing conducted at NOAA Ground Stations sites would use a signal generator capable of emulating LTE signals mounted in a vehicle with a rooftop antenna with GPS and a PC for recording and time synchronization with receiver recordings. Measurements would be taken to test long-term fading statistics. Transmission of the offending signal would be done during quiet times between POES passes and during some scheduled passes to better understand the impact of a LTE signal on the downlink data and receiver operation.

Stress testing measurements would be conducted both inside and outside of the defined Protection Zone at the sites on predefined routes that take advantage of the local population density and topography.

Testing at operational Ground Stations would require coordination with NOAA operators and transmit authorization from the FCC.

Team and Responsibilities If approved by the RFIMS Project, these tests will be performed by ITS and/or Aerospace. They will write the test plans, execute the tests, and issue a final report. The RFIMS Systems Engineering Team will review the plans and reports and witness the test activities.

4.4.5. Initial Concept Factory Test

Test Description and Objective During the Proof of Concept Phase, the Vendors will develop a low-fidelity system to prove initial technology selection and software systems (algorithms). Prior to their demonstration test to the Government, the Vendor will conduct in-house testing on the software builds and component integration.

These tests are performed at the early stages of the RFIMS development by the Vendors to validate the algorithms of their early prototype systems. If applicable to their concept, the Vendors can use Government-provided test data during the test. The Factory Test will be conducted at each Vendor’s site.

Any external interfaces will be simulated.

The objective of these tests is for the Vendors to validate their approach for the software code to perform the detect function of the RFIMS.

Teams and Responsibilities The RFIMS Vendors teams will perform these tests. The RFIMS Team, to include ITS and Aerospace, will write the test plans, and Vendors will execute the tests and issue a final report. The RFIMS Systems Engineering Team, with help from ITS and Aerospace, will review the test cases, procedures, and reports and witness the test activities.

4.4.6. Proof of Concept Demonstration

Test Description and Objective At the conclusion of the Proof of Concept development phase, the Vendors will conduct a demonstration of their initial concept. No requirements will be formally verified during the demonstration. Rather, the Vendors will demonstrate capabilities for detection of RF Interference as part of the Alternate System Review (ASR), as further described in the RFIMS SEP. The ASR will assess that the:

• Technical approach is credible and responsive to the identified requirements

• Tradeoffs are completed, and those planned adequately address the option space

• Operations concept is consistent with proposed design concept and is in alignment with the segment requirements

• Demonstration shows system capability to detect RF interference within specification tolerances

In this test activity, Vendors demonstrate that their systems can perform at least the detection functionality of RFIMS. The detection needs to happen at -161.4 dBW/180 kHz IPSD at the receiver both in the presence of a strong satellite downlink signal and noise only. At this stage, Vendors will also demonstrate their concept system’s capabilities for the other RFIMS functionalities (classify, identify, and notify); these will be assessed as a best effort result. The demonstrations will be conducted at the Vendors’ factories. Any external interfaces will be simulated. Test data, if needed, for the demonstration will be provided by the Government.

The objective of these demonstrations is to show the RFIMS Project the state of the concept to determine the capabilities and limitations of the Vendor’s RF Interference technology and algorithms. Based on the results of the demonstrations, the RFIMS Project might choose to waive, deviate or change the requirements baseline prior to the next phase of development (Prototyping). Additionally, the RFIMS Project will use their assessments of the demonstrations as input into the acquisition process for additional Contract Line Item Numbers (CLINs) for each Vendor, as specified in the RFIMS Acquisition Plan.

Teams and Responsibilities Vendors will provide their test plan to the RFIMS Systems Engineering Team as well as ITS and Aerospace Corporation teams. These teams will review the proposed test plans and provide their feedback to the Vendors. These demonstrations will be performed by the RFIMS Vendors and witnessed by the RFIMS Systems Engineering Team, and potentially ITS and Aerospace team. Vendors will provide a final test report and the RFIMS Systems Engineering team will review the report. The RFIMS Project will issue their own report assessing the performance of each demonstration.

4.4.7. Prototype Factory Demonstration

Test Description and Objective Vendors will develop a prototype of the RFIMS in the second phase of the project development that has increased capabilities and technical maturity. During the factory test, they will assess the functionality of the integrated prototype at their factory. This test will not have formal verification of requirements, but will demonstrate system performance. Specifically, in this test activity, Vendors will show system performing the detection, classification, and identification functions for RFIMS. The proposed systems will need to demonstrate detection at -161.4 dBW/180 kHz IPSD and classification and identification at -

161.4 dBW/180 kHz IPSD at the receiver both at presence of strong satellite downlink signal and noise only. At this stage, Vendors are also expected to provide proof for their best effort on the notification functionality.

This demonstration will be a precursor for the Prototype Site Test at the test range and will confirm proper integration of software and hardware of the prototype system. It will occur at the factory; all external interfaces will be simulated. The RFIMS Team will provide test data if needed.

Teams and Responsibilities These demonstrations will be performed by the RFIMS Vendors. They will write the demonstration plans, execute the demonstrations, and issue a final report. RFIMS Systems Engineering Team, and potentially ITS and Aerospace, will review the plans and reports and witness the demonstration activities.

4.4.8. Prototype Site Test

Test Description and Objective Following successful factory testing of the prototype system, the Vendors will move their prototype to the ITS test range for site testing. In this test, at least two sources of LTE interference, with controlled transmit power, will radiate emissions in the AWS-3 band from two different locations. The Vendors’ prototype monitoring systems will monitor the functional test antennas while receiving operational weather satellite downlink signals. It is expected that the monitoring system would detect, classify, and identify these interfering LTE signals. Additionally, the prototypes will undergo a demonstration of the notify functionality. The Government will provide test data if needed.

The objective of the prototype tests is to show the RFIMS Project the capabilities of the Vendor system to perform the detection, classification and identification functions as well as the best effort for notification functionality. Based on the results of the demonstrations, the RFIMS Project might choose to waive, deviate, or change the requirements baseline prior to the next phase of development (Production & Deployment). Additionally, the RFIMS Project will use their assessments of the demonstrations as input to the acquisition process for awarding optional Contract Line Item Numbers for each Vendor, as specified in the RFIMS Acquisition Plan.

Teams and Responsibilities The RFIMS Vendors will perform these demonstrations. The RFIMS Team, to include ITS and Aerospace, will write the test plans, and Vendors will execute the tests and issue a final report. The RFIMS Systems Engineering Team, ITS, and Aerospace will review the plans and reports and assess pass and fail criteria for test activities. Wireless Carrier teams will review the final report and will evaluate the Vendors’ notification schemes and usability of each Vendors’ approach for the notification functionality.

4.4.9. First Article Factory Testing

Test Description and Objective Prior to the installation of the final RFIMS build at a NOAA Ground Station operational facility, the build must undergo factory testing to demonstrate function, performance, and readiness for site installation and use. The Vendor will verify those Level 3 requirements that have the First Article Factory Test designated as the Verification event as specified in the RVM. This will include component functionality and other requirements that are not site or interface-dependent. The RFIMS Project will confirm the status of each verified requirement and give approval for site installation.

The test will be conducted at the Vendor’s factory. External interfaces will be simulated. The test data will be provided by the Government. This test will demonstrate that the system is integrated and performing to the requirements and is ready for site testing.

Teams and Responsibilities The Vendor will plan and conduct these pre-ship tests. They will write the test plans, execute the tests, and issue a final report. RFIMS Systems Engineering Team, along with ITS and Aerospace team, will review the plans and reports and witness the test activities.

4.4.10. First Article Site Testing

Test Description and Objective Following successful factory testing of the First Article RFIMS, the system will be moved to a NOAA Ground Station operational facility for site testing. During this test, the system will be fully integrated with the Wireless Carrier interface and all functionality will be tested. Applicable Level 3 requirements will be verified during this test. Any deviations or waivers will follow the prescribed process. This objective of this test is to fully sign off on all the levied requirements for the RFIMS system.

Teams and Responsibilities The RFIMS Vendors teams will perform these tests. They will write the test plans, execute the tests, and write the final report. The RFIMS Systems Engineering Team, ITS, and Aerospace will review them and witness the test activities. The Wireless Carriers will support interface testing.

4.4.11. End-to-End Test

Test Description and Objective Following a successful site test of the First Article, the RFIMS Project will conduct end-to-end tests at the NOAA Ground Station. This test will validate the system functionality and performance against user expectations. End-to-end testing demonstrates the full system operational capability. The RFIMS Project’s Level 1 and Level 2 requirements will be verified as applicable. Those requirements that involve multiple instances of system deployment will be tested prior to System Acceptance. The system will be fully integrated with the Wireless Carrier interface.

Teams and Responsibilities These tests will be designed by the RFIMS Project; they will write the test plans containing the user scenarios and objectives and will execute the tests, assess pass and fail criteria, then issue a final report.

The Vendors will support the testing as directed. Aerospace and ITS will witness the tests.

4.4.12. Production Testing

Test Description and Objective Following testing and deployment of the First Article RFIMS, the Vendors will roll out subsequent systems to the NOAA Ground Stations. These systems will undergo less strenuous testing than the First Article testing, while still demonstrating successful builds of the hardware and software components.

At each site prior to operations, the subsequent products will undergo a 24-hour demonstration of functionality. The purpose of this testing is to verify manufacturing of system components prior to operations. Once all the systems are deployed, system-wide testing will be conducted to ensure the entire system-of-systems is integrated correctly with each other and the centralized monitoring location. At this test, the remaining system-level requirements (Levels 1, 2, and 3) will be verified. This test will be prior to System Acceptance by the Government.

Teams and Responsibilities These tests will be designed by the RFIMS Project; they will write the test plans, execute the tests, and issue a final report. The Vendors and Wireless Carriers will support the testing as directed. Aerospace and ITS will witness the tests as needed.

5. Reporting

5.1. Requirements Verification Matrix

The RFIMS Project will develop and maintain a comprehensive RVM. This matrix will track the following information for each requirement:

• Verification Method (Test, Analysis, Inspection, Demonstration)

• Test at which verification is performed

• Success criteria

• Verification status

• Parent and child requirement traceability

The verification status of each requirement is updated following each test and documented as part of the post-test report.

5.2. Anomaly Reporting and Resolution

During testing, all anomalies will be tracked using a discrepancy reporting system (specified in the Vendors’ V&V Plan). The RFIMS Team will track the anomalies until a satisfactory resolution has been achieved. This could include repeating some test activities or adding additional tests. The timeframe will be decided by the RFIMS Project with input from the Vendors.

5.3. Waivers and Deviations

As discussed, the Proof of Concept Phase and Prototyping Phase might identify requirements that are not technically feasible as specified. These requirements might then be waived or be deviated. All waivers and deviations will follow the RFIMS Configuration Plan and the RFIMS Mission Assurance Plan, as applicable.

6. Resource Summary

6.1. Testbed/ Test Range

The ITS testbed/ test range will enable testing and simulations without use of actual operational equipment at the NOAA Ground Stations. This testbed/ test range will be used during testing (as specified in previous sections). The ITS team will maintain the testbed/ test range through the system development cycle.

6.2. Test Data

Trusted, high-fidelity test data sets are important to have meaningful testing of the RFIMS products. The Government will provide test data to the Vendors and the Wireless Carriers. The Aerospace Corporation through the NRE creates this data. Data will be generated by two methods – software-based using MATLAB toolboxes and lab-generated using LTE signal generators and commercial-off-the-shelf modems.

The test data will include various scenarios for testing use cases; such as signals with background-noise only, a single interfering LTE uplink signal, and multiple LTE uplink signals. More information about these scenarios and the data generation can be found in the RFIMS Test Data Approach document.

6.3. Government Furnished Items

The Government might provide items to the Vendors as Government Furnished Information or Government Furnished Property to be used during testing.

7. Appendix A: Acronym List

AWS-3 Advanced Wireless Service-3

ASR Alternate System Review

CFR Code of Federal Regulations

CLIN Contract Line Item Number

CONOPS Concept of Operations

DOC Department of Commerce

FCC Federal Communications Commission

GEO Geostationary Earth Orbit

GOES Geostationary Operational Environmental Satellite

INR Interference-to-Noise Ratio

IPC Interference Protection Criteria

ITS Institute for Telecommunication Sciences kHz Kilohertz

LEO Low Earth Orbits

LTE Long-Term Evolution

MHz Megahertz

MetOp Meteorological Operational

MEF Mission Essential Functions

MRD Mission Requirements Document

NESDIS National Environmental Satellite, Data, and Information Service

NOAA National Oceanic and Atmospheric Administration

NRE Non-Recurring Engineering

NTIA National Telecommunications and Information Administration

PMP Project Management Plan

POES Polar-orbiting Operational Environmental Satellites

RFI Radio Frequency Interference

RFIMS Radio Frequency Interference Monitoring System

RFMD Radio Frequency Management Division

RTM Requirements Traceability Matrix

RVM Requirements Verification Matrix

SBIR Small Business Innovative Research

SOO Statement of Objectives

TEMP Test and Evaluation Master Plan

UE User Equipment

V&V Verification and Validation