Attachment_G_-_LMOC_Scalability_Study_1.pdf

1 | P a g e

Excerpts from Special Study Report:

LMOC Scalability for Landsat Next

Performed Summer, 2023

Notes for LNext MOC:

This document is provided for historical LMOC reference and informational purposes ONLY.

This document is ONLY to be used for the purpose of the LNext MOC Studies and IS NOT to be shared or used for any other purposes. It SHOULD NOT be construed as direction or guidance for the LNext MOC. This study was one of many similar activities performed in 2023 during

Phase A for Landsat Next, which included a MOC RFI and other special studies on key Ground needs for LNext. The analysis and information included below were based on the state of the

MOC in summer of 2023 at the time the study was performed and has not been updated or modified to reflect any changes made since then.

To reiterate, DO NOT utilize this document as an expected architecture for LNext, but only as historical reference of L8 and L9 systems and processes in 2023. The USGS will consolidate the results of all Phase A activities and vendor studies and from them will formulate the LNext

MOC.

2 | P a g e

3 | P a g e

Table of Contents

1 Introduction

1.1 Introduction

1.2 Proposed Near Term Actions

1.2.1 Cloud Migration

1.2.2 Standardizing MPS activity schedule to WRS cycle

2 LMOC Scalability

2.1 High-Level Architecture

2.2 Telemetry and Command Subsystem Scalability

2.2.1 Galaxy Pressure Points

2.2.2 Fleet Operations Considerations

2.2.3 T&C Database

2.2.4 Commanding Script and Procs

2.2.5 Telemetry Archive

2.2.6 Telemetry Decom / Display

2.2.7 Command Generation

2.2.8 Command Encryption

2.2.9 Limit Monitoring

2.2.10 Schedule Executor

2.2.11 Mission Data Management

2.2.12 Load Compiler

2.3 Trending and Analysis Subsystem Scalability

2.3.1 Telemetry Archive

2.3.2 User Interface

2.3.3 Data Export

2.3.4 Data Ingest

2.3.5 Limit Monitoring

2.3.6 Event Message and Command History

2.3.7 Statistics

2.3.8 Routine Products

2.3.9 Data Quarantine

2.3.10 Housekeeping Completeness

2.3.11 Data Replication

2.3.12 Cloud Migration

2.3.13 Machine Learning

2.4 Data Management Subsystem Scalability

2.4.1 Controlled Repository

2.4.2 Product Approvals

2.4.3 Controlled Repository Management

2.4.4 Web Portal

4 | P a g e

2.4.5 Mercury Model

2.4.6 Product Processing Glueware

2.4.7 Cloud Migration

2.5 Mission Planning Subsystem Scalability

2.5.1 Contact Scheduling

2.5.2 Routine Calibration Scheduling

2.5.3 Interval Planning (NADIR Imaging)

2.5.4 Collection Requests

2.5.5 Housekeeping Operations

2.5.6 Recorder Management Planning

2.5.7 Activity Compilation

2.5.8 Command Load Verification

2.5.9 Mission Planning to DPAS

2.5.10 Calibration Opportunity Scheduling Tool

2.5.11 Attitude Planning

2.5.12 Automation

2.6 Flight Dynamics Subsystem Scalability

2.6.1 Orbit & Attitude Determination

2.6.2 Orbit Control and Prediction

2.6.3 Conjunction Assessment

2.6.4 Attitude Prediction

2.6.5 Attitude Control

2.6.6 Attitude Sensor Alignment and Calibration

2.6.7 Product Generation

2.6.8 Propellant Bookkeeping

2.6.9 Fleet Data Management

2.6.10 Orbit and Attitude Validation

2.6.11 Station Visibilities

2.7 LMOC Hardware Scalability

2.7.1 Virtual Cluster

2.7.2 Storage

2.7.3 Networking

2.7.4 User Interface

2.7.5 Management

2.8 LMOC Software Scalability

2.8.1 COTS Software

2.9 Analytical Modeler

2.10 Configuration Management

List of Figures Figure 1: Current LMOC Hardware Resource Utilization

5 | P a g e

Figure 2: LMOC Hardware Costs and Annual Maintenance Costs

Figure 3: LMOC Memory and CPUs

Figure 4: LMOC Storage

Figure 5: LMOC Architecture Diagram

Figure 6: Notional L8/L9/LNext : Updated LMOC Architecture Diagram with LNext flows

Figure 7: Load Planning Flow Diagram

Figure 8: Load Building Flow Diagram

Figure 9: Load Execution Flow Diagram

Figure 10: CPAW Load Buildling Details Pt 1

Figure 11: CPAW Load Building Details Pt 2

Figure 12: CPAW Load Building Details Pt 3

Figure 15: Current LMOC Galaxy Context Architecture

Figure 16: Proposed Galaxy Context Architecture for Augmented LMOC

Figure 17: Realtime, RAM SSOH, and SSR HKDB Telemetry Paths, Use, and Storage

Figure 18: Command Generation, Transmission, and Processing

Figure 19: Landsat 8 Science Scheduling, Collection, and Product Generation High-Level

Overview

Figure 21: MPS Manual Edits made since 3/13/23

Figure 22: LMOC Virtual Cluster Resource Utilization

List of Tables

Table 3: Landsat 8 Activity Table

Table 6: ROC-Spacecraft Mapping in Current LMOC

Table 11: MPS Scaling and Reuse by Function

Table 15: LMOC Hardware Scaling and Reuse by Function

Table 16: LMOC Software Scaling and Reuse by Function

6 | P a g e

1 INTRODUCTION

1.1 Introduction

This document is provided for historical LMOC reference and informational purposes ONLY. It

SHOULD NOT be construed as direction or guidance for the LNext MOC. This study was one of many similar activities performed in 2023 during Phase A for Landsat Next, which included a

MOC RFI and other special studies on key Ground needs for LNext. The analysis and information included below were based on the state of the MOC in summer of 2023 at the time the study was performed and has not been updated or modified to reflect any changes made since then.

To reiterate, DO NOT utilize this document as an expected architecture for LNext, but only as historical reference of L8 and L9 systems and processes in 2023. The USGS will consolidate the results of all Phase A activities and vendor studies and from them will formulate the LNext

MOC.

The Landsat Multi-satellite Operations Center (LMOC) system provides all telemetry, command, trending, flight dynamics, and mission planning services for Landsat 8 and 9 operations, as well as infrastructure, automation, and alerting to support those services. The LMOC has a loosely coupled, highly scalable architecture that was designed to be multi-mission capable and can accommodate new Landsat observatories. Each new mission added to the LMOC will require some development checkout of mission specific capabilities and integration into the multi-mission environment. This special study will discuss the areas of the LMOC where observatory-specific software and processes exist, and what development would be required to support new observatories or operations concepts.

1.2 Proposed Near Term Actions

During the analysis and evaluations a few items have been identified which can be applied now to both improve L8 and L9 operations and prepare the LMOC the for LNext. These projects were evaluated based on the immediate positive impact they can make for the operational costs today.

1.2.1 STK Scheduler Replacement

The software which currently handles contact scheduling, STK Scheduler, still requires manual interactions and verification due to reliability concerns. In the legacy MOC, the FOT developed a

Perl script that handled all the scheduling tasks, using the still available view period products for ground contacts generated by the FDS software Meridian. There are open discrepancy uncovered reports (DUR’s) regarding the orbit number handling and spurious dropped contacts in STK

Scheduler. In addition, some of the operational processes must be handled manually due to reliability concerns, particularly any contact modifications and deletions are handled manually by mission planners. Developing the new tool provides an opportunity to streamline this process, eliminate the discrepancy impacts, have less proprietary code, and save on license fees with immediate effect.

Secondly, one of the new features which is needed for LNext, independent high rate / low rate scheduling, has immediate value for L8 and L9 as well. During MDM anomalies and periods of high image requests, having the ability to scheduling Xband contacts with our ground assets

7 | P a g e without having to deconflict the Sband command and telemetry would add functionality and flexibility to the operations teams.

1.2.2 Cloud Migration

There are two main cost drivers with the current LMOC on-premises implementation which we would like to rectify with cloud migration. If these two items are not addressed with LNext, then onsite hardware and software costs will continue to increase to meet the additional processing and storage needs of the LNext Fleet. Figure 1 illustrates the current LMOC cluster utilization.

As illustrated the LMOC is oversized for current operations. The CPU, Memory, and Network were all sized to support the strains of Launch and Commissioning which drove upfront equipment costs and continues to drive maintenance costs. The Storage is purchased in discreet

5-year increments and is sized to support the existing data archives and 5 additional years of collection.

Figure 1: Current LMOC Hardware Resource Utilization

The first cost driver is the compute resources and network / infrastructure to support them. The

LMOC needs to export as many offline subsystems, underutilized VM’s, and launch surge systems to the cloud as possible. This applies for LNext but also for existing Landsat 8 and 9 operations since most of these subsystems are fleet capable and will be augmented with LNext capabilities. The correct approach for performing this migration without incurring higher cloud service costs can be found in Section 3.3. This will manifest as cost savings in two ways. The first being the overall reduction in hardware needs for the LMOC cluster. As seen in the figures below (Figure 2) the costs for the compute resources and subsequent networking to support it are substantial. Currently the LMOC onsite compute resources could be reduced by half with no cloud migration and with additional cloud migrations possibly ¼ of the original. However, if

LNext does not support this cloud migration, then the compute needs to support LNext launch within the LMOC will drive these hardware and software costs past the original implementation.

8 | P a g e

Figure 2: LMOC Hardware Costs and Annual Maintenance Costs

Migration of compute resources will also reduce the LMOC infrastructure and networking needs.

As seen in the figures below the support infrastructure for the onsite systems consumes approximately 44% of the compute resources. Similarly, to how the hardware was sized to support launch, the infrastructure and network was sized to support the subsystems and the surge of users at launch. As we reduce the onsite footprint and export those infrastructure functions to cloud provider services, the onsite infrastructure requirements and needs will reduce.

Figure 3: LMOC Memory and CPUs

The second cost saving measure with cloud migration is to reduce the onsite data storage at the

LMOC. As seen in the figure below, over half of the onsite storage array is consumed just with

Virtual Machine Backups. This represents almost 150TB of storage and represents a significant cost investment. More efficient and cost-effective cloud solutions are available, and they provide far more data resiliency then an on-prem implementation could achieve. The other 42% of the onsite storage belongs to the STARS trending archive. Currently this archive represents close to

9 | P a g e

120TB of onsite storage. Within the next 5 years this is expected to exceed 200TB and which will drive backup needs past 250TB. With a few years of LNext data for 3 spacecraft the LMOC could easily exceed 1PB of onsite housekeeping storage. Most of this housekeeping and backup data is infrequently or outright never accessed. This makes it a prime candidate for cloud migration since most data storage costs are for egress, not ingress. More details on STARS cloud migration can be found in section 2.3.12.

Figure 4: LMOC Storage

1.2.3 Standardizing MPS activity schedule to WRS cycle

A major factor that drives the complexity of mission planning in today’s concept of operations is the lack of standardization of activities and how they interact with each other. The result is that each load build is a totally custom build, where we often experience new edge cases due to activities occurring near each other for the first time. To mitigate this, there should be all effort put into scheduling as many activities as possible according to the WRS grid, and therefore increase repeatability and decrease the discovery of new edge cases.

Table 1 gives a summary table of all the L8 activities and how they are scheduled.

Activity WRS2 Grid Based Cadence

OLI Dark Calibrations No, Currently scheduled by

CPAW around normal imaging

1-2 per orbit, 2nd one skipped on orbits with night imaging

OLI Working Stim Lamp Yes, South Pacific, range of paths, same row each time

Daily

OLI Backup Stim Lamp Yes, South Pacific, range of paths, same row each time

One per WRS cycle

10 | P a g e

OLI Pristine Stim Lamp Yes, Fixed Path/Row Biannual

OLI Solar Working Cal Yes, Fixed Path/Row Twice per WRS cycle

OLI Nonroutine solar cals

(Linearity, Pristine, Extended)

Yes, Fixed Path/Row Replace normal solar working at set cadences. Each of 3 types of non-routine solars occur biannually.

OLI/TIRS Lunar Cal No Once per lunar cycle, ~28 days

OLI/TIRS Lunar Cal + Temp

Sweep

No Replaces regular lunar cal once every 3 lunar cycles

Other Non-routine TIRS Cals:

FPA Transmit All, ITS Science

Mode, BB Stability, 36 min

DS, 36 min BB, TIRS

Recovery

Yes, Range of Paths, Fixed

Row

Variable, driven by lunar cal placement, 3 cals are placed as evenly as possible in between lunar calibrations

IMU Diagnostic No Every 14 days

Routine Off nadir collects Yes, Fixed Path/Row Salton Sea and Lake Tahoe each once per WRS cycle

Side Slither Collects Yes, Fixed Path/Row 3 slithers throughout year each with its own target

Orbit Adjustments No As needed, typically one DMU per 3-4 weeks, one INC per year

Table 1: Landsat 8 Activity Table

Two items stand out from this list: the non-routine cals driven by Lunar cal placement and the

IMU diagnostic telemetry downlinks. The latter is easily handled by moving it to a once per

WRS cycle cadence. The former is a bit trickier to resolve and requires buy in from the CalVal team. Obviously lunar calibrations are always going to be independently scheduled without any ties to the WRS grid based on lunar phase angle requirements. However, the other TIRS calibrations are scheduled by the following rules:

1. Place lunar calibrations on the full moon day (or the following day) to meet the lunar phase angle constraint of +5 to +9 degrees

11 | P a g e

2. Place the 1.5 Orbit Stability, aka Blackbody Stability calibration on the precise day that contains its desired singular WRS path

3. Place the following TIRS cals in between lunar calibrations evenly spaced: ITS Science

Mode, FPA Transmit All, 36 min DB, 36 min BB. These calibrations are placed on particular months throughout the year

4. Place TIRS recovery cals whenever there is a gap of more than 8 days between TIRS cals

As the calibration strategy for Landsat Next is developed, the activities should be set to the

WRS grid to the highest degree possible. This will simplify the activity deconfliction process of

MPS and allow the operations to be maximally automated.

2 LMOC SCALABILITY

2.1 High-Level Architecture

The Landsat Mission Operations Center (LMOC) is the segment of the Landsat Program that performs observatory operations. The LMOC is a combination of 4 separate facilities and includes the entire hardware and software infrastructure needed to support Landsat 8 and 9

Operations. The Primary LMOC is located at Goddard Space Flight Center, which includes both an operations area (LMOC) and the Landsat Mission Equipment Room (LMER) for server and rack housing. During Launch, Early Operations, and Commissioning (LEO&C) a Launch

Support Room (LSR) was present to handle Landsat 9 launch surge. This facility was decommissioned shortly after Post Launch Acceptance Review. Finally, the single string Backup

LMOC (bLMOC) facility provides physical redundancy in the event of a LMOC multi-thread failure. The bLMOC is located at the EROS Data Center in Sioux Falls, SD with a remote access point at NASA Goddard.

The LMOC consists of six primary services or functionalities:

• Telemetry, Command and Control (T&C)

• Mission Data Management (MDM)

• Mission Planning and Scheduling (MPS)

• Trending & Analysis (T&A)

• Flight Dynamics (FDS)

• Data Management (DMS)

Additional LMOC functionality supports the primary functions, including:

• Ground System Infrastructure (GSI)

• Alerting and Notification System (ANS)

• Government-furnished ground station & spacecraft simulators

• Analytical Modelling (AM)

• Work Management System (WMS): Jira, Confluence, Bitbucket

12 | P a g e

A high-level architecture of the LMOC with associated data flows can be seen in Figure 5:

LMOC Architecture Diagram below.

Looking ahead to the incorporation of Landsat Next, the T&C and MPS system are listed generically as we execute trade studies for software solutions. The ground element is also genericized as there are TBD’s for the assets to be used, such as GNE, Near Space Network

(NSN). Galaxy, CPAW, and STK Scheduler are shown retained for use on Landsat 8 and 9, and

Freeflyer, Mercury, and Stars are shown as retained for the entire fleet.

Flight Dynamics:

Freeflyer & MTASS

Mission Planning:

Scheduler & CPAW

TLM

Master Activity Lists

Ephemeris, Attitude slews

Telemetry:

Galaxy

TLM

Telemetry & Command: Galaxy

Events

Alerting & Notifications:

What s Up Gold

Events

Events

Events

Events Notifications

AWT

Data

Mgmt:

Mercury

CMD / TLM

Resource Model

L8 / L9

Simulators

Trending:

Stars

Spacecraft

Vendor

TLM

LCR, SCR

DPAS

GNE/LGN/iLGN

Pass schedules

FDS

products

GNE

CMD/TLM

SIFMT CFDP

SN / NEN

CMD/TLM

SSOH

AM: JASON

CMD/TLM

Legend

DMS (Mercury)

Other socket

LMOC Subsystem

External

Spacecraft-Unique Instances

(all others are spacecraft agnostic)

Figure 5: LMOC Architecture Diagram

As a part of this study we will begin to explore the addition of new subsystems to the current

LMOC architecture as well as migration of subsystems into the cloud. The LMOC was built on the principle of modularity with loosely coupled interfaces. This design is well suited for adding additional subsystems that interface with existing capabilities and for migration of subsystems to the cloud by extension of our network file systems. As noted earlier, the T&C subsystem and the

MPS subsystem currently have the lowest reusability, so a notional architecture has been provided in Figure 6 to illustrate what a modified LMOC infrastructure may look like if new

LNext software was brought in for these two subsystems.

These new subsystems can be integrated within the LMOC with little to no disturbance to the other existing systems. Some mercury modifications would be needed to handle the new product flows but these are all done in configuration space and can be easily handled. The FOT still make regular mercury product updates when they have new products or would like to add new recipients to existing products. We would perform the necessary development work to make the new subsystems compatible with the existing FDS and DPAS products to concentrate development efforts on the new systems instead of opening source code on existing proven

13 | P a g e systems. Since all of the LMOC’s interfaces are file based, the integration of new subsystems can be handled in a straightforward fashion.

Figure 6: Notional L8/L9/LNext : Updated LMOC Architecture Diagram with LNext flows

A medium resolution set of flow diagrams describing the full end to end process for the generation, approval, and activation of the current command loads are given below in Figure 7 through Figure 12.

14 | P a g e

Figure 7: Load Planning Flow Diagram

Figure 8: Load Building Flow Diagram

See Below for Details

See Below for Details

15 | P a g e

Figure 9: Load Execution Flow Diagram

Figure 10: CPAW Load Buildling Details Pt 1

16 | P a g e

Figure 11: CPAW Load Building Details Pt 2

Figure 12: CPAW Load Building Details Pt 3

The majority of the tasks listed in the above figures are automated. Those which are not are noted by the description indicated tasks done by the mission planners (MPers).

2.2 Telemetry and Command Subsystem Scalability

The Telemetry and Command functionality is provided by the GALAXY software developed by the Hammers Company. The Hammers Company also develops and maintains the Mission Data

17 | P a g e

Management tool, commonly called the Landsat File Accounting Tool (LFAT) which is paired with the GALAXY System. GALAXY also contains the Stored Command Compiler (SCS) which compiles and builds the observatory command loads. The T&C system establishes, configures, and maintains the data communications paths between the LMOC components and the Networks

(Ground Network Element [GNE], Near Earth Network [NEN] and the Space Network [SN]) or

Landsat Simulators. The Telemetry function processes Housekeeping (S-band) real-time telemetry to assess spacecraft state of health and supports command functions. This function evaluates any housekeeping (HK) data: real-time (R/T), stored state of health (SSOH), and station playback (P/B). The Command function generates, manages transmission, and verifies all Landsat commands. This function supports command encryption, authority management, and constraint checking prior to transmission of the commands to all Landsat observatories.

Command verification is performed following issuance of commands. Command loads are built to implement observatory activity schedules and flight software updates.

2.2.1 Galaxy Pressure Points

The following sections will look at each functional area within the Telemetry and Command system with an explanation of the readiness assessment for each. This section will give an overview of the major pressure points that impact both current L8/L9 operations as well as those which need to be addressed to support the LNext triplet. The augmentation plan for the LMOC will include:

1. Command Encryption: We have experienced six GEDU crashes since 2023 while operating Landsat 9, which all resulted in missed ground contacts and emergency anomaly response from the realtime ops and GSE teams. This is related to the hybrid infrastructure of a virtual machine ROC interfacing with a physical EDT card.

2. Mission Data Management: The LNext conops will require a total rework of the LFAT tool..

3. Non-automated tasks: In an effort to reduce staffing the operations should be automated to the highest degree. There are a few tasks which has remained manual operations in the

LMOC due to an absolute minimum risk stance or due to lack of return on investment to automation. These include:

a. Orbit adjustments: These could be automated today, there’s nothing technical in the way. The reason it’s not currently is minimum risk posture. The savings in terms of team effort is high between the simulations and realtime support during the burn.

b. Monthly EOP (drag and timing parameters) uplinks: could be automated, trigger off the delivery of the EOP file to the ROC. Automating this would save a few hours a month so the return on investment is low.

c. Proficiency exercises (connectivity tests with stations, handover contacts, blind acquisitions, low rate passes): To some extent these activities cannot be fully

18 | P a g e automated. Handover contacts have to be manually coordinated with the ground station operators. But the team is implementing processes to ease the planning aspects of these and can continue to refine these processes going forward.

d. Anomaly responses: Again this cannot be fully automated across the board, but with an increase in the use of configuration monitors and telemetry monitors some anomaly responses can be automated. This has already been used to effect for a few anomalies that have occurred after launch of L8 and L9, including PIB-T lockup mitigation for L8, RFI response for L9, and SSR bad block response for

L9, among many others.

4. Database Translation: Assuming the T&C database input file format will be significantly different than the format used by Northrup Grumman for L8 and L9, then there will be significant rework on the PRDS translation tool.

5. Flight Procs: With the anticipated little heritage with Landsat 8 and 9 for spacecraft flight hardware, almost no reuse of flight procs for Landsat Next.

2.2.2 Fleet Operations Considerations

As the LMOC expands to support a growing fleet there are a few major considerations as it pertains to the T&C system design and development. First is the consideration of the context structure for the fleet. There are currently three levels that are considered for the context (see

Figure 13 below). All the operations products (procs, pages, sequential prints, configuration monitors, record language files, other config files) are assigned a context (L08, L09, FLEET, L8L9, ALL) which is determined by its directory location within the Galaxy repository.

Whenever the Galaxy application is started, a context is specified. Only the lowest level green boxes are permitted as selectable contexts in which to run Galaxy (L08, L09, or fleet). For each of these selections, the Galaxy session has access to operations products which are specific to that context, but also inherits the operations products for any linked context at higher levels. For example, L08 context has access to L08 products, L8L9 products, and ALL products.

Figure 13: Current LMOC Galaxy Context Architecture

19 | P a g e

The bottom right corner of this chart shows L8 and L9 individual products and shared products in the L8L9 bin. The FLEET context is used very rarely, but the intent was to provide a context which could connect to telemetry streams for either L8 or L9. It has no access to any commanding procs, but has a few pages which contain both L8 and L9 telemetry. It primarily is configured for displaying ground configuration information. The ALL context has no Landsat specific operations products and is the location for Galaxy application-level products (start up and tear down scripts for example).

The proposed context structure for the combined L8, L9, LNext fleet is shown in Figure 14.

Notice that the second yellow layer differentiates the two generations of the fleet. There are likely to be a significant amount of operations products which live at this level. Most of the current L8L9 products are unlikely to be applicable to the LNext fleet, and will remain in the same bin. And most of the LNext flight and ground procs are likely to be consistent for the entire

LNext triplet and live in the LNext bin. The LNext1,2,3 boxes are expected to contain only a small number of deltas for each spacecraft within the triplet (e.g., differences in telemetry limits).

The LSAT level is added in the gray box to contain products relevant for all five spacecraft. The

FLEET context remains for telemetry monitoring of multiple spacecraft. The ALL context also remains for Galaxy application-level products.

Figure 14: Proposed Galaxy Context Architecture for Augmented LMOC

Another fleet operations consideration is how we manage the fleet mapping to real-time operations consoles (ROC’s). Currently the LMOC has enforced that each ROC is able to command only one spacecraft, L8 or L9, with no crossover potential (See Table 2).

ROC Commanding Ability

LMOC ROC A L08 (Primary)

LMOC ROC B L08 (Secondary)

20 | P a g e

LMOC ROC C L09 (Primary)

LMOC ROC D L09 (Secondary)

LMOC ROC E L09 (Contingency)

LMOC ROC F L08 (Contingency)

BLMOC ROC A L08 (Tertiary)

BLMOC ROC B L09 (Tertiary)

Table 2: ROC-Spacecraft Mapping in Current LMOC

2.2.3 T&C Database

The T&C database functionality involves the ability of the ground system to translate a command and telemetry database from the spacecraft and instrument vendors into a format useable by the T&C system. For Galaxy this means generating record language files (rec files) for the telemetry mnemonics, command mnemonics, telemetry limits, and mnemonic conversions.

For L8 and L9, the format of the T&C database files was in XML coding, one file per command and telemetry mnemonic. The XML file would prescribe all the pertinent information for that mnemonic: APID, byte location, variable type, conversion polynomial if applicable, etc.

One further method to improve efficiency is to have a single set of database files for the LNext triplet which then only needs to be translated once. If there are any small changes between each of the triplets, those can be captured in separate rec files which are then compiled into a Galaxy database specific for that spacecraft.

2.2.4 Commanding Script and Procs

The effort of writing and/or updating the procedure (Procs) and commanding scripts for Landsat

9 launch was a significant percent of the overall Galaxy I&T effort. LNext has significantly less heritage to the on-orbit observatories than L9 did to L8. Therefore, it is largely expected that the majority of the flight procs will have to be rewritten. However, the team can leverage the established process for formatting the procs, testing them, and managing configuration.

In addition to flight procs, there are ground procs which are largely multi-mission in nature.

These procs would require significantly less rework than the flight procs. Overall, L8 has 465 flight procs, while L9 has 439. For ground procs there are 45 multi-mission procs, along with 24 specifically for L8 and 19 for L9.

In preparation for the L9 launch, 421 procs were written and certified for operations as the team successfully executed 8 MRTs, 7 GRTs, and 5 Mission Simulations in a span of 10 months.

21 | P a g e

2.2.5 Telemetry Archive

Galaxy’s telemetry archive subcomponent contains no mission-unique elements and could be readily used with the new LNext triplet of observatories without modification. Figure 15 shows telemetry paths for real-time data, RAM SSOH telemetry downlink via S-band, and HKDB file downlink via X-band, applicable to L8 and L9. Depending on LNext comm systems (especially

Ka-band vs X-band), some changes to the inputs to the network layer may be required, but the

LMOC systems can easily be adapted for maximum reusability and cross-mission support.

Figure 15: Realtime, RAM SSOH, and SSR HKDB Telemetry Paths, Use, and Storage

2.2.6 Telemetry Decom / Display

Galaxy’s decom and telemetry display subcomponents are driven by the T&C database and contain no mission-unique elements which make it ready for the LNext observatories without modification.

2.2.7 Command Generation

Galaxy’s command generation subcomponent is driven by the contents of the T&C database and contains no mission-unique elements which make it ready for the LNext observatories without modification. A high-level command flow diagram is shown in Figure 16, with functions in dark blue provided by Galaxy in the LMOC (and available in most other T&C software systems). One important note is that Galaxy does not perform command encryption, but it does encapsulate encrypted command data for network routing; it is the sole input and output for the ground encryptors.

22 | P a g e

Figure 16: Command Generation, Transmission, and Processing

2.2.8 Command Encryption

The LMOC utilizes hardware command encryption solutions for both Landsat 8 and 9. These hardware encryptors are interfaced to the Realtime Operations Controller (ROC) virtual machine.

Another factor to consider during the augmentation plan of the LNext is to ensure there are enough encryption keys to support the full expected lifetime of the fleet under the key management plan.

2.2.9 Limit Monitoring

Galaxy’s limit monitoring subcomponent is driven by the contents of the T&C database and contains no mission-unique elements and could be readily used for the LNext observatories without modification.

2.2.10 Schedule Executor

Galaxy’s schedule executor subcomponent is driven by a schedule file generated by the mission planning system and contains no mission-unique elements which make it ready for the LNext observatories without modification.

2.2.11 Mission Data Management

Mission Data Management (MDM) was a significant portion of both the system development and the I&T effort for T&C. The current high-level MDM concept is shown in Figure 17, which shows the sequence of key events and file transfers in the generation of a science product for

Landsat 8:

23 | P a g e

• MPS generates command load inputs and a matching Scene Interval Mapping Table

(SIMT), which will correlate SSR file ID numbers with path/row intervals.

• The ROC generates the spacecraft command load and ingests the SIMT file into LFAT

(operating on the ROC).

• The spacecraft collects the image and records it to the SSR.

• On a realtime contact, the spacecraft transmits science data over X-band directly to the ground station, while sending the directory listing of files in the recorder to the ROC.

• The ground station sends the list of acknowledged files (ACKs) to the ROC.

• LFAT correlates the lists of recorded files and science intervals, generates the Scene

Interval File Mapping Table (SIFMT), and sends it to EROS for use in reconstructing the science image files from the raw X-band files transmitted from the ground station.

This process is much the same for Landsat 9, with a key terminology difference: the Image

Window Table (IWT) command load for L8 is called the Mission Absolute Time Sequence

(MATS) on L9, though both contain timestamped record commands and other science activity sequences. For the sake of simplicity, a number of internal steps are omitted, to allow the reader to concentrate on interfaces between mission elements.

Figure 17: Landsat 8 Science Scheduling, Collection, and Product Generation High-Level Overview

Figure 17 also illustrates the importance and complexity of LFAT, as it provides correlation between three different science data accounting systems:

24 | P a g e

• The spacecraft SSR, which identifies files only by Root File ID (RFID, the SSR file ID name commanded by the IWT/MATS)

• The ground science collection plan in MPS, which identifies intervals to be collected from various inputs and assigns RFIDs to the record commands in the IWT/MATS

• DCRS, which works on interval IDs and has no direct insight into RFID planning. LFAT is crucial to this science processing by providing the SIFMT that maps RFID from the ground station and scene IDs planned in MPS.

2.2.12 Load Compiler

The reuse for the load compiler software (SCS) is very high, however the development of the activity definitions which link an activity to the corresponding sequence of commands to include in the Absolute Time Sequence (ATS) stored load are very mission dependent. The main difference is that usually the demonstrations of the product to the vendor is done by tabletop reviews of the ATS and simulation logs as opposed to live simulator demonstrations for practical time reasons.

2.3 Trending and Analysis Subsystem Scalability

The Trending & Analysis (T&A) functions process near-term and long-term observatory telemetry data from all Landsat observatories. This function is used to trend and analyze the performance of each observatory for state of health monitoring using all collected housekeeping data (e.g., potential problems with the observatory attitude, power, temperature, or other subsystems). T&A provides the capability to report data in various formats to meet engineering needs. The LMOC already has the ability to access this functionality remotely.

2.3.1 Telemetry Archive

Archival of telemetry data in the STARS database is driven by the mission T&C database (the same database used by Galaxy) and therefore requires no modification to support additional missions. Incorporating LNext into the LMOC would require obtaining more storage arrays, either physical hardware or via cloud storage services. Currently Landsat 8 and 9 are generating approximately 20.5 TB of housekeeping data annually combined. The LMOC storage system is currently sized to accommodate L8/9 data through 2028.

LNext would likely require its own instance of STARS with a separate database from Landsat 8 and 9. This would mean that there would be potential of process collision between the two generations. It also is likely required from a practical standpoint in terms of size of database, failovers, database scrubbing, etc.

2.3.2 User Interface

The STARS user interface contains no mission-unique elements and could be readily used for the

LNext observatories without modification.

25 | P a g e

2.3.3 Data Export

The STARS data export features contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.4 Data Ingest

To support simultaneous ingestion of L8 and L9 housekeeping data, the STARS ingest software has been threaded. Theoretically a single STARS server could be resource loaded to handle additional missions. The system has demonstrated up to five parallel ingests during operations

(concurrent L9 ops, 3 parallel L8 legacy ingests, L9 simulation). STARS current ingests a rate

14x the housekeeping rate for L8 and 12x for L9 and has been shown to perform well even with months of continual 100% duty cycle usage. To routinely support five spacecraft plus simulations plus readiness testing, STARS would likely need a resource increase in RAM and

CPU cores.

However, there is a trade study assessing a single STARS server vs. separate servers for L8/L9 and for LNext. Having separate servers would allow partial failovers of operations from LMOC to BLMOC. In addition, it would enable an upgrade in the Postgres database version to the latest version without potential impacts to historical data. Separate servers mean higher licensing fees.

A hybrid approach is also under consideration where LNext begins in a new cloud-based server and then L8/L9 merge into that server (or its own cloud-based server) when the on-prem hardware is due for a refresh.

There are two modes for data ingests and how to keep servers in sync between LMOC, BLMOC, and during LSR (during LEOC). The preferred method is to have a primary and backup system and sync the database from primary to backup. The second method is to set up parallel ingests to all environments, which requires increased monitoring to ensure the servers stay in sync. The default configuration would be that the LMOC is primary and BLMOC is receiving database syncs from the LMOC. During most BLMOC failovers, the configuration would be toggled to parallel ingests in both environments. This would remain until operations is restored in the

LMOC. The hybrid approach allows each spacecraft to failover when the comm schedule determines it to be convenient. For longer duration LMOC outages, once all spacecraft were transitioned over, the BLMOC would instead become the primary and would provide syncs back to the LMOC.

2.3.5 Limit Monitoring

The STARS limit monitoring and reporting features are all driven by limits in the T&C database and therefore contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.6 Event Message and Command History

Command history validation is performed using the load report and the command echoes from housekeeping telemetry archives. A simple script is utilized to perform this check. It is anticipated that only small modifications to this script will be necessary to support new missions.

26 | P a g e

2.3.7 Statistics

The STARS statistical computation and reporting features contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.8 Routine Products

Generation of routine trending products from STARS is not driven by any mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.9 Data Quarantine

The STARS data quarantine features contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.10 Housekeeping Completeness

Currently mission housekeeping data is stored onboard the L8/L9 observatories on the Solid

State Recorder (SSR) in file format, and is played back to the ground as a CADU file, which is then captured by the ground station and sent to the LMOC for processing. This requires that the

LMOC determine the time period that the data covers from the command load and the packet sequence.

This process is handled by housekeeping completeness software in STARS. While this tool is generic enough to be used on future missions, the process would be more efficient and consistent if instead the onboard CCSDS File Delivery Protocol (CFDP) software modules were used to package and downlink housekeeping data files.

2.3.11 Data Replication

The STARS data replication features contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.3.12 Cloud Migration

There are two approaches for migrating STARS to the Cloud. The first approach involves little to no software modifications but will yield little to no cost savings. The second approach will involve some modest software improvements and enhancements but may yield significant savings with hardware and data storage costs.

The first approach is to stand up a second STARS system in the cloud to be used as a long-term cold archive. This STARS instance would be a lift and shift application and we would anticipate rare access by design. Once every year the cold archive would be powered up to ingest the previous year’s data, this ingestion would coincide with the Mission Annual Review when the

FOT typically looks at performance metrics across the life of the mission. Once all the data had been ingested and the cold archive was up to date, annual metrics would be generated and the

VM would be powered off. The on-prem STARS system would currently be holding 2 full years of mission data and could be pruned down to 1 year of mission data and continue serving FOT

27 | P a g e needs. The offline STARS system in the cloud could still be accessed by the FOT, it may just require additional time to bring the system online, which is acceptable for long term trending.

The On-Prem system would maintain a lean storage archive improving performance of the more often looked at data.

The second approach to STARS migration to the cloud involves software updates and enhancements to make the application more cloud native. In this implementation the entire

STARS trending system and archive would be moved into the cloud and no on-prem system would exist. The STARS software would be containerized, and the STARS data files would be reformatted to cloud native storage formats.

2.3.13 Machine Learning

Telemetry Analysis with Learning Algorithms (TALA) is an anomaly detection system. It was tested on data from the L8 Legacy MOC and implemented in the LMOC for both L8 and L9 missions. TALA uses Python’s scikit-learn machine learning packages to evaluate deviations in telemetry based on previous mnemonic behavior. TALA is trained on past data to learn expected behavior, then evaluated on new data to detect anomalies. TALA is designed to detect more subtle behavioral changes and anomalies than we could detect with our other tools (config check, telemetry limits).

TALA is fully fleet capable: each spacecraft has config files defining the predictive and monitored mnemonic lists. Scaling up to support the LNext fleet is simply a matter of defining new mnemonic lists for the new spacecraft and setting new cron jobs to train and evaluate the data.

2.4 Data Management Subsystem Scalability

Product, event, and report handling is the process by which the products, reports and events generated by the LMOC Subsystems are delivered to the intended recipients. Products are considered any file or information set generated by a LMOC subsystem intended for use by another LMOC subsystem or external element. Reports are human readable documents or displays intended for use by the Flight Operations Team (FOT). In this context, events are considered the warning or error messages generated by the various subsystems and used to alert the FOT. The ability to monitor these event messages remotely is a function of the LMOC system.

2.4.1 Controlled Repository

The Mercury Controlled Repository (CR) is organized by mission and the Mercury model.

Automated scripts are used to build the repository structure given a mission name and can be used to add additional missions.

The I&T rework needed in this area is focused on establishing new Mercury mounts and network file systems (NFS’s) for the LNext fleet, as well as controls to ensure proper segregation of products between the missions.

28 | P a g e

2.4.2 Product Approvals

The Mercury Product Approval processes contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.4.3 Controlled Repository Management

The Mercury CR Management tools contain no mission-unique elements and could be readily used for the LNext observatories without modification.

2.4.4 Web Portal

The Mercury Web Portal is a HTML representation of the archive and contains records on product movements throughout the LMOC. There is no mission specific code in the web portal, and it could be used for the LNext observatories without modification.

2.4.5 Mercury Model

All products within the LMOC are labeled by the Mission and Observatory source (Flight or

Simulator). A new suite of products would be generated for additional Landsat missions. These product naming conventions would need to be added to the Mercury model. The Mercury model is a FOT configurable XML file defining all the product movements in the LMOC.

The I&T reuse is driven by updates required to the Mercury model for new products defined by the next generation of the mission. There were some significant new products and interfaces established during the generation of the LMOC and the expectation is that there will be a more modest adjustment for LNext.

All products within the LMOC are labeled by the Mission and Observatory source (Flight or

Simulator). For existing products in the Mercury model, the regular expressions must be updated to add in the new spacecraft. The Mercury model is a FOT configurable XML file defining all the product movements in the LMOC.

2.4.6 Product Processing Glueware

There are a number of support scripts that reside within the Mercury subsystem which are used to manage modifications to products to meet ICD requirements or to support operations ease automation. With a modest expectation of new products required for the LNext fleet, some of these glueware scripts may require rework.

2.4.7 Cloud Migration

The Mercury controlled repository can easily be migrated to the cloud since it is all file based.

Once products hit their pruning age in Mercury (which is approximately 30 days) they should be moved to cold storage in the cloud such as S3 Glacier, EFS or FSx. These files are accessed extremely infrequently and could be stored in a much more cost-effective manner.

29 | P a g e

2.5 Mission Planning Subsystem Scalability

The current software solution in operations in the LMOC for mission planning includes the following components:

• Systems Toolkit (STK) Scheduler: Handles contact scheduling and forecasting, including

Landsat Ground Network (LGN), Near Earth Network (NEN), Space Network (SN), and

International Collaborators (IC). The base software is proprietary provided by Orbit

Logic, but the Landsat Interface and scheduling scripts are not proprietary.

• Collection Planning and Analysis Workstation (CPAW): Handles activity deconfliction, image collection planning, routine calibration scheduling, solar/offnadir/side slither attitude maneuver planning, MP report generation. The base software is proprietary provided by Orbit Logic, but the Landsat Plugin and default task orders are not proprietary.

• Calibration Operations Schedule Tool (COST): Non-proprietary FOT-built Python script for calibration scheduling, forecasting, and deconfliction.

• Report Builder and Checker (RBC): Non-proprietary FOT-built package of Python scripts for automated load constraint checking

• Master Activity List (MAL) Post-processor (MPP) and Edit_SIMT scripts: Non-proprietary FOT-built Python scripts for automated load postprocessing

• File Monitor: Non-proprietary FOT built Python package for MPS automation. Handles all file manipulation, triggers some post processing scripts, and generates the final load package for FOT review.

The planning and scheduling tools in the LMOC will require the most changes when adding new missions. Since mission planning is the primary implementation of the flight segment operations concept, changes in imaging and calibration concepts, or changes to the WRS grid will have impacts to the planning and scheduling tools and require re-development. Since it is the most vulnerable to mission changes, it is also the area where it would be most logical to simplify by implementing more mission planning capabilities onboard the observatories. By migrating some key functionality and processes onboard we can remove some of the mission-unique aspects from the LMOC MPS tools and make them more readily scalable for additional missions.

The following table provides a high-level overview of the key functionality provided by CPAW

& STK Scheduler, its potential reuse for additional missions, and some particulars for where mission specific functionality has been configured or developed.

30 | P a g e

Table 3: MPS Scaling and Reuse by Function

2.5.1 Contact Scheduling

Given a triplet of observatories for LNext, it is anticipated that more LMOC internal station management and deconfliction activities will be required. The lower altitude and longer repeat cycle for the LNext triplet compared to L8/L9 dictates that there will be periodic conflicts for shared resources due to the relative precession of the two generations of observatories. With the assumption that some of the ground assets will be shared across the entire Landsat fleet, the

LMOC MPS tools may need to be able to segment up station contacts and perform internal mission prioritization. This is not a functionality currently available in the LMOC.

The switch to Ka-band antenna (from X-band) for the high-rate downlinks may result in a significantly different set of ground antenna resources supporting LNext, which would somewhat relieve the impacts and frequency of the shared resource deconfliction. However, the RFI material proposed 5-8 S-band contacts and 10-12 Ka-band contacts per day per LNext bird.

Separating the S-band assets from the high-rate downlink assets (X-band…