SDSDv1.3_14Dec2021_signed_ADDM_A.pdf

Systems Development Digital Division 14 December 2021

Software Development Strategy Document (SDSD)

Version 1.3

Prepared by

Systems Development Digital Division

Patrick Space Force Base, Florida 32925-3516

Distribution authorized to US Government agencies for administrative or operational use. Other requests for this document shall be referred to AFTAC/SDD, 1020 South Patrick Drive Building 10989, Patrick Space Force Base, Florida 32925-3516

JOSEPH A. WEHLEN III, GG-15, DAF

Chief, Digital Division, AFTAC/SDD

SDSD V1.3

REVISIONS

Version Date Author/Team Revision Description Authorized by

1.0 28 May 20 SDD Initial Lt Col Paz

1.1 29 Jun 20 SDD Changed ELK to EFK; Logstash to Fluentd

Joseph Wehlen

1.2 9 Nov 20 SDD Added versioning description Joseph Wehlen

1.3 14 Dec 21 SDD Included latest guidance on containers and FOSS

Joseph Wehlen

TABLE OF CONTENTS

1.0 Context

2.0 Functional Overview

3.0 AFTAC Information Technology Strategic Planning

3.1 Roles and Responsibilities

3.2 Information Technology Infrastructure Implementation

3.3 Software Planning

4.0 Principles

4.1 Services

4.2 Architectural Layering

4.3 Adopt, Buy, Create

5.0 Constraints

5.1 Containerization

5.2 Virtual Machines

5.3 Databases

5.4 Languages

5.5 Classification

5.6 Versioning

5.7 Security

5.8 Messaging Formats

6.0 Quality

6.1 Attributes

6.2 Monitoring

6.3 Code Quality Scans and Reporting

7.0 Infrastructure

7.1 Development/Staging/Production

7.2 Development Environment

7.3 Staging Environment

7.4 Production Environment

7.5 Virtual Machines

7.6 Databases

8.0 Data

8.1 AFTAC Data Governance Council

8.2 Storage Requirements

8.3 Data in Motion

9.0 Deployment

9.1 Deployment Strategy

9.2 Contingency Plans

9.3 Operations and Support

10.0 References

11.0 Acronyms

TABLE OF FIGURES

Figure 1 – Common Interface Concept Figure 2 – Architecture Layers and IT Strategies Figure 3 – Differences between Software Services Paths

Figure 4 – Structure of Vitual Machines and Containers Figure 5 – Example Versioning Scheme Figure 6 – Development/Staging/Production Environments Figure 7 – Continuous Integration Pipeline

1.0 CONTEXT

This software development strategy provides guidance on the architecture, design, development, and delivery of software for use in the Air Force Technical Applications Center (AFTAC). This strategy encompasses both software used for 24/7 operational missions, as well as Research and

Development (R&D) tools and applications. The intent of this document is not to provide an exact architecture or coding standards, but to provide strategic guidance on AFTAC software development efforts. The intended audience includes AFTAC’s internal workforce and external government and contractor teams performing software development for or on behalf of AFTAC.

Adherence to this strategy’s guidance will enable faster adoption of new technologies and provide new capabilities to the mission user through efficiencies gained by common architectures, software components, and coding conventions.

Recommendations on changes to this document should be submitted in writing to AFTAC/SDD.

2.0 FUNCTIONAL OVERVIEW

AFTAC’s legacy software architecture is built around a number of disparate systems lacking commonality of infrastructure. As a result, these systems became stovepipes without any communication between systems. To address these shortcomings, AFTAC is embarking upon a strategic plan to develop a core set of capabilities, such as logging, auditing, data access, messaging, etc. Once developed, this core functionality will be accessible to software focused on specific mission areas and techniques, such as Geophysical or Materials. The end-goal is to extrapolate as much as possible into the core capabilities to reduce the footprint of mission-specific software. Figure 1 illustrates this concept where custom-developed applications leverage common functionality and access disparate databases through a common interface.

Figure 1 – Common Interface Concept

3.0 AFTAC INFORMATION TECHNOLOGY STRATEGIC PLANNING

As the center is the leading federal organization for supplying technical information to the United

States government, it must maintain an integrated approach to leveraging its unique datasets and analysis tools to deliver timely information for expanding decision space to policy makers.

3.1 ROLES AND RESPONSIBILITIES

The success of any process to meet an organization’s vision and needs is dependent upon clear guidance and direction. To accomplish this, AFTAC will assign technical experts to specific roles with the responsibility for defining and guiding this process. These Information Technology (IT) domain leads below will serve as a collective for charting the center’s overall IT strategy in terms of the center's data, software, and IT hardware needs. The roles and responsibilities of these individuals will include, but not be limited to:

IT strategic planning IT investment planning IT transitioning to enable operations IT requirement forecasting

3.1.1 Chief Software Architect

The AFTAC Chief Software Architect (CSA) oversees and manages the software process and procedures and provides the strategic vision for AFTAC software. The CSA’s primary role is to provide hands-on software architecture leadership, guidance, and mentoring to all distributed software engineering teams, and to be a close collaborator and technical advisor to the Systems

Development Directorate (SD) to ensure the delivery of high-quality, robust, and scalable solutions. The CSA is responsible for ensuring AFTAC has a strategy for software architecture to manage current and future technologies. The CSA chairs the AFTAC Software Strategy Council.

3.1.2 Chief Technology Officer

The AFTAC Chief Technology Officer (CTO) oversees and manages the AFTAC computing infrastructure. The CTO’s primary role is to provide hands-on computing hardware architecture leadership, guidance, and mentoring to the engineering teams, and to be a close collaborator and technical advisor to the 709th Cyberspace Squadron (709 CYS) to ensure the delivery of high-quality, robust, and scalable solutions. The CTO is responsible for ensuring AFTAC has a strategy for the computing architecture to manage current and future technologies.

3.1.3 Center Data Officer

The AFTAC Center Data Officer (CDO) is responsible for providing enterprise-wide governance and utilization of information as an asset, via data processing, analysis, data mining, information trading, and other means. The CDO serves as the chairman for the AFTAC Data Governance

Council (DGC), which is composed of data stewards from across AFTAC’s mission areas.

3.2 INFORMATION TECHNOLOGY INFRASTRUCTURE IMPLEMENTATION

To support a truly center-wide IT process, the data, software and hardware strategies and visions must align. AFTAC is focusing on a layered architecture approach to software as the base for all these strategies. This approach provides a separation between differing components and organizes them in horizontal layers. This method of design is meant to be self-independent meaning that components are interconnected but do not depend on each other. While this approach can have n-tiers, AFTAC is focused on four primary layers. The Presentation, Business, Persistence, and Data layers.

The Presentation layer focuses on how data is presented to the users. This may be either with GUIs, file outputs, maps, or any other method users utilize to extract or interpret data.

The Business layer contains business or scientific logic. This layer is where the bulk of scientific processing lives and any other transformational or processing algorithms are placed.

The Persistence layer provides a layer of abstraction between the Data layer and the

Business logic layer. This layer is intended to allow the upper layers to query, access, and store data through common Application Programming Interfaces (APIs) without detailed knowledge of the storage mechanism or organization. This is where AFTAC will place a Data

Fabric to provide a single method for data access.

The Data layer is the lowest layer in this approach and is where all the data is stored.

In aligning the strategies with this architectural approach, each of the three strategies helps to define and plan for one or more layers.

The Software Strategy focuses primarily on the Presentation and Business layers, defining the technical decisions and architectures that go into those layers. The Software Strategy also helps to define and manage the Persistence layer by ensuring the API allows appropriate access and structure for the needs of the upper layers.

The Hardware Strategy is primarily responsible for defining the Data layer and what types of databases and technologies are preferred by the center for storing and backing up data.

Although a database is essentially software, AFTAC’s database administrators fall within

709 CYS and as a result database software is managed by the AFTAC Infrastructure Strategy

Council (AISC). The Hardware Strategy must also define and provide an infrastructure that can support all four layers of the architecture.

The Data Strategy is primarily focused on the Persistence and Data layers. For the Data layer, the Data Strategy is concerned with ensuring the data policies are met. For the

Persistence layer, the Data Strategy defines the minimum metadata requirements as well as the requirements for data access, discoverability, and use in multi-phenomenology analysis.

Figure 2 – Architecture Layers and IT Strategies

3.3 SOFTWARE PLANNING

All AFTAC software projects shall have planning sessions allowing for early and continuous discussions on architecture, security, requirements, and priorities. The following items should be discussed early at technical meetings to define runway and frameworks for software development.

Identification of requirements/use cases (stakeholders should include both end users as well as other AFTAC units requiring engagement).

Potential use of existing Commercial Off The Shelf (COTS), Government Off The Shelf

(GOTS), or Free and Open Source Software (FOSS).

Identification of the solution’s technologies (databases, programming languages, frameworks, etc.).

Data required for solution (including data flow diagrams detailing new data acquisition or modification of existing data/data flows).

Deployment strategy (delivery schedule, licensing costs, etc.).

4.0 PRINCIPLES

All software written specifically for AFTAC should follow the below principles.

4.1 SERVICES

4.1.1 All new software should utilize services, which may be Service Oriented

Architecture (SOA) based to support large system development, or the finer grained microservices to support smaller-scale web applications. Whichever services path taken, SOA or microservices, AFTAC is working towards an end-goal of sunsetting its legacy, monolithic systems in favor of services-based architecture. AFTAC plans to extrapolate as much as possible into core infrastructure provided capabilities to reduce the footprint of mission-specific software. Figure 3 illustrates the differences between these services-based architectures.

Figure 3 – Differences between Software Services Paths

4.1.2 Software services should adhere to industry best practices, such as statelessness, loose coupling achieved through dependency injection, and dependency inversion through the

Hollywood principle (“don’t call us, we’ll call you”).

4.1.3 Production microservice databases for a given project should be combined on a single Oracle database instance using separate table spaces or accounts, rather than creating a new

Oracle server for each microservice. This prevents sprawl of database instances and reduces maintenance costs.

4.2 ARCHITECTURAL LAYERING

4.2.1 All new software should implement a multi-tier/multi-layer architecture and must enforce the following architectural layers when required as shown in Figure 2: Presentation, Application/Business, Data/Persistence, and Database. Application layering facilitates maintenance by ensuring services can be tested independently, and changes to one layer have little or no impact on services in other layers.

4.2.1.1 Of course, not all applications will require every layer. For example, applications without a user interface do not require a Presentation layer, and simplistic applications may not require a database.

4.2.1.2 However, for applications that do require database access, this should be performed through a Data/Persistence layer to disconnect database access from application/business logic.

4.3 ADOPT, BUY, CREATE

4.3.1 Software development is an expensive endeavor, which does not simply end at product deployment but continues through sustainment and maintenance efforts while the software continues to be used operationally. Every effort, including a risk assessment, should be made to determine if a suitable off-the-shelf product, whether FOSS, COTS, or GOTS, is available before the decision is made to embark upon a software development project. AFTAC follows the same approach as the Air Force and Department of Defense (DoD) to software, preferentially adopting existing solutions before buying commercial offerings, and only creating new software when no off-the-shelf solutions are adequate.

4.3.2 Do not reinvent the wheel. If the decision has been made to develop custom software, take advantage of the trove of existing solutions (or partial solutions) that may already be available. Evaluate existing frameworks, libraries, and technologies for applicability to the problem at hand. The AFTAC Software Strategy Council can provide guidance on technologies that already exist within AFTAC’s technology ecosystem.

5.0 CONSTRAINTS

5.1 CONTAINERIZATION

AFTAC is shifting to a containerization strategy to facilitate software deployment across multiple environments (such as Development, Staging, and Production) and multiple networks, some of which may be geographically separated (such as the case between onsite production systems and hot-backup contingency systems). Therefore, all AFTAC software should be targeted to run as

Docker containers. Additionally, AFTAC utilizes Kubernetes as the preferred container orchestration tool. Exceptions should be presented to the AFTAC Software Strategy Council.

While any container can be hardened for use on production systems, AFTAC recommends using containers from DoD Centralized Artifact Repository (DCAR)/IronBank. AFTAC can immediately use secure, hardened containers from DCAR/IronBank as base containers. At a future date, AFTAC will require a “sidecar” security stack (such as what is provided using a Kubernetes container orchestration platform + Istio service mesh + appropriate logging) and is working with high-level approvals on detailed guidance for establishing sidecars in production containers.

5.2 VIRTUAL MACHINES

Although Docker containerization is preferred, there are some cases where it is not practical. In this case, AFTAC software should be targeted to run within a virtual machine. When utilizing virtual machines, Red Hat Enterprise Linux is the preferred operating system for AFTAC systems.

AFTAC recognizes there are special-case applications utilizing high-performance computing clusters that are recommended to run on bare metal instead of through virtualization. Exceptional use cases shall be presented to the AFTAC Software Strategy Council for review.

Figure 4 – Structure of Vitual Machines and Containers

5.3 DATABASES

The primary database management system used and maintained by AFTAC is the Oracle RDBMS.

Unfortunately, databases accessed externally by non-Air Force personnel do not fall within the Air

Force’s Oracle site-license. In these cases, the licensing costs inhibit the use of Oracle as a practical solution. In order to comply with existing Oracle licensing requirements, MySQL

Enterprise Edition is the preferred database for systems accessed by non-Air Force personnel as it includes features mandated by Defense Information Systems Agency (DISA) Security Technical

Implementation Guides (STIGs), such as data-at-rest encryption, auditing, etc. In an effort to limit personnel required for database administration, AFTAC is reluctant to increase the number of distinct database technologies. Therefore, specialized databases other than Oracle (for internal systems) or MySQL (for external systems) must be approved by both the AFTAC Software

Strategy Council and the AISC.

An alternative option for contractors may be to develop using an alternative database, such as

Postgres or MariaDB. If the solution is destined for the production system, sufficient time must be allocated during deployment to migrate to Oracle for production use.

5.3.1 AFTAC is currently exploring other, non-relational database types, such as

NoSQL, time-series, graph, etc. Use cases for non-relational databases should be presented to the

AFTAC Software Strategy Council.

5.4 LANGUAGES

As with limiting database technologies to facilitate maintenance, the number of software development languages should also be constrained. AFTAC’s legacy systems have been implemented using many software development languages over the course of several decades. In an effort to manage this and improve maintainability, AFTAC is recommending software development planned for production be limited to the following languages: Java, Python, JavaScript, TypeScript, and Go/Golang. Additional languages should be presented to the AFTAC

Software Strategy Council.

5.5 CLASSIFICATION

Although AFTAC systems may span multiple classification levels, every effort should be made to keep custom-developed software for AFTAC unclassified. Any classification considerations should be segregated from source code into either configuration parameters or isolated libraries/plug-ins. Although unclassified software will facilitate future maintenance, AFTAC recognizes that some software development must occur on classified systems and infrastructure exists on higher levels for full software versioning and Continuous Integration/Continuous

Delivery (CI/CD) pipelines.

5.6 VERSIONING

All software will utilize a semantic versioning scheme. This document recommends systems development utilize a services-based architecture, resulting in systems being composed of many components that can be deployed separately. Therefore, we provide for slightly different versioning schemes for components and the overarching systems. Figure 5 illustrates how a system is composed of varied component versions and increments a patch outside of a release based on internal component releases. AFTAC/SDD is responsible for the definition of system versus component in AFTAC software. New software versions shall include a version description explaining the version increment(s) and why the specific component or system number was incremented.

5.6.1 Systems

Systems will utilize a 4 (four) number semantic versioning scheme defined as Release.Major.

Minor.Patch as shown in Figure 5 below.

Figure 5 – Example Versioning Scheme

Release versions are incremented when enough components have changed as to require complete system tests to validate the system as defined by Subject Matter Experts (SMEs).

Additionally, because many projects are tied to contractual efforts, the release number may also change upon the first release after a new contract is assigned to develop or maintain the system.

Major versions will increment when significant architectural changes are made that may have potential increased risk or provide a change in security posture. Examples of these are changes in database or authentication technologies.

Minor versions will increment with the cadence typical of Agile software development deliveries, and represent incremental modification to the system.

Patch numbers will increment whenever a new component version is promoted outside of a formal release. These are typically hotfixes requiring an urgent or expedited release.

Note: All release and major system increments will include a document explaining the reason for the version increase, which will be included in Certificate To Field (CTF) artifacts.

5.6.2 Components

Components will utilize a 3 (three) number semantic versioning scheme defined as

Major.Minor.Patch. Each component (e.g., service, artifact, GUI) will have its own version number for separate security scanning and testing allowing them to be released outside of a complete system release. Some organizations choose to include a fourth number to represent the build version, such as Major.Minor.Patch.Build.

Major numbers will increment for any backward-incompatible change, significant architectural change, or change to the security technologies of the component.

Minor numbers increment with backward-compatible changes that also do not change the security posture of the component. Minor changes should be deployable without changes to any other components.

Patch numbers are incremented for backward-compatible changes to existing functionality and are typically reserved for bug fixes or maintenance releases.

5.7 SECURITY

5.7.1 All software destined for Production must pass DoD Assured Compliance

Assessment Solution (ACAS) scanning and DISA STIG inspection, to include containers and virtual machines hosting that software. Layered container images shall be initiated using DoD-approved hardened container base images, which have STIG rules pre-applied and are available on DoD repositories. The “Application Security and Development” STIG is applicable to

“browser-based applications comprised of numerous web technologies and architectures” and therefore pertinent to services-based software developed for AFTAC. Additionally, AFTAC-developed software, GOTS software, and FOSS must pass both SonarQube quality scans and

Micro Focus Fortify security scans of source code in the Development environment before moving to the Staging environment and then eventually to Production. All “Critical” and “High” Fortify vulnerabilities, and all “Blocker” and “Critical” SonarQube findings, should be addressed. Third-party software packages, frameworks, and libraries must come from reliable, trusted sources adhering to secure coding standards as determined by AFTAC’s Information Assurance (IA) and the Software Solution Management Team (SSMT) via software composition analysis scans.

Software security does not end with the resolution of “Critical” and “High” vulnerabilities, as continued software Operations and Maintenance (O&M) should strive to improve AFTAC’s security posture. Although “Medium” vulnerabilities represent a lower threat, reducing risk and securing software should be a continuing component of any R&D or O&M effort. “Medium” findings should be mitigated as much as possible or identified with a Plan of Action and Milestones

(POAM) to be addressed in a future release.

5.7.2 In addition to SonarQube and Fortify scans occurring within the Development environment, software containers and virtual machines will be scanned via DISA’s ACAS software within the Staging environment. AFTAC has obtained an Automatic Certificate To Field

(Auto-CTF) for software that successfully passes the SonarQube quality scan, the Fortify security scan, and the ACAS scan. This Auto-CTF will enable rapid deployment of software and reduce the time from Development to Production. The DoD Enterprise DevSecOps Reference Design is an excellent reference for integrating security into rapid development.

5.7.3 Software peer reviews are an industry best practice, and tools exist to enforce peer review prior to code commits into repositories (e.g., Gitlab merge requests). AFTAC-developed software, either written by AFTAC’s internal workforce or contracted out, must be peer reviewed prior to promotion to production systems. Peer reviews should not only focus on quality, such as adherence to coding standards and best practices, but also ensure code is secure. Peer reviews provide accountability and limit the risk of harmful software written by rogue personnel. As it is not practical to manually inspect delivered software, AFTAC assumes software developed externally through a contract or third-party organization has been peer reviewed throughout its course of development (language regarding this may be added to future contracts).

5.7.4 The following is an example path to Production:

[1] Software source code is scanned via code analyzers for quality and security in the

Development environment.

[2] Software source code is compiled and packaged into hardened container(s) and deployed into the Development environment.

[3] Software is run through automated tests.

[4] Software container(s) moved into the Staging environment.

[5] Software container(s) are security scanned via DISA ACAS in Staging environment.

[6] Software container(s) are moved into the Production environment.

5.8 MESSAGING FORMATS

AFTAC’s accredited cross-domain solutions pass information between networks and permit only pre-approved data types. Although the full list of pre-approved data types is CUI and therefore only available upon request, JavaScript Object Notation (JSON) is one of these pre-approved formats and it is the preferred method for messaging between networks. In instances where messages have been designed to be very large, and as a result performance necessitates a binary transmission, NetCDF3 format is available for transmission. It is important to consider that messages and data not pre-approved must be (1) serialized/deserialized into the JSON ASCII format, or (2) in the NetCDF3 binary format in order to enable transmission through cross-domain solutions. All data formats mapped to the common messaging formats must also go through a data validation check prior to conversion and movement through AFTAC guards. The AFTAC

Software Strategy Council will address any questions regarding messaging and data types approved for cross-domain transmission.

6.0 QUALITY

6.1 ATTRIBUTES

The following attributes should be taken into consideration when developing new software:

Performance (e.g., latency and throughput)

Scalability (e.g., data and traffic volumes)

Availability (e.g., uptime, downtime, scheduled maintenance, 24x7, 99.9%, etc.)

Security (e.g., authentication, authorization, data confidentiality, etc.)

Extensibility

Flexibility

Auditing

Monitoring and management

Reliability

Failover/disaster recovery (e.g., manual vs automatic, how long will this take?)

Interoperability

Accessibility

Usability

Adaptability (e.g. microservices may be used by multiple systems)

Abstractability (e.g. not vendor locked)

6.2 MONITORING

Software should be instrumented to allow for the monitoring of quality attributes. This monitoring allows AFTAC to ensure software is being used to the greatest impact of the mission. The following attributes are some of those that could potentially be monitored: Performance, Scalability, Availability, and Failover.

6.3 CODE QUALITY SCANS AND REPORTING

AFTAC utilizes the SonarQube source code analyzer and Fortify Static Code Analyzer to inspect developed software. The intent of Fortify and SonarQube scanning is to ensure quality software is promoted to production systems to enhance software robustness and maintainability. SonarQube provides a freely available Community Edition supporting all the recommended software development languages provided by this document. AFTAC also utilizes the Synopsys Black

Duck tool for validating open source tools.

6.3.1 At the very least, all “BLOCKER” and “CRITICAL” severity level findings by

SonarQube and Fortify should be addressed. Software containing numerous “MAJOR” severity level findings may be requested for rework by the AFTAC Software Strategy Council prior to promotion to production systems.

6.3.2 SonarQube will also be used to analyze source code unit test coverage, with 70% or higher as the goal based on industry studies. Each piece of business logic in the software should be covered by automated unit tests to reduce maintenance costs. Software containing insufficient test coverage will be requested for rework by the AFTAC Software Strategy Council prior to promotion to production systems.

7.0 INFRASTRUCTURE

7.1 DEVELOPMENT/STAGING/PRODUCTION

AFTAC provides separate Development, Staging, and Production environments at all 3 (three) classification levels. The Development environment is flexible enough to allow for rapid prototyping, analysis, and R&D. The Staging environment is an exact copy of Production where testing and security scans will be executed prior to promoting to operations. The Production system is the operational mission system. Figure 6 illustrates these environments.

Figure 6 – Development/Staging/Production Environments

7.2 DEVELOPMENT ENVIRONMENT

7.2.1 A CI/CD pipeline is available for personnel to leverage within AFTAC’s

Development environment. The Development environment is ever evolving and growing, and below is a current sampling of available technologies:

Integrated Development Environments (IDEs) and code editors, such as IntelliJ, Eclipse, NetBeans, Atom, PyCharm, and others

Compilers and debuggers via Redhat Developer Toolset (gcc, gdb, valgrind, etc.)

Source repository via Gitlab CE

CI/CD coordination via Gitlab Runner

Code quality scanning via SonarQube

Code security scanning via Fortify

Artifact repository via Artifactory Pro

Containerization via Docker and Kubernetes/Rancher

Common libraries and software packages applicable to AFTAC’s problem sets

A CI/CD pipeline is a series of steps that reduce the time to deploy software by forcing security considerations and automation into the process. The AFTAC pipeline follows the procedure portrayed in Figure 7.

Figure 7 – Continuous Integration Pipeline

7.2.2 Automated software unit and integration tests should be implemented in the CI/CD pipeline using built-in modules and third-party tools. The list of tools continues to evolve with the software. AFTAC currently recommends the following automated testing tools:

Junit – Java unit test

“unittest“ module – Python unit test

Jest/Enzyme – Javascript/Typescript unit test

“testing” module – Golang unit test

Wiremock/Mockito – Java service unit test

Testcontainers.java – containerized Java integration/db test support

Gherkin/Cucumber – integration test

Cypress – web UI integration test

Additional automated testing tools may be added to the centrally managed list via AFTAC

Software Strategy Council.

7.3 STAGING ENVIRONMENT

The Staging environment closely mirrors the Production environment. This environment provides a place for soak testing, end-to-end testing, validation testing, and failure testing by software developers and government SMEs. The Staging environment also provides a place for focused security testing by the Infrastructure as a Service (IaaS) group, who executes DISA’s ACAS software scans. AFTAC has obtained an Auto-CTF for software that successfully passes both the

SonarQube quality scan and the Fortify security scan with the Development environment, and then the ACAS scan within the Staging environment.

7.4 PRODUCTION ENVIRONMENT

Not all software developed by/for AFTAC will be planned for installation within the Production environment. AFTAC’s Production environment is a robust, mission-critical environment from which AFTAC’s 24/7 missions are executed. Additionally, the Production environment is duplicated at AFTAC’s geographically-separated hot-backup location for contingency plans. The following tech stack composes AFTAC’s preferred solution for production software:

OS – RedHat Enterprise Linux

Container – Docker CE

Orchestration – Kubernetes

Service Mesh – Istio

Authentication/SSO – Keycloak

Logging – EFK stack (Elasticsearch, Fluentd, Kibana)

Metrics/Alerting – Prometheus+Grafana, Kafka

Database – Oracle

Web Map Server – Geoserver

Refer to AFTAC Enterprise Infrastructure Baseline documentation for details about environment options.

7.5 VIRTUAL MACHINES

Although Docker containers are AFTAC’s preferred method for delivery and deployment of software, AFTAC recognizes there are cases for which containers are not optimal and, in these cases, software may be targeted for deployment on a RedHat Enterprise Linux virtual machine.

The underlying hypervisor infrastructure executing virtual machines is managed by 709 CYS and thereby falls outside the purview of this document.

7.6 DATABASES

If possible, a microservice database is preferable. This allows the best database to be utilized for each problem set, but data must be shared across services through APIs. Otherwise, Oracle is the preferred database with PostgreSQL as an option for smaller projects and MySQL as a fallback when access is required by non-Air Force personnel. To facilitate database maintenance, effort should be made to reduce or limit the database footprint by installing application metadata within existing databases rather than introducing a new database infrastructure.

8.0 DATA

8.1 AFTAC DATA GOVERNANCE COUNCIL

The AFTAC DGC tracks and catalogs data flowing into and out of AFTAC. Any new data acquisition or distribution systems should be coordinated with the council. Questions presented by the council may include:

Why do we need the data?

Where is the data coming from?

Who owns the data? Who is the data steward?

How is the data acquired?

What is the format of the data?

What are the archiving and back-up strategies?

Are there any regulatory requirements for the long-term archive of the data?

Is the data distributed outside of AFTAC?

Is the data required for production/operations?

Is data replication to AFTAC’s geographically-separated hot-backup contingency site required?

8.2 STORAGE REQUIREMENTS

Storage space should be considered for any new data acquisition. Potential storage requirements should be shared with 709 CYS early in the development lifecycle. Below are several questions that should be addressed regarding data storage:

What is the format for storage (databases, flat files, etc.)?

How much storage space is needed for data? (especially if you are dealing with “big data”).

How much storage space is needed for log files and auditing?

What is the projected 3-to-5-year growth horizon for storage needs?

What is the required data backup frequency?

Data at rest should follow the principles of the Zero Trust security framework, which requires all users to be authenticated, authorized, and continuously validated before being granted or keeping access to applications and data.

8.3 DATA IN MOTION

The AFTAC enterprise solution for managing data in motion is Apache NiFi. Any solutions requiring data movement, especially across classification domains, must be discussed at the

Technical Interchange Meetings.

9.0 DEPLOYMENT

9.1 DEPLOYMENT STRATEGY

Similar to data storage requirements, deployment requirements should be shared with AFTAC early in the development lifecycle. Consultation with AFTAC’s SSMT is highly encouraged in the development of deployment plans, and the AFTAC Software Strategy Council can be the intermediary to facilitate coordination with the appropriate personnel. The deployment strategy should be documented to answer the following questions:

Topology: How and where is the software installed and configured? How many containers? Virtual Machines?

What tools are recommended/required to execute the deployment procedure?

What are the requirements for memory, CPU, and networking configuration? Do certain containers/processes have unusually large resource demands?

What existing system(s) must the new software be integrated with?

What classification levels are required? Is data planned to move across networks?

9.2 CONTINGENCY PLANS

Deployment to the AFTAC production network must also take into account contingency planning in the event of failures and/or failover to AFTAC’s geographically-separated hot-backup location.

This poses additional questions to address:

Has a rollback strategy been defined?

What happens in the event of a software or infrastructure failure?

Is it clear how data is replicated across sites (e.g., data and analysis results replication to hot-backup contingency location)?

9.3 OPERATIONS AND SUPPORT

Software planned for the AFTAC production network must consider concerns for operations and support. Developers should be mindful of how the software will be used during 24/7 mission execution. For example, robustness should be built in to handle error conditions that may occur outside of normal business hours. Below are additional questions that need to be addressed regarding operating and supporting production software:

Who will monitor and manage software? Is additional effort required to allow AFTAC’s

Maintenance Operations Control Center to monitor status?

Where are logs and auditing stored? Are they accessible to 24/7 personnel for troubleshooting?

How are configuration changes deployed? Are outages required for configuration changes?

Has appropriate training been developed and provided? Troubleshooting instructions?

10.0 REFERENCES

AFI 17-130 (section 3.3.7) https://static.e-publishing.af.mil/production/1/saf_cn/publication/afi17-130/afi17-130.pdf

AFMAN 17-1203

https://static.e-publishing.af.mil/production/1/saf_cn/publication/afman17-1203/afman17-

1203.pdf

AFTAC Enterprise Infrastructure Baseline documentation

S:\709SPTG\709 CYS\Architecture Board\Hardware Baseline\Final

DevSecOps Initiative Documentation https://software.af.mil/dsop/documents

DISA Security Technical Implementation Guide (STIG) https://iase.disa.mil

DoD Enterprise DevSecOps Reference Design https://dodcio.defense.gov/Library/

IronBank - DoD Centralized Artifact Repository (DCAR) https://ironbank.dso.mil/

Software Factories https://software.af.mil/software-factories https://static.e-publishing.af.mil/production/1/saf_cn/publication/afi17-130/afi17-130.pdf https://static.e-publishing.af.mil/production/1/saf_cn/publication/afman17-1203/afman17-1203.pdf https://static.e-publishing.af.mil/production/1/saf_cn/publication/afman17-1203/afman17-1203.pdf https://software.af.mil/dsop/documents https://iase.disa.mil/ https://dodcio.defense.gov/Library/ https://ironbank.dso.mil/ https://software.af.mil/software-factories

11.0 ACRONYMS

ACAS ............................................................................. Assured Compliance Assessment Solution

AFTAC ............................................................................ Air Force Technical Applications Center

AISC ................................................................................. AFTAC Infrastructure Strategy Council

API ........................................................................................... Application Programming Interface

Auto-CTF ......................................................................................... Automatic Certificate To Field

CDO ................................................................................................................... Center Data Officer

CI/CD ......................................................................... Continuous Integration/Continuous Delivery

COTS ......................................................................................................Commercial Off The Shelf

CSA ........................................................................................................... Chief Software Architect

CTF .................................................................................................................... Certificate To Field

CTO.......................................................................................................... Chief Technology Officer

DCAR .................................................................................... DoD Centralized Artifact Repository

DGC ......................................................................................................... Data Governance Council

DISA .....................................................................................Defense Information Systems Agency

DoD .............................................................................................................. Department of Defense

FOSS .............................................................................................. Free and Open Source Software

GOTS ..................................................................................................... Government Off The Shelf

IA .................................................................................................................. Information Assurance

IaaS .......................................................................................................... Infrastructure as a Service

IDE ........................................................................................ Integrated Development Environment

IT ................................................................................................................ Information Technology

JSON ...................................................................................................... JavaScript Object Notation

O&M ....................................................................................................Operations and Maintenance

POAM ............................................................................................... Plan of Action and Milestones

R&D .......................................................................................................Research and Development

SD ............................................................................................... Systems Development Directorate

SME ............................................................................................................... Subject Matter Expert

SOA....................................................................................................Service Oriented Architecture

SSMT ....................................................................................Software Solution Management Team

STIG ............................................................................... Security Technical Implementation Guide

709 CYS ............................................................................................... 709th Cyberspace Squadron

This page was intentionally left blank.