IPAWS-OPEN GovCloud Recompete - BPA SOO.pdf

Statement of Objectives (SOO) Integrated Public Alert and Warning System (IPAWS)

IPAWS-OPEN GovCloud Recompete Blanket Purchase Agreement

IPAWS BACKGROUND:

The Federal Emergency Management Agency (FEMA) Integrated Public Alert and Warning System (IPAWS) program was established under Executive Order (E.O) 13407 in 2006, incorporating insights and capabilities from decades of national alerting experience with a mandate to make alerts more available and useful for local emergencies. IPAWS provides the capability to notify people in America and visitors of impending natural and man-made disasters, and ensure that under all conditions, the President of the United States can alert and warn people in America and transient populations. Through two systems, IPAWS Open Platform for Emergency Networks (IPAWS-OPEN), and the National Public Warning System (NPWS), the program delivers timely, geographically targeted messages during a crisis to both save lives and protect property.

The program integrates services to support Federal, State, Tribal, Territorial, and Local (FSTTL) Alerting Authorities (AAs) that use the systems to send alerts to their communities through a range of dissemination channels, increasing the likelihood a given message will reach people in an affected area through the national alert and warning infrastructure. IPAWS has and will continue to expand the delivery of alert and warning information over additional channels and platforms to more people before, during, and after an emergency. In this way, IPAWS will increase resilience to local systems and provide additional means by which officials and emergency managers can distribute life-saving information to all people during a crisis.

IPAWS-OPEN is FEMA's national system for local alerting that provides authenticated emergency and life-saving information to the public through mobile phones using Wireless Emergency Alerts (WEA), to radio and television via the Emergency Alert System (EAS), and on the National Oceanic and Atmospheric Administration's (NOAA) Weather Radio. The National Public Alert and Warning System (NPWS) ensures that the President can broadcast a national warning message in the most severe conditions.

In accordance with Department of Homeland Security (DHS) policy for major systems, IPAWS-OPEN has a number of performance measures, identified as Key Performance Parameters (KPPs), which establish baseline thresholds and objectives for various performance areas. As directed by DHS Under Secretary of Management and via Congressional oversight, IPAWS seeks to meet its 99.9% availability KPP threshold given the criticality of the IPAWS life-saving emergency alerts to the public and first responders.

PURPOSE:

In 2021, FEMA migrated the IPAWS-OPEN application to Amazon Web Services (AWS) GovCloud Cloud Service Provider (CSP); this migration effort included modernizing the application to be cloud-optimized. The IPAWS Program Management Office (PMO) intends to continue improving and maintaining the IPAWS-OPEN system. To meet this requirement, FEMA is issuing a solicitation to award a Blanket Purchase Agreement (BPA) to a single cloud services integrator to support the continued modernization, development, and sustainability of the IPAWS- OPEN application and cloud infrastructure. The BPA is designed to consolidate systems engineering, architecture and infrastructure, and cloud management services under a single contractual effort. By consolidating cloud services and engineering, it is anticipated to generate cost efficiencies; promote a more tightly integrated management approach across system development, infrastructure, and maintenance operations; and allow IPAWS PMO management to focus on the mission and strategy, rather than technical management.

Statement of Objectives

SCOPE:

FEMA intends to award a BPA to provide IPAWS-OPEN application and cloud support; specifically in the areas of systems engineering and development, cloud infrastructure management (Infrastructure as a Service [IaaS]), Managed Services, and Operations and Maintenance (O&M) Sustainment services. The Period of Performance for this BPA will be twelve (12) months from date of award with four (4) one (1) year options. The Period of Performance is anticipated to start no later than TBD and continue sixty (60) months if all four (4) option years are exercised. The primary place of performance will be the Contractor’s facilities, with visits, as needed, to FEMA facilities in the Washington Metro Area.

IPAWS-OPEN is a mission critical system that cannot afford even minor interruptions of service. The application has a 99.99% availability KPP objective given the criticality of the IPAWS life-saving emergency alerts to the public and first responders. To achieve and maintain the 99.9% availability threshold, the Government requires the services of an experienced systems engineering and cloud operations contractor. To manage the risk to the IPAWS Program and public, the Government requires the Contractor possess and maintain good standing within the AWS Partner Network (APN) program as an AWS Advanced Tier Consulting Partner (Advanced Tier at a minimum).

Though the primary scope of this acquisition focuses on systems engineering, IaaS, Managed Services, and O&M sustainment, it is envisioned that the Contractor will be called upon to assist the Government as it implements network changes that impact IPAWS-OPEN. IPAWS-OPEN connects to the DHS Trusted Internet Connection (TIC).

While network design and implementations outside of AWS GovCloud are not a direct responsibility of the Contractor, the IPAWS Program anticipates that during the contract period of performance there may be the need for network changes driven by DHS or FEMA agency directives, new industry standard, or other factors. These network changes will require the Contractor to support the IPAWS Program as it collaborates with DHS, FEMA, or other parties for the purposes of planning, impact analysis to IPAWS-OPEN, and sharing technical information regarding the IPAWS-OPEN architecture, and configuration changes necessary to interface with the federal network components. The Contractor may also be called upon to add new services and support tests following network changes to verify IPAWS alerts are traversing the TIC and other networks properly. While the implementation of network changes is outside the scope of this contract, the Contractor, however, will be responsible for AWS end-point connection changes/implementations - the Contactor is responsible for design and implementation of CSP hosted environment network changes. The Government recognizes the CSP will implement the CSP end-point change, but within the scope of System Engineering, IaaS/Managed Services, the Contractor shall design and coordinate the changes with the CSP.

SYSTEMS ENGINEERING: The migration to GovCloud included incorporating system and code changes to enable the application to operate in the cloud, but the scope of that effort did not include completely rewriting all of the legacy application code. FEMA is in the process of refactoring the remaining legacy application code to address technical debt, as well as incorporating enhancements for the next major release. It is anticipated that planning and design for the next major release will be underway at the time of contract award. Development is expected to take place after contract award. Over the contract period of performance, it is anticipated the IPAWS PMO will develop and deploy multiple minor and major releases. As previously mentioned, Systems Engineering responsibilities may include supporting the Government with network planning and testing (from an IPAWS-OPEN perspective). CSP hosted end-point network changes are the responsibility of the Contractor (in collaboration with the CSP).

The government requires Agile development for all DHS and FEMA systems. IPAWS is on an evolutionary path to Agile development. The IPAWS development methodology currently consists of Waterfall/Agile hybrid development approach, but is in the process of maturing its Agile development approach to comply with DHS/FEMA Agile requirements. The Contractor shall support the IPAWS PMO with compliance with Agile development requirements, to include DHS Agile reporting requirements. Contractor support for maturing the IPAWS-OPEN Agile development methodology (to include meeting DHS reporting requirements), is within scope of this contract.

IaaS: The Contractor shall continually optimize the IPAWS-OPEN cloud infrastructure with respect to performance and cost efficiency by recommending and managing the appropriate IaaS services and levels based on the IPAWS requirements and objectives. IPAWS-OPEN system in AWS contains multiple cloud environments, consisting of Production, Disaster Recovery (DR), Development, Test, Shared Services, Staging, Security, and IPAWS Demo Lab.

MANAGED SERVICES: FEMA requires Managed Services support for effective and optimized operations of the IPAWS architecture stack, cloud environment, and application. The shall Contractor to provide day-to-day management of the IPAWS-OPEN infrastructure environments and cloud services. The IPAWS PMO currently has a Managed Services approach in place modeled after IT Service Management (such as Information Technology Infrastructure Library [ITIL]) guidance to align IT services with business needs and ability to scale services to those needs.

O&M SUSTAINMENT: The Contractor shall maintain the system architecture and operations within the IPAWS- OPEN system boundary, as well as continue to identify opportunities to improve system operations and maintenance procedures.

IPAWS-OPEN OPERATIONS AND ARCHITECTURE:

The fundamental purpose of IPAWS is to securely host web services that enable routing of standards-compliant emergency alerts and warning messages between disparate third-party applications, systems, networks and devices. Federal, state, local, tribal, and territorial alerting authorities can use third-party alerting tools that comply with Common Alerting Protocol (CAP) standards to sends alerts to IPAWS for dissemination. By making unique alerting services CAP-compliant and integrating them with IPAWS, alerting authorities will be able to send a single alert from or to their unique system through IPAWS that will reach radio, television, wireless phones, unique state and local alert systems, internet-based services, and all future CAP-compliant IPAWS connected technologies.

IPAWS-OPEN serves as the IPAWS Alerts Aggregator. The primary focus is dissemination of Alerts and Warnings;

IPAWS collects the alerts from the Alert Originating Systems and routes it to the alert dissemination emergency systems that serve the public. Additional IPAWS-OPEN system overview includes the following:

• IPAWS-OPEN receives and authenticates messages sent by Alerting Authorities. IPAWS-OPEN then routes the messages to private and public sector communications gateways that disseminate the emergency messages to the public.

• Alerting Authorities use private-sector software or hardware alert origination tools that are compatible with the IPAWS-OPEN interface.

• Participating private sector communication providers disseminate emergency messages to the public through radio, television, mobile phone providers, other systems such as electronic sign boards and sirens and internet base applications, and the National Oceanic and Atmospheric Administration Weather Radio network.

• Private or public sector developers can obtain access to the IPAWS test and development environment when they are designing IPAWS-OPEN-compatible systems. They may also access the IPAWS All-Hazards Information Feed of live alerts for redistribution and/or situational awareness.

The FEMA-managed IPAWS OPEN system is designed to have IPAWS AAs create CAP messages and send them through various alert dissemination pathways. The Alert Gateway Aggregator acts as a CAP message broker and dissemination gateway. It authenticates the alert message based on Memorandum of Agreement (MOA) signed with FEMA, and disseminates messages simultaneously to multiple public alerting systems. The IPAWS process flow for an incoming alert message from an IPAWS AA: A standard alert flow from an alert originator reaching the public is depicted in the following simplified diagram and further described below:

1) Authenticate the message is from the AA by verifying the digital signature.

2) Ensure the message is valid to the CAP standard and IPAWS specification.

3) Verify if the type of alert is in accordance with the pre-established agreement between FEMA and the AA.

a. Validate the alert target geo-location

b. Validate the alert type and dissemination method

4) Confirm non-repudiation from step 3 via digital signature.

5) IPAWS-OPEN does not actually disseminate the alerts to the public, but routes them to the appropriate interfaces for dissemination. Authenticated alerts that are meant for public dissemination are routed to the following outbound message dissemination interfaces.

a. EAS Feed: AM, FM and satellite radio, as well as broadcast, cable and satellite TV

b. Non-Weather Emergency Message (NWEM) Feed: NOAA Weather Radio

c. Wireless Emergency Alert (WEA) gateway: Mobile Carriers

i. This is the system interface to the WEA service that wireless carriers rolled out across the nation in 2012. WEA allows public safety authorities to use IPAWS-OPEN to send geographically targeted, text-like WEAs to the public

d. Internet-based service providers can connect to the IPAWS All-Hazards Information Feed to consume alerts and warnings and disseminate via their communications pathways

The following architecture diagram depicts the normal flow from alert originator to dissemination (as described above). It shows the standards based alert message protocols, authenticated alert message senders, shared, trusted access and distribution networks, alerts delivered to more public interface devices.

IPAWS-OPEN Production Cloud Environment (CE) is designed using AWS along with Apache Tomcat to accomplish message processing and support interoperability between different systems. Key design patterns include:

• AWS Simple Message Queue (SQS) – A reliable, highly-scalable buffer for a message que, microservices, batch tasks, and storing notifications

• Tomcat Server – Used by the CAP Aggregator Service along with other IPAWS components. Handles web application hosting, access, and general logging

• Spring Boot is an open source Java-based framework that is used for message process execution; it takes requests from the CAP service and processes them (prior to the cloud migration, IPAWS-OPEN operated on an Oracle SOA platform hosted in DHS and FEMA data centers, and used Apache ODE to manage Business Processes Execution Language (BPEL); IPAWS-OPEN operating in the cloud uses Spring Boot as a replacement to Apache ODE)

• CloudTrail and CloudWatch – Provides activity logs and CloudWatch monitors resources, sends alerts and performs some automated tasks to respond to events

• Route 53 – Domain Name System (DNS) service with a reliable infrastructure to ensure fast, secure routing of web service and traffic to the application load balancer where health checks are conducted. Utilized for internal FEMA Enterprise Network (FEN) routing only

• Digital Certificate – Validated when a user sends a message

• Amazon Relational Database Services (RDS) – Hosts the PostgreSQL database; configured with high availability

• Amazon EC2 – Using t3.xlarge instances in the auto scaling group with an additional m5.xlarge instance to handle spillover traffic

• AWS S3 Storage – Used for archival of IPAWS-OPEN legacy data (IPAWS-OPEN pre-cloud migration data), and AWS EC2 data storage

• DHS TIC – Edge DNS handles secure external DNS resolution to TIC endpoints

The following architecture diagram and descriptions provide additional information regarding the current IPAWS- OPEN technology stack. Included in the diagram Data Tier section is the Commercial off-the-shelf (COTS) Retool application management tool, which is an user interface connected to the IPAWS database for access to all message data and COG profiles in order to conduct troubleshooting and information gathering. Retool has a connection to the IPAWS database to run queries and pull information into a user friendly, readable format. It provides the capability for IPAWS-OPEN cloud environment IPAWS PMO administrators to manage the dynamic profiles associated with Aggregator Service dissemination channels.

1. Alert Originators and EAS devices access https://integration.aws.fema.gov and https://apps.fema.gov respectively via DHS HSEN proxy servers

2) Incoming messages are routed over AWS Direct Connect hosted by AT&T to the IPAWS Transit Gateway (hosted in IPAWS Transit account) and then route traffic to IPAWS Production environment in GovCloud US-West

3) Traffic from the Transit Gateway is directed to an Application Load Balancer which will distribute traffic to IPAWS servers for message processing via round robin distribution

4) The first of three tiers, this is known as the Web Tier

5) IPAWS Cap Aggregator servers run on an underlying Red Hat Enterprise Linux (RHEL) server deployed with an Apache Tomcat servlet container

6) Autoscaling is configured to increase and decrease resources dependent on server utilization.

7) Messages are placed in AWS SQS (Message Queuing Service) – if it’s a Presidential alert it is placed in the Presidential Queue, all other alerts are placed in the non-Presidential Queue

8) The second of three Tiers; this is known as the Application Tier

9) The IPAWS Database tables are hosted in AWS RDS PostgreSQL. RDS has a primary database in one availability zone, a standby database in a separate availability zone and a read replica in a third availability zone. PostgreSQL tables will be managed by Retool for Administrative and Customer Support modifications

10) The third of the three tiers; this is known as the Database Tier

11) The CMAS Interface receives Commercial Mobile Alerting System C-Interface Format (CMAC) https://integration.aws.fema.gov/ https://apps.fema.gov/ messages submitted by the Aggregator, these are converted from original CAP messages in the CAP aggregator, then sent to the Gateways

12) Hosted in the Application Tier, the Commercial Mobile Service Provider (CMSP) Gateways will interface with CMAC systems based on the context of the web-service call

13) Validated EAS CAP Alert Messages are posted to the EAS/PUBLIC FEED Platform for retrieval by broadcasters. Alert Networks retrieve multiple EAS and IPAWS Public Alerts in a single retrieval based on date/time. Alert Networks retrieve all available un-expired alerts and select Alerts of interest based on ATOM XML document which contains a catalog of those messages

14) In the event the entire GovCloud-West infrastructure is unavailable, including multiple Availability Zones located in independent data centers, incoming traffic will be routed manually via DHS TIC to GovCloud-East where the same standby architecture would automatically scale up to manage incoming traffic demands

REQUIREMENTS:

As FEMA matures the IPAWS cloud solution, it requires Contractor services to extend results-driven improvements in efficiency, agility, and innovation with little or no negative impact to users. To achieve this, the Contractor shall provide services to meet the following business/administration, technical, and security requirements.

Business and Administration Requirements

• Provide quality end-to-end project management services by adhering to industry recognized best practices and Project Management principals

Provide an experienced project manager to serve as the lead for all contractor activities and performance (cost, schedule, and quality) outlined in this BPA document and subsequent Call Orders, and serve as the interface between the government and contractor and sub-contractor personnel

Contractor Project Manager shall possess and maintain a Project Management Institute (PMI) Project Management Professional (PMP) certification for the duration of the contract

Provide Technical Account Management (or Project Manager) in support of all technical and service delivery with a focus on CSP liaison management and all areas of professional cloud services/engineering and support; POC to ensure Contractor performance is meeting expectations

Employ rigor in the tracking of performance requirements summary, deliverables, and risks for full life-cycle project management from planning to implementation

All development initiatives shall include a project plan, with milestones, sequencing, deliverables, resource loaded, and other schedule management best practices (all documented in Microsoft Project): figure out where to put this bullet

Identify and recommend new and emerging cloud services, standards, and best practices for incorporation into IPAWS services with minimal delay throughout the BPA period of performance

Actively participate in capturing lessons learned; apply lessons learned for areas within Contractor’s scope

Provide consolidated and timely monthly invoice statements that are presented in a logical order (i.e., broken down by CLIN, by role/labor category), are easy to understand, and support “drilling down” for additional details

Provide archive and records retention management; maintain a central repository accessible by FEMA for artifacts, such as metrics, application code, system requirements, documentation, testing data, etc; support data retrieval in support of data calls, program initiatives, and oversight requirements (NARA); conduct data analysis upon request

Collaborate with existing incumbent contractor and Government during the transition period following contract award; report progress and status of transition activities to include hand-off milestone for the transfer of AWS GovCloud account, third-party tools, establishment of Service Desk, and all IPAWS-OPEN contractor support activities/responsibilities

Provide controls to submit timely, accurate deliverables; maintain appropriate staffing; effectively manage staff; execute task area work; and ensure quality to ensure the efficacy of deliverables achieve cost, schedule, and performance objectives

• Ensure staffing, recruiting, and retaining staff possessing senior level knowledge and capabilities to address highly complex challenges in IT engineering, architecture, and cloud related services

Ensure staff possess the knowledge, skills, education, and experience to successfully meet the requirements

Employ a rigorous screening and selection criteria enabling superior talent for FEMA and associated contract teams;

Hire personnel that meet the labor category requirements, but also hire thought leaders with the requisite skills to be effective; be diligent in their pursuit of delivering employees that perform at the highest levels and providing avenues for effective collaboration and partnership

If applicable, document (within a project management plan) Subcontractor relationship and quality management; Contractor shall outline the roles and responsibilities per party involved in the service and where key responsibilities reside

Provide experienced personnel to properly manage IT lifecycle, risk, and Service Desk o IT Service Management (ITSM) framework approach required o IT Infrastructure Library (ITIL) 4 Foundation (at a minimum) certification required for personnel leading ITSM management responsibilities, to include the Service Desk

• Provide Managed Services that adheres to the principles of ITSM/ITIL for the optimization of operations management and IPAWS-OPEN performance and operations management to achieve and maintain cost and performance efficiencies, architecture and application operational stability, and cybersecurity compliance:

Maintain optimized provisioning, processing, storage, networks, and other fundamental computing resources, including the IPAWS application operating system

Optimize IPAWS performance, efficiency, operations, maintenance, and management

Provide meaningful reporting and analytics that provide FEMA with up-to-date and comprehensive information regarding technical and management performance of the services and solutions provided

Establish 24/7 Service Desk (existing Service Desk is managed by incumbent contractor so new Service Desk will need to be established following contract award)

Provided Service Level Agreement (SLA) management and compliance o Monitor and report AWS SLA performance to ensure compliance and receipt of service credits in if AWS infrastructure outage exceeds GovCloud regional service level objective o Comply with (or propose improvements) to existing IPAWS-OPEN Incident Management SLA requirements:

- Manage Contractor 24x7x365 support desk services and Contractor-supported Tiers to respond and resolve break fix issues related to the IPAWS-OPEN application and all components within AWS GovCloud

- Follow and comply with Incident Management Process Flow to meet SLA incident response times and reporting requirements based on incident severity o Maintain role/responsibilities and escalation contact list for incident management o Contractor shall maintain account with AWS Enterprise Support for the duration of the contract to support SLA compliance regarding incident management o Propose remediations if SLA requirements are not being met

Provide dynamic pricing models that allow for future cost savings derived from the cloud’s evolving economies of scale and take advantage of IaaS price reductions as they occur with minimal delay;

the savings shall be passed on to FEMA as price reductions

Provide consumption based commercial cloud services that enable on-demand provisioning of compute and storage resources and are accurately billed in increments no larger than one hour

Provide licenses and certificates, as needed, for the entire stack as needed (all layers of architecture and IaaS) and environment; Contractor may need to acquire tools via Bring Your Own License (such as CloudCheckr), conduct license research, and support tool implementation

Technical Requirements

• The Contractor shall possess AWS corporate-level certification, experience, expertise, and qualifications necessary to proficiently provide technical services addressed in this SOO

Contractor company must possess (at a minimum) Advanced Tier as an AWS Partner Network (APN) Consulting Partner

Provide Subject Matter Expertise (SME) for AWS GovCloud architecture engineering and as-needed maintenance, to include systems engineering services for consultation, upgrades, continued support, and incident response (that requires systems engineering expertise)

Contractor must demonstrate ability to manage AWS GovCloud IaaS operations on Day-One of contract period of performance (upon completion of transition period); this is a non-negotiable requirement – the Contractor must be able to restore IaaS services expeditiously during an outage or degradation of cloud performance

• The Contractor shall apply systems management disciplines and controls for the full systems development lifecycle and operations

Contractor shall validate assumptions well in advance of design and development to reduce risk of faulty assumptions (provide time for contingency planning)

Support the IPAWS PMO in preparations for all gate, oversight, and governance reviews; prepare gate review briefings, artifacts, and acceptance letters for SELC control gates

The Contractor shall follow DHS and FEMA SELC guidelines, to include developing and delivering SELC documentation prior to the corresponding SELC control gates; this requirement is expected to evolve as the Contractor implements steps to mature the IPAWS-OPEN Continuous Integration and Continuous Delivery (CI/CD) and Agile system development methodology o Conduct planning and create/maintain technical architecture and design documents for IPAWS capabilities using industry best practices o Follow SELC process to capture and document all Engineering development actions o Align development tasks and deliverables with corresponding control gates o Contractor shall provide support to FEMA for maintenance/renewals for program documents, such as:

- IPAWS-OPEN Disaster Recovery Plan

- IPAWS-OPEN Contingency Plan

- IPAWS-OPEN Privacy Impact Assessment

- IPAWS-OPEN Privacy Threshold Assessment

- IPAWS-OPEN System of Record Notice

- IPAWS-OPEN User Acceptance Test Plan

- IPAWS-OPEN User Acceptance Test Report

• The Contractor shall support and document release planning; collaborate with Government to identify requirements and conduct release planning

Conduct release planning analysis, to include impact analysis (IA) and level of effort (LOE) analysis;

assess risks, challenges, feasibility, and dependencies during release planning process

Document deferred requirements from prior release(s) and new capabilities/enhancements

Provide recommendations for sequencing, prioritizing, and types of requirements (such as emergency release, minor release, major release)

Based on Product Manager’s/System Owner’s prioritization of requirements, Contractor shall conduct sprint planning

• Proactively follow Requirements Management best practices, such as:

Documenting system/technical requirements that are traceable, clear, comprehensive, and testable for each release

Developing Requirements Traceability Matrix (RTM) for each release that documents and traces inception/identification of requirements through the incorporation into a release; RTM shall document:

o Business/functional requirements translated to technical requirements, with associated test cases and results, as well as which release is associated with each requirement o Deferred requirements

Develop and document quality requirements, by at a minimum considering:

o Need and priority (ensure this a priority and needed for IPAWS-OPEN to operate) o Verifiable (ensure that the requirement is met in the system) o Unambiguous (ensure documented requirement is not ambiguous; ambiguous or poorly worded requirements can lead to serious misunderstandings) o Complete (ensure all conditions under which the requirement applies stated) o Consistent (ensure the requirement doesn’t conflicting with any other requirement) o Traceable (ensure the origin/source of the requirement known, and is there a clear path from the requirement back to its origin) o Concise (ensure requirement is stated simply and clearly) o Standard constructs (ensure requirements are stated as imperative needs and the word “shall” must be used)

• The Contractor shall comply with FEMA Configuration Management guidelines, as well as follow Configuration and Change Management best practices (such as ITIL processes)

Ensure software within CSP boundary remains compliant with FEMA software configuration and version management policies

Contractor shall support an internal IPAWS Change Control Board (CCB) process and comply with FEMA and IPAWS PMO CCB procedures; follow the change management process to ensure all changes are vetted and tested within the test environment prior to production deployment

Provide an impact analysis and level of effort analysis for each change request (CR) and defect report. The IA and LOE analysis will help the Government in its evaluation and decision if the change is economically and technologically justifiable

Contractor shall select from the FEMA Technical Reference Model (TRM) before proposing a non-approved tool; if no viable tool is available on the TRM, evaluate and recommend appropriate tools

For all release types (e.g. major, minor, emergency, patch), the Contractor shall maintain version and change control, to include properly applying the IPAWS naming convention, storing the release and baseline in version-controlled repositories, and providing the Government with applicable release documentation

• Comply with FEMA Incident Management guidelines, as well as follow Incident Management best practices (such as ITIL processes) to restore IPAWS-OPEN system outages and degradation to fully functional operations.

The Contractor shall provide 24/7 Service Desk Support and Incident Management services based on ITIL best practices tailored to IPAWS-OPEN needs. The Contractor shall follow an Incident Management lifecycle and workflow approach for identifying, categorizing/prioritizing, diagnosing, assigning (escalating through tiers as needed), resolving, and closing incidents to restore operations as quickly as possible while limiting impacts on the business. IPAWS PMO has an established Incident Management process flow but encourages continuous evaluation and identification of lessons learned for making improvements. The Contractor may be required to work with other teams and vendors to resolve issues. The Contractor shall provide ITIL-like service delivery framework to include technical support or and service desk for incidents, requests, and problem management in event of service downtime and degradation:

o Provide 24x7 operational status monitoring, operational issue alerts/notifications, and break-fix action o Monitor system performance and provide monthly status reports o Resolve issues impacting performance o Manage incidents via the management process flow, include assignment of incident priority and escalations within the tiers (as needed) o Provide tracking of issues until resolved; provide status report updates for active incidents

- Updates will include key information, a timeline where applicable, and steps being taken to remediate the incident, to keep management, system owners, and admins informed. A list or table of key information as it pertains to the incident will be provided.

o Comply with IPAWS Incident Management Procedures documented in the below table:

Priority/ Severity Ticket Assignment Updates Resolution Times

1 No Later Than 30 Min. Hourly Within 4 Hours

2 No Later Than 1 Hr. No Later Than 2 Hr. Within 8 Hours (1 Day)

3 No Later Than 1 Hr. No Later than 4 Hr. Within 16 Hours (2 Days)

4 No Later Than 2 Hr. No Later Than 3 Working business Days Within 32 Hours (4 Days)

5 No Later Than 2 Hr. No Late Than 5 Working Business Days Within 160 (20 Days)

• Contractor shall comply with IPAWS-OPEN Production environment requirements for Recovery Time Objective (RTO) and Recovery Point Objective (RPO), as depicted below:

Feature RTO RPO

Automated backups 1 hour Since last backup

Manual snapshots 15 minutes Since last snapshot

Read Replicas Support 99.99% uptime No data loss

• Support the IPAWS PMO in the development of technical reporting and briefings, such as briefings for DHS, FEMA, monthly IPAWS-OPEN performance, oversight bodies, conferences, etc.; reporting may include (as well as other technical topics):

Sprint/test/release status and schedule

Architecture

Technical risks/mitigations

Technical collaboration status/results

• The Contractor shall provide technical and management services to optimize IPAWS-OPEN system operations, architecture, and maintenance

Provide cloud architecture services and IaaS management (in collaboration with CSP); provide Managed Services to optimize cloud operations

Maintain logically isolated environments for Production, DR, Development, Test, Shared Services, Staging, Security, and IPAWS Demo Lab; number and scope of environments is expected to evolve over the life of the contract

Support, as necessary, implementation planning (or changes) for virtual private network (VPN) connections in collaboration with the CSP or DHS (depending on solution) and the cell phone carriers’ gateways (rather this task may entail a coordination role among DHS or FEMA-managed effort, depending on solution)

Continue to improve and maintain resilient IT architecture designed to tolerate system or hardware failures (either within or external to the system boundary) with minimal customer impact and eliminate single points of failure

The Contractor shall optimize the performance of the IPAWS-OPEN architecture, services, and processes o Provide IaaS technical support established by the CSP o Dynamically scale IaaS architecture solution up or down, adding or modifying stacks, storage, and other technical resources in order to efficiently meet current and future IPAWS capacity requirements o Provide scaling automation services with built-in resiliency; shall have the ability to be programmatically configured and scale based on a multitude of configuration options including processor & memory and shall auto-scale o Database services: Relational databases shall be geographically dispersed and be able to scale storage on demand; optimize RDS architecture (e.g. CPU and memory) and provision additional RDS resources as needed o Ensure continuous IT system availability and performance through maintainability and improvements of redundancy, load balancing, and DR measures o Manage and provision infrastructure and services available within the cloud environment >99% of the time. These configurable services shall include but are not limited to all networking components and routing within AWS boundary, AWS Web Application Firewall (WAF), security, keys, certificates, users, roles, virtual computing instances, micro-services, storage, databases, load balancers, messaging services, monitoring o Provide automated monitoring of performance, resource utilization, and other events such as failure of service, degraded service, network availability, API access, operating systems, application, security, etc., via a service dashboard or other electronic methods o Cloud Service Monitoring (CSM) through logs, network traffic, events and API calls to enable governance, compliance, and auditing of all actions and services within the cloud environment. Features for event-based triggers to enable automatic actions based on changes in the environment shall be deployed into Production Environment (e.g. auto-scaling) o Maintain and optimize existing configurable AWS Canaries scripts in the Production environment using CloudWatch Synthetics o When relevant, identify and propose additional metrics for technical performance, business success, and user satisfaction

• The Contractor shall provide support to PMO in the areas of User and Customer Experience regarding IPAWS-OPEN system and operations management

Support the Government in developing user and customer experience monitoring and tracking techniques; recommend system enhancements to improve user/customer experience

Support translating user/customer experience data and feedback into technical requirements

• DHS/FEMA requires systems to implement Observability capabilities

Conduct analysis to evaluate where Observerability would benefit IPAWS-OPEN operations;

recommend observerability tools and processes

Implement Government-approved Observability tool(s) and processes

• Provide end-to-end IPAWS-OPEN system monitoring capability

Provide automated and standardized monitoring of IT systems (to include the application) for availability and performance in order to improve overall system performance

Maintain monitoring and continue improvements during the duration of the contract

• Provide backup and restore services with minimal loss of data

Provide/maintain managed backup storage

Provide mirroring or copying upon request of the Government

Provide an orderable (back-up) bulk retrieval of all data, scripts, software, and virtual machine images, to include:

o Automated backup, archival, and redundant storage o Import/export with reliable, responsive, and cost effective cloud-based storage service, as well as back-up service o Block-level storage volumes for use with compute instances that provide: 1) off-instance storage that persists independently from the life of an instance; and 2) the ability to quickly move large amounts of data in and out of the cloud infrastructure o At FEMA’s request, create full snapshots for the platform and application, content, and related data – to be retrieved at FEMA’s request within 24 hours, up to a period to be determined by FEMA o Long-term storage and Contingency Plan / DR services o Utilize AWS backup/restore services and capabilities for system data, virtual machines, configurations/design, etc., to include:

- Backup of version management/configuration management system for source code in Development and Staging Test environments

- Backup of Production application code such that it is retrievable in an expeditious manner

- DR backup/restore services

- Amazon Machine Image (AMI) on a scheduled basis, as well as event-triggered

• The Contractor shall provide expertise and services to manage the IPAWS-OPEN system which operates in a dynamic emergency management domain and technologically evolving environment; to include other factors that have the potential to impact IPAWS-OPEN architecture and operations (such as emerging technologies, new mandates, new architecture improvements)

Investigate emerging technologies and industry best practices relevant to IPAWS-OPEN and provide recommended solutions for FEMA to consider. Once the Government agrees on the recommendation, the Contractor shall incorporate emerging technologies into updated architecture designs for Government approval before implementing

IPAWS operations and system architecture are impacted by external requirements: dynamic emergency management environment; emerging technologies; and policy and standards changes from Federal, DHS, FEMA and cellular industry o In response to the unforeseen requirements, the Contractor shall possess the ability to analyze emerging requirements, assess level of effort, and develop new services in support of unique alerting requirements (e.g. low latency processing for earthquake early warning, large volume processing for National Weather Service) and policy/standards changes o Identify architecture options and recommendations to address the future requirements, as they emerge, that may require significant architectural and process changes

Develop approved application capabilities and enhancements (may require development of supporting applications necessary for the operation of IPAWS-OPEN), such as interfaces, automated test tools, and web applications that integrate other IPAWS software packages developed by third parties

Implement REpresentational State Transfer (REST) Interface and ability to implement other interfaces, as needed

IPAWS PMO intention is to redesign the system to improve performance, as well as perform code refactoring and address technical debt for performance improvements on a continual basis o Future (To-Be-Determined/unknown) requirements may include need to make significant changes or complete redesign of the IPAWS-OPEN application o Contractor shall have ability to onboard (in a timely manner) appropriately skilled resources to address new requirements, as needed o Contractor shall possess experience and skills to be able to redesign the IPAWS-OPEN application and support functions, additionally Contractor shall possess knowledge of the alert and warning domain, standards, and interfaces/interoperating systems

- ATIS C-Interface Specification (https://www.techstreet.com/atis/standards/atis- 0700037-v003?product_id=2248537)

- ATIS C-Interface Test Specification (https://www.techstreet.com/atis/standards/atis- 0700038-v003?product_id=2248538)

- OASIS CAP 1.2 Standard (http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2.html)

- OASIS IPAWS Profile (http://docs.oasis-open.org/emergency/cap/v1.2/ipaws-profile/v1.0/cs01/cap-v1.2-ipaws-profile-cs01.html)

- ECIG EAS CAP Implementation Guide (http://www.eas-cap.org/ECIG-CAP-to- EAS_Implementation_Guide-V1-0.pdf)

- FCC Regulation for WEA (https://www.ecfr.gov/current/title-47/chapter- I/subchapter-A/part-10)

- FCC Regulation for EAS (https://www.ecfr.gov/current/title-47/chapter- I/subchapter-A/part-11) o Contractor shall possess ability to perform code refactoring and address technical debt to improve the performance of IPAWS-OPEN application

Support the IPAWS PMO in complying with existing and emerging/changing DHS and FEMA Enterprise Architecture (EA) and cybersecurity mandates during the development lifecycle and O&M:

o Create and maintain a baseline for system architecture/IT assets, interrelationships, topology o DHS Chief Technology Officer and DHS EA recommend IPAWS incorporate The Open Group Architecture Framework (TOGAF) approach into IPAWS-OPEN architecture planning and documentation, to include such disciplines/documentation as:

- Architecture View

- Business Architecture

- Data Architecture

- Transition Roadmap o Ensure IPAWS-OPEN system compliance with DHS/FEMA cybersecurity directives to satisfy EA and cybersecurity requirements during system engineering development, and continued architecture compliance under O&M management o IPAWS anticipates continuous changes in enterprise architecture and cybersecurity requirements, which will have impacts to IPAWS-OPEN; IPAWS needs support in anticipating and reacting to those changes, to include keeping a clear inventory of IPAWS-OPEN architecture o Support preparedness for technology insertion or proposed or mandated EA and cybersecurity changes; the Contractor shall:

- Conduct impact/LOE analysis to determine the impact to IPAWS-OPEN system architecture and performance, (such as anticipated Zero Trust cybersecurity architecture policies and mandates), and how the IPAWS Program software and architecture assets will be impacted

- Conduct feasibility assessment for complying with new or changed EA and cybersecurity policies

Due to the dynamic nature of network management within DHS and FEMA, the Contractor needs to be able to support planning, collaboration, and testing/troubleshooting with Government network management, to include migrations to either DHS or FEMA TIC 3.0 o Support the IPAWS Program, as needed, with IPAWS-OPEN network configuration planning and testing; DHS and FEMA are implementing next generation TIC solutions (DHS TIC 3.0;

FEMA TIC 3.0); TIC 3.0 solution implementation dates are to-be-determined, but anticipated to be implemented at some point during the BPA period of performance o Depending on which TIC 3.0 solution IPAWS-OPEN migrates to, the IPAWS Program will need to collaborate with DHS or FEMA to configure the network solution customized to IPAWS- OPEN requirements o Contractor shall support the IPAWS Program with developing IPAWS-OPEN network configuration requirements; collaborating with DHS/FEMA network management and engineers regarding IPAWS-OPEN end-point interfacing with TIC 3.0; and performing tests (from IPAWS-OPEN perspective that the application AWS end-point is successfully receiving and sending messages over TIC 3.0) o Coordination, configuration, and testing of network implementation/changes with cell carriers

Contractor may be required to integrate third-party tools and/or software and provide supporting architecture to support those tools/software; Contractor shall assume responsibility to maintain the third-party tools/software

Contactor shall support required migration from IPv4 to IPv6, if directed by FEMA

FEMA Office of the Chief Information Officer (OCIO) is in the process of implementing the FEMA Enterprise Cloud (FEC); if IPAWS-OPEN is required to migrate to the FEMA FEC, Contractor shall:

o Support IPAWS PMO for application and data migration planning o Support migration cutover validation testing from IPAWS-OPEN operational, data, and performance testing perspective

• The Contractor shall provide expertise and implementation steps to mature the IPAWS-OPEN system development methodology, to include CI/CD and Agile practices tailored to IPAWS-OPEN development management requirements

Assess IPAWS-OPEN system components (such as the application and custom developed support tools/apps) to determine the most appropriate development methodology for each system component; support Government in developing and continuously maturing the development approach for each component, to include testing and reporting o IPAWS PMO currently employs a hybrid Waterfall-Agile development approach (Waterfall for planning, requirements, design, and system integration; Agile for sprint development) o To comply with DHS and FEMA development guidelines, the IPAWS PMO requires the Contractor to propose and support an approach to mature the IPAWS-OPEN Agile methodology

Provide Continuous Integration and Continuous Delivery (CI/CD): Contractor shall support IPAWS PMO’s goal for maturing its CI/CD process to achieve fully automated development phase for building and testing code; manage development pipeline so builds are ready for deployment into Production upon FEMA approval of deployment readiness o IPAWS PMO currently employs AWS CodeCommit, CodeDeploy, and CodePipeline CI/CD services, however the PMO is relying on the Contractor’s experience and expertise to select the most optimal CI/CD services and tools for maturing the CI/CD and DevOps IPAWS practices; note that the Government is open to recommendations for new CI/CD methodologies and tools but expects justifications for the recommended changes o The Contractor shall follow industry best practices to mature IPAWS-OPEN development to achieve continuous integration and continuous delivery; these efforts will include the management and maintenance of the AWS GovCloud infrastructure and the DevSecOps process o All code changes made by developers will immediately be checked in and integrated into the main branch so automated testing can run to quickly identify code issues so the developer can remediate issues earlier in the development lifecycle than traditional development methodology o The Contractor shall design, develop, and deploy builds according to the development priorities established by the IPAWS-OPEN Product Owner o Provide development and test tools, security tools, and ability to mature processes to provide continuous integration, and continuous delivery/deployment processes

Support maturation of development process, to include ability to implement Agile development for the full lifecycle development and…