Attachment J-2 Draft PWS.pdf

U.S. Department of Homeland Security

Office of the Chief Information Officer

Draft Performance Work Statement for

Data Center and Cloud Optimization

DRAFT ‐ October 2020

This page intentionally left blank

1 General The Department of Homeland Security (DHS) Data Center and Cloud Optimization (DCCO) acquisition plans to acquire information technology services to manage and operate the DHS Hybrid Computing Environment (HCE). The HCE is a collection of enterprise computing resources including a data center, colocation sites, and commercial and private cloud services. In addition, DHS will acquire associated professional services to automate, optimize, and modernize the HCE.

1.1 Background

The main source of enterprise computing resource for DHS has been Data Center 1 (DC1), a Government Owned Contractor Operated data center located at the National Aeronautics and Space Administration (NASA) Stennis Space Center in Mississippi. NASA owns and manages the data center facility. DHS is one among a number of tenants within this facility. DHS leases approximately 36,000 square feet of raised floor space in the data center. NASA provides power management, raised floor space, physical security, office space, environmental control, and fire suppression to support DHS IT needs.

In addition to DC1, DHS will add Infrastructure as a Service (IaaS) offered by Cloud Service Providers (CSPs) and colocation facilities to its portfolio of enterprise resources. The Contractor shall operate, optimize, and automate the HCE as an integrated entity. The addition of colocation and CSP environments will extend the capacity, scalability, and redundancy of DHS enterprise computing capacity by providing additional physical, virtual, and cloud resources.

As of September 2020, the operating environment for DC1 can be characterized by the following:

14 DHS Components hosting systems in DC1

109 Hosted customer Federal Information Security Management Act (FISMA) systems

2009 Physical servers; 1660 virtual machines (VM)

Installed networking devices:174 Firewalls; 74 Load balancers; 46 Routers; 1,183 Switches

599 Racks populated with DHS equipment, 975 racks installed, total rack capacity 1049

765 Terabytes (TB) of provisioned storage, 432 TB utilized

7,267 Devices installed supporting Component-owned systems

1,425 Devices installed supporting data center management and operations

Monthly workload statistics, January through June 2020 o 2,694 Service desk tickets generated, ranging from 2,467 low to 2,940 high o 264 Change requests processed, ranging from 225 low to 288 high o 4,062 Incidents managed, ranging from 2,488 low to 7,395 high

Two General Support Systems (GSS) operate in DC1 that are essential for the management and operations of the current DC1 and the future HCE:

1. Data Center 1 GSS (DC1 GSS) is a support system providing the underlying hardware infrastructure for the data center consisting of hardware switches, routers, network devices, firewalls, load balancers, and WAN optimization appliances and respective operating systems for the data centers.

2. Data Center 1 Management Zone GSS (Management Zone) is a support system providing the support mechanism (management, monitoring, and storage) for DC 1 systems consisting of the appliances, hardware (e.g., Storage Area Network (SAN) storage, storage hardware, encryption devices, tape libraries, tape backups, servers, blades, network devices, monitoring appliances), software, and applicable reporting mechanisms and enterprise solutions dedicated to supporting the availability of DC1 Component systems and services.

The HCE contains physical compute servers organized into five zones in DC1. It also includes physical compute server resources placed in colocation facilities. The servers include but are not limited to x86 blades from HP, Dell, and other vendors. Server operating systems include but are not limited to Windows, Red Hat Enterprise Linux (RHEL), Community Enterprise Operating System Linux distribution (CentOS), and IBM Advanced Interactive eXecutive (AIX). Some of the servers are configured with hypervisors, including VMware and IBM Power Virtualization Center (VC), to provide VMs. The HCE also contains storage servers and SANs including but not limited to products from EMC and Hitachi.

This requirement focuses on information technology services required to support the DHS HCE including those DC1 support services not provided by NASA and infrastructure support services at colocation sites. It also includes the migration, operation and maintenance of systems and applications among the data center, colocation, and cloud environments that constitute the HCE.

For DC1, NASA coordinates with the Federal Data Center Operations team for all facility support activities. DHS Federal staff act as the intermediary between NASA and the data center support services Contractor.

The purpose of this requirement is to acquire Contractor support for the operation, maintenance, automation, optimization, and modernization of the DHS HCE and to offer an efficient, responsive Information Technology (IT) hosting environment that serves as the foundation for continued computing operations in support of the DHS mission. The HCE shall include unclassified and classified IT infrastructure, applications, and data.

The Contractor shall provide data center-based hosting and IaaS obtained from both commercial and Gov Cloud CSP service offerings. The Contractor shall provide professional services to support application operation, migration, and Operations and Maintenance (O&M) Orders. The Contractor shall provide customer service including an introduction to HCE environments and services for DHS customers.

DHS intends to operate the HCE as a high‐priority 24x7x365 computing environment supporting critical mission and business needs for the Department. This includes, but is not limited to, maintaining Service Level Agreements / Service Level Objectives (SLA/SLO) and having the appropriately cleared and qualified staff and support services necessary to maintain operations.

The HCE will offer a more efficient, responsive IT hosting environment that serves as the foundation to ensure continued operations in support of the DHS mission.

1.2 Scope

The scope of this Performance Work Statement (PWS) is to provide operations, hosting and professional services for the DHS HCE. The Contractor shall identify, provide and manage personnel, processes, tools, and reporting to facilitate transparency and decision‐making as well as to support the transition to the future state HCE. This requirement also includes the continuing support of current operations and services.

The Contractor’s responsibility within DC1 includes the maintenance, operation, and organization of the IT infrastructure technologies. Power, environmental controls, and facility modifications come under the purview of NASA Shared Services Center National Center for Critical Information Processing and Storage (NCCIPS) management.

1.3 Objectives

The DCCO requirements will be the core support services to drive a more efficient, responsive hybrid IT environment that serves as the foundation for the management and integration of on-premises, colocation, and cloud-based environments.

These support services must optimize and ensure continued DC1 operations, while implementing and managing the future state HCE in support of the DHS mission and, where appropriate, migrate infrastructure and applications within the HCE (e.g., from DC1 to CSP environments).

The objective of DHS’s DCCO support service is to expedite the transformation of IT capabilities from an asset‐based model to a service‐based, customer‐centric IT business model;

provide transparent operational expenditures; and reduce both capital expenditures and time‐to‐ delivery for new capabilities. To accelerate this transformation, DHS will continue existing initiatives associated with cloud optimization efforts as well as leverage new approaches. The business objectives for this requirement include:

1. Improved Total Cost of Ownership (TCO) and Cost Transparency.

2. Modernization and optimization - Operations within the HCE should reflect state-of-the-art, optimized, automated, modern processes that rely upon automatic processes, to the maximum extent practicable, with the minimum required human intervention.

3. Responsiveness ‐ Maximize the customer experience, improve trust, reduce the time to deliver, and enhance operational performance.

4. Scalability ‐ Elastically scale hosting environments and supporting services quickly to meet mission operations and surge capabilities.

5. Improved Reliability and Availability ‐ Monitor and improve Mean‐Time‐Between‐ Failures and Mean‐Time‐To‐Repair metrics.

6. Secure HCE Environment ‐ Monitor and evaluate systems, capabilities, interfaces, applications and data transactions to assess and mitigate cybersecurity threats.

7. Continual Operational Assessment and Improvement ‐ Continuously collect and analyze performance metrics – both technically and administratively – to identify and implement optimizations that improve products services, and processes reflected by those metrics, and lead to continuing performance improvements as processes achieve greater levels of optimization.

8. Architectural approach - Identify objectives for further development of the HCE technical architecture that enhance interoperability, improve standards, reduce risks, improve security, and support emerging technologies and capabilities.

9. Service Governance ‐ Enforce a strong governance mechanism to ensure consistent interpretation of policy, monitor DHS enterprise computing performance, adhere to DHS policy requirements and address stakeholder issues.

10. Simplified Management ‐ Improve communication through a data‐driven, transparent management approach that allows for clear focus and attainment of management and technical goals.

1.4 Applicable Documents

See Appendix A for a full list of Applicable Documents.

1.5 Performance Requirements Summary

This PWS includes a Performance Requirements Summary (PRS) in Section 12. The PRS plays an integral role in the administration of the resulting contract. In addition to any applicable inspection clauses or other related terms and conditions contained in the contract, the PRS shall serve as a primary tool for inspection and acceptance of services as facilitated by the Contracting Officer’s Representative (COR). Evaluation of the Contractor’s overall performance shall be in accordance with the performance standards set forth in the PRS, and will be conducted by the COR. The PRS constitutes a material aspect of the PWS and will not be changed or otherwise modified without prior written approval of the Contracting Officer.

1.6 Definitions

Incident Severity. Categorization of the level of negative impact that service outages have on the DHS enterprise.

a. Severity 1 – production systems or applications not operating or unavailable causing end users to be unable to work in multiple sites for one or more DHS Components.

b. Severity 2 – development environments unavailable with no workaround or alternate with intermittent or no functionality for multiple users in multiple sites for one or more DHS Components; or production environments operating in failover or redundancy mode with intermittent or no functionality for multiple users in multiple sites for one or more DHS Components.

c. Severity 3 – production systems or applications operating in failover or redundancy mode with no impact to multiple users in one or more sites for one or more Components; or testing, preproduction, or development environments operating in failover or redundancy mode with intermittent or no functionality for multiple users at one or more sites for a single Component.

Inventory. Inventory shall consist of a complete list of all equipment (i.e., hardware), applications (i.e., software), and resources within the HCE, in all physical, virtual, colocation and cloud environments. For software inventory includes; applications, system, owner, software type, description, type of license (e.g., enterprise, single site), version, cost, date of acquisition, date of installation, and anticipated end of software assurance date.

d.

2 Management Requirements and Tasks The Contractor shall support the requirements in this Section across the HCE. The support includes physical and virtual assets in the DC1 facility in Stennis, MS. It includes the addition of private cloud assets in DC1 and in colocation facilities. The support also includes the addition of public cloud and Gov Cloud assets from commercial CSPs. All these assets shall be considered as part of the HCE.

2.1 DHS Information Technology Infrastructure Management

The Contractor shall manage DHS IT infrastructure. As required, the Contractor shall interface with DHS Data Center Operations Team to address any issue that requires modification to general facility services. The Contractor shall be responsible for providing operational best practices including the following:

HCE Operations, Maintenance, and Management Network Communications LAN & WAN Governance Service Classification Hybrid Computing Environment Management Automation Policies and Procedures

General Support Systems Customer Dashboard and Information Repository Acquisition Colocation Services Cloud Services System Architecture / Engineering Contract Transition Additional Management Services

2.1.1 HCE Operations, Maintenance, and Management

The Contractor shall operate, maintain, and manage the DHS HCE which includes DC1, current and future colocation facility installations, and cloud resources. The Contractor shall operate the HCE on a 24x7x365 basis with at least 99.9% availability. Within the HCE the Contractor shall manage, maintain, and operate Component owned equipment, systems, and software as specified on Task Orders and the GSS described in Section 1.1. The HCE shall provide compute, storage, network, database, and security services for DHS and Component use. Storage services include file, object, block and archival storage. The Contractor shall work with the DHS Data Center Operations Team and COR for any facility management support.

2.1.2 Network Communications LAN & WAN

The Contractor shall maintain and operate Local Area Networks (LAN) within HCE facilities and environments that connect to the DHS OneNet Wide Area Network (WAN). DHS provides all WAN connectivity points and equipment. The LAN shall provide dual ports to each hosted server for redundancy and an overall redundant architecture.

The Contractor shall monitor and administer network traffic throughout the HCE, forecast bandwidth requirements associated with service requests, and implement network modifications to accommodate demand. The Contractor shall design the network architecture to support networking requirements including provision for a Zero Trust model and maintain as-designed and as-installed architecture drawings and descriptions in the Information Repository. The Contractor shall also provide dashboard displays of real time network performance metrics and posts notices regarding unplanned network outages as well as any planned service disruptions.

The Contractor shall provide any access or network monitoring required by DSH management or programs and shall support any required integration with enterprise monitoring

2.1.3 Governance

The Contractor shall operate under the umbrella of DHS governance and oversight processes that dictate interactions among DHS Headquarters, DHS Components, the Contractor Team, colocation providers, and CSPs, and all other DHS contractors addressing interrelated requirements.

The Contractor shall adhere, follow, and support all DHS IT and Information Security Policy in governing change and configuration management.

The Contractor shall obtain Government approval for all solutions proposed to satisfy requirements and for all purchases required to fulfill those requirements.

2.1.4 Service Classification

The Contractor shall be capable of providing Level 1 services (see Section 3.8.1), Level 2 services (see Section 3.8.2, and Professional Services (see Section 4) at the classified level (i.e.

Top Secret) at DC1 and at all current and future colocation facilities.

The Contractor shall provide services at the following classification levels across the HCE:

Sensitive But Unclassified (SBU) Level. Including FedRAMP certified services from commercial and Gov cloud CSPs assessed at FIPS 199 Low-Low-Low categorization, up to a High-High-High categorization, and variations of FIPS 199 categorization.

Secret Level. Services at the Secret level including services in Secret CSP environments.

TS/SCI Level. Services at the TS/SCI level including services in TS/SCI CSP environments.

This includes services from public CSPs and includes commercial and gov-cloud environments.

The Contractor shall support Sensitive Compartmentalized Information Facilities (SCIFs) in DC1 in compliance with DHS and Government Security Directives. The Contractor shall maintain SCI accreditation, in coordination with DHS.

2.1.5 Hybrid Computing Environment Management Automation

The Contractor shall automate processes throughout the HCE to maximize operational efficiency, included but not limited to:

- Alert Monitoring

- Circuit Monitoring

- Change Management

- Incident Management

- CSP-based Monitoring

- Configuration Management

- Infrastructure Monitoring

- Security and Network Monitoring

- Hardware and Software Monitoring

- Analytics, Intelligence, and Reporting

2.1.6 Policies and Procedures

The Contractor shall create, update, and maintain Contractor policies and procedures that govern its operation of the HCE and store those policies and procedures in the Information Repository.

2.1.7 General Support Systems

The Contractor shall manage, operate, and maintain the existing equipment, software, and other assets that constitute the GSS described in Section 1.1 to manage the existing HCE environment.

The Contractor shall extend GSS management, operation, and maintenance capabilities beyond DC1 to all HCE resources, current and future. The Contractor shall provide dashboard access to GSS-generated reporting, metrics, and statistical data. The Contractor shall be responsible for populating various reports and interfacing GSS operational data to enterprise dashboards as requested.

2.1.8 Customer Dashboard and Information Repository

The Contractor shall establish and maintain a centralized, integrated dashboard and Information Repository with real-time, on-demand, 24x7x365 access by DHS and Components. Role-based access shall be available to authorized DHS users and groups, with privileges allowing viewing, data retrieval, and data download in formats (e.g., Microsoft EXCEL, PDF) suitable for analytical processing.

The Contractor shall use the dashboard to track and manage assets and services provisioned and consumed in the HCE, including DC1, colocation centers, and CSPs. The dashboard shall enable DHS to manage, use, report, audit, and track assets and services. The dashboard shall include inventory and management of physical and virtual assets and cloud services, usage and performance metrics for all assets and services, security metrics such as cyber threats and vulnerabilities, and metrics for billing and accounting. The dashboard shall capture and report actual and anticipated expenditures and forecast future spending to support financial, budget, audit and benchmarking activities. The dashboard shall also provide visibility into resource and asset tagging. Financial management shall be based on the Information Technology Infrastructure Library (ITIL).

The Contractor shall Implement a Financial Management solution based on ITIL to provide cost-effective management of HCE resources in accordance with awarded Task Orders. The Contractor shall provide a dashboard displaying all spending and purchasing under awarded Task Orders and forecasts for future spending.

The Contractor shall establish a centralized knowledge management capability, in conjunction with the dashboard and Information Repository, to share deliverables and related documentation about the HCE. The Contractor shall populate the Repository with information and documentation during the Transition-in Period. The Contractor shall maintain current versions during the contract performance period and shall provide final versions during the Transition-Out period. The Contractor shall provide DHS with access to the information throughout the period of performance.

The Contractor shall work with DHS to modify the dashboard and Repository as requested. The Contractor shall perform periodic quality assurance reviews on the dashboard data to ensure accuracy and necessary modification. The Contractor shall ensure DHS can collect, analyze and synthesize the data (e.g., usage, security, performance) along with system metrics and SLA results on the services being provided. The Contractor shall also enable DHS to collect, analyze and synthesize cost data, including non-labor or professional services, to ensure accurate billing, configuration, and cost transparency. The Contractor shall enable the dashboard data to be exported into formats, such as Microsoft Office and Adobe PDF for download.

The Contractor shall upload deliverables, work products and related material to the Information Repository. These documents shall include environment descriptions and operations manuals, describing the HCE data center, colocation, and cloud IaaS environments, usage, processes and procedures, with updates when there are significant changes in those environments, processes or procedures. The Contractor shall provide separate addendums to these documents for customer unique configurations, processes, or procedures.

2.1.8.1 Real Time Resource Consumption and Cost Tracking

The dashboard shall enable DHS to use, track, report and audit all assets and services, including compute instances, storage volumes, and third-party products. The dashboard shall enable DHS to set and manage manual and automated approval of workloads, quotas, and thresholds for asset and service usage across the HCE for organizational and project accounts. The dashboard shall enable DHS to provision, deploy, deprovision, and decommission assets, infrastructure, and services manually or automatically. The dashboard shall enable DHS to start, stop, terminate, and reboot virtual machines. The dashboard shall separate usage costs into billable groups for reporting purposes; and shall provide DHS with the ongoing capability to set thresholds and limits to usage, deployment, and provisioning of resources.

2.1.8.2 Capacity Management

The Contractor shall monitor the capacity utilization of HCE resources and provide real-time dashboard displays and periodic reports to DHS regarding capacity utilization. The Contractor shall also develop future capacity requirement projections for each type of resource provided within the HCE. The contractor dashboard shall integrate with DHS enterprise-level systems and dashboards and upload data to and accept downloads of data from those systems and dashboards.

2.1.9 Acquisition

Requirements may demand the acquisition of computing, communications, and data storage assets necessary to host and operate systems applications that DHS Components want to implement and operate. The Contractor shall assess the Task Order requirements specified by Components, determine whether resources are available within the HCE, and either allocate resources if available or procure additional resources as necessary to satisfy those requirements.

2.1.10 Colocation Services

The Contractor shall provide colocation services to DHS customers and will support the process to obtain physical security authorization and Authority to Operate (ATO) at these colocation locations. Colocation services are defined as leased data center services that provide facility management, connectivity, cloud enablement, and security services. The Contractor shall provide the same operations, maintenance, and management services in colocation facilities as provided in DC1 including on-site hands-on services.

2.1.11 Cloud Services

The Contractor shall obtain CSP public cloud and Gov cloud IaaS services on behalf of DHS customers. The Contractor shall identify FedRAMP certified IaaS services offered by CSPs that meet DHS customers’ technical, security, and business objectives as defined in Task Orders and subsequently acquire, integrate, and manage those services as components of the HCE on behalf of the Government.

2.1.12 Intermediary for Software Licenses

The Contractor shall acquire, manage, and provide visibility of software licenses on behalf of the

DHS.

2.1.13 System Architecture / Engineering

The Contractor shall provide system architecture and engineering services in order to maintain and continuously improve the HCE. The Contractor shall document the HCE system architecture including DC1, colocation centers, CSP infrastructure, the interconnections among HCE elements, and HCE security provisions. The Contractor shall develop and maintain associated operations manuals to optimize operations by DHS and store those architectural descriptions and manuals in the Information Repository.

2.1.14 Contract Transition

The Contractor shall assume responsibility for the management and operation from the existing contractor and shall ensure that all operations continue with minimal to no disruption during the contract transition.

2.1.14.1 Transition-In

The Contractor shall develop and implement an Incoming Transition Plan to accomplish the transition of operations from the current service provider to the DCCO Contractor within 120 days from Transition Task Order award without any disruption of service. At a minimum, the Transition shall include:

1. Establishing a Contractor Transition Team within 5 business days of Task Order award.

2. Conducting and completing a site survey and assessment of DC1 within 15 business days of Task Order award.

3. Creating a Draft contract-level Incoming Transition Plan, to include an initial staffing plan, for the overall contract effort within 30 business days of Task Order award for assuming operational and managerial control of the HCE and a final Transition Plan, to include final staffing plan, within 90 business days of Task Order award. Task Order transition will be addressed on an order by order basis. Task Orders under the predecessor contract all expire with that contract and must be established under the new contract.

4. Conducting and documenting an inventory of all equipment (i.e., hardware) and applications (i.e., software), and resources within the HCE – in all physical, virtual, colocation and cloud environments within 60 days of Task Order award.

5. Establishing Task Orders under this new contract for continuing existing customer data center support and Task Orders for new customer support requirements in the HCE.

2.1.14.2 Transition-Out

The Contractor shall:

1. Create, deliver and execute an Exit Transition Strategy Plan within 60 days of Transition Task Order Award.

2. Deliver final updates on all policies and procedures governing current operations within the HCE not less than 90 days prior to the end of the period of performance or last exercised option period.

3. Deliver a current, final outgoing inventory of all equipment (i.e., hardware) and applications (i.e., software), and resources within the HCE – in all physical, virtual, colocation and cloud environments, not less than 90 days prior to the end of the period of performance or last exercised option period.

2.1.14.3 Task Order Transition Exit

Shall be specified at the Task Order Level.

2.2 Security Services – HCE Level

The Contractor shall monitor the physical and virtual security of HCE components, and coordinate with DHS as appropriate, by performing the following work activities:

2.2.1 Security Operations Management

The Contractor shall provide security management services for the HCE and shall also provide specific security-related services in accordance with awarded Task Orders. The security services include maintenance of Zero Trust through identity and access management (role-based or attribute-based) with multifactor authentication and also include encryption to protect data at rest and in transit. The contractor shall ensure any access requirements or security monitoring needed by DHS management or programs, or integration with enterprise monitoring requirements, is provided.

The Contractor shall implement the security controls required for DHS servers and implement security mitigations as recommended by the Enterprise Security Operations Center (ESOC) or other enterprise-level authority. The Contractor shall conduct security testing to verify that the servers are protected against potential cyber threats.

Personnel security services are addressed in Section 5.5.2.

2.2.1.1 Vulnerability Assessments

The Contractor shall implement and deploy the tools, toolsets, and staff to support, operate, and maintain vulnerability assessment services in the HCE for systems and services delivered and operated by the Contractor. The Contractor shall develop a Vulnerability Assessment Plan and conduct routine, network-based vulnerability scans and assessments.

The Contractor shall track and collect threat and vulnerability data, report threats and vulnerabilities in the centralized dashboard, and implement mitigations as a defense from potential attacks.

2.2.1.2 Intrusion Detection and Prevention Systems (IDS/IPS)

The Contractor shall deploy, operate and maintain an Intrusion Detection System (IDS) to include host-based and network-based detection and an Intrusion Prevention System (IPS) to include host-based and network-based prevention for networks and systems resident within the

HCE.

2.2.1.3 Firewall Management

The Contractor shall operate and maintain firewall software and hardware components and implement and deploy the tools, toolsets, and staff to support, operate, manage and maintain firewalls for applications, systems, and services delivered and operated by the Contractor within the HCE.

2.2.1.4 Anti-Virus Management

The Contractor shall operate and maintain anti-virus protection for the HCE and ensure that signatures and databases are the latest approved and tested versions.

2.2.2 Equipment Access and Control

The Contractor shall:

1. Ensure that all delivery and removal of equipment within the data center facility is authorized by DHS personnel.

2. Ensure that Contractor-issued and owned electronic devices meet DHS configuration guidance, otherwise the equipment shall not be permitted in the data center.

3. Prevent personally-owned electronic devices (e.g., laptops, portable storage media, cell phones, etc.) from being taken into restricted areas within the data center.

4. Scan all electronic devices (e.g., government-issued devices, and Contractor-furnished devices) to identify vulnerabilities, verify the existence of up-to-date virus definitions, and ensure compliance with DHS configuration and policy guidance.

5. Dispose of or destroy media containing sensitive information in accordance with National Institute of Standards and technology (NIST) Special Publication 800-88 Revision 1, Guidelines for Media Sanitization and DHS Information Security Policy. Inventory tracking status and approval chain of custody shall be updated in the dashboard.

6. Allow the DHS-appointed Communications Security (COMSEC) custodian to execute the responsibilities as outlined in DHS National Security Systems (4300B.200) - COMSEC.

2.2.3 Authority to Operate

The Contractor shall obtain and maintain a formal ATO as described in the DHS System Authorization Guide for the two GSS described in Section 1.1.

2.2.4 Risk Management

The Contractor shall create a Risk Management Plan and conduct and document Risk Assessments (RA) for the GSS operated by the Contractor after completing a NIST 800-53 evaluation, Contingency Plan Testing, and assisting the OCISO in conducting Security Tests and Evaluation (ST&E). Risk Assessments shall identify threats and vulnerabilities, assess the impacts of the threats, evaluate in-place countermeasures, and identify additional countermeasures necessary to ensure an acceptable level of security. Risk Assessments shall also address the cost and schedule of mitigation activities. The Contractor shall update Risk Assessments annually.

2.2.5 Common Controls

The Contractor shall:

1. Enter security controls for each GSS that the Contractor operates and for the infrastructure supporting DC1 private cloud service offerings in the DHS Enterprise Cyber Risk Management tool, currently Xacta, and update those controls annually. The controls shall consist of controls provided FISMA systems as well as physical controls inherited or provided by NCCIPS and by Contractor processes and services such as Change Management and anti-virus deployment.

2. Provide inheritable controls to be used by tenants for their security documentation for systems and applications that those tenants implement on the HCE and enter into the DHS Enterprise Cyber Risk Management tool. These tenant systems and applications will inherit the controls entered by the Contractor in the DHS Enterprise Cyber Risk Management tool for the private cloud infrastructure.

3. Provide additional security controls for systems and services delivered and operated by the Contractor.

2.2.6 Access Management

The Contractor shall provide DHS with full inquiry, retrieval, and data download user access to all approved service offerings. In addition to user access, the Contractor shall enable application access to the HCE by enterprise applications. This shall include allowing applications to use the Application Programming Interfaces (APIs) and Software Development Kits (SDKs) available from the CSPs, to directly interact with CSP services.

The Contractor shall ensure DHS can implement permissions and restrictions, as required, for authorized users to access CSP marketplaces. The Contractor shall ensure the access permits authorized DHS users to download, install and run approved marketplace applications in the CSP environments.

3 Operations Support Requirements and Tasks

3.1 Operations Services

Contractor shall provide HCE operations services on a 24x7x365 basis. The Contractor shall provide real-time system status to DHS customers and monthly service level reports. Contractor shall develop and maintain Standard Operating Procedures (SOPs) governing infrastructure operations to align with relevant SLAs and shall provide access to the documentation following commercial and federal standards and best practices.

3.2 Service Desk

The Contractor shall maintain and operate a 24x7x365 Service Desk for infrastructure and applications residing within the HCE. The Contractor shall respond to and manage incidents occurring within the HCE. The Contractor shall provide efficient management of incident response and maintain communication with DHS stakeholders during response and resolution and provide an escalation path for the resolution of complex issues. The Contractor shall:

1. Track and manage service requests from receipt through closure, problem reports to closure, and provide statistics for service desk performance reporting and analysis.

2. Accept and process incoming service requests.

3. Provide enterprise help desk support for GSS applications that the Contractor operates in on-premises locations, colocation environments, approved CSP platforms and environments, and internal private cloud infrastructure supported by the Contractor.

4. Provide service and incident management support to Components for systems and applications hosted within the HCE.

5. Integrate solutions (e.g., service management software), processes, and procedures with overall DHS IT enterprise reporting and incident management processes and functions as components of enterprise DHS incident management.

6. Provide documented root cause analysis on “problem cases” with recommendations for remediation and shall provide initial and final analysis reports to DHS, load reports to the Information Repository, and provide role-based access to the root cause reports.

7. Track ongoing infrastructure operation metrics and Service Desk performance metrics in the dashboard, in the monthly Service Quality Review (SQR), and in the formal quarterly Service Level Agreement Level of Service Report.

3.3 Inventory Control and Asset Management

The Contractor shall use the dashboard to manage the inventory of all equipment (i.e., hardware) and applications (i.e., software), and resources within the HCE – in all physical, virtual, colocation and cloud environments in the HCE. The Contractor shall:

1. Maintain a Configuration Management Database (CMDB) with the complete HCE hardware and software asset inventory in the dashboard and provide integration and access to enterprise CMDB solution(s).

2. Record receipt of all incoming hardware and software items within an asset tracking and inventory management system and record changes in location for each item as they occur.

3. Provide retrieval, view, and download access to all asset information indicating location and status of each inventory item as well as maintenance and licensing agreements and expiration dates for all inventory items. Download access shall enable the export of data in processable formats (e.g., Microsoft EXCEL, PDF) for offline analysis.

4. Deliver a complete, fully validated inventory of all equipment, applications and system software within the HCE not less frequently than at four-month intervals in both the dashboard and as a separate analytical data file.

3.4 Change Management

The Contractor shall control change within the HCE in compliance with the enterprise change management process to ensure the completeness and integrity of all changes implemented within the HCE. The Contractor shall follow established DHS processes and procedures for adequate change management control and implementation.

The change processes shall manage the following components that may change in fulfillment of Task Orders:

• Hardware

• System software

• Application software

• Cloud IaaS services and resources

All documentation and procedures associated with the running, support, and maintenance of live systems shall be stored in the Information Repository.

3.5 Customer Satisfaction

The Contractor shall solicit an assessment of Customer satisfaction from each Task Order owner.

Contractor shall design and implement surveys to capture and document customer satisfaction using Net Promoter Score to gauge Customer satisfaction. The Contractor shall develop and execute corrective action plans to remedy deficiencies identified through these surveys.

3.6 Quality Control

The Contractor shall develop and implement a Quality Management Program. The Contractor shall create and submit a Quality Management Plan (QMP) describing the standards, processes and procedures used to support the consistent delivery of high-quality, professional products and services provided in support of 24x7x365 HCE operation. The QMP shall be based on the Quality Assurance Surveillance Plan (QASP) provided by the Government. The Contractor shall deliver the initial QMP 60 days after Transition Task Order award and review and update the QMP at least annually.

The Contractor shall implement processes and procedures described in the QMP to meet the Government-provided SLAs. The Contractor shall not diminish the service offerings from the CSPs to a level lower than the published commercial cloud service, or standard SLAs (Acceptable Quality Levels (AQLs) in Section 12 correspond to SLAs), unless otherwise specified and approved by DHS.

3.6.1 Application Quality Control

The Contractor shall provide Application Quality Control services for applications hosted in the HCE in order to maintain application integrity. The Contractor shall manage Quality Control according to industry best practices and shall detect and report quality problems per contract SLAs and any SLAs particular to Task Orders.

3.6.2 Independent Verification and Validation Support

As directed by the COR, the Contractor shall provide cooperation and support to any Independent Verification and Validation (IV&V) project performed by or commissioned by DHS or the Department’s designated representative.

3.7 Availability Management

The Contractor shall ensure CSPs provide a minimum availability of 99.9% for each service (e.g., virtual machine, object storage, virtual private cloud) incorporated into the HCE, unless otherwise published by the CSP. The Contractor shall ensure each service provided to DHS, meets or exceeds, the commercially-advertised level or published SLA. The Contractor shall ensure that services launched by DHS in multiple cloud zones or regions continue to operate and remain available when any of the CSP’s data centers are offline or unavailable. The Contractor shall ensure CSP services acquired for DHS have redundancy characteristics that ensure the capability for maintaining the minimum availability requirement.

3.8 Operations and Maintenance Services

The Contractor shall follow the industry-recognized ITIL version 4 practices to provide lifecycle O&M support for the HCE. This includes installing and configuring hardware and system software on physical equipment, system software on cloud-based IaaS services, integrating, testing and securing physical and cloud resources, and conducting ongoing maintenance. The ongoing maintenance shall include asset management, change management, problem and incident management, continuity of operations, continuous monitoring, and decommissioning.

The Contractor shall apply a similar approach to providing lifecycle O&M support for customer applications in the HCE. The Contractor shall apply this lifecycle approach at the required classification levels: Unclassified, SBU, Secret, and Top Secret (TS), including Sensitive Compartmented Information (SCI).

The Contractor shall provide tiered service levels:

Level 1 services apply to physical assets located in DC1 or in colocation centers -compute, storage, database, and networking devices.

Level 2 services apply to all system software, including operating systems, database management systems, data backup systems and network software, in any HCE environment. System software also includes tools for managing and monitoring hardware and software components, problems and incidents, and resource management.

Level 2 services may apply to physical assets within the HCE, virtual assets within the HCE, or to assets placed in HCE cloud environments.

3.8.1 Basic Level Service (Level 1)

Basic Level Service (Level 1) is a hosting service offered for physical assets installed within the HCE. Level 1 services include hardware maintenance and network monitoring for equipment.

This is the minimum level of service that is provided for all physical assets residing within the HCE. The Contractor shall ensure that equipment is installed in DC1 or in colocation facilities as appropriate and that the equipment is brought up to an operational state; the Contractor shall also provide the services described in this section. Basic Level Service provides network connectivity from the servers to the WAN point of demarcation for all systems hosted in the environment.

3.8.1.1 Installation

The Contractor shall perform all tasks necessary to properly install equipment to ensure all items are installed in the state as required by Task Orders. The Contractor shall test and document equipment and ensure that the enterprise change management process is followed. The Contractor shall ensure proper communication with the DHS COR, HCE service desk, DHS Data Center Operations Team, and facility management whether NASA for DC1 installations or colocation facility management for colocation installations, as appropriate.

3.8.1.2 Hardware Maintenance

For all equipment within the HCE the Contractor shall:

1. Monitor equipment

2. Ensure that maintenance agreements are renewed in a timely manner

3. Activate maintenance agreements as and when service is scheduled or needed

4. Record equipment identification, location, status and updates in the CMDB

5. Document as-installed configurations

6. Document as-installed network schematics

7. Store all documentation in the Information Repository

3.8.1.3 Configuration (Hardware/System)

The Contractor shall:

1. Configure the hardware systems based on DHS Headquarters (HQ) standards and as specified on Task Orders by the Ordering Component (if consistent with DHS HQ provided configuration guidelines).

2. Document and provide updates to the as-built documentation to the DHS Project Manager and store those updates in the Information Repository.

3. Maintain configuration information resulting from maintenance or change implementations in the HCE Information Repository.

3.8.1.4 Incident and Problem Management

The Contractor shall:

1. Conduct automated monitoring of the operational status of each equipment item in the

HCE and implement a data feed to the enterprise network operations center and to other enterprise systems if required.

2. Provide real-time operational status of each system and respond if the system performance degrades (e.g., providing integration with existing solutions and implementation of timely reporting and automation notification (s) to enterprise stakeholders for outages, as approved by DHS).

3.8.1.5 Decommissioning

The Contractor shall provide decommissioning services. The Contractor shall:

1. Retain images, settings, and data from the device to be decommissioned until there is written permission to delete/dispose of the data from the Task Order COR not to exceed 60 days.

2. Describe the provisioning, de-provisioning, and decommissioning status.

3. Track status changes in the Asset Inventory Tracking System in the dashboard.

4. Power down, de-rack, degauss, and sanitize decommissioned physical equipment and/or

VMs in accordance with NIST Special Publication 800-88 Revision 1, Guidelines for Media Sanitization, DHS Information Security Policy, contract(s), and task order requirements.

5. Inventory, de-install, pack, and ship DHS Component GFE equipment, and prepare for pickup by either the customer for reuse or for disposal based on the customer’s requirement.

6. Maintain and provide a clear chain of custody for all equipment movement and hard disk disposal.

7. Certify equipment throughout the operations of the system while hosted in the HCE in accordance with DHS Security Authorization (formerly Certification and Accreditation) policies.

3.8.2 Level 2 Support Services

Level 2 Managed Services are additional services offered for systems, software, and applications residing in the HCE. Level 2 services may be elected individually. For equipment resident in physical locations, individual Level 2 services represent additional services in addition to the basic Level 1 services. However, Level 2 services are not contingent upon ordering Level 1 services (i.e., in the case of services performed for cloud resources). Individual Level 2 services may be elected for systems, services, or applications implemented or managed in cloud environments – whether on-premises private cloud or off-premises public cloud or Gov Cloud environments provided by CSPs.

3.8.2.1 Operating System Installation, Configuration, & Management

The Contractor shall perform all the necessary functions to install, configure, maintain, and manage operating systems.

3.8.2.2 Software Patch and Release Management

The Contractor shall provide software patch and release management and application services to protect software from potential threats, maintain software operational functionality, comply with DHS Configuration Management Processes and Procedures, including, but not limited to, patch testing, installation, fixture, and deployment, and the support and implementation of DHS Data Center Service Resource Management Process. The Contractor shall comply with DHS Configuration Management Processes and Procedures in all HCE environments including cloud and colocation. The Contractor shall apply operating system, application, solution and security patches as received from software vendors and approved by DHS and shall support and manage processes for waivers, exceptions, and Plans of Action and Milestones (POA&Ms) in a timely manner.

3.8.2.3 Storage Management

The Contractor shall provide storage services and utility offerings for managing customer-owned data storage assets residing in HCE physical locations and for managing cloud-based storage assets or services. All DHS data shall be stored in facilities located within the Continental United States. The Contractor shall ensure that DHS data placed in cloud storage is not backed up, stored, replicated, or transmitted in any manner outside the boundaries of the Continental United States and that CSPs ensure that only U.S. citizens provide support services for resources acquired for DHS. The Contractor shall ensure the operations of each application, storage device, or service as required on Task Orders.

The Contractor shall:

1. Install and configure physical storage devices to include SAN, network attached storage (NAS), and tape libraries.

2. Configure cloud storage resources including any management software and manage the allocation and retraction of storage in a dedicated and/or virtualized storage environment.

3. Monitor and manage storage capacity to keep utilization below the ceiling specified in Section 12.

4. Use automated tools to perform data compaction, compression, and migration tasks.

3.8.2.4 Backup and Restore

The Contractor shall perform backup, archival and restoration services to include incremental backups daily and full backups weekly and manage backup scripts to backup critical operating system and system data files including all system batch processes, as required by the respective system. Backup and restoration may be elected for systems, applications, and data stores residing in any HCE environment.