Draft - Performance Work Statement .pdf

U.S. Department of Homeland Security

Office of the Chief Information Officer

Performance Work Statement for

Data Center and Cloud Optimization Support Services

DRAFT

July 10, 2020

This page intentionally left blank

1 General The Department of Homeland Security (DHS) Data Center and Cloud Optimization (DCCO) acquisition plans to acquire information technology services to operate a data center, colocation sites, and provide cloud services, as part of the Hybrid Computing Environment (HCE). In addition, DHS will acquire associated professional services to automate, optimize, and modernize the HCE.

1.1 Background

The DHS currently locates its enterprise level computing environments in the enterprise data center known as Data Center 1 (DC1) located in Stennis MS. DC1 is located at the National Aeronautics and Space Administration (NASA) Stennis Space Center in Mississippi. This is a government‐owned, contractor-operated data center. NASA owns and manages the data center facility. DHS is a tenant within this facility and uses approximately 35,000 square feet of raised floor space in the data center. NASA provides power management, raised floor space, physical security, office space, environmental control, and fire suppression to support DHS IT needs.

The future DHS HCE, that will be managed by the Contactor receiving the award of this contract, will include the DC1 but will expand to include colocation environments, either leased or provided by DHS, where DHS chooses to locate IT infrastructure equipment. The HCE will also include cloud environments provided by multiple Cloud Service Providers (CSPs).

This requirement focuses on information technology services required to support the DHS HCE including DC1 support services not provided by NASA, infrastructure support services at colocation sites and in cloud implementations, as well as the migration, operation and maintenance of systems and applications among the data center, colocation, and cloud environments that constitute the HCE. For DC1, NASA coordinates the Federal Data Center Operations team for all facility support activities. DHS Federal staff act as the intermediary between NASA and the data center support services Contractor.

As of June 2020 the operating environment for DC1 can be characterized as follows:

• 14 DHS Components hosting systems in DC1

• 109 hosted customer FISMA systems

• 588 racks populated with DHS equipment

• 7,267 Government-furnished devices installed

• 1,425 Contractor-furnished devices installed The purpose of this requirement is to acquire Contractor support for the operation, maintenance, automation, optimization, and modernization of the DHS HCE. The HCE shall include unclassified and classified Information Technology (IT) infrastructure, applications, and data.

The DHS HCE is a hybrid IT environment including DC1, colocation centers and CSPs. The Contractor shall provide operation and maintenance of the physical and virtual servers in DC1 and colocation centers. The Contractor shall offer private cloud services in DC1 and colocation centers and commercial, public cloud, and Gov Cloud services from approved CSPs.

The Contractor shall provide professional services to support application, migration, and Operations and Maintenance (O&M) projects. The Contractor shall provide customer support including onboarding, training, and service desk.

DHS intends to operate the HCE as a high‐priority 24x7x365 computing environment supporting critical mission and business needs for the Department. This includes, but is not limited to, maintaining Service Level Agreements / Service Level Objectives (SLA/SLO) and having the appropriately cleared and qualified staff and support services necessary to maintain operations.

The HCE will offer a more efficient, responsive IT hosting environment that serves as the foundation to ensure continued operations in support of the DHS mission.

1.2 Scope

The scope of this requirement is to provide operations, hosting and professional services for the DHS HCE. The Contractor shall identify, provide and manage personnel, processes, tools, and reporting to facilitate transparency and decision‐making as well as to support the transition to the future state HCE. This requirement also includes the continuing support of current operations and services.

The Contractor’s responsibility within DC1 includes the maintenance, operation, and organization of the IT infrastructure technologies. Power, environmental controls, and facility modifications come under the purview of NASA Shared Services Center (SSC) NCCIPS (National Center for Critical Information Processing and Storage) management.

Services to be provided include but are not limited to those listed in the following table:

Hosting/Computing Services Application Services

Performance and Capacity Monitoring Network and Security Service

Data Storage Services Operation/Monitoring and Maintenance services

Data Archiving Disaster Recovery/Continuity Services/Reconstitution

Technical Guidance Classified Computing Services

Network and Security Infrastructure Program and Project Management

Cloud and Colocation Services Virtualized and Physical Compute Services

Professional Services supporting hybrid operating environments (e.g., Common Dashboard for Common Operating Picture)

Transitions Among Environments (e.g., Application Rationalization, Migration, etc.)

1.3 Objectives

The DCCO requirements will be the core support services to drive a more efficient, responsive hybrid IT environment that serves as the foundation for the management and integration of on-premises, colocation and cloud-based environments.

These support services must optimize and ensure continued DC1 operations, while implementing and managing the future state HCE in support of the DHS mission and, where appropriate, migrate infrastructure and applications within the HCE.

The objective of DHS’s DCCO support services is to expedite the transformation of IT capabilities from an asset‐based model to a service‐based, customer‐centric IT business model;

provide transparent operational expenditures; and reduce both capital expenditures and time‐to‐ delivery for new capabilities. To accelerate this transformation, DHS will continue existing initiatives associated with cloud optimization efforts as well as leverage new approaches. The business objectives for this requirement include:

1. Improved Total Cost of Ownership (TCO) and Cost Transparency ‐ DHS requires cost management and transparency to better understand how to reduce costs, improve efficiencies, and provide accountability. The reporting and management of costs should comprise DC1, cloud and colocation services, and enable feasible adjustments as new technical and management methods are implemented.

2. Modernization and optimization – Operations within the HCE should reflect state-of-the-art, optimized, automated, modern processes that rely to the maximum extent practicable upon automatic processes with the minimum of required human intervention.

3. Responsiveness ‐ DHS must improve its customer service approach to maximize the customer experience, improve trust, reduce the time to deliver, and enhance operational performance while fostering improved communication, innovation, and accountability.

4. Scalability ‐ The target hosting environment and supporting services must quickly scale and provide the elasticity to meet mission operations and surge capabilities. Additionally, the environment and services must adapt to the organic growth of operational and data needs.

5. Improved Reliability and Availability ‐ DHS needs to implement and manage an ecosystem of DC1, CSPs, and co‐location providers to ensure the Mean‐Time‐Between‐ Failures and the Mean‐Time‐To‐Repair are continually monitored and improved throughout the period of performance. This also includes the ability to provide disaster recovery and continuity services and shall apply to CSPs and services added to the HCE.

6. Continuous Security Monitoring and Mitigation ‐ The complexity of the target hybrid IT environment and the critical nature of the DHS mission requires the continuous monitoring and evaluation of systems, capabilities, interfaces, applications and data transactions to assess and mitigate threats to cybersecurity that may affect confidentiality, integrity and availability.

7. Security countermeasures are integrated from the beginning to protect critical data and maintain a Zero Trust model.

8. Continual Operational Assessment and Improvement ‐ Because of evolving mission, IT, and industry capabilities, DHS must continually analyze and deliver an environment that supports the optimization of service delivery. A continuous improvement methodology should bring gradual, ongoing improvement to products, services, and processes through constant review, measurement, and action. This methodology should feature the collection of performance metrics, analysis of the performance – both technical and administrative – reflected by those metrics, and lead to continuing performance improvements as processes approach greater levels of optimization.

9. Architectural Approach ‐ DHS must maintain an understanding of its current state, including a variety of architectures, configuration items, services, deployment models, and technologies. Also, DHS must identify opportunities and improvements to transition to a target state that maintains architectural principles to enhance interoperability, improve standards, reduce risks, improve security, and support emerging technologies and capabilities.

10. Service Governance ‐ A strong governance mechanism is needed to ensure consistent interpretation of policy, monitoring of DHS enterprise computing performance, adherence to policy requirements and addressing consumer and provider issues. Service governance will ensure alignment of DHS investments, policies, processes, and standards.

11. Simplified Management ‐ Improved communication through a data‐driven, transparent management approach that allows for clear focus and attainment of management and technical goals. This includes the management and total cost of ownership of asset and application lifecycles.

12. Continuity – Minimal to no disruption of operations during the hand-off from the existing service provider to the incoming Contractor and the seamless transfer of responsibility for on-going data center projects to the incoming Contractor without schedule delays or other disruption.

13. Visibility – DHS needs an efficient centralized mechanism to view, manage and optimize service inventory, usage, performance, security and cost across the HCE based on evolving mission needs.

1.4 Applicable Documents

See Appendix A for a full list of Applicable Documents.

1.5 Performance Requirements Summary

This contract includes a Performance Requirements Summary (PRS) in Section 10.0. The PRS plays an integral role in the administration of the contract. In addition to any applicable inspection clauses or other related terms and conditions contained in the contract, the PRS shall serve as a primary tool for inspection and acceptance of services as facilitated by the Contracting Officer’s Representative (COR). Evaluation of the Contractor’s overall performance shall be in accordance with the performance standards set forth in the PRS, and will be conducted by the COR. The PRS constitutes a material aspect of the contract and will not be changed or otherwise modified without prior written approval of the Contracting Officer.

2 Management Requirements and Tasks The Contractor shall support the requirements in this Section across the HCE. The support includes physical and virtual assets in the DC1 facility in Stennis, MS. It includes the addition of private cloud assets in DC1 and in colocation facilities. The support also includes the addition of public cloud and Gov Cloud assets from commercial CSPs. All these assets shall be considered as part of the HCE.

2.1 DHS Information Technology Infrastructure Management

The Contractor shall manage DHS IT infrastructure. As required the Contractor shall interface with DHS Data Center Operations Team to address any issue that requires modification to general facility services. The Contractor shall be responsible for providing operational best practices including but not limited to:

• Governance

• Facility Management and Interface Support

• Hybrid Computing Environment Management Automation

• General Support Systems

• Policies and Procedures

• Customer Dashboard and Repository

• Acquisition

• Colocation Services Resale

• Cloud Services Resale

• Network Communications LAN & WAN

• System Architecture / Engineering

• Contract Transition

• Additional Management Services

2.1.1 Governance

The Contractor shall operate under the umbrella of DHS governance and oversight processes that dictate interactions among DHS Headquarters, DHS Components, the Contractor Team, colocation providers, and CSPs, and all other DHS contractors addressing interrelated requirements. The Contractor shall provide DHS with the capability to exert appropriate management, governance, oversight and visibility of all assets and services within the HCE.

The DHS governance model is led and directed by DHS management. The Contractor shall support and adhere to the DHS Infrastructure Change Control Board (ICCB) governance process.

The ICCB is responsible for managing the risk assessment and decision making with respect to all changes proposed to be implemented throughout the entire DHS IT infrastructure to include infrastructure installed in any HCE environment supporting DHS. The Contractor shall also interact with and cooperate with the appropriate DHS security and network operations centers and directed by DHS in response to security alerts and managing and recovering from security incidents.

The Contractor shall obtain Government approval for all solutions proposed to satisfy requirements and for all purchases required to fulfill those requirements.

2.1.2 Facility Management Interface and Support

Contractor shall collaborate with DHS who will work with the NASA Team on potential changes to infrastructure and work with the team to recommend activities to maximize efficiencies in the center. Contractor shall work with the Data Center Operations Team and COR when any reconfiguration or repositioning is considered to ensure all appropriate approvals are in place before proceeding.

2.1.3 Reserved

2.1.4 Financial Management

The Contractor shall Implement a Financial Management solution based on Information Technology Infrastructure Library (ITIL) to provide cost-effective management of HCE resources in accordance with awarded Task Orders. The Contractor shall execute processes in three main Financial Management process areas: Budgeting, Accounting, and Charging. The Contractor shall make all data/information available via a centralized dashboard and access should be restricted to authorized personnel.

The Contractor shall provide a dashboard that provides a complete depiction of spending. The dashboard shall show spending per reporting period, cost metrics, and cloud spending per reporting period with period over period cost trends.

2.1.5 Hybrid Computing Environment Management Automation

The Contractor shall:

1. Provide the management software tools to manage the HCE enterprise architecture and automate administrative, monitoring, and technical maintenance processes.

2. Provide computing infrastructure support necessary to support the selected management software.

3. Ensure compliance with the DHS Security Policies and posture when implementing and utilizing their equipment for the Management tools function.

4. Include the Contractor Provided Equipment in the (CMDB) Configuration Management Database.

2.1.6 Policies and Procedures

The Contractor shall create, update, and maintain policies and procedures that govern its operation of the HCE. The Contractor shall adhere to the DHS guidance and policy process. The Contractor shall create formal documentation for all procedures, obtain DHS approval for those procedures, retain that documentation in a repository and shall provide DHS access to the information repository. The Contractor shall deliver an initial set of policies and procedures.

2.1.7 General Support Systems and Major Applications

The Contractor shall select and implement those information systems or applications that the Contractor will use to manage and operate the HCE. DHS will provide no systems or applications for the management and operation of the HCE. The Contractor shall be responsible for the selection, procurement, hosting, operation, security, and maintenance of selected General Support Systems and Major Applications. The Contractor shall be responsible for populating various reports and the centralized dashboard with operational data extracted from the Contractor’s various general support and major applications system(s). Functions to be performed by the General Support or Major Application Systems may include but are not limited to the following examples:

• Service desk operation

• System Security Plans (SSP) for the GSS of the Data Center

• Ticketing

• Workflow management

• Inventory control

• Configuration control

• Software installation and patch management

• System monitoring

• Network monitoring & traffic analysis

• Vulnerability detection

• Capacity management

• Financial management and billing

• Trend analysis

2.1.8 Customer Dashboard and Repository

The Contractor shall establish and maintain a centralized, integrated dashboard and information repository with real-time, on-demand, 24x7x365 access by DHS and Components. Access shall be available to authorized DHS users and groups, with view, modify and download privileges.

The Contractor shall use the dashboard to track and manage assets and services provisioned and consumed in the HCE, including DC1, colocation centers, and CSPs. The dashboard shall enable DHS to manage, use, report, audit, and track assets and services. The dashboard shall include inventory and management of physical and virtual assets and cloud services, usage and performance metrics for all assets and services, security metrics such as cyber threats and vulnerabilities, and metrics for billing and accounting. The dashboard shall capture and report actual expenditures to support financial, budget, audit and benchmarking activities, and shall provide visibility about asset tagging of resources.

The Contractor shall establish a centralized knowledge management repository, in conjunction with the dashboard, to share deliverables and related documentation about the HCE. The Contractor shall populate the repository with information and documentation during the Transition-in Period. The Contractor shall maintain current versions during the contract performance period and shall provide final versions during the Transition-Out period. The Contractor shall provide DHS with access to the information throughout the period of performance.

The repository shall record installation dates, maintenance agreements, maintenance performed, configuration management data, capacity forecasts, status reports, software licenses, lessons learned, operations run books, architectural drawings and diagrams, and standard operating procedures (SOPs) for all assets and services in the HCE. The repository shall include the results of cost-benefit analyses and Return on Investment studies as required by DHS.

The Contractor shall work with DHS to modify the dashboard, as requested. The Contractor shall perform periodic quality assurance reviews on the dashboard data to ensure accuracy and necessary modification. The Contractor shall ensure DHS can collect, analyze and synthesize the data (e.g., usage, security, performance) on the services being provided. The Contractor shall also enable DHS to collect, analyze and synthesize cost data, including non-labor or professional services, to ensure accurate billing, configuration, and cost transparency.

2.1.8.1 Real Time Resource Consumption and Cost Tracking

The Contractor shall provide real-time, on-demand 24x7x365 integrated dashboard access to DHS for managing and tracking usage, performance, security, tagging, and associated costs for all assets and services in the HCE.

The dashboard shall enable DHS to use, track, report and audit all assets and services. The dashboard shall enable DHS to set and manage approval workloads, quotas, and thresholds for asset and service usage across the HCE for organizational and project accounts. The dashboard shall also enable DHS to provision, deploy and de-provision assets and services, manually or automatically, and shall provide DHS with the ongoing capability to set thresholds and limits to usage, deployment, and provisioning of resources.

The dashboard shall allow DHS to separate usage costs into billable groups for reporting purposes. The dashboard shall provide DHS with a summary of the assets or services in a designated group, as defined by Task Orders. The dashboard shall include environment-specific information, such as billing points of contact, resource names, length of time the resource has been running, total resource costs, consumption and cost forecasts, current and total monthly burn rates, and related metrics.

The Contractor shall perform periodic quality assurance reviews of dashboard data and summaries to ensure accuracy and completeness. The Contractor shall enable DHS to collect, download, and analyze usage data and costs for all assets and services whether non-labor or professional services to ensure accurate billing, configuration, and cost transparency.

2.1.8.2 Capacity Management

The Contractor shall monitor the capacity utilization of HCE resources and provide real-time dashboard displays and periodic reports to DHS regarding capacity utilization. The Contractor shall also develop future capacity requirement projections for each type of resource provided within the HCE. The contractor dashboard shall integrate with DHS enterprise-level dashboards, upload data to and accept downloads of data from those dashboards.

2.1.9 Acquisition

Requirements may demand the acquisition of computing, communications, and data storage assets to host and operate systems applications that DHS Components want to implement and operate. The Contractor shall determine whether resources are available within the HCE to meet the requirement, allocate resources if available, or procure additional resources as necessary to satisfy the requirement.

2.1.10 Colocation Services

The Contractor shall have the capability to provide colocation services to DHS customers. The Contractor shall identify colocation services that meet DHS customers’ technical, security, and business objectives as defined in Task Orders and subsequently execute agreements with the DHS-selected colocation facility provider.

2.1.11 Cloud Services

The Contractor shall have the capability to provide cloud services to DHS customers. The Contractor shall identify cloud services that meet DHS customers’ technical, security, and business objectives as defined in Task Orders and subsequently acquire, integrate, and manage services from the FedRAMP certified CSPs.

2.1.12 Intermediary for Software Licenses

The Contractor shall provide software licenses from software vendors for applications that are required for the HCE, as requested by DHS. The Contractor shall ensure DHS has visibility, via the dashboard, into the licenses of any software product acquired from the vendors.

2.1.13 Network Communications LAN & WAN

The Contractor shall provide separated Local Area Networks (LAN) within HCE facilities that connect to the DHS OneNet Wide Area Network (WAN). DHS will provide all WAN connectivity. The Contractor shall be responsible for the local networking within all HCE facilities and environments. The Contractor shall acquire and install all necessary LAN devices (routers, switches, cabling, etc.) and install and maintain all components of the HCE LAN. The Contractor shall design the LAN architecture and install, maintain, and operate the network infrastructure such that the Contractor provides dual LAN ports to each hosted server for redundancy. The Contractor shall configure the LAN architecture such that the loss of an individual component does not result in the loss of connectivity.

The Contractor shall maintain network architecture drawings and descriptions of the as-designed and as-installed architecture. The Contractor shall monitor and administer network traffic, forecast bandwidth requirements associated with service requests, and implement network modifications to accommodate demand. The Contractor shall optimize all network traffic between HCE and other DHS facilities, colocation facilities, and CSPs across WAN replication circuits with redundant architectures capable of supporting up to 200,000 TCP connections.

The Contractor shall support the Zero Trust model for the design and implementation of the HCE network, to include trust policies, micro segmentation, software defined network infrastructure, and multi-factor authentication.

The Contractor shall implement and maintain a dashboard that displays real time network performance metrics and posts notices regarding unplanned network outages as well as any planned service disruptions.

2.1.14 System Architecture / Engineering

The Contractor shall provide system architecture and engineering services to include engineering assessments, analyses and studies. The specifics of these activities will be documented and awarded in Task Orders.

2.1.15 Contract Transition

The Contractor shall assume responsibility for the management and operation from the existing contractor and shall ensure that all operations continue without disruption during the contract transition.

2.1.15.1 Transition-In

The Contractor shall develop and implement a plan to accomplish the transition of operations from the current service provider to the DCCO Contractor within 120 days from award without any disruption of service. At a minimum, the Transition shall include:

1. Establish a Contractor Transition Team within 5 business days of contract award.

2. Conduct and complete a site survey and assessment within 15 business days of award.

3. Create a Draft Incoming Transition Plan within 30 business days of contract award for assuming operational and managerial control of the HCE and a final Transition Plan within 90 business days of award.

4. Conduct an inventory of all hardware and software assets/licenses and cloud resources within 60 days of award.

2.1.15.2 Transition-Out

The Contractor shall:

1. Create, deliver and execute an Exit Transition Plan not less than 90 days prior to the end of the period of performance or last exercised contract option, whichever is later, or when requested by the COR.

2. Deliver final updates on all policies and procedures governing current operations within the HCE not less than 90 days prior to the end of the period of performance or last exercised option period.

3. Deliver a current, final outgoing inventory of all equipment (i.e., hardware) and applications (i.e., software) within the HCE – in all physical, virtual, and cloud environments, not less than 90 days prior to the end of the period of performance or last exercised option period. The final inventory shall indicate whether the CFE will be transferred to DHS or be removed.

4. Remove all CFE not being transferred to DHS from DC1 or any colocation facility.

2.1.15.3 Task Order Transition Exit

Shall be specified at the Task Order Level.

2.2 Security Services – HCE Level

The Contractor shall monitor the physical and virtual security of HCE components, and coordinate with DHS as appropriate, by performing the following work activities:

2.2.1 Network Operations Management

The Contractor shall maintain and manage the internal HCE networks on a 24x7x365 basis. The Contractor shall implement and sustain a network management program that ensures uninterrupted network services throughout the HCE. The Contractor shall monitor HCE network traffic and maintain network capacity required by HCE processing. The Contractor shall ensure that all network-based management software operates successfully and that all software updates and security patches are successfully applied. The Contractor shall identify, respond to, and resolve any network traffic outages and provide all necessary touch labor necessary to maintain network operations.

2.2.2 Security Operations Management

The Contractor shall provide security management services for the HCE and shall also provide specific security-related services in accordance with awarded Task Orders. The HCE contains resources, including firewalls, gateways and proxy servers, to provide endpoint security. The security services include maintenance of the Zero Trust model through identity and access management (role-based or attribute-based) with multifactor authentication to enable authorized user access to applications and data sources. The services also include encryption to protect data at rest and in transit.

The Contractor shall implement the security controls required for DHS servers. The Contractor shall conduct security testing to verify that the servers are protected against potential cyber threats. The Contractor shall also complete the process required to achieve an ATO. The Contractor shall develop a plan to resolve open issues affecting the ATO.

The Contractor shall track and collect threat and vulnerability data. The Contractor shall report threats and vulnerabilities in the centralized dashboard for visibility and execute automated mitigation steps to respond to and recover from potential attacks.

The Contractor shall implement security mitigations as recommended by the Enterprise Security Operations Center (SOC) and implement measures consistent with ensuring HCE security. The Contractor shall monitor endpoints within the HCE to detect any anomalous activity among endpoints, applications, and databases and to ensure secure user and application traffic to and from the data center. The Contractor shall report any detected anomalies to the Enterprise SOC in accordance with standard reporting instructions and assist the Enterprise SOC or Component SOCs as required in responding to security incidents and in the recovery from such incidents.

The Contractor shall provide touch labor within the HCE associated with incident response and recovery. The Contractor shall control change within the HCE through a change management process to ensure the completeness and integrity of all changes implemented within the HCE shall be integrated with the enterprise change management solutions.

Personnel security services are addressed in Section 5.5.2.

2.2.2.1 Vulnerability Assessments

The Contractor shall implement and deploy the tools, toolsets, and staff to support, operate, and maintain vulnerability assessment services in the HCE for systems and services delivered and operated by the Contractor, develop a Vulnerability Assessment Plan and conduct routine vulnerability scans to determine the vulnerabilities and associated risks to operating systems, software and hardware and the infrastructure architecture, and perform network-based vulnerability scans and assessments.

2.2.2.2 Intrusion Detection and Prevention Systems (IDS/IPS)

The Contractor shall deploy, operate and maintain Intrusion Detection System(s) (IDS) to include host-based and network-based detection systems and Intrusion Prevention System(s) (IPS) to include host-based and network-based prevention systems for networks and systems resident within the HCE.

2.2.2.3 Firewall Management

The Contractor shall provide firewall software and hardware components and implement and deploy the tools, toolsets, and staff to support, operate, and manage and maintain firewalls for systems and services delivered and operated by the Contractor within the HCE.

2.2.2.4 Anti-Virus Management

The Contractor shall provide anti-virus protection for the HCE and ensure that signatures and databases are the latest approved and tested versions. In certain circumstances, DHS may provide anti-virus protection that the Contractor shall operate and maintain.

2.2.3 Equipment Access and Control

The Contractor shall:

1. Ensure that all delivery and removal of equipment within the data center facility is authorized by DHS personnel.

2. Ensure that Contractor-issued and owned electronic devices meet DHS configuration guidance, otherwise the equipment shall not be permitted in the data center.

3. Prevent personally-owned electronic devices (e.g., laptops, portable storage media, cell phones, etc.) from being taken into restricted areas within the data center.

4. Scan all electronic devices (e.g., personal devices, government-issued devices, and

Contractor-furnished devices) to identify vulnerabilities, verify the existence of up-to-date virus definitions, and ensure compliance with DHS configuration and policy guidance.

5. Dispose of or destroy media containing sensitive information in accordance with NIST Special Publication 800-88 Revision 1, Guidelines for Media Sanitization. Inventory tracking status shall be updated in the dashboard.

6. Allow the DHS-appointed Communications Security (COMSEC) custodian to execute the responsibilities as outlined in DHS National Security Systems Handbook (4300B) -

COMSEC.

2.2.4 Authority to Operate

All systems connected to DHS networks require an Authority to Operate (ATO) issued by a responsible Authorizing Official (AO) who has the authority to accept any risks posed by a candidate system. The Contractor shall:

1. Obtain and maintain authorization for connection to DHS networks for each CFE system that it operates, when approved to use the CFE.

2. Obtain ATO for each approved CFE system that the Contractor connects to DHS networks following the guidance provided in the DHS System Security Authorization Guide.

a. Generate all artifacts required by the Security Authorization Guide.

b. Develop and execute Plans of Action and Milestones (POA&M) for remediating security shortcomings and risks in preparation for requesting

ATO.

3. Maintain authorization for operating each CFE system by:

a. Renewing the ATO at three-year intervals OR

b. Enrolling systems that qualify in the DHS Ongoing Authorization (OA)

Program that monitors security controls on a continuing basis.

2.2.5 Risk Assessment

The Contractor shall create a Risk Assessment Plan and conduct Risk Assessments (RA) for systems and services delivered and operated by the Contractor after completing a NIST 800-53 evaluation, Contingency Plan Testing, and assisting the OCISO in conducting Security Tests and Evaluation (ST&E). Risk Assessments shall identify threats and vulnerabilities, assess the impacts of the threats, evaluate in-place countermeasures, and identify additional countermeasures necessary to ensure an acceptable level of security. Risk Assessments shall also address the cost and schedule of mitigation activities. The Contractor shall update system and service Risk Assessments annually.

2.2.6 Common Controls

The Contractor shall:

1. Enter security controls for each of the systems that the Contractor installs and operates and for the infrastructure supporting DC1 private cloud service offerings in the DHS Enterprise Cyber Risk Management tool, currently Xacta, and update those controls annually. The controls shall consist of controls provided by Federal Information Security Management Act FISMA systems as well as physical controls inherited or provided by NASA National Center for Critical Information Processing and Storage (NCCIPS) and by Contractor processes and services such as Change Management and anti-virus deployment.

2. Provide inheritable controls to be used by tenants for their security documentation for systems and applications that those tenants implement on the HCE and enter into the DHS Enterprise Cyber Risk Management tool. These tenant systems and application will inherit the controls entered by the Contractor in the DHS Enterprise Cyber Risk Management tool for the private cloud infrastructure.

3. Provide additional security controls for systems and services delivered and operated by the Contractor.

2.2.7 Access Management

The Contractor shall ensure DHS is provided full administrative and management user access to all approved service offerings. In addition to user access, the Contractor shall enable application access to the HCE. This shall include allowing applications to use the Application Programming Interfaces (APIs) and Software Development Kits (SDKs) available from the CSPs, to directly interact with CSP services.

The Contractor shall ensure DHS can implement permissions and restrictions, as required, for authorized users to access CSP marketplaces. The Contractor shall ensure the access permits authorized DHS users to download, install and run approved marketplace applications in the CSP environments.

3 Operations Support Requirements and Tasks

3.1 Operations Services

Following commercial and federal standards and best practices the Contractor shall provide HCE operations services on a 24x7x365 basis. The Contractor shall provide real-time system status to DHS customers and monthly service level reports. Contractor shall develop and maintain SOPs governing infrastructure operations to align with relevant SLAs and shall provide access to the documentation.

3.2 Service Desk

The Contractor shall maintain and operate a 24x7x365 Service Desk for infrastructure and applications residing within the HCE. The Service Desk shall:

1. Implement and maintain an Information Technology Service Management (ITSM) System to track and manage service requests from receipt through closure, problem reports to closure, and provides statistics for service desk performance reporting and analysis.

2. Accept and process service requests received through:

a. Telephone contact (available 24x7x365)

b. Email submission

c. Centralized dashboard

3. Track ongoing infrastructure operation metrics and Service Desk performance metrics in the dashboard and monthly in the formal monthly Service Level Agreement Level of Service Report.

4. Provide full Tier 1, 2, and 3 help desk support for applications that the Contractor implements and operates in on-premises locations, colocation environments, approved CSP platforms and environments, and internal private cloud infrastructure, provided and supported by the Contractor.

5. Provide Tier 2 service desk support for Component-level help desks that provide the first level of support for systems and applications hosted within the HCE infrastructure.

6. The Contractor shall provide documented root cause analysis on “problem cases” and shall provide initial and final analysis reports to DHS and shall maintain the repository and provide role-based access to the information. The Contractor shall describe what led to the creation of the problem (root cause) and describe the solution to be implemented.

As needed, the Contractor shall make root cause analysis discussion part of weekly and monthly reporting. If requested by the COR, the Contractor shall host a specific meeting to discuss problems, analysis and solutions. The Contractor shall lead the agenda and discussion. Part of the agenda will be dedicated to review the history of the incident(s) that lead to the creation of a “problem” case and the root-cause analysis that ensued. Part of the agenda will be dedicated to describing the “solution” to be implemented. The solution may have technical, procedural, and organizational actions as part of the comprehensive fix. And, changes to Contractor infrastructure services may only be part of the comprehensive solution. Contractor shall document attendance and minutes and make them available in electronic format.

3.3 Inventory Control and Asset Management

The Contractor shall use the dashboard to manage the inventory of all physical assets, virtual assets, colocation assets, and cloud services in the HCE. The Contractor shall:

1. Record receipt of all incoming items within an asset tracking and inventory management system.

2. Record the location of all hardware and software items - including both operating system and application level software - within the IT infrastructure in an asset management system and track changes in location for each item.

3. Maintain a Configuration Management Database (CMDB) with the complete hardware and software asset inventory in the dashboard and provide integration and access to enterprise CMDB solution(s).

4. Provide access to all information indicating location and status of each inventory item as well as maintenance and licensing agreements and expiration dates for all inventory items.

5. Deliver a complete, fully validated inventory of all equipment, applications and system software within the HCE not less frequently than quarterly.

3.4 Change Management

The Contractor shall follow and recommend enhancements to established DHS processes and procedures for adequate change management control and implementation. Change management ensures that standardized methods and procedures are used for the efficient and prompt handling of all changes, to minimize the impact of changes upon any related service. This shall be achieved by facilitating efficient and prompt handling of all changes and by maintaining the proper balance between the need for change and determining the impact of changes.

The change processes shall manage the following components that may change in fulfilment of Task Orders:

• Hardware

• System software

• Application software

• Configuration

All documentation and procedures associated with the running, support and maintenance of live systems.

3.5 Customer Satisfaction

Customer Satisfaction includes all activities required to ensure DHS and the Component satisfaction with Contractor performance. Customer satisfaction shall be solicited from each Task Order owner. The Contractor shall assign a Component advocate to assist in planning and implementation. Contractor shall design and implement the use of surveys to capture and document customer satisfaction. The Contractor shall develop corrective action plans that are shown in the surveys.

3.6 Quality Control

The Contractor shall provide a Quality Management Plan (QMP) describing the standards, processes and procedures used to support the consistent delivery of high-quality, professional products and services provided in support of a 24x7x365 HCE. The QMP shall be based on the Quality Assurance Surveillance Plan (QASP) provided by the Government. The quality assurance process establishes the authority of the Quality Control function, quality assurance standards, procedures, policies, and monitoring and evaluation processes to determine quality in relation to established standards. Quality Control provides standards against which the quality of the product/service being provided can be measured. Quality Control activities concentrate on the prevention of problems through the continuous improvement of processes. The HCE QMP shall serve as a guide for Quality Control activities and shall fit the services being provided to DHS and their specific activities.

The Contractor shall implement processes and procedures to meet the Government-provided SLAs to include but not limited to; metrics, target values, penalties, and incentives as mutually agreed to by DHS. SLAs shall provide all the detail necessary for calculation and SLA interpretation; for example, SLA description, assumptions, exceptions, data sources, calculation methods weighting, and thresholds. The Contractor shall not diminish the service offerings from the CSPs to a level lower than the published commercial cloud service, or standard service level agreement (SLA), unless otherwise specified and approved by DHS.

The Contractor shall propose, receive DHS approval and implement, Incentive/Disincentive Plan(s) providing the incentive/disincentive approach for no cost, earn back, frequency, process for determining value, of the invoice pool and other language required to identify the approach for earning incentives/disincentives.

3.6.1 Application Quality Control

The Contractor shall provide Application Quality Control services in order to maintain application integrity. Application Quality Control shall meet the quality standards as set forth in the Task Orders.

The Contractor shall provide support to System Assurance services. System assurance provides certifiable operational capability in a 24x7x365 environment.

In a System Assurance project, the Contractor shall address operational capabilities including but not limited to system backups, disaster recovery, and continuity of operations to ensure data center operations are performed in accordance with DHS requirements and ITIL processes.

The Contractor shall manage Quality Control according to industry best practices, for example, with processes and staff that are functionally and administratively independent from the product lines and the services delivered to the DHS customer. The Contractor shall detect and report quality problems per the SLAs and maintain a Quality Control Plan.

3.6.2 Independent Verification and Validation Support

As directed by the COR, the Contractor shall provide cooperation and support to any Independent Verification and Validation (IV&V) project performed by or commissioned by DHS or the Department’s designated representative.

3.7 Availability Management

The Contractor shall ensure all CSPs provide a minimum availability of 99.9% for each service (e.g., virtual machine, object storage, virtual private cloud), unless otherwise published by the CSP. The Contractor shall ensure each service provided to DHS, meets or exceeds, the commercially-advertised level or published SLA. The Contractor shall ensure that services launched by DHS in multiple cloud zones or regions continue to operate and remain available when any of the CSP’s data centers is offline or unavailable. The Contractor shall ensure services acquired for DHS maintain the capability for 100% uptime redundancy.

The Contractor shall:

1. Develop processes and procedures for adequate system availability as directed by the Service Level Agreements. Availability Management shall include activities to ensure that appropriate availability metrics are actively met, on a per application (system) basis.

Depending on the criticality of the system, different levels of availability for any system may be required.

2. Conduct maintenance requiring system or application downtime during time windows specified by system owners. Any maintenance required on systems shall be scheduled during the maintenance window. If the required maintenance eclipses the scheduled maintenance, even if it is still in the maintenance window, the time required to bring the system online shall be counted against the availability metric. For example, a weekly maintenance window is allotted from 1:00 – 3:00 AM every Sunday. If the scheduled maintenance is to occur from 1:00 – 2:00 AM and the system is not live until 2:15 AM, then 15 minutes shall be counted against the availability metric.

3.8 Operations and Maintenance Services

The Contractor shall follow the industry-recognized ITSM approach and the associated ITIL practices to provide lifecycle O&M support for the HCE. This includes installing and configuring hardware and system software on physical equipment, system software on cloud-based IaaS services, integrating, testing and securing physical and cloud resources, and conducting ongoing maintenance. The ongoing maintenance shall include asset management, change management, problem and incident management, continuity of operations, continuous monitoring, and decommissioning.

The Contractor shall apply a similar approach to providing lifecycle O&M support for the customer applications in the HCE as required on Task Orders. The Contractor shall also apply this lifecycle approach at the required classification levels: unclassified, Sensitive But Unclassified (SBU), Secret, and Top Secret (TS), including Secure Compartmented Information

(SCI).

The Contractor shall provide tiered service levels for GFE and CFE.

• Level 1 services for all compute, storage, database, networking, and network hardware in the CSP environments, data center location, and colocation centers.

• Level 2 services for all system software, including operating systems, database management systems, data backup systems and network software, in the HCE. System software also includes tools for managing and monitoring hardware and software components (e.g., ServiceNow), problems and incidents (e.g., Remedy), and resource management (e.g., Cloud Management Platforms).

• Level 3 services for all customer applications across the HCE. This includes installation, monitoring, integration, testing, migration, maintenance, user training and security authorization support for all environments.

The Contractor shall document architectural and environmental descriptions of the HCE and develop and maintain associated operations manuals to optimize use by DHS.

3.8.1 Basic Level Service (Level 1)

Basic Level Service (Level 1) is a hosting service offered to DHS Components that desire to place physical assets within the HCE. Level 1 services include hardware maintenance and network monitoring for equipment. This is the minimum level of service that is provided for all Component-owned and data center owned assets residing within the HCE–. The Contractor shall ensure that equipment is installed in DC1 or in colocation facilities as appropriate and that the equipment is brought up to an operational state; the Contractor shall also provide O&M support.

Basic Level Service provides network connectivity from the servers to the WAN for all systems hosted in the environment. The Contractor shall provide personnel, processes, and technology to support hosting services for DHS systems and applications.

3.8.1.1 Installation

The Contractor shall perform all tasks necessary to properly install equipment to ensure all items are installed in the state as required by Task Orders.