DCSS_PWS_.docx

Performance Work Statement (PWS) DCSS (draft)

Contents

SECTION 1 – SCOPE AND BACKGROUND OVERVIEW

1.1 Introduction

This is a single award to a Contractor to provide Data Center Support Services (DCSS) and related services for all Divisions and Offices of the U.S. Customs and Border Protection (CBP) both in Washington, DC and its regional locations and data centers.

General Description of Services Provide comprehensive Data Center Support Services (DCSS) in support of CBP Office of Information Technology (OIT) in the area of Enterprise Operations, Enterprise Infrastructure, Enterprise Architecture, and Common Services. Support shall span across all locations, to include CBP Headquarter Offices in Washington, DC, CBP Data Centers, the eleven Regional Offices, and in any location where CBP operates in the future.

Limits of Services This is not a Personal Services Contract.

Background The mission of the Office of Information and Technology (OIT) is to provide U.S. Customs and Border Protection (CBP) with information, services, and technology solutions to secure the border, prevent the entry of terrorists or terrorist weapons, and to ensure the efficient flow of lawful people across U.S. borders. The OIT provides the U.S. Department of Homeland Security (DHS) and CBP with a 24x7x365 secure, stable and high-performance information technology infrastructure, scientific solutions, and forensic services.

The OIT has the responsibility for identifying and evaluating new technologies for applications in support of CBP business processes. A new era of economic downturn and terrorist threats require systems that can support the future vision of a unified, integrated, intelligence-driven agency to protect the United States against all threats against us. In order to be successful, the new Government and Contractor team must identify and acquire highly-skilled professionals from across industry that are capable of leading the technological revolution currently underway within OIT systems applications without abandoning and losing all existing critical corporate knowledge.

Needs CBP requires Data Center Support Services (DCSS), to provide the OIT with support in all aspects of information technology. These services primarily include, but are not limited to, the management and operations of all OIT IT Infrastructure, support within the Enterprise Operations Center (EOC), End User Computing, Asset Management, as well as Infrastructure Support for Engineering solutions and ongoing Operations.

Objectives The Contractor shall provide OIT through the EDME Directorate with Infrastructure Support Services in the following areas described in detail in Section 5.

With information technology pervading all aspects of its business, CBP depends on Contractor support for problem resolution and support services to meet its business needs. CBP expects the Contractor to deliver optimal daily performance of CBP IT operations and take day-to-day responsibility for CBP IT operations, achieving the minimum performance objectives on a daily basis, and taking a proactive approach with CBP in improving the operations and efficiencies of CBP IT environment.

The Contractor shall perform functions that are operational in nature, and shall not be expected or permitted to perform any inherently government functions.

1.1.1 Scope

The Office of Information Technology (OIT) provides CBP with support in all aspects of information technology. OIT has overall management responsibility for IT program including, but not limited to, infrastructure operations and engineering, user support, and enterprise architecture.

CBP data center processing is performed at multiple data centers within the continental United States. Data centers currently providing services to CBP include the following - CBP National Data Center (NDC) Complex located in Springfield, Virginia; the Stennis Data Center (DC1) at Space Center in Mississippi (supports multiple DHS Agencies, including CBP); the DHS Data Center 2 (DC2) in Clarksville, Virginia (supports multiple DHS Agencies, including CBP); the Finance Center located in Indianapolis, Indiana; and other locations throughout the Washington, D.C. metropolitan area. Additionally, CBP utilizes public cloud service providers (e.g., Amazon Web Services), in which the contractor shall support.

The Contractor shall respond to current requirements and proactively anticipate new requirements. The Contractor shall work with CBP to assess the impact of new requirements on CBP operating environment. The Contractor shall handle requirements for new services through CBP Change Control process and shall adjust services accordingly.

The Contractor shall transition the current CBP IT Infrastructure Support to the contractor's Support while ensuring the continuity of operations at each CBP location. The Contractor shall minimize disruption to the existing operations, optimize the use of existing assets, and improve customer satisfaction.

1.2 Service Levels

The Contractor shall develop and maintain effective service levels as defined in <update with section>. The Contractor shall measure and report on the Service Levels as defined in <update with section>. The Contractor shall develop and implement procedures to identify, prevent, and ensure non-recurrence of defective services. The Contractor’s quality control program is the means by which the Contractor ensures that work complies with the requirement of the contract.

Security Requirements The security classification for work performed under this Contract is Unclassified. The documents and information that will be reviewed and produced are non-public and sensitive in nature and shall be protected from unauthorized disclosure. Work on this Contract requires that personnel may have access to Privacy Act information. Personnel shall adhere to the Privacy Act, 5 U.S.C. Section 552a and applicable agency rules and regulations.

Physical Security The Contractor shall be responsible for safeguarding all government equipment, information and property provided for contractor use.

Key Control The Contractor shall establish and implement methods of making sure all keys/key cards issued to the Contractor by the Government are not lost or misplaced and are not used by unauthorized persons. NOTE: All references to keys include key cards. No keys issued to the Contractor by the Government shall be duplicated. The Contractor shall develop procedures covering key control that shall be included in the Quality Control Plan. Such procedures shall include turn-in of any issued keys by personnel who no longer require access to locked areas. The Contractor shall immediately report any occurrences of lost or duplicate keys/key cards to the Contracting Officer.

In the event keys, other than master keys, are lost or duplicated, the Contractor shall, upon direction of the Contracting Officer, re-key or replace the affected lock or locks; however, the Government, at its option, may replace the affected lock or locks or perform re-keying. When the replacement of locks or re-keying is performed by the Government, the total cost of re-keying or the replacement of the lock or locks shall be deducted from the monthly payment due the Contractor. In the event a master key is lost or duplicated, all locks and keys for that system shall be replaced by the Government and the total cost deducted from the monthly payment due the Contractor.

The Contractor shall prohibit the use of Government issued keys/key cards by any persons other than the Contractor’s employees. The Contractor shall prohibit the opening of locked areas by Contractor employees to permit entrance of persons other than Contractor employees engaged in the performance of assigned work in those areas, or personnel authorized entrance by the Contracting Officer.

Staff Controls Using CBP established procedures, the Contractor shall report weekly the staff on board and changes in staff working under the contract.

Quality Assurance/Government Surveillance The government shall evaluate the Contractor’s performance under this contract in accordance with a Quality Assurance Surveillance Plan (QASP). This plan sets forth specific surveillance methods the Government will perform to ensure that the Contractor has performed in accordance with the performance standards. The QASP is authored by the government upon acceptance of the Contractor’s Quality Control Plan and is shared with the Contractor.

Post Award Conference/Periodic Progress Meetings The Contractor agrees to attend any post award conference convened by the contracting activity or contract administration office in accordance with Federal Acquisition Regulation Subpart 42.5. The Contracting Officer, Contracting Officers Representative (COR), and other Government personnel, as appropriate, may meet periodically with the Contractor to review the Contractor's performance. At these meetings the Contracting Officer will apprise the Contractor of how the Government views the Contractor's performance and the Contractor shall apprise the Government of problems, if any, being experienced. Appropriate action shall be taken to resolve outstanding issues. These meetings shall be at no additional cost to the government.

Transition In and Transition Out The Contractor shall provide a comprehensive Transition In Plan facilitating a smooth transfer of responsibility from the current provider of Infrastructure Support Services.

The Contractor's transition plan shall include the following at a minimum:

· A comprehensive communications plan to inform CBP OIT and other management of transition objectives and on-going status and key deliverables identification of all in-flight project and the underlying technical requirements to keep the projects on schedule

· Identification of any change management barriers that could complicate the transition and remedial plans to effectively overcome the barriers

· Risks and risk mitigation strategies associated with the change in contractors

· Plan for knowledge transfer

· Define the general means for measuring the on-going progress against the Transition Plan including, measurement criteria and tools; milestone dates and deliverables

· Recommendations for improving CBP IT infrastructure architecture support

· Review, evaluation, and transition of current CBP hardware, software, data, and workflow

· Orientation phase and program to introduce CBP personnel, programs, and users to the Contractor's team, tools, methodologies, and business processes

The Contractor shall adapt its Transition Plan as required to address any changes resulting from additional information and due diligence.

Successful transition will be measured by the seamless transfer of responsibility from the current providers of support services to the new contractor, without impact to end users, and while meeting agreed upon service levels.

The Contractor shall provide a Transition Out Plan 120 days prior to the end of each Period of Performance that shall identify transition support activities for all transition efforts for follow-on requirements to minimize disruption of services. The Exit Transition Plan and support shall include the following:

· Inventory all Government Furnished Equipment (GFE) and Government Furnished Information (GFI) in Contractor possession

· Status of all deliverables

· Current issues, problems, or activities in process that require immediate action

· The most current and up to date Policy and Procedures Manual

· The most current and up to date Operations Manual for each sub task

· The plan on how the Contractor intends to support the transition to the new follow-on contractor, and

· the list of key personnel involved in this effort

SECTION 2 – PERFORMANCE REQUIREMENTS (TASKS)

The Contractor shall perform management support services. Contractor shall assist CBP with developing project and program plans according to the target project management methodology (e.g., Agile, traditional project management), making recommendations to CBP in improving existing management processes and practices, and executing government decisions upon approval of the plan(s) by CBP officials. The Contractor cannot exercise discretion in applying Federal government authority or commit the government to a course of action without approval.

2.2 Task 1 – Enterprise Operations Center Support (EOC)

The Contractor shall operate a 24x7x365 EOC providing Tier 2 incident management and response managers and Tier 2 Technical Support (with expertise in technology as outlined throughout the remainder of tasks). This team is primarily responsible for monitoring, analyzing, troubleshooting and taking action to resolve and/or escalate all critical, major, and minor events as they are reported via phone, work tickets, or monitoring services.

The EOC shall have multiple teams in order to provide the following:

· General EOC Support

· Advanced Monitoring Engineering

· Root Cause Analysis

The following table identifies general roles and responsibilities the Contractor will be responsible to support the EOC under the DCSS contract.

Contractor Roles and Responsibilities for General EOC Support

1. Monitor and support all applications and systems in production, development, and test environments at all Data Centers used by CBP.

2. Provide a central point for coordination of incidents that arise in all environments. Establish and orchestrate bridge calls with emphasis on restoring service to users as quickly as possible, facilitate and troubleshoot toward resolution of incidents, and manage incidents to completion.

3. Coordinate, escalate, and/or resolve operational system/application/network events that have the potential of negatively impacting system and application availability to the user community.

4. Create and provide metrics to judge efficiency and effectiveness of Incident Management Process.

5. Create, update and maintain Standard Operating Procedures, Technical User Guides, Troubleshooting Guides, and Customer Contact Database. Conduct quarterly reviews of all documents.

6. Develop and maintain the Incident Management process, to include: incident identification, incident logging, incident categorization, incident prioritization, initial diagnosis, incident escalation, investigation and diagnosis, incident resolution and closure.

7. Develop and maintain web-based reporting for incident management as well as all facets related to systems availability.

8. Develop and populate Knowledge Management Database with known troubleshooting procedures. Develop "lessons learned" on all escalated incidents.

9. Escalate incidents in accordance with established CBP/OIT escalation procedures.

10. Identify and respond to potential problems or trends before failure or degradation occurs. Provide information on scenarios for monitoring, automation, and adjusted threshold levels to allow appropriate action to be taken before adverse impact occurs. Develop and/or enhance processes to detect and prevent future incidents and allowing for quicker diagnosis and resolution if further incidents occur.

11. Provide broadly skilled technical staff to maximize single point of contact resolution and to serve as the level II support for potential or confirmed issues with information systems owned and operated by CBP.

12. Report on previous business day’s EOC call volume and SLAs to be incorporated into the OIT Morning Meeting report slides. Content may change as the Government reporting requirements change over time. Due daily by 7:30am.

13. Report monthly on outstanding tickets dependent on third party action. Report to include ticket, item awaiting action, third party, duration and if known estimated resolution time

14. Develop monthly EOC reporting for SLAs and KPIs. Can be delivered as a report or government accepted dashboard

15. Develop a EOC SOP and updated Incident Response Plan to be delivered to CBP within 90 days after contract award date (or as agreed upon by the Government).

Root Cause Analysis The Contractor shall be responsible to provide Problem and Risk Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Root Cause Analysis

1. Acting as primary POC for Program Directorate Interaction.

2. Identify the root cause of Infrastructure problems as well as resolve the problem following standard processes, policies, procedures, and practices

3. Perform a root-cause analysis (RCA) of events that are mission critical, have recurring negative effects on CBP IT Infrastructure, or when requested by the CBP

4. Maintain ownership of an incident or event as well as coordinate with various teams to discover how and why an incident or event occurred, identify the actions, lessons learned to correct or eliminate the root cause of an event, propose the corrective actions to CBP, obtain approval from CBP to take the corrective actions, implement the changes required to eliminate the root cause following the Change Management processes

5. Develop and maintain a depository of root cause analyses that is easily accessible to authorized Government personnel for viewing

6. Develop and provide daily accounts of its Problem Management activities, and deliver monthly Problem Management Reports to the CBP

7. Develop a Risk Management Plan to outline plans and processes to identify, track, and mitigate risks to the CBP

8. Provide CBP with a monthly report of risks identified, mitigated, and open for the previous month. For all open risks, provide a plan of action to mitigate those risks

Advanced Monitoring Engineering The Contractor shall provide advanced monitoring engineering support in order to modernize CBP’s monitoring capabilities for both current and future system states to include adaptive systems which automatically change system parameters such as increasing RAM/CPU, adding storage, etc. The Contractor shall work seamlessly with EOC, EDMED teams, and other stakeholders as directed to ensure the comprehensive monitoring needs are met.

· Advanced Monitoring Engineering

1. Engineer, implement, and support using current and future CBP identified tools (e.g., AppDynamics, Oracle Enterprise Manager, SCOM) to provide real-time system availability monitoring and reporting. Monitoring shall include the end-to-end system, from regional user sites (e.g., remote synthetic transactions) to the backend databases.

17. Implement real-time system health views through online dashboards.

18. Implement automated application discovery and topology mapping.

19. Establish monitoring capabilities necessary to support current and future Service Level Agreements (e.g., system availability, performance).

20. Establish dynamic thresholds for proactive alerting before incidents occur.

21. Identify and recommend to EDMED management modernized monitoring standards, best practices, and new capabilities which meet ongoing needs.

2.3 Task 3 – Enterprise Data Center Operations Branch

Enterprise Data Center Operations provides support services that entail data center management, server and storage support, network administration, and system administration. Additionally, Enterprise Infrastructure monitors and oversees the operations of the IT hardware, software, and facility infrastructure within the NDC and CBP enclaves within DC1 and DC2.

Note: Performance will be evaluated according to Service Level Agreements and the modernization and automation of current processes which reduces the required Full Time Employees necessary for the associated task.

Subtask Areas: The Contractor shall provide Server and Storage support in the following areas:

· General Servers and Virtualization

· EDMED LAN

· Enterprise Storage Management

· Database Services

· Mainframe Operations

· DCIM

An overview of the current CBP Server and Storage environment can be found within each subtask.

Current CBP Server and Storage asset counts and volumes are provided in <update with section>.

2.3.1 Task 3.1 – General Servers and Virtualization

EDMED manages 4,500+ Linux servers, 1,200+ Windows servers, and multiple AIX environments. The majority of the servers are virtualized, and support application and database hosting services in both non-production and production environments. The following table identifies general roles and responsibilities the Contractor would be responsible to provide:

· Contractor Roles and Responsibilities for General Servers and Virtualization Support

1. Provide system administration and operational support for server infrastructure, hypervisor, and Operating Systems. Contractor shall design and implement automated fail-over and automated recovery solution.

2. Contractor shall implement, and maintain a Automated Configuration Management, by using tools such as Puppet, Chef, Ansible, or other according to EDMED management direction to ensure consistent configurations on all server operating systems.

3. Manage and monitor the performance, utilization of the various server resources, support the analysis as well as correction of all end-user computing and network problems that may be associated with server processing. Contractor shall design and implement automated resource management of virtual assets to include automated corrective actions for systems experiencing performance issues related to the technical stack from Operating System to Infrastructure (i.e., does not include middleware/Java/etc).

4. Design, implement, and maintain a hybrid cloud utilizing EDMED directed hypervisor and cloud management suite(s) and integrate with multiple public cloud service providers.

5. Automate the end-to-end service fulfilment for application hosting (e.g., automate virtual machine creation, OS installation and configuration, IP addressing and DNS registration, Java container deployment and configuration, and integration with access management solution).

6. Contractor shall design, implement, and maintain a Configuration Management Database (CMDB). Update the documentation for all server operations procedures and services and maintain accurate information about all Configuration Items (CIs) in the CMDB.

7. Utilize tools and processes, such as software distribution, backups, performance measurement, tuning, license and asset management to allow automated as well as remote systems management of the server environment.

8. Provide Tier 3 support to resolve incident, problem, or service requests that require subject matter expertise and cannot be resolved by the EOC’s Tier 2 support. Tier 3 support personnel shall provide final problem resolution.

9. Develop and maintain automation (e.g., scripts, existing product features) to replace processes which are manually performed as directed by management.

10. Work with EOC to implement automated preventative measures, proactive monitoring and system self-healing capabilities to limit outages/performance issues which impact service levels.

11. Provide a weekly and roll-up monthly Performance Baseline and Operational Metrics report to include metric name/type, Acceptable Quality Levels (AQL), actual performance (raw performance), vendor performance (excludes issues outside of vendor control), vendor rating based on vendor performance, running 3 month average performance, details and explanations supporting exclusions and related information, comments, planned actions for next period, relevant supporting data, event log to include outage time, exclusion time, total availability, and details related to exclusion

12. Provide a monthly Resource Allocation report to include metrics associated with support staff (by technical area and skill set). This report shall provide utilization of resource to include hours expended against normal operations, hours expended against incident resolution, hours expended against problem resolution, and hours expended against project support (including technical reviews, testing, documentation, etc.). These reports will also be utilized for performance evaluation of service levels to include the workforce reduction as a result of implementing automated and improved processes.

13. Provide a detailed Root Cause Analysis report to include actions associated with unplanned outages delivered within 5 working days of Root Cause Identified.

14. Provide a detailed Service Improvement Plan (SIP) to include the process, procedure, training, or personnel issues identified during normal and emergency operations

15. Coordinate with the Budget and Strategy (Capacity Planning) Team, and other stakeholders as directed to establish an automated and on-demand capability for the reporting of Capacity/Performance to include the measurement and analysis of resource utilization and trends for both physical and virtual assets. Contractor shall also establish a dashboard for performance monitoring and resource utilization including Real-Time and Historical trends.

16. Provide remote support for OS and standard software build installation activities as required to meet SLAs.

17. Perform system or component configuration changes necessary to support Server Virtualization products.

18. Manage storage on the virtualization environment to minimize cost and overhead. Contractor shall ensure automated processes do not over allocate and negatively impact the stability of the environment.

19. Provide and maintain security hardened database baseline configurations (software) and deploy security (patches), software maintenance (patches) and new software releases across the Database and supporting products infrastructure

2.3.2 Task 3.2 – EDMED LAN

The Contractor shall provide CBP support in operating 24x7x365 for those assets in which the EDMED LAN has operational responsibility, to include the monitoring of the availability, performance, capacity, and events/alerts.

The Contractor shall be responsible to provide EDMED LAN support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for EDMED LAN Support Services

1. Manage system hardware and software releases, changes, moves, and installation to minimize disruption

2. Monitor and identify problem areas and coordinate corrective action with appropriate CBP SMEs and personnel proactively

3. Develop and provide recommendations to improve the EDMED LAN procedures and documentation.

4. Participate in root-cause analysis of problems that are mission critical or have recurring negative effects on CBP Information Technology Infrastructure

5. Review the availability, capacity, and performance reports to identify the problem areas and coordinate with various teams to plan, design, and implement corrective actions

6. Maintain a complete list of all the thresholds configured in the monitoring tools and ensure list is updated to reflect any changes

7. Provide an Executive Management and Trending Analysis Report that includes high-level analysis and trending of key CBP system metrics.

8. Plan, design, and configure the tools used by EDMED LAN and ensure these tools can generate a daily, monthly, and yearly availability, performance, and capacity reports to be provided to CBP

9. Perform integration between tools when possible to reduce the number of false and duplicate alerts and events, and generate a consolidated reporting of the infrastructure from the perspective of service availability, capacity, and performance

10. Proactively monitor current network utilization and provide information to CBP for use in determining future capacity requirements.

11. Install, configure, and manage software defined network solutions.

12. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP policy, or the Cybersecurity Directorate.

2.3.3 Task 3.3 – Storage Management

CBP’s Storage Environment has a capacity of more than eighteen (18) petabytes (PB) of enterprise class storage on various vendor solutions (e.g., Hitachi, EMC, NetApp) across three datacenters to include multiple cloud service providers. CBP’s Storage Environment leverages multiprotocol arrays that utilize Fiber Channel (FC), Ethernet, and Fiber Channel over Ethernet (FCoE) connectivity to provide FC SAN and both CIFS and NFS NAS storage. The Contractor shall provide CBP support in operating 24x7x365 for the responsibilities for this task.

The Contractor shall be responsible to provide Storage Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Storage Management

1. Design, implement, configure, and maintain non-virtual and virtual storage solutions for other EDMED managed hosting environments. Note: This is a separate task from other references of storage within virtualized environments, databases, and Mainframe.

2. Assign and initialize online storage volumes, as well as manage online storage thresholds and data archives.

3. Provide data migration/archive management, and provide documentation support as well as maintenance.

4. Backup production systems as well as development systems where needed to backup media and alternate processing/recovery sites as required by CBP backup and recovery policy.

5. Monitor backup jobs and provide a periodic Backup Report to CBP.

6. Support service requests for the restoration of data from the backups as required by EDMED teams or system owners.

7. Process, manage, and resolve all event, warning, alert, as well as alarm messages of the backup jobs and schedules. Contractor shall work with other EDMED teams in order to properly establish automated remediation activities pertaining to storage issues.

8. Provide proactive storage capacity management to meet CBP requirements.

9. Monitor and control storage performance according to CBP requirements and perform tuning as required.

10. Maintain and improve storage resource efficiency in accordance with CBP approved thresholds.

11. Provide multiple tiered storage options as well as local storage options where appropriate.

12. Decommission storage and backup environment in accordance with CBP policies and procedures at the data centers.

13. Maintain online Tapeless backup solution in accordance with CBP policies and procedures.

14. Support the testing of end-to-end system recovery process and remediate flaws.

15. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP policy, or the Cybersecurity Directorate.

2.3.4 Task 3.4 – Database Services

The primary focus of Database Services is to proactively design, implement and support all areas and components of Database infrastructure. Database Services currently supports and manages Datacom/DB, IBM DB2, Microsoft SQL Server, Oracle Database, and Informatica. These services include reliability, availability, scalability, capacity on demand, performance optimization, inter-operability, problem analysis/resolution, configuration management, disaster recovery, capacity planning, operational redundancy, contingency planning, integration, and operations of all components across the CBP Database enterprise architecture.

The Contractor shall be responsible to provide database support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Database Services

1. Engineer, integrate, implement, and operate CBP’s Database systems (Datacom/DB, IBM DB2, Microsoft SQL Server, and Oracle) across the entire enterprise and with cloud service providers to support 24x7x365 availability.

2. Provide problem determination/analysis and isolate system problems utilizing diagnostic and system management tools (i.e. system dumps and system/component traces, monitoring tools, vendor problem data bases).

3. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP policy, or the Cybersecurity Directorate.

4. Engineer, integrate, implement, and operate a virtualization strategy to include Database server consolidation, storage consolidation, and automation. Automation includes seamless Database provisioning, and non-disruptive workflow/application/data migrations.

5. Maintain up-to-date documentation of all procedures, infrastructure diagrams, system configuration and baseline standards, implementation plans, system back out plans, system recovery plans, maintenance plans, system upgrades, and all system changes.

6. Process, manage, and resolve all event, warning, alert, as well as alarm messages. Contractor shall design and implement automated responses to system issues (e.g., automatic failover, resource allocation adjustments).

2.3.5 Task 3.5 – Mainframe Operations

CBP operates critical high-transaction applications on two (2) IBM z-series mainframes. This platform supports key core services for CBP’s mission critical systems including passenger and cargo processing. The z/OS production systems are fully replicated at our continuity site with automated failover. CBP’s mission critical data repositories reside in Datacom, DB2, and Oracle databases and utilize Customer Information Control System (CICS) and WebSphere Message Queue (MQ) transactions. As CBP migrates to our open systems target architecture, this function will be critical to the planning and execution of migration and retirement activities.

The Contractor shall be responsible to provide support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Mainframe

1. Engineer, install, maintain and customize IBM z-series or later processors running the z/OS operating system in a Parallel Systems Complex (Sysplex) environment to support 24x7x365 availability. Expert knowledge is required for core z/OS functions, mainframe network products, and related subsystems.

2. Provide configuration, implementation and maintenance support for z/OS-related ISV products and apply latest maintenance concurrent with the z/OS bi-monthly maintenance cycle. Support required for, but not limited to, the following products: CA-Top Secret, CA-OPS/MVS, CA-MIM, CA-1, CA-7, CA-11, CA-Dispatch, CA-Sysview, CA-ROSCOE, CA-Auditor, Content Manager, ASG-TMON, JES2, Syncsort, FDR products, Hitachi Business Continuity Manager, TREX, VPS, Phoenix, Dimensions, VTAM, TCPIP, Netview, Pagent, TPX, Zen Exigence, Implex and FTPAlert, TN3270, VPS Anyqueue, etc

3. Create and maintain an up-to-date mainframe knowledge base to include Mainframe Standard Operating Procedures (SOPs), infrastructure logical and physical diagrams, system configuration baselines, implementation plans, back-out plans, system recovery plans, maintenance plans, system upgrades, off hours callout and escalation procedures, and a record of all system changes.

4. Provide technical support for the upgrade, installation, configuration, implementation, and maintenance of the CBP CICS transaction processing systems.

5. Provide 24x7x365 integration support for all Mainframe components (including networking) and CBP-supported applications interfaces, such as MQ/DataPower support products (COTS and proprietary), networking technology, and storage management.

6. Respond to incidents according to the government approved Incident Response Plan developed as part of the EOC Task.

2.3.6 Task 3.6 – Data Center Infrastructure Management (DCIM) The DCIM teams support the activities performed within the NDC. All aspects of server, storage, and network hardware deployments, as well as all design and operation of power, cooling, and rack infrastructure are provided by DCIM. Furthermore, DCIM is accountable for IT infrastructure location mapping and planning, and facility monitoring, implementation of changes, release deployment support, and management as well as execution of IT business continuity, regardless of the physical location of the equipment.

2.3.6.1 Data Center Infrastructure Management

The Contractor shall be responsible to provide services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Data Center Infrastructure Management

1. Provide a daily Activities Report to CBP to account for the all daily activity status and to report any issues

2. Develop proactive monitoring processes and procedures that prevent and minimize any service impacts

3. Recommend changes to improve the monitoring and recommend ways to alleviate environmental issues and improve the monitoring of environmental elements

4. Monitor IT Facility equipment including, racks, power strips, metered power module units (MPMU), uninterruptible power supplies (UPS), computer room air conditioning (CRAC), keyboard-video-mouse switches (KVM), LCD-keyboard-monitors (LKM), power distribution units (PDU), water and fire detection systems, humidity detectors, temperature sensors, and any other new technology that is implemented that is necessary to support the overall data center.

5. Visually inspect data center equipment on an annual basis. The equipment requiring visual inspection includes, at a minimum, racks, patch panels, cabling plant, keyboard-video-monitor (KVM) units, LCD-keyboard-mouse (LKM) units, computer room air conditioning (CRAC) unit, and power distribution units (PDU). Document all issues discovered in the EOC ticketing system as an Incident and promptly notify EDMED management. Maintain inspection logs in electronic format.

6. Maintain a list that documents the temperature thresholds of equipment and sensors for each environmental / physical threat sensor.

7. Maintain a list that documents the power usage and equipment lifespan thresholds of UPS and power distribution equipment for each rack and piece of power distribution equipment.

8. Perform preventive maintenance and take proactive actions to minimize damage to equipment if the in-service usage, operational capacity, or temperatures exceed the established thresholds.

9. Provide a threshold list of all equipment in monitored spaces and sensors, with the appropriate recommended actions for each threshold violation. The threshold list and actions shall be provided to the operations center.

10. Track maintenance performed on IT Facility equipment in a provided central location that is well organized so that information can be readily searched. Contractor shall work with other EDMED teams to have the information ingested into the CMDB.

11. Deliver periodic Environmental Trending and Analysis reports for all sites and all rooms and racks. Reports shall include the Events and Incidents generated when environmental thresholds were exceeded.

12. Produce inventory reports of the equipment. The inventory report shall include equipment name, physical location (room number, rack number, site abbreviation, and unit location within a rack), operating BTU output, operating amperage (AMP) draw, operating kilowatt (kW) or kilovolt (kV) requirement, serial number, and CBP barcode.

13. Perform monthly reconciliations of the inventory report with CBP Asset management system of record.

14. Update and maintain the Data Center management software instances with inventory and configuration information based on monthly reconciliations with CBP Asset Management inventory tracking system of record. Contractor shall design and implement the integration of data into the target CMDB.

15. Create and deliver capacity reports to CBP monthly, which clearly indicate the capacity of individual units and systems supporting the data center equipment. Capacity report totals shall be expressed in relative values of the infrastructure component (Watts, Amps, U-space, Square Feet, BTU/hr., etc.). The reports shall indicate:

· The usage expressed as both actual units and a percentage of total capacity

· Detail and summary indicating the availability of supporting systems expressed as N redundancy

· Annual and monthly uptime expressed both hourly and as percentage

16. Create and deliver performance reports to CBP monthly, which clearly indicate the nominal and maximum performance characteristics of all systems and sub-systems, monitoring points, and alert thresholds. Reports shall indicate the number of exceptions that exceeded the thresholds, identify the problem areas within the Infrastructure, and show the trends, as well as delta from the last established baseline

17. Create and deliver availability reports to CBP monthly, that include power maps and breaker studies sufficient to verify the correct implementation of availability requirements for redundancy and fault tolerance. The reports shall include:

· Cross reference of the power circuits from PDUs to each individual rack in each of the Data Centers

· Cooling system performance analysis sufficient to assure the Government the requirements for heat rejection are met

· Analysis of redundancy and fault tolerance

18. Create (using CBP approved software) and maintain clear rack elevation drawings for each rack in all sites showing BTU usage, AMP draws, KW or KV draws, and similar data for each rack

19. Create (using CBP approved software) and maintain accurate wire maps of all low voltage signaling cabling (i.e. twin-ax, fiber, Cat-6, TP, AU, etc.) for all network, storage, and security, and management equipment, including all voice and data connections. The report shall be provided in human and machine readable electronic format compatible with Microsoft Excel. The format of wire maps shall contain all from/to information, including cable/wiring type and style, including connector type and style. The wire maps shall clearly indicate and all intermediate patch panels and patch connections. Wire maps must include cable type, label contents, length, and termination type. Wire maps shall be provided for all power connections and must include the panel number, breaker number, cable length, receptacle type, power strip rating and type, and receptacle outlet number for each cord set

20. Provide ad hoc reports as requested by CBP. Ad Hoc reports may be out of routine cycle snapshots of a standard report, a custom report of equipment and infrastructure operating conditions. Ad Hoc reports would typically be in the nature of, but not be limited to, air flow, plenum temperatures, pressure variations, power quality, power trends, battery monitoring, battery age, environmental sensor testing, feasibility studies, incident analysis investigation, project workflow. Additional reports may require the contractor to obtain and supply additional information, formatting, presentation materials, inspection artifacts, or require special monitoring equipment

21. Create and deliver to CBP weekly project and activity status reports

22. Develop and document a manual processing procedure to use in an event the scheduling software and tools are not available.

2.4 Task 4 – Enterprise Systems Engineering Branch

The Contractor shall provide CBP with support for business needs with appropriate technology (hardware or software). This includes architectural design, engineering design, proof of concept development, pilots, analysis, results, and documentation.

2.4.1 Task 4.1 - Desktop Management Team (DMT)

The Desktop Management environment encompasses approximately 65,000 employees and contractors, and includes over 70,000 desktops managed by Microsoft System Center configuration Manager (SCCM) for Windows-based desktops and Casper for Mac OS X based desktops.

The Contractor shall review and recommend updates to CBP existing technology, processes, and reviews new and emerging technology trends in the attempt to identify opportunities for CBP to satisfy business requirements in a more efficient manner. The Contractor under this task shall provide CBP with support services to manage the desktop baseline images and associated patching of the desktop baseline images.

2.4.1.1 General DMT Support

The following table identifies general roles and responsibilities the Contractor would be responsible for to support the DMT under the DCSS contract:

· Contractor Roles and Responsibilities for General DMT Support

1. Provide recommendation to improve services and standards for supporting the desktop computing and End-User environment

2. Provide all desktop, tablet, and laptop Tier 3 support.

3. Provide Tier 3 resolution for incidents and problems associated with DMT equipment and software

4. Provide both proactive and reactive troubleshooting to effectively identify potential incidents or problems, and attempt to eliminate them before they occur

5. Support authorized users (e.g., Field Technical Officers) in correctly using DMT software, and in accessing and using related technologies and services

6. Support Incident Determination, Resolution and Root Cause Analysis

7. Maintain a document that describes the minimum equipment and software requirements and/or the specific equipment and software designated for standard use within CBP (the “Standard Baseline Software”)

8. Develop recommendations based on strategies and approaches that shall result in the elimination of unnecessary equipment or software, or modifications to existing equipment and software that shall provide CBP with competitive advantages, increased efficiency, increased performance, or cost savings

9. Build, test, support, and maintain CBP computer image(s)

10. Conduct system-level and End-User testing of computer image(s) to validate that they perform in accordance with the approved specifications

11. Conduct deployment testing to verify that images can be deployed successfully and operate with all supported applications, hardware and Software

12. Automate computer image deployment processes (e.g., remote electronic upgrading of desktop images)

13. Monitor patch activity on a daily basis, providing CBP with Patch Management Reports, and deploy patches requested by management

14. Establish contingency plans and rollback plans to effectively and efficiently restore network deployed software patches to their original state as necessary

15. Schedule software distributions to support time-critical release upgrades, to deploy urgent patches, and to maintain current projects while minimizing the disruption to authorized users

16. Utilize an electronic software distribution system that effectively and efficiently distributes new software to LAN-connected workstations located at CBP sites and to remotely connected workstations in a timely manner using CBP approved tools (BigFix and SCCM)

17. Employ an automated software distribution solution and procedure that provides consistent and efficient software deployments for software configuration changes

18. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP policy, or the Cybersecurity Directorate.

2.4.1.3 Technology Refresh

The Contractor shall support CBP in deploying new hardware, software and operating systems as part of CBP standard and emerging technology refreshment cycle as designated by the CBP.

The Contractor shall be responsible to provide Technology Refresh support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Technology Refresh

1. Identify and develop Enhancement and Upgrade Recommendations to be presented to CBP management that shall include: potential improvements to current CBP technology and business processes, forecasting of viable future technologies, and new business processes that may be considered

2. Develop and maintain a Technology Refresh Plan and schedule

3. Assess the usability of the assets and review alternatives to replace, re-lease, consolidate, or retain the assets. Deliver an initial recommendation regarding the assets to CBP for review and approval

4. Implement a Technology Refresh Plan and provide status reports regarding implementation status as required

5. Perform quarterly reviews of technology refresh plans to ensure they properly support CBP business requirements

2.4.2 Task 4.2 - Systems Integration Management

The main goal of SIM is to facilitate the alignment of customer requirements with computing infrastructure business and operational strategies. SIM focuses on business outcomes by linking service assets, services and customers. SIM is intent on employing infrastructure standards, performance engineering and capacity planning practices to reduce and prevent duplication of data, applications, systems, workload and services.