Draft_DCSS_PWS_(11.10.16).docx

Performance Work Statement (PWS) DCSS (draft)

Contents

SECTION 1 – SCOPE AND BACKGROUND OVERVIEW

1.1 Introduction

This is a single award to a Contractor to provide Data Center Support Services (DCSS) and related services for all Divisions and Offices of the U.S. Customs and Border Protection (CBP) both in Washington, DC and its regional locations and data centers.

General Description of Services Provide comprehensive DCSS in support of CBP Office of Information Technology (OIT) in the area of Enterprise Operations, Enterprise Infrastructure, Enterprise Architecture, and Common Services. Support shall span across all locations, to include CBP Headquarter Offices in Washington, DC, CBP Data Centers, and in any location where CBP operates in the future.

Limits of Services This is not a Personal Services Contract.

Background The mission of the OIT is to provide CBP with information, services, and technology solutions to secure the border, prevent the entry of terrorists or terrorist weapons, and to ensure the efficient flow of lawful people across U.S. borders. The OIT provides the U.S. Department of Homeland Security (DHS) and CBP with a 24x7x365 secure, stable and high-performance information technology infrastructure, scientific solutions, and forensic services.

The OIT has the responsibility for identifying and evaluating new technologies for applications in support of CBP business processes. A new era of economic downturn and terrorist threats require systems that can support the future vision of a unified, integrated, intelligence-driven agency to protect the United States against all threats. In order to be successful, the new Government and Contractor team must identify and acquire highly-skilled professionals from across industry that are capable of leading the technological revolution currently underway within OIT systems applications without abandoning and losing all existing critical corporate knowledge.

Needs CBP requires DCSS, to provide the OIT with support in all aspects of information technology. These services primarily include, but are not limited to, the management and operations of all OIT IT Infrastructure, support within the Enterprise Operations Center (EOC), End User Computing, Asset Management, as well as Infrastructure Support for Engineering solutions and ongoing Operations.

Objectives

The objective of the contract is to seamlessly transition the current dispersed family of operations and support services under a single integrated contract. During the transition the contractor must ensure the critical mission support services included in this contract continue to function uninterrupted. Once transition is complete the contractor shall take all actions necessary to maintain optimal daily performance of CBP IT operations, taking day-to-day responsibility for all established performance standards. Further, the contractor shall proactively assess and recommend changes to current operations the gain improved performance and greater efficiencies in the CBP OIT environment. Finally, the contractor shall closely monitor the current operations, and collect and report a full range of performance baseline data during the initial period of performance. This comprehensive collection of “current state” performance data will be used as the contract matures to measure the impacts of improvements such as automating manual process and integrating like services, and others that will be considered and incorporated through the anticipated life of this contract.

1.1.1 Scope

The OIT provides CBP with support in all aspects of information technology. CBP OIT has overall management responsibility for IT program activities including, but not limited to, infrastructure operations and engineering, user support, and enterprise architecture. Attachment 2 provides a high-level summary of the inventory/scale.

CBP OIT data center processing is performed at multiple data centers within the continental United States. Data centers currently providing services to CBP OIT include the following: CBP National Data Center (NDC) Complex located in Springfield, Virginia; the Stennis Data Center (DC1) at Space Center in Mississippi (supports multiple DHS Agencies, including CBP); the DHS Data Center 2 (DC2) in Clarksville, Virginia (supports multiple DHS Agencies, including CBP); the Finance Center located in Indianapolis, Indiana; and other locations throughout the Washington, D.C. metropolitan area. Additionally, CBP utilizes public cloud service providers (e.g., Amazon Web Services), in which the contractor shall support.

The Contractor shall respond to current requirements and proactively anticipate new requirements. The Contractor shall work with CBP OIT to assess the impact of new requirements on CBP operating environment. The Contractor shall handle requirements for new services through CBP OIT Change Control process and shall adjust services accordingly.

The Contractor shall perform transition activities with the intent to minimize disruption to the existing operations.

1.2 Service Levels

The Contractor shall develop and maintain effective service levels according to the requirements in the Tasks and as outlined in Attachement 1 – Performance Requirements Summary (PRS) table. The Contractor shall develop and implement procedures to identify, prevent, and ensure non-recurrence of defective services. The Contractor’s Quality Control Plan is the means by which the Contractor ensures that work complies with the requirement of the contract. CBP OIT expects non-production and production environments to have the same levels of operational support.

Security Requirements The security classification for work performed under this Contract is Unclassified. The documents and information that will be reviewed and produced are non-public and sensitive in nature and shall be protected from unauthorized disclosure. Work on this Contract requires that personnel may have access to Privacy Act information. Personnel shall adhere to the Privacy Act, 5 U.S.C. Section 552a and applicable agency rules and regulations.

Physical Security The Contractor shall be responsible for safeguarding all government equipment, information and property provided for contractor use.

Key Control The Contractor shall establish and implement methods of making sure all keys/key cards issued to the Contractor by the Government are not lost or misplaced and are not used by unauthorized persons. NOTE: All references to keys include key cards. No keys/key cards issued to the Contractor by the Government shall be duplicated. The Contractor shall develop procedures covering key/key card control that shall be included in the Quality Control Plan. Such procedures shall include the Government receiving any previously issued keys/key cards by personnel who no longer require access to locked areas. The Contractor shall immediately report any occurrences of lost or duplicate keys/key cards to the Contracting Officer.

In the event keys, other than master keys, are lost or duplicated, the Contractor shall, upon direction of the Contracting Officer, re-key or replace the affected lock or locks; however, the Government, at its option, may replace the affected lock or locks or perform re-keying. When the replacement of locks or re-keying is performed by the Government, the total cost of re-keying or the replacement of the lock or locks shall be deducted from the monthly payment due the Contractor. In the event a master key is lost or duplicated, all locks and keys for that system shall be replaced by the Government and the total cost deducted from the monthly payment due the Contractor.

The Contractor shall prohibit the use of Government issued keys/key cards by any persons other than the Contractor’s employees. The Contractor shall prohibit the opening of locked areas by Contractor employees to permit entrance of persons other than Contractor employees engaged in the performance of assigned work in those areas, or personnel authorized entrance by the Contracting Officer.

Quality Assurance/Government Surveillance All Service Levels defined within the Performance Requirements Summary (Attachment 1) must be addressed in the Contractors Quality Control Plan (QCP). The Contractor shall develop a draft Quality Control Plan (QCP) and deliver it to the CBP within 30 days of contract award. The QCP will address how quality control activities will be documented, and the data stores for maintaining documentation.

The government shall evaluate the Contractor’s performance under this Contract in accordance with a Quality Assurance Surveillance Plan (QASP). This plan sets forth specific surveillance methods the Government will perform to ensure that the Contractor has performed in accordance with the performance standards.

Post Award Conference/Periodic Progress Meetings The Contractor agrees to attend any post award conference convened by the Contracting Officer in accordance with Federal Acquisition Regulation Subpart 42.5. The Contracting Officer, Contracting Officers Representative (COR), and other Government personnel, as appropriate, may meet periodically with the Contractor to review the Contractor's performance. At these meetings the Contracting Officer will apprise the Contractor of how the Government views the Contractor's performance and the Contractor shall apprise the Government of problems, if any, being experienced. Appropriate action shall be taken to resolve outstanding issues. These meetings shall be at no additional cost to the government.

Transition In and Transition Out The Contractor shall provide a comprehensive Transition-In Plan facilitating a smooth transfer of responsibility from the current providers within 20 days after Contract award date.

The Contractor's Transition-In Plan shall include the following at a minimum:

· A comprehensive communications plan to inform CBP OIT and other management of transition objectives and on-going status and key deliverables and the underlying technical requirements to keep the projects on schedule

· Identification of any change management barriers that could complicate the transition and remedial plans to effectively overcome the barriers

· Risks and risk mitigation strategies associated with the change in contractors

· Plan for knowledge transfer

· Define the general means for measuring the on-going progress against the Transition Plan including, measurement criteria and tools; milestone dates and deliverables

· Recommendations for improving CBP OIT infrastructure architecture support

· Review, evaluation, and transition of current CBP OIT hardware, software, data, and workflow

· Orientation phase and program to introduce CBP OIT personnel, programs, and users to the Contractor's team, tools, methodologies, and business processes

The Contractor shall adapt its Transition-In Plan as required to address any changes resulting from additional information and due diligence.

Successful transition will be measured by the seamless transfer of responsibility from the current providers of support services to the new contractor, without impact to end users, and while meeting agreed upon service levels.

The Contractor shall provide a Transition-Out Plan within 120 days prior to the end of each Period of Performance that shall identify transition support activities for all transition efforts for follow-on requirements to minimize disruption of services. The Transition-Out Plan and support shall include the following:

· Inventory all Government Furnished Equipment (GFE) and Government Furnished Information (GFI) in Contractor possession

· Status of all deliverables

· Current issues, problems, or activities in process that require immediate action

· Most current and up to date Policy and Procedures Manual

· Most current and up to date Operations Manual for each sub task

· Plan on how the Contractor intends to support the transition to the new follow-on contractor, and

· List of key personnel involved in this effort

SECTION 2 – PERFORMANCE REQUIREMENTS (TASKS)

The Contractor shall perform management support services. Contractor shall assist CBP OIT with developing project and program plans according to the target project management methodology (e.g., Agile, traditional project management), making recommendations to CBP OIT in improving existing management processes and practices, and executing government decisions upon approval of the plan(s) by CBP OIT officials. The Contractor cannot exercise discretion in applying Government authority or commit the Government to a course of action without approval.

2.1 Task – Enterprise Operations Center Support (EOC)

The Contractor shall operate a 24x7x365 EOC providing Tier 2 incident management and response managers and Tier 2 Technical Support (with expertise in technology as outlined throughout the remainder of tasks). This team is primarily responsible for monitoring, analyzing, troubleshooting and taking action to resolve and/or escalate all critical, major, and minor events as they are reported via phone, work tickets, or monitoring services.

The EOC shall have multiple teams in order to provide the following:

· General EOC Support

· Advanced Monitoring Engineering

· Root Cause Analysis

The following table identifies general roles and responsibilities the Contractor will be responsible to support the EOC under the DCSS contract.

Contractor Roles and Responsibilities for General EOC Support

1. Monitor and support all applications and systems in production, development, and test environments at all Data Centers used by CBP.

2. Provide a central point for coordination of incidents that arise in all environments. Establish and orchestrate bridge calls with emphasis on restoring service to users as quickly as possible, facilitate and troubleshoot toward resolution of incidents, and manage incidents to completion.

3. Coordinate, escalate, and/or resolve operational system/application/network events that have the potential of negatively impacting system and application availability to the user community.

4. Create and provide metrics to judge efficiency and effectiveness of Incident Management Process. Examples: Mean Time to Repair, Mean Time Between Failures, Repeat Incidents

5. Create, update and maintain Standard Operating Procedures, Technical User Guides, Troubleshooting Guides, and Customer Contact Database. Conduct quarterly reviews of all documents.

6. Develop and maintain the Incident Management process, to include: incident identification, incident logging, incident categorization, incident prioritization, initial diagnosis, incident escalation, investigation and diagnosis, incident resolution and closure.

7. Develop and maintain web-based reporting for incident management as well as all facets related to systems availability.

8. Develop and populate Knowledge Management Database with known troubleshooting procedures. Develop "lessons learned" on all escalated incidents.

9. Escalate incidents in accordance with established CBP/OIT escalation procedures.

10. Identify and respond to potential problems or trends before failure or degradation occurs. Provide information on scenarios for monitoring, automation, and adjusted threshold levels to allow appropriate action to be taken before adverse impact occurs. Develop and/or enhance processes to detect and prevent future incidents and allowing for quicker diagnosis and resolution if further incidents occur.

11. Provide broadly skilled technical staff to maximize single point of contact resolution and to serve as the level II support for potential or confirmed issues with information systems owned and operated by CBP.

12. Report on previous business day’s EOC call volume and SLAs to be incorporated into the OIT Morning Meeting report slides. Content may change as the Government reporting requirements change over time. Due daily by 7:30am.

13. Report monthly on outstanding tickets dependent on third party action. Report to include ticket, item awaiting action, third party, duration and if known estimated resolution time

14. Develop monthly EOC reporting for SLAs and KPIs. Can be delivered as a report or government accepted dashboard.

15. Develop a EOC SOP and updated Incident Response Plan to be delivered to CBP within 90 days after contract award date (or as agreed upon by the Government).

16. Retrieve hardware deliveries as part of emergency product replacement from the front gate of NDC 1 and 2.

Root Cause Analysis The Contractor shall be responsible to provide Problem and Risk Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Root Cause Analysis

1. Act as primary POC for CBP OIT interaction.

2. Identify the root cause of Infrastructure problems as well as resolve the problem following standard processes, policies, procedures, and practices.

3. Perform a root-cause analysis (RCA) of events that are mission critical, have recurring negative effects on CBP IT Infrastructure, or when requested by the CBP.

4. Maintain ownership of an incident or event as well as coordinate with various teams to discover how and why an incident or event occurred, identify the actions, lessons learned to correct or eliminate the root cause of an event, propose the corrective actions to CBP, obtain approval from CBP to take the corrective actions, implement the changes required to eliminate the root cause following the Change Management processes.

5. Develop and maintain a repository of root cause analyses that is easily accessible to authorized Government personnel for viewing.

6. Develop and provide daily accounts of Problem Management activities, and deliver monthly roll-up Problem Management Reports to management.

Advanced Monitoring Engineering The Contractor shall provide advanced monitoring engineering support in order to modernize CBP OIT monitoring capabilities for current and future system states, to include adaptive systems which automatically change system parameters such as increasing RAM/CPU, adding storage, etc. The Contractor shall work seamlessly with EOC, CBP OIT teams, and other stakeholders as directed to ensure the comprehensive monitoring needs are met.

· Advanced Monitoring Engineering

1. Engineer, implement, and support using current and future CBP OIT identified tools (e.g., AppDynamics, Oracle Enterprise Manager, SCOM) to provide real-time system availability monitoring and reporting. Monitoring shall include the end-to-end system, from regional user sites (e.g., remote synthetic transactions) to the backend databases.

18. Implement real-time system health views through online dashboards.

19. Implement automated application discovery and topology mapping.

20. Establish monitoring capabilities necessary to support current and future Service Level Agreements (e.g., system availability, performance).

21. Establish dynamic thresholds for proactive alerting before incidents occur.

22. Identify and recommend to CBP OIT modernized monitoring standards, best practices, and new capabilities which meet ongoing needs as part of the Change Management process.

2.2 Task – Enterprise Data Center Operations Branch

CBP OIT Enterprise Data Center Operations provides support services that entail data center management, server and storage support, network administration, and system administration. Additionally, Enterprise Infrastructure monitors and oversees the operations of the IT hardware, software, and facility infrastructure within the NDC and CBP enclaves within DC1 and DC2.

Note: Performance will be evaluated according to Service Level Agreements and the modernization and automation of current processes which reduces the required Full Time Employees necessary for the associated task.

2.2.1 Task – EDMED LAN

The Contractor shall provide CBP support in operating 24x7x365 for those assets in which the EDMED LAN has operational responsibility, to include the monitoring of the availability, performance, capacity, and events/alerts. This task includes the management of local hardware at NDC, and remote management of infrastructure at DC1/DC2 and Cloud-based networking services (e.g., elastic loadbalancing).

The Contractor shall be responsible to provide EDMED LAN support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for EDMED LAN Support Services

1. Manage system hardware and software releases, changes, moves, and installation to minimize disruption.

2. Monitor and identify problem areas and coordinate corrective action with appropriate CBP OIT SMEs and personnel proactively.

3. Develop and provide recommendations to improve the EDMED LAN procedures and documentation.

4. Participate in root-cause analysis of problems that are mission critical or have recurring negative effects on CBP information technology infrastructure.

5. Review the availability, capacity, and performance reports to identify the problem areas and coordinate with various teams to plan, design, and implement corrective actions.

6. Maintain a complete list of all the thresholds configured in the monitoring tools and ensure list is updated to reflect any changes.

7. Plan, design, and configure the tools used by CBP OIT LAN and ensure these tools can generate daily, monthly, and yearly availability, performance, and capacity reports to be provided to CBP OIT.

8. Perform integration between tools when possible to reduce the number of false and duplicate alerts and events, and generate a consolidated reporting of the infrastructure from the perspective of service availability, capacity, and performance.

9. Proactively monitor current network utilization and provide information to CBP OIT for use in determining future capacity requirements.

10. Install, configure, and manage software defined network solutions.

11. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP OIT policy, or the Cybersecurity Directorate.

2.2.2 Task – Storage Management

CBP OIT storage environment has a capacity of more than eighteen (18) petabytes (PB) of enterprise class storage on various vendor solutions (e.g., Hitachi, EMC, NetApp) across three datacenters to include multiple cloud service providers. CBP OIT storage environment leverages multiprotocol arrays that utilize Fiber Channel (FC), Ethernet, and Fiber Channel over Ethernet (FCoE) connectivity to provide FC SAN and both CIFS and NFS NAS storage. The Contractor shall provide CBP OIT support in operating 24x7x365 for the responsibilities for this task.

The Contractor shall be responsible to provide Storage Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Storage Management

1. Design, implement, configure, and maintain non-virtual and virtual storage solutions for CBP OIT managed hosting environments. Note: This is a separate task from other references of storage within virtualized environments, databases, and Mainframe.

2. Assign and initialize online storage volumes, as well as manage online storage thresholds and data archives.

3. Provide data migration/archive management, and provide documentation support as well as maintenance.

4. Backup production systems as well as development systems where needed to backup media and alternate processing/recovery sites as required by CBP OIT backup and recovery policy.

5. Monitor backup jobs and provide a periodic Backup Report as requested to management.

6. Support service requests for the restoration of data from the backups as required by CBP OIT teams or system owners.

7. Process, manage, and resolve all event, warning, alert, as well as alarm messages of the backup jobs and schedules. Contractor shall work with other CBP OIT teams in order to properly establish automated remediation activities pertaining to storage issues.

8. Provide proactive storage capacity management to meet CBP OIT requirements.

9. Monitor and control storage performance according to CBP OIT requirements and perform tuning as required.

10. Maintain and improve storage resource efficiency in accordance with CBP OIT approved thresholds.

11. Provide multiple tiered storage options as well as local storage options where appropriate.

12. Decommission storage and backup environment in accordance with CBP OIT policies and procedures at the data centers.

13. Maintain online Tapeless backup solution in accordance with CBP OIT policies and procedures.

14. Support the testing of end-to-end system recovery process and remediate flaws.

15. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP OIT policy, or the Cybersecurity Directorate.

2.2.3 Task – Database Services

The primary focus of Database Services is to proactively design, implement and support all areas and components of Database infrastructure. Database Services currently supports and manages Datacom/DB, IBM DB2, Microsoft SQL Server, Oracle Database, and Informatica. These services include reliability, availability, scalability, capacity on demand, performance optimization, inter-operability, problem analysis/resolution, configuration management, disaster recovery, capacity planning, operational redundancy, contingency planning, integration, and operations of all components across the CBP Database enterprise architecture.

The Contractor shall be responsible to provide database support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Database Services

1. Engineer, integrate, implement, and operate CBP OIT Database systems (Datacom/DB, IBM DB2, Microsoft SQL Server, and Oracle) across the entire enterprise and with cloud service providers to support 24x7x365 availability.

2. Provide problem determination/analysis and isolate system problems utilizing diagnostic and system management tools (i.e. system dumps and system/component traces, monitoring tools, vendor problem data bases).

3. Maintain security hardened database baseline configurations and deploy security (patches), software maintenance (patches) and new software releases and supporting products infrastructure as directed by the system owner, DHS and CBP OIT policy, or the Cybersecurity Directorate.

4. Engineer, integrate, implement, and operate a virtualization strategy to include Database server consolidation, storage consolidation, and automation. Automation includes seamless Database provisioning, and non-disruptive workflow/application/data migrations.

5. Maintain up-to-date documentation of all procedures, infrastructure diagrams, system configuration and baseline standards, implementation plans, system back out plans, system recovery plans, maintenance plans, system upgrades, and all system changes.

6. Process, manage, and resolve all event, warning, alert, as well as alarm messages. Contractor shall design and implement automated responses to system issues (e.g., automatic failover, resource allocation adjustments).

2.2.4 Task – Mainframe Operations

CBP OIT operates critical high-transaction applications on two (2) IBM z-series mainframes. This platform supports key core services for CBP OIT mission critical systems including passenger and cargo processing. The z/OS production systems are fully replicated at our continuity site with automated failover. CBP OIT mission critical data repositories reside in Datacom, DB2, and Oracle databases and utilize Customer Information Control System (CICS) and WebSphere Message Queue (MQ) transactions. As CBP OIT migrates to our open systems target architecture, this function will be critical to the planning and execution of migration and retirement activities.

The Contractor shall be responsible to provide support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Mainframe

1. Engineer, install, maintain and customize IBM z-series or later processors running the z/OS operating system in a Parallel Systems Complex (Sysplex) environment to support 24x7x365 availability. Expert knowledge is required for core z/OS functions, mainframe network products, and related subsystems.

2. Provide configuration, implementation and maintenance support for z/OS-related ISV products and apply latest maintenance concurrent with the z/OS bi-monthly maintenance cycle. Support required for, but not limited to, the following products: CA-Top Secret, CA-OPS/MVS, CA-MIM, CA-1, CA-7, CA-11, CA-Dispatch, CA-Sysview, CA-ROSCOE, CA-Auditor, Content Manager, ASG-TMON, JES2, Syncsort, FDR products, Hitachi Business Continuity Manager, TREX, VPS, Phoenix, Dimensions, VTAM, TCPIP, Netview, Pagent, TPX, Zen Exigence, Implex and FTPAlert, TN3270, VPS Anyqueue, etc.

3. Create and maintain an up-to-date mainframe knowledge base to include Mainframe Standard Operating Procedures (SOPs), infrastructure logical and physical diagrams, system configuration baselines, implementation plans, back-out plans, system recovery plans, maintenance plans, system upgrades, off hours callout and escalation procedures, and a record of all system changes.

4. Provide technical support for the upgrade, installation, configuration, implementation, and maintenance of the CICS transaction processing systems.

5. Provide 24x7x365 integration support for all Mainframe components (including networking) and CBP OITsupported applications interfaces, such as MQ/DataPower support products (commercial-off-the-shelf and proprietary), networking technology, and storage management.

6. Respond to incidents according to the government approved Incident Response Plan developed as part of the EOC Task.

2.2.5 Task – General Servers and Virtualization

CBP OIT manages 4,500+ Linux servers, 1,200+ Windows servers, and multiple AIX environments. The majority of the servers are virtualized, and support application and database hosting services in both non-production and production environments. The Contractor shall provide CBP OIT support in operating 24x7x365 for the responsibilities for this task.

· Contractor Roles and Responsibilities for General Servers and Virtualization Support

1. Provide system administration and operational support for server infrastructure, hypervisor, and Operating Systems. Contractor shall design and implement automated fail-over and automated recovery solution.

2. Implement, and maintain a Automated Configuration Management, by using tools such as Puppet, Chef, Ansible, or other according to CBP OIT direction to ensure consistent configurations on all server operating systems.

3. Manage and monitor the performance, utilization of the various server resources, support the analysis as well as correction of all end-user computing and network problems that may be associated with server processing. Contractor shall design and implement automated resource management of virtual assets to include automated corrective actions for systems experiencing performance issues related to the technical stack from Operating System to Infrastructure (i.e., does not include middleware/Java/etc).

4. Design, implement, and maintain a hybrid cloud utilizing CBP OIT directed hypervisor and cloud management suite(s) and integrate with multiple public cloud service providers.

5. Automate the end-to-end service fulfilment for application hosting (e.g., automate virtual machine creation, OS installation and configuration, IP addressing and DNS registration, Java container deployment and configuration, and integration with access management solution).

6. Design, implement, and maintain a Configuration Management Database (CMDB). Update the documentation for all server operations procedures and services and maintain accurate information about all Configuration Items (CIs) in the CMDB.

7. Utilize tools and processes, such as software distribution, backups, performance measurement, tuning, license and asset management to allow automated as well as remote systems management of the server environment.

8. Provide Tier 3 support to resolve incident, problem, or service requests that require subject matter expertise and cannot be resolved by the EOC’s Tier 2 support. Tier 3 support personnel shall provide final problem resolution.

9. Develop and maintain automation (e.g., scripts, existing product features) to replace processes which are manually performed as directed by management.

10. Work with EOC to implement automated preventative measures, proactive monitoring and system self-healing capabilities to limit outages/performance issues which impact service levels.

11. Provide a weekly and roll-up monthly Performance Baseline and Operational Metrics report to include metric name/type, Acceptable Quality Levels (AQL), actual performance (raw performance), vendor performance (excludes issues outside of vendor control), vendor rating based on vendor performance, running 3 month average performance, details and explanations supporting exclusions and related information, comments, planned actions for next period, relevant supporting data, event log to include outage time, exclusion time, total availability, and details related to exclusion

12. Provide a monthly Resource Allocation report to include metrics associated with support staff (by technical area and skill set). This report shall provide utilization of resource to include hours expended against normal operations, hours expended against incident resolution, hours expended against problem resolution, and hours expended against project support (including technical reviews, testing, documentation, etc.). These reports will also be utilized for performance evaluation of service levels to include the workforce reduction as a result of implementing automated and improved processes.

13. Contribute to the Root Cause Analysis efforts when applicable.

14. Coordinate with CBP OIT as directed to establish an automated and on-demand capability for the reporting of Capacity/Performance to include the measurement and analysis of resource utilization and trends for both physical and virtual assets. Contractor shall also establish a dashboard for performance monitoring and resource utilization including Real-Time and Historical trends.

15. Provide remote support for OS and standard software build installation activities as required to meet SLAs.

16. Perform system or component configuration changes necessary to support Server Virtualization products.

17. Manage storage on the virtualization environment to minimize cost and overhead. Contractor shall ensure automated processes do not over allocate and negatively impact the stability of the environment.

18. Provide and maintain security hardened database baseline configurations (software) and deploy security (patches), software maintenance (patches) and new software releases across the Database and supporting products infrastructure

2.3 Task 3 – Enterprise Systems Engineering

The Contractor shall provide CBP OIT with support for business needs with appropriate technology (hardware or software). This includes architectural design, engineering design, proof of concept development, pilots, analysis, results, and documentation.

2.3.1 Task - Technology Refresh

The Contractor shall support CBP OIT in deploying new hardware, software and operating systems as part of CBP OIT standard and emerging technology refreshment cycle as designated by the CBP OIT.

The Contractor shall be responsible to provide Technology Refresh support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Technology Refresh

1. Support the identifaction and development of Enhancements and Upgrade Recommendations to be presented to CBP OIT that shall include: potential improvements to current CBP OIT technology and business processes, forecasting of viable future technologies, and new business processes that may be considered.

2. Support the development and maintenance of a Technology Refresh plan and schedule.

3. Assess the usability of the assets and review alternatives to replace, release, consolidate, or retain the assets. Deliver an initial recommendation regarding the assets to CBP OIT for review and approval.

4. Provide a Technology Refresh status report regarding implementation status as required.

5. Perform quarterly reviews of technology refresh plans to ensure they properly support CBP OIT business requirements.

2.3.2 Task – Systems Integration Management (SIM)

The main goal of SIM is to facilitate the alignment of customer requirements with computing infrastructure business and operational strategies. SIM focuses on business outcomes by linking service assets, services and customers. SIM is intent on employing infrastructure standards, performance engineering and capacity planning practices to reduce and prevent duplication of data, applications, systems, workload and services. One of SIM’s primary objectives is to utilize existing resources (infrastructure, services, applications, data, and human) for initiatives wherever possible.

The Contractor staff in all tasks areas shall be responsible to provide Overarching Process support to the CBP OIT. The overarching support shall span across the following subtasks:

· Business Relationship Management (BRM)

· Release Management (RM)

· United

· Capacity Performance Management

2.3.2.1 Business Relationship Management (BRM)

The Contractor shall be responsible to BRM support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Business Relationship Management

1. Serve as the primary point of contact for CBP OIT customers.

2. Communicate Service Level Agreements with CBP OIT customers.

3. Assist CBP OIT teams with meeting facilitation, QA, POC tracking and any other activities needed to ensure satisfaction with the quality and delivery of infrastructure needs.

4. Provide Agile services and expertise such as Scrum Master and Product Owner roles as requested by CBP OIT teams.

5. Communicate Service Catalog features to CBP OIT customers.

6. Receive, analyze, and verify requested services meeting customer requirements as part of Service Delivery Management.

7. Administer and report results of Customer Surveys.

8. Facilitate the remediation of Third Party VAT SCAN findings as necessary including care and feeding of the SharePoint site.

2.3.2.2 Release Management

The Contractor shall take a holistic view of a change to an CBP OIT service and shall verify that all aspects of a release, technical as well as non-technical, are considered together.

The Contractor shall be responsible to provide Release Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Release Management

1. Plan, oversee the successful roll-out of new/changed hardware and software, associated documentation in accordance with CBP OIT policies and procedures, including security, CBP OIT Release Management policy, processes, as well as associated procedures.

2. Validate that the release and deployment has produced the intended outcome. Review the release and deployment for improvements as well as document any work-around or correction.

3. Reevaluate the release and deployment process for failures as well as communicate results back to CBP OIT along with all relevant parties involved.

4. Provide a monthly Release report that includes the number of releases grouped by category as well as status, a planned release schedule, and progress reports against projects. For each release, provide CBP OIT with the reports and notices detailed in the Release Management process documentation.

2.3.2.3 United

United is a collection of applications which support the development community and is primarily composed of the entire Atlassian suite, and includes various third-party add-ons and other vendor solutions for Continuous Integration and Continuous Deployment.

The Contractor shall be responsible to provide support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Release Management

1. Atlassian tool (JIRA, Confluence, Bamboo, BitBucket, FishEye, etc.) management and administration, to include support for the Good Technologies and AirWatch MDM/MAM suite.

2. Perform tasks for Continuous Integration (CI), Continuous Delivery (CD) and Continuous Monitoring, including implementing, configuring and managing new tools in the UNITED environment.

3. Management and administration of other, non-Atlassian, CI/CD tools in the UNITED environment.

4. Offers and administers shared code repository (Subversion/Git/BitBucket).

5. Offers and administers automated testing tools in support of CI/CD.

6. Administer, manage and assist developer usage with code security testing (Fortify).

7. Provide 24x7 support for UNITED tool outages.

8. Maintain a comprehensive Knowledge Management repository.

2.3.2.4 Capacity and Performance Management

The Contractor shall be responsible to provide Capacity and Performance Management support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Capacity and Performance Management

1. Develop, operate, and maintain a Capacity and Performance Management process to plan, implement, measure, as well as manage the capacity of the services to confirm that the levels of capacity consistently meet CBP OIT business requirements and objectives.

2. Maintain the current capacity and utilization of each of the CBP OIT components that the Contractor manages, including hardware, software, licenses, voice and data circuits, IT Facility-related equipment and their subcomponents. Current tools consist of Oracle Enterprise Manager (OEM), Oracle Business Intelligence Enterprise Edition (OBIEE), AppDynamics

3. Support CBP OIT in understanding and assessing the future business requirements as well as forecasting CBP OIT capacity requirements.

4. Support CBP OIT in monitoring and validating the capacity forecast against CBP OIT actual utilization. Coordinate with CBP OIT infrastructure engineering staff to ensure that all current as well as future capacity and performance aspects of the business requirements are provided in a cost effective manner.

5. Validate whether CBP OIT as well as facility infrastructure has adequate capacity to meet the required demand in supporting the changes introduced through releases and deployments.

6. Develop, operate, as well as maintain an Availability Management process to plan, implement, measure, and manage the availability and reliability of the services to confirm that the levels of availability as well as reliability consistently meet CBP OIT business requirements and objectives.

7. Develop and deliver a Capacity Management Plan to CBP OIT every six (6) months, to include a forecast of growth, along with other changes in response to the projected business and operational needs.

8. Provide CBP OIT with written Capacity, Performance, and Availability reports encompassing the entire for all services supported by CBP OIT. The reports shall clearly indicate the number of exceptions that exceeded the thresholds, identify the problem areas within the Infrastructure, and show the trends, projected growth, as well as delta from the last established baseline per each Service.

2.3.3 Task – Enterprise Middleware Services

The primary goal of Enterprise Middleware Services is to provide engineering and operational support for Middleware technologies (e.g., XML, Web Services, MQ, DataPowers, and Service-Oriented Architecture), Web Services products (e.g.,WebLogic, WebSphere, Apache Web Server, Tivoli Access Manager and WebSEAL) and Enterprise Applications and Technical Projects based on customer requirements and mission needs while ensuring 24x7x365 O&M support for its CBP mission-critical systems relating to MQ/Middleware Operations and Web Services Operations.

The Contractor staff in all task areas shall provide overarching process support to the CBP. The overarching support shall span across the following subtasks:

· Transaction Messaging Services (TMS)

· Web Services

· Enterprise Applications and Technical Projects Services

2.3.3.1 Transaction Messaging Services

The contractor shall provide middleware support services for a standards-based global Middleware infrastructure that is designed to provide CBP OIT business partners, users, and Other Government Agencies (OGAs) the ability to process information rapidly and securely while ensuring data integrity, data security, and delivery. TMS provides engineering solutions for MW technologies, such as Web Services, MQ, DataPower SOA Appliances and Service-Oriented Architecture, based on customer requirements and mission needs while ensuring 24x7x365 O&M support for its CBP mission-critical systems. Additionally, MW supports enterprise-wide mission critical systems and applications as well as MQ Appliance testing and Mainframe migration to commodity cloud while supporting the IBM Websphere product suite (i.e., MQ, IIB, WDI, WTX, DataPower XML Security Gateway and IBM Connect Direct). TMS monitors the CBP OIT enterprise network in an effort to detect issues, minimize outages and deliver data messaging services while supporting the CBP OIT, airlines, trade partners, Value-Added-Networks and participating OGAs.

The Contractor shall provide support services to meet, at a minimum, the roles and responsibilities outlined in the following table:

· Contractor Roles and Responsibilities for Transaction Messaging Services

1. Implement and resolve connectivity with trade partners and Value Added Network (VAN) providers.

2. Provide data translation/transformation maps and resolve data translation/transformation mapping problems.

3. Engineer, implement and support custom coded and COTS-based system interfaces supporting the Middleware infrastructure. Expertise in Assembler Language, COBOL, C, C++, Perl, eSQL, SQL, Java, Unix Scripting (Bash & ksh), Java Script, Gateway Script, XLST, XML and XML Security Gateway appliances is required to troubleshoot technical issues and support custom coded software.

4. Track, maintain, and continuously improve upon system availability, system security, performance, and overall health of CBP OIT Enterprise Middleware Systems.

5. Respond to problems and work tickets within 24 hours for non-critical issues and 15 minutes for production critical issues from developers and users via phone calls, emails, or remedy tickets. Provide written resolution and/or updates to problems within 1 hour of closure or significant update.

6. Participate in CBP OIT outage calls within 5 minutes of notification for production environment during core business hours and within 15 minutes outside of core business hours during on-call support (phone, remote) for production environment to provide 24x7x365 support.

7. Maintain up-to-date documentation of all Middleware, MQ/DataPower, IBM product suite specific procedures and processes, system configuration baselines, implementation plans, system back-out plans, system recovery plans, maintenance plans, system upgrades, and all system changes. Maintain all project management and system documentation such as:

· Operational procedures

· Ad-hoc service procedures

· Off-Hours/emergency team contact list

· User manuals

· Announcements of new features, usage “tips”, and other similar information for timely distribution to user

· Configuration documentation

· Contingency planning and system recovery procedures

8. Review, evaluate and modernize current Messaging designs and architecture to ensure efficient and highly available services are provided to customers.

9. Proactively address MW infrastructure issues to decrease the potential of any service impacts and failures.

10. Validate whether the CBP OIT MW Infrastructure has adequate capacity to meet the required demands of CBP OIT and business growth.

11. Monitor the CBP OIT enterprise network in an effort to detect issues, minimize outages and deliver data messaging services while supporting the CBP OIT, airlines, trade partners and Value-Added-Networks.

12. Be a network subject matter expert to Airline Operators that CBP OIT interfaces with in support of various DHS Aviation and Trade applications.

13. Provide configuration services and Tier 3 support for all network hardware, software and protocols across multiple platforms and vendors.