DRAFT_-_Enterprise_Data_Center_Support_Services_(for_RFI)_060816_(10Jun16).docx

Statement of Work

Enterprise Data Center Support Services

For U.S. Customs and Border Protection Office of Information and Technology Enterprise Data Management and Engineering (EDME) Directorate

CBP Enterprise Data Center Support Services

Source Selection Sensitive See FAR 2.101 and 3.104 1

1. Background The mission of the Office of Information and Technology (OIT) is to provide U.S. Customs and Border Protection (CBP) with information, services, and technology solutions to secure the border, prevent the entry of terrorists or terrorist weapons, and to ensure the efficient flow of lawful people across U.S. borders. The OIT provides the U.S. Department of Homeland Security (DHS) and CBP with a 24x7x365 secure, stable and high-performance information technology infrastructure, scientific solutions, and forensic services.

The OIT has the responsibility for identifying and evaluating new technologies for applications in support of CBP business processes. A new era of economic downturn and terrorist threats require systems that can support the future vision of a unified, integrated, intelligence-driven agency to protect the United States against all threats against us. In order to be successful, the new Government and Contractor team must identify and acquire highly-skilled professionals from across industry that are capable of leading the technological revolution currently underway within OIT systems applications without abandoning and losing all existing critical corporate knowledge. Efforts are underway to include:

· Improving system availability to include re-engineering the current CBP infrastructure environments to provide local and geographic redundancies at the server, storage, network, database, and application layers. The availability and reliability of CBP’s mission critical application services is vital to supporting the screening of passengers and cargo coming into the United States.

· Implementing the ITIL framework to establish best practices through a set of concepts, strategies, policies and procedures for managing the CBP information technology infrastructure, development and operations environments;

· Implementing Agile practices, in conjunction with our required DHS System Engineering Life Cycle, supports the practice of shorter and more succinct software/system delivery;

· Modernizing storage and media providing cost effective highly reliable, highly scalable, highly available, cost effective storage and backup solutions with capacity on demand;

· Continuing to modernize the CBP monitoring systems to provide auto discovery, automatic business processing groupings and threshold and failure alerts to enable the Enterprise Data Center Operations Group to proactively and un-disruptively respond to capacity and performance thresholds and hardware failures;

· Supporting modernization of systems such as the Treasury Enforcement Communications System (TECS), Passenger systems, Cargo systems and Automated Commercial Environment (ACE);

· Engineering, implementing and operating clusters of highly available and scalable commodity servers that host discrete services which are leveraged by multiple projects (i.e. Structured Query Language (SQL), Oracle, WebSphere, Message Queue (MQ), etc.); and

· Transitioning and migrating production CBP systems and applications to operate at multiple data centers (including but not limited to Department of Homeland Security Data Centers, CBP Sponsored Data Centers, and commercially provided Data Centers).

2. Scope The scope of this Task Order (TO)contract will is to acquire services to support the engineering and operations of the CBP enterprise infrastructure. The primary, mandatory objective is to provide24 hours a day, seven days a week, 365 days a year (24x7x365) systems availability and reliability. Work shall be conducted in accordance with all DHS/CBP/OIT policies, standards, directives, processes, procedures and practices. This includes working with CBP Government and Contractor personnel, other government agencies, and vendors to support DHS/CBP/OIT mission and goals. The Agile methods of project management of automated systems shall be used to develop work products / artifacts that are compliant with current and future DHS/CBP/OIT policies, standards, processes, procedures, and guidelines.

CBP data center processing is performed at multiple data centers within the continental United States. Data centers currently providing services to CBP include the following - the CBP National Data Center (NDC) Complex located in Springfield, Virginia; the Stennis Data Center (DC1) at Space Center in Mississippi (supports multiple DHS Agencies, including CBP); the DHS Data Center 2 (DC2) in Clarksville, Virginia (supports multiple DHS Agencies, including CBP); the Finance Center located in Indianapolis, Indiana; and other locations throughout the Washington, D.C. metropolitan area.

The Government will retain all authority over access to systems and applications.

3. Applicable Documents The following DHS/CBP/OIT policies, standards, directives, processes and procedures are applicable to performing work under this Task Order.

· DHS Acquisition Instruction/Guidebook #102-01-001: Appendix B, Systems Engineering Life Cycle, and support CBP processes found in online Process Asset Library (PAL);

· DHS Sensitive Systems Policy 4300A;

· DHS National Security Systems Policy Handbook 4300B;

· DHS National Security Systems Policy 4300 B;

· DHS National Security Systems Handbook 4300 B;

· CBP Information Systems Security Policies and Procedures Handbook (HB 1400-05D);

· Federal Information Processing Standards (FIPS) 199, Standards for Security Categorization of Federal Information and Information Systems;

· CBP OIT Change Management Handbook;

· CBP Technical Reference Model (TRM);

· DHS Enterprise Service Bus (ESB) Standards;

· DHS Service Oriented Architecture (SOA) Standards;

· Federal Enterprise Architecture;

· National Information Exchange Model (NIEM);

· International Electronic Data Interchange (EDI) Standards; and

· Information Technology Infrastructure Library (ITIL) Best Practices;

· Agile Development Methodology.

4. Specific Tasks The service requirements included in this Performance Based Statement of Work (PWSSOW) are summarized in a Performance Requirements Summary (PRS) which is included as Appendix E. The PRS lists:

1. Each performance requirement (task);

2. SOW reference;

3. Performance standard for each requirement/task;

4. Acceptable quality level (AQL) corresponding to the performance standard;

5. CBP person(s) responsible for monitoring Contractor performance; and method of assessment.

The PRS provides a brief description of the minimum performance standards, called the AQL that the Contractor must maintain for acceptable accomplishment of the task requirements. CBP will use these performance standards to assess the overall performance of the Contractor against the SOW requirements. Performance requirements listed in the PRS relate directly to mission-essential services. These thresholds are considered critical to project success. Prioritization and tier definitions are included in Appendix D.

In coordination with and support of the migration of CBP applications off of the Mainframe, the Contractor shall support the migration legacy applications and infrastructure to the enterprise target standards. Migration of Applications will also include engineering and supporting of Applications located in Commercial Cloud infrastructure as a hybrid expansion of the CBP Cloud Computing Environment. This task includes the development and deployment of automation (i.e., scripts and/or software products) that further automates manual recurring business processes with the goal of reducing the manpower needed to support Database Operations.

The tasks in the PWSSOW are labor hours and materials (T&M) requirements.

Task 1: Enterprise Middleware Services Task 2: UNIX/Linux Services Task 3: Network Services Support Task 4: Enterprise Storage Task 5: Enterprise Virtualization Services Task 6: Windows Server Services Task 7: Enterprise Operation Testing & Evaluation Support Task 8: Mission Resumption Support Task 9: IT Continuity Management Support Services & Continuity of Operations Liaison Support Task 10: Program Management and Infrastructure Delivery Services

Task 13: Mainframe OS and Network Operations and Technical Support Task 14: Mainframe Operations and Technical Support - CICS Task 15: Enterprise MQ/DataPower Support Services Task 16: IT System Security Task 17: Database Services Task 18: Identity Credential Access Management (ICAM) Task 19: Oracle Enterprise Manager (OEM)

4.1 Task 1: Enterprise Middleware Services

Provide middleware support services for a standards-based global Middleware infrastructure that is designed to provide CBP’s business partners, users, and other government agencies the ability to process information rapidly and securely while ensuring data integrity, data security, and delivery. Engineer, implement and operate all components of the CBP enterprise Middleware infrastructure 24x7x365. Ensure uninterrupted systems availability, improved system response times, improved processing throughput, and support CBP agency-wide performance improvement goals. When applicable, Agile methodologies will be used to shorten the time it can take operations to bring a process into production, as well as include Middleware engineering services into Agile efforts to expedite product/service delivery to the customer.

4.1.1 Performance Requirements

The Contractor shall perform the following:

4.1.1.1 Support Middleware services on WebLogic, WebSphere, JavaBeans Open Source Software Application Server (JBOSS), IBM zOS, IBM AIX, Sun Solaris, Redhat Linux, Oracle Linux and Microsoft Windows servers. Provision and maintain WebSphere Application Server (WAS) services, WAS instances, data translation and data transformation maps.

4.1.1.2 Implement and resolve connectivity with trade partners and Value Added Network (VAN) providers.

4.1.1.3 Provide data translation/transformation maps and resolve data translation/transformation mapping problems.

4.1.1.4 Respond to problems and work tickets within 24 hours for non-critical issues and 15 minutes for production critical issues from developers and users via phone calls, emails, or remedy tickets. Provide written resolution and/or updates to problems within 1 hour of closure or significant update.

4.1.1.5 Create, maintain and update Standard Operating Procedures (SOPs), system documentation, infrastructure logical and physical diagrams, and maintain/update the Middleware knowledge base.

4.1.1.6 Engineer, monitor, and maintain all CBP enterprise integration services, reliable messaging, J2EE application servers and Enterprise Portals.

4.1.1.7 Track, maintain, and continuously improve upon system availability, system security, performance, and overall health of CBP enterprise Middleware systems.

4.1.1.8 Troubleshoot operational problems and issues, escalate problems to software vendors when necessary, and provide system diagnostics to vendor for problem analysis and resolution.

4.1.1.9 Provide system management/monitoring and capacity planning support for all components of the middleware infrastructure.

4.1.1.10 Engineer, implement and maintain external/internal data interconnectivity between CBP systems, the commercial trade community, the airline community, US government agencies, and foreign government agencies to facilitate the collection and exchange of data.

4.1.1.11 Engineer, implement and operate a reliable, highly available, secure, and scalable Enterprise Application server environment.

4.1.1.12 Engineer, implement and support custom coded and COTS/GOTS based system interfaces supporting the Middleware infrastructure. Expertise in Assembler Language, COBOL, C, C++, Oracle BPEL, J2EE, Java, and XML Security Gateway appliances is required to troubleshoot technical issues and support custom coded software.

4.1.1.13 Provide technical expertise throughout project development scrums (i.e. designing, testing, and production and operational readiness reviews) to ensure operational efficiencies are incorporated into the design and also to ensure seamless transition of middleware infrastructure components into the production environment.

4.1.1.14 Maintain up-to-date documentation of all Middleware specific procedures and processes, system configuration baselines, implementation plans, system back-out plans, system recovery plans, maintenance plans, system upgrades, and all system changes. Maintain all project management and system documentation such as:

· Operational procedures

· Ad-hoc service procedures

· Off-Hours/emergency team contact list

· User manuals

· Announcements of new features, usage “tips”, and other similar information for timely distribution to user

· Configuration documentation

· Contingency planning and system recovery procedures

4.1.1.15 Participate in CBP outage calls within 5 minutes of notification for production environment during core business hours and within 15 minutes outside of core business hours during on-call support (phone, remote) for production environment to provide 24x7x365 support.

4.1.1.16 Maintain an on-call rotation schedule that must be provided to CBP management and provide on-call phone/pager support 24x7x365.

4.1.1.17 Provide standard templates for requirements gathering.

4.1.1.18 Provide audit remediation support as needed.

4.1.1.19 Engineer mid to long term solutions to operational issues by providing corrective action plans, documenting lessons learned, tiger team and Root Cause Analysis support. Provide system enhancement recommendations to continuously improve efficiency, and availability.

4.1.1.20 Track license usage for all products utilized.

4.1.1.21 Engineer solutions and provide operational support for enterprise business continuity requirements which include the support for multiple geographically disperse data centers in either active/active or active/passive architecture.

4.1.1.22 Participate in and actively support continuity of operations plan (COOP) and disaster recovery (DR) Plan exercises.

4.1.1.23 Support and comply with all DHS/CBP/OIT IT project management methodologies and best practices (e.g. ITIL, SELC, Agile, etc.).

4.1.1.24 Provide operations status reports pertaining to summary of all incidents, system utilizations, patch/upgrade plans, recommendations, required capacity projections, and other information relevant to operation and maintenance of the systems.

4.1.1.25 Provide briefings, presentations, and reports that will be presented to all Middleware internal and external stakeholders to include CBP and DHS Executive Management.

4.1.1.26 Research, test, evaluate and provide technical recommendations for emerging technologies.

4.1.1.27 Provide support for Quality Assurance and Configuration Management (CM) activities to include:

· Manage and optimize CM plans, principles, processes, and implementation.

· Propose naming conventions of Configuration Items (CI) as uniquely identifiable; report on user compliance with identification standards for object types, environments, processes, lifecycles, documentation, versions, formats, baselines, releases, and templates.

· Create processes and procedures, which include CI registration procedures and access control and privileges; and ensure that roles and responsibilities are defined in the asset and CM plans and procedures.

· Manage document baselines and coordinate with Operations and Engineering to update documents after changes.

· Ensure that only authorized and identifiable configuration items are accepted and recorded in the CMDB throughout its lifecycle.

· Verify correct information is recorded and follow up with regularly scheduled audits.

· Participate in root cause analysis (RCAs) from unsuccessful or partially successful changes. Findings from the RCAs will be used to improve or enhance quality assurance processes.

· Inspect, audit, and certify all implementation and change activities for comprehensive test/validation, implementation and back out plans.

4.1.1.28 Ensure that all systems are engineered and operationally maintained to comply with any and all CBP/DHS/Federal Government IT security policies, procedures, and standards including patch/release versions and hardened security configurations.

4.1.1.29 Support the development and on-going maintenance of enterprise security agreements and system integration policies such as Memorandums of Understanding (MOU), Interagency Security Agreements (ISA), and Interagency Connection Documentation (ICD).

4.1.2 Deliverables and Delivery Schedule

The work products shall be delivered in accordance with the schedule set forth in the Table below:

4.1.2.1 System Outage Report: The Contractor shall provide a single page report outlining date/time/ problem description and resolution if available.

4.1.2.2 Consolidated system outage report and trends: The Contractor shall provide a consolidated system outage report and trends that shall include all planned and unplanned outages describing root cause (human or technology), duration of outage and any potential corrective action.

4.1.2.3 Status Reports: The Contractor shall provide a Status Report depicting work activities conducted during a weekly timeframe.

4.1.2.4 Incident Notification: The Contactor shall provide a brief description of an incident sufficient to alert stakeholders and coordinate a rapid response.

4.1.2.5 System Enhancement Recommendations: The Contractor shall provide recommended enhancements to the system to improve efficiency, performance, usability, and availability.

4.1.2.6 Standard Operating Procedures: The Contractor shall develop / maintain SOPs that define system, procedures, steps, and points of contact.

Incident Notification

Within 10 minutes of the incident occurrence or immediately if the system is completely down and not functioning

System Enhancement Recommendations

Standard Operating Procedures

4.2 Task 2: UNIX/LINUX Services

The primary focus of this task is to proactively design, implement and support all areas and components of the UNIX/Linux infrastructure. Unix/Linux Services currently supports and manages Oracle Solaris, Oracle Linux, SUSE Linux Enterprise Server, and Red Hat Enterprise Linux. These services include reliability, availability, scalability, capacity on demand, performance optimization, inter-operability, problem analysis/resolution, configuration management, patch management, disaster recovery, capacity planning, operational redundancy, contingency planning, integration, and operations of all components (hardware and software) across the CBP enterprise architecture.

4.2.1 Performance Requirements

The Contractor shall perform the following:

4.2.1.1 Standard Engineer, integrate, implement, and operate CBP’s UNIX/Linux operating systems (Oracle Linux & Solaris, Red Hat Enterprise Linux and SUSE Linux Enterprise Server) and commercial off the shelf (COTS) products deployed across the entire enterprise, to support 24x7x365 availability for CBP’s mission critical applications. Expert knowledge is required for core UNIX/Linux operating systems in functional areas of administration, backup, configuration standards, monitoring, tuning (to obtain optimum system efficiency and system utilization), virtualization (that includes server consolidation, storage consolidation, and automation that includes seamless server/software/storage provisioning and non-disruptive workflow/application/data migrations) and scripting for languages in C, C++, J2EE, Java, various UNIX shells, and Perl but not limited to these.

4.2.1.2 Provide support services to ensure that all UNIX/Linux hardware, firmware, and software interfaces associated with the CBP network are designed, deployed and maintained in a manner that optimizes the processing of mission critical application and data.

4.2.1.3 Provide problem determination/analysis to isolate system problems utilizing diagnostic and system management tools (i.e. system dumps and system/component traces, monitoring tools, vendor’s knowledge base). Provide system enhancement recommendations to improve efficiency, performance, usability, and availability. Provide audit remediation support as needed and develop corrective action plan for remediation.

4.2.1.4 Perform server hardware/software patching (service packs/security patches) within 30 calendar days of notice for critical patches, and as directed by the Government for all other patches. Use Government provided, vendor specific or Commercial Office the Shelf (COTS) tools to automate scheduling and deployment of hardware/software patches across the UNIX/Linux infrastructure.

4.2.1.5 Develop and maintain up-to-date documentation of all procedures, infrastructure diagrams, system configuration and baseline standards, implementation plans, system back-out plans, system recovery plans, maintenance plans, system upgrades, and all system changes.

4.2.1.6 Ensure completion of deliverables, track milestones and adherence to timeliness across projects, and institute necessary workflow alterations.

4.2.1.7 Participate in CBP outage calls within 5 minutes of notification for production environment during core business hours and within 15 minutes outside of core business hours during on-call support (on-site, phone, remote) for production environment to provide 24x7x365 support.

4.2.1.8 Maintain an on-call rotation schedule that will be provided to CBP management and provide on-call phone/pager support 24x7x365 for the production environment.

4.2.1.9 Provide support services using the Remedy ticketing system.

4.2.1.10 Design solutions that support business continuity with the ability that provide to transfer services from site to site.

4.2.1.11 Participate in Continuity of Operations Plan (COOP) and Disaster Recovery (DR) Plan exercises.

4.2.1.12 Implement and provide documentation of server and software system changes; submit all software and hardware changes to production environment to the Change Control Board (CCB) for approval prior to implementation.

4.2.1.13 Develop the following documentation and ensure it is continually updated as the affected systems change. Documentation requirement may be database, web-based, or in Word or Excel files at the determination of the Government.

4.2.1.14 Operational procedures

4.2.1.15 Ad-hoc service procedures

4.2.1.16 Off-Hours/emergency team contact list

4.2.1.17 User manuals

4.2.1.18 Announcements of new features, usage “tips”, and other similar information for timely distribution to user

4.2.1.19 Configuration documentation

4.2.1.20 Contingency planning and system recovery procedures

4.2.1.21 When directed by the Government, coordinate and work directly with hardware and software vendors to identify and recommend system solutions.

4.2.1.22 Provide technical expertise throughout system lifecycle stages (i.e. designing, testing, and production and operational readiness reviews) to ensure operational efficiencies are incorporated into the design and also, to ensure seamless transition of infrastructure components into the production environment.

4.2.1.23 Follow all standard processes outlined in the CBP OIT Change Management Handbook for all change requests.

4.2.1.24 Research, test, evaluate and provide technical recommendations for emerging technologies. Follow CBP processes for selected emerging products to submit Decision Request and Technology Insertions.

4.2.1.25 Provide support for all CBP Production, Development, TEST, System Acceptance Test (SAT), Disaster Recovery and Quality Assurance Testing (QAX) environments and other strings (e.g. test, DR).

4.2.2 Deliverables and Delivery Schedule

The work products shall be delivered in accordance with the schedule set forth in the Table below:

4.2.2.1 Incident Notification: The Contractor shall provide a brief description of an incident sufficient to alert stakeholders and coordinate a rapid response.

4.2.2.2 Operations and Engineering Report: The Contractor shall provide a report that includes Operations and Engineering weekly significant accomplishments.

4.2.2.3 O/S Patching Status Report: The Contractor shall provide a report and maintain master inventory of server O/S patching status.

4.2.2.4 O/S Configuration Baseline Report: The Contractor shall provide an O/S Configuration baseline report that captures changes made to O/S parameters. Document monitoring and tuning of O/S parameters; all systems will be evaluated against these parameters.

4.2.2.5 Task and Project Schedule: The Contractor shall develop and maintain a backlog, list of tasks, deliverables, milestones (i.e., Project or Excel). The Contractor shall also be required to integrate schedule with upper level enterprise schedules as well.

4.2.2.6 Contractor On-Call Schedule: The Contractor shall develop and maintain an on-call schedule on SharePoint accessible by Technology Operations Center (TOC) staff.

4.2.2.7 Meeting Minutes / Reports: The Contractor shall document meeting minutes for planning sessions and engineering review meetings; provide reports such as lessons learned or hardware/firmware reports.

4.3 Task 3: Network Services

The Contractor shall provide services to support the engineering and operations of all data center LAN infrastructure, management tools, and Software Defined Network (SDN). Including but not limited to the design, configuration, documentation and maintenance of all LAN network devices supporting servers and applications hosted within the CBP data center, DC1 and DC2 enclave. Integrate, implement, operate, administer and, provide on-call support along with maintaining all components of the CBP LAN network 24x7x365. Work shall be conducted in accordance with all DHS/CBP/OIT policies, standards, directives, processes, and procedures. The CBP LAN infrastructure currently consists of Cisco switches and Load Balancers, Brocade switches, Juniper firewalls and F5 load balancers running Enterprise operating systems.

4.3.1 Performance Requirements

The Contractor shall perform the following:

4.3.1.1 Provide on-site engineering support during core business hours.

4.3.1.2 Participate in CBP outage calls within 5 minutes of notification for non-production and production environment during core business hours and within 15 minutes outside of core business hours during on-call support (on-site, phone, remote) for production environment to provide 24x7x365 support.

4.3.1.3 Provide CBP 24x7x365 on call production LAN phone/remote access support.

4.3.1.4 Report all detected LAN faults/failures to CBP/OIT management and duty officers within 10 minutes of detection.

4.3.1.5 Develop, maintain, and provide the government documentation for all data center LAN hardware and SDN, connection configurations, and cable lists. This includes, but is not limited to maintaining LAN configuration artifacts detailing hardware configurations, rack elevations, connectivity, and software configurations.

4.3.1.6 Provide support services using the Remedy ticketing system.

4.3.1.7 Perform weekly backups of all switch configurations and store them in a central repository accessible by all LAN engineers.

4.3.1.8 Ensure all LAN devices are entered properly into Network monitoring tools including but not limited to CISCO Works, Opsware Solarwinds and eHealth.

4.3.1.9 Coordinate and work closely with various government organizations (such as the wide area network (WAN) teams, CBP OIT management, and Duty Officers) and hardware and software vendors on the delivery of new network capabilities/services, system changes, and problem resolution, which include troubleshooting, isolating, and remediating network performance and network faults.

4.3.1.10 Maintain, support and provide 24x7x365 availability of redundant hardware load balancing for CBP production applications. All system maintenance will be implemented non-disruptively.

4.3.1.11 Non-disruptively upgrade and maintain all data center LAN hardware BIOS, OS and firmware on a weekly, monthly or as needed basis support to deliver high availability and security.

4.3.1.12 Provide, update and maintain pro-active and fault monitoring for all Data Center LAN Components

4.3.1.13 Maintain tools (hardware, software) that provide rapid troubleshooting and fault / problem isolation.

4.3.1.14 Capture provisioning and change metrics and provide daily, weekly and monthly capacity utilization reports for all LAN network components.

4.3.1.15 Support and maintain network security systems to provide LAN security and network segregation and isolation between the production, test, and development networks.

4.3.1.16 Test power connection redundancy on all new hardware installations and test legacy switches.

4.3.1.17 Adhere to data center working SOPs (e.g., access control, work area cleanliness)

4.3.1.18 Perform network hardware/software patches (service packs / security patches) within three (3) calendar days of notice for critical patches; monthly for all other patches.

4.3.1.19 Provide audit remediation support as needed.

4.3.1.20 Provide corrective action plan support as needed.

4.3.1.21 Perform classified spillage clean up if required.

4.3.1.22 Participate in continuity of operations plan (COOP) and disaster recovery (DR) Plan exercises.

4.3.1.23 Perform quarterly redundancy tests and document results.

4.3.1.24 Provide a weekly status report that details key meetings, work performed, work planned, issues, hardware / software requirements, and performance / availability metrics.

4.3.1.25 Analyze, engineer, coordinate, test and implement network configurations for new systems to include IP addressing, sub-netting, VLAN management / routing, local and geographic server and application load balancing, network redundancy, network traffic segregation (application, backup, monitoring, management) in support of CBP applications and infrastructure requirements.

4.3.1.26 Non-disruptively deploy new switches to provide additional capacity and redundancy.

4.3.1.27 Deploy redundant connections from servers to redundant switches. The Data Center LAN All servers will be connected with redundant application, backup, monitoring and management connections. Heartbeat connections are provided only when required by the application.

4.3.1.28 Confirm all system changes and upgrades have been rigorously tested and coordinated with program offices prior to presenting change request to the Change Control Board. All changes must contain implementation plans and back-out plans that have been tested.

4.3.1.29 Ensure system changes do not compromise the integrity of the foundation system redundancy capabilities.

4.3.1.30 Pro-actively engineer and implement changes to support growth and capacity demands ensuring the ability to transfer services from site to site.

4.3.1.31 Use industry best practices (e.g. ITIL) to compile, maintain current, and make accessible to IT staff a repository of all specifications, manuals, drawings, repair standards, parts lists, inventories, CM documentation, Contractor-developed government owned software programs, capacity/usage data, and service request tracking system documentation to properly maintain, supply, support and upgrade the infrastructure.

4.3.1.32 Develop the following documentation and ensure it is continually updated as the affected systems change. Documentation requirement may be database, web-based, or in Microsoft Word or Excel files at the determination of the Government.

· Operational procedures

· Ad-hoc service procedures

· Off-hours/emergency team contact list

· User manuals

· Announcements of new features, usage “tips”, and other similar information for timely distribution to user

· Configuration documentation

· Contingency planning and system recovery procedures

4.3.1.33 Provide technical expertise throughout project scrum activities (i.e. designing, testing, production and operational readiness reviews) to ensure operational efficiencies are incorporated.

4.3.1.34 Provide feedback on all activities involving systems administration and information system design and development.

4.3.1.35 Research, test, evaluate and provide technical recommendations for emerging technologies. Provide system enhancement recommendations to improve efficiency, performance, usability and availability.

4.3.1.36 Perform quality assurance validation for all installations, initiate actions to correct deficiencies, and capture metrics on deficiencies.

4.3.1.37 Provide operations status reports pertaining to summary of all incidents, system utilizations, patch/upgrade plans, recommendations, required capacity projections, and other information relevant to operation and maintenance of the systems.

4.3.2 Deliverables and Delivery Schedule

The work products shall be delivered in accordance with the schedule set forth in the Table below:

4.3.2.1 Status Report: Report that provides status of all EDME LAN Tasks. This report shall include the project name, requester, engineer assigned, project implement data and miscellaneous comments.

4.3.2.2 Network Switch Log & Host Mapping: The Contractor shall develop and maintain a log of hosts attached to network switches and application impact map. This shall include a host based list of all devices connected to the network to identify application impact during network configuration changes.

4.3.2.3 Error Trend and Remediation Report: An analysis report to include but not limited to network traffic utilization and errors and recommended solutions.

4.3.2.4 Incident Notification: A brief description of an incident sufficient to alert stakeholders and coordinate a rapid response along with completing the incident questionnaire.

4.3.2.5 System Enhancement Recommendations: Describes recommended enhancements to the system to improve efficiency, performance, usability, and availability.

4.3.2.6 Configuration Management Support Plan: The plan shall establish a baseline and include system design, hardware and software configurations.

4.3.2.7 Standard Operating Procedures: SOPs define system, procedures, steps, and points of contact.

4.3.2.8 Incident Notification: The Contractor shall provide a brief description of incident sufficient to alert stakeholders and coordinate a rapid response.

4.3.2.9 Operations and Engineering Report: The Contractor shall provide a report that includes Operational and Engineering weekly significant accomplishments, deliverables and milestones and monthly goals.

4.3.2.10 Contractor On-Call Schedule: The Contractor shall develop and maintain an on-call schedule on SharePoint accessible to the Government LAN staff.

Status Report

Network Switch Log & Host Mapping

4.4 Task 4: Enterprise Storage

Availability and resiliency of our storage is critical to CBP meeting its mission. CBP operates close to 16 petabytes of enterprise class storage across multiple data centers. As part of our modernization initiatives CBP has implemented both a virtual and tiered storage approach from high speed fiber channel drives to dense economy storage. We have implemented two 8-gig fabrics with over 4,000 ports of active capacity. Several implementations and critical applications have moved to boot and data on storage area network (SAN) with diskless servers. Our reliance on this technology demands that maintenance be non-disruptive and that any changes ensure no interruption of service within the environment. CBP has recently moved toward Cloud Based architectures which include, but are not limited to, Oracle Exadata and ZFS storage appliances. As CBP continues to modernize, enterprise storage continues to escalate in its criticality to every minute of CBP’s operating ability globally.

4.4.1 Performance Requirements

The Contractor shall perform the following:

4.4.1.1 Provide storage management support services for the enterprise infrastructure to assure acceptable levels of space utilization, space availability, direct access storage devices (DASD) performance, data availability, and catalog management.

4.4.1.2 Provide technical management for related hardware devices which include Storage Subsystems (Appliance based, SAN, NAS, DASD, directors, tape libraries, tape robots, and communications switches). Expertise with Sun/Oracle, Hitachi, EMC, NetApp storage, and Storage Virtualization will be required to perform this task.

4.4.1.3 Provide integration of a commercial cloud into storage architecture.

4.4.1.4 Troubleshoot and repair host bus adapter (HBA) and switch network problems.

4.4.1.5 Test hardware/software patches within the lab and at our continuity site prior to deployment to production.

4.4.1.6 Monitor storage consumption and provision storage on demand when necessary in a non-disruptive manner.

4.4.1.7 Provide capacity forecasting for SAN and switch fabrics.

4.4.1.8 Implement, maintain, and optimize storage replication for all platforms.

4.4.1.9 Participate in new application service planning sessions.

4.4.1.10 Coordinate hardware and software issues to resolution.

4.4.1.11 Document standard procedures for provisioning, maintenance (including patches/upgrades), and optimization of Storage Platforms and components.

4.4.1.12 Support the evaluation and implement new SAN products and integrate into CBP enterprise solution offering.

4.4.1.13 Implement, configure, and validate patches/upgrades to all SAN and Network Attached Storage (NAS) components.

4.4.1.14 Provide performance and status reports - daily, weekly, monthly.

4.4.1.15 Maintain an on-call rotation schedule that will be provided to CBP management and provide on-call phone/pager support 24x7x365 for the production environment.

4.4.1.16 Participate in CBP outage calls within 5 minutes of notification for production environment during core business hours and within 15 minutes outside of core business hours during on-call support (on-site, phone, remote) for production environment to provide 24x7x365 support.

4.4.1.17 Participate in new application service planning sessions and engineer performance optimized solutions.

4.4.1.18 Provide audit remediation support as needed.

4.4.1.19 Track license usage for all products utilized.

4.4.1.20 Perform system preventative maintenance including hardware, software, BIOS and firmware on a daily, weekly, monthly or as needed basis to maintain system performance.

4.4.1.21 Perform data spillage clean up if required in accordance with established procedures.

4.4.1.22 Provide support services using Standard CBP Reporting tools.

4.4.1.23 Design solutions that support business continuity by providing for the ability to transfer services from site to site.

4.4.1.24 Support continuity of operations plan (COOP) and disaster recovery (DR) Plan exercises and develop contingency plans and disaster recovery procedures.

4.4.1.25 Use industry best practices (e.g. ITIL) to compile, maintain current, and make accessible to IT staff a repository of all specifications, manuals, drawings, repair standards, parts lists, inventories, change/configuration management documentation, Contractor-developed software programs, capacity/usage data, and service request tracking system documentation to properly maintain, supply, support and upgrade the infrastructure.

4.4.1.26 Provide operations status reports pertaining to summary of all incidents, system utilizations, patch/upgrade plans, recommendations, required capacity projections, and other information relevant to operation and maintenance of the systems.

4.4.1.27 Provide system enhancement recommendations to improve efficiency, performance, usability, and availability.

4.4.1.28 Provide standard operating procedures which define storage systems, operational procedures and points of contact for each storage vendor.

4.4.1.29 Follow all processes outlined in the CBP OIT Change Management Handbook for all change requests.

4.4.1.30 Ensure that storage frames and switch fabric are maintained in a fully redundant configuration to prevent any operational disruptions that may arise from hardware or software failure.

4.4.1.31 Ensure all hosts use multi-path Input/Output (I/O) drivers.

4.4.1.32 Use CBP best practices for storage provisioning (currently “Thin Provisioning”).

4.4.1.33 Coordinate tuning of storage I/O parameters with server administrators.

4.4.1.34 Ensure that storage is monitored via CBP provided automation tools and that the tool feeds into the CBP monitoring system.

4.4.1.35 Provide a Daily Storage Report via a web-based tracking tool. The report will include: usable storage, host allocations by server (allocated, used, free), consumption and performance trends and projections.

4.4.1.36 Provide support for Quality Assurance and Configuration Management (CM) activities to include:

· Manage and optimize CM plans, principles, processes, and implementation.

· Propose naming conventions of Configuration Items (CI) as uniquely identifiable; report on user compliance with identification standards for object types, environments, processes, lifecycles, documentation, versions, formats, baselines, releases, and templates.

· Create processes and procedures, which include CI registration procedures and access control and privileges; and ensure that roles and responsibilities are defined in the asset and CM plans and procedures.

· Manage document baselines and coordinate with Operations and Engineering to update documents after changes.

· Ensure that only authorized and identifiable configuration items are accepted and recorded in the CMDB throughout its lifecycle.

· Verify correct information is recorded and follow up with regularly scheduled audits.

· Participate in root cause analysis (RCAs) from unsuccessful or partially successful changes. Findings from the RCAs will be used to improve or enhance quality assurance processes.

· Inspect, audit, and certify all implementation and change activities for comprehensive test/validation, implementation and back out plans.

4.4.2 Deliverables and Delivery Schedule

4.4.2.1 The work products shall be delivered in accordance with the schedule set forth in the Table below. Daily Storage Report: Report shall include the daily status; any storage entities that are at risk due to consumption, imminent component failure or any other issues. Report shall also include updated status of any outstanding storage tasks and pending requests.

4.4.2.2 Weekly Storage Maintenance, Provision, and Issue Report: Report shall include the weekly maintenance that is performed; how much space has been provisioned and to who; any issues that need to brought to the Governments' attention; outstanding storage tasks; and report on pending requests.

4.4.2.3 Weekly Storage Utilization Slides: The Contractor shall develop a high level storage utilization slides that are presentation ready for weekly briefings to Senior Management.

4.4.2.4 Standard Tiered Solution Offering Report: Periodic reports illustrating enhancements in Vendor offerings slanted to tiering (i.e., recommendations that would improve the cost or performance of the storage infrastructure).

4.4.2.5 Map hosts to storage device: Report shall include every application server and illustrate all storage logical unit number (LUN), the array that it is provisioned on and the size and physical characteristics.

4.4.2.6 Error Trend and Remediation Report: Report shall include Weekly, Monthly, Quarterly and Annual metrics regarding storage errors. Report shall provide metrics for trending by type, by vendor and by device and include all remediation efforts addressing what corrective actions were taken with each error.

4.4.2.7 Incident Notification: Notification shall include a brief description of an incident sufficient to alert stakeholders and coordinate a rapid response.

4.4.2.8 Device Driver Distribution Plan: Report shall detail all failover testing done to ensure that CBP retains redundancy throughout its storage architecture.

4.4.2.9 Failover Testing Report: Report shall detail all failover testing done to ensure that CBP retains redundancy throughout its storage architecture. System Enhancement Recommendations: Annual report shall include recommended changes to storage architecture that would improve performance, usability, efficiency or availability.

4.4.2.10 Standard Operating Procedures: SOPs shall define system, procedures, steps, and points of contact.