Draft NOC NMSS PWS for Initial RFI release 11_06_2024.docx

Performance Work Statement (PWS) Networks Operation Center (NOC) Network(s)Management Support Services (NMSS)

U.S. Customs and Border Protection

Networks Operations Center Network(s) Management Support Services

PR # TBD

November 06, 2024

Background U.S. Customs and Border Protection (CBP) serves as a premier law enforcement agency enhancing the nation's safety, security, and prosperity through collaboration, innovation, and integration. With approximately 64,000 employees, CBP is one of the world's largest law enforcement organizations and is charged with intercepting illegal goods and entrants while facilitating lawful international travel and trade.

Purpose This contract requirement will provide support services for the Office of Information and Technology’s (OIT) Network Management that provides Network(s) support and is responsible for Network(s) Management, Operations, and Maintenance for all CBP Network(s) infrastructure. Support includes providing real-time incident monitoring, predictive and proactive analysis and reporting, incident resolution and escalation, problem management and resolution, coordinating of events related to network communications, network utilization and availability analysis and reporting. CBP Network Management also provides Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), Enterprise Data Management and Engineering (EDME) Directorate LAN, and Wireless Local Area Network (WLAN). The CBP Network is the focal point for network troubleshooting, updating, router and domain name management, and coordination with affiliated networks.

Enterprise Network Services Infrastructure CBP has a global presence with connectivity to approximately 1300 field sites as well as Other Government Agencies (OGAs) and trade partners. CBP conducts operations in 22 countries, 328 ports of entry, 129 border patrol stations within 22 sectors, and 75 Air and Marine Operations (AMO) locations. To support its mission, CBP needs a robust, resilient, and high performing enterprise network and supporting services 24x7x365 throughout its global footprint.

Historical Metrics The Network Services Division within the Office of Information and Technology (OIT) Enterprise Infrastructure and Operations Directorate (EIOD) provides network services and oversight of the CBP enterprise network infrastructure. The Network Services Division comprises the Network Architecture and Engineering Division (NAED) and Network Operations Division (NOD).

NOD provides 24x7x365 Network Operations Center (NOC) and Wireless Network Operations Center (WNOC) support services that include providing real-time incident monitoring, predictive and proactive analysis and reporting, incident resolution and escalation, problem management and resolution, coordinating of events related to network communications, network utilization and availability analysis and reporting. CBP Network Management also provides support to Enterprise Infrastructure in excess of 14000 devices (at the time this PWS was created) including Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), and Homeland Security Data Network (HSDN) devices all which service Department of Homeland Security (DHS) and its components, Trade partners, and other Federal agencies.

The CBP NOC/WNOC is staffed 24x7x365 with three (3) shifts of mid to senior level contract technicians. The facilities are located the Ashburn, Virginia, Springfield, Virginia, and in Orlando, Florida, and staffed as active/active operations centers. The active/active operation provides for increased situational awareness, increased availability, and the staffing to ensure continuity of operations should something happen to the other location.

Shift turnover is conducted between each shift and includes a comprehensive list of open events, scheduled/upcoming changes and other “critical” items as defined in other sections of this PWS. The Contractor shall support in excess of 14000 devices (at the time this PWS was created) via automated alerts as well as customer calls, emails and trouble tickets transferred from external support groups. Historical weekly tickets/ contact type volume estimates are listed below.

CBP must be on the cutting edge of emerging network technologies, while maintaining an integrated robust Network, secure, reliable WAN/LAN, and an infrastructure that delivers improved network reliability and performance (99.99%) to support all traffic that may traverse the network to meet mission requirements. CBP must leverage industry standard Information Technology Service Management (ITSM) service model utilizing Information Technology Infrastructure Library (ITIL) methodologies. CBP requires support to recommend changes that provide efficiency and cost savings and support a seamless transition to any planned software (e.g., enterprise monitoring, deep packet analysis) implementation or approaching initiatives.

Scope The Contractor shall provide NOC/WNOC management support services described herein to ensure that CBP is able to maintain operations on a 24x7x365 basis. This includes providing support at facilities located in Ashburn, Virginia, Springfield, Virginia, and in Orlando, Florida. The Contractor shall operate in the confines of a CBP Personal Identification Verification (PIV) card secured facility. The contractor must be organized, staffed, and equipped to manage CBP’s Network Enterprise Infrastructure consisting of more than 14000 Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), and Homeland Security Data Network (HSDN) devices (at the time this PWS was created) all which service Department of Homeland Security (DHS) and its components, Trade partners, and other Federal agencies; providing real time network operation monitoring and maintenance. Devices inclusive of but not limited to:

· Device cabling

· Routers

· Switches

· Firewalls

· Load-Balancers

· Specialized appliances or hardware that provide network services (e.g., Domain Name Services/Dynamic Host Configuration Protocol (DNS/DHCP), Load Balancing Support Services, Proxy Services, and Security Services)

· Wireless LAN Controllers

· Wireless Access Points

· Wide Area Network (WAN) Optimization

· Video

· Voice

· Cloud

· Software Defined Networks (SDN)

· Surveillance Cameras

· Cellular

· Network Security

· Emerging Technologies

· Secure Sockets Layer (SSL) Certificates

WNOC

· Analog Modem

· Base Radio

· Channel Service Unit/Data Service Unit (CSU/DSU)

· Core Gateway

· Device cabling

· Digital Interface Unit (DIU)

· GPS

· Microwave Radio

· Mobile Radio

· Portable Radio

· Radio Gateway

· Radio-net rack

· Repeater

· Routers

· Switches

· UPS

· VPM

These support services include Transition, Program Management, Network(s) Infrastructure Monitoring and Operational 24x7x365, Event Management, Incident Management, Problem Management, Application Management, Request Fulfilment, Technical Support, Knowledge Management and Performance Reporting Services, Information Systems Security Officer (ISSO) Support, and Surge Support, The Contractor shall support Virtual Private Networking (VPN) solutions, remote access services support, access Operational, and identity management tasks. The Contractor shall provide trained, qualified, and Background Investigation (BI) cleared staff to support these functions 24x7x365. The Contractor shall be held accountable for the performance requirements outlined in Appendix A, Performance Standards.

Applicable Documents The Contractor shall perform in accordance with the Applicable Documents outlined in this section. Specific to safeguarding CBP generated documents, which include policies and procedures that may contain sensitive information. These Documents will be made available to the Contractor for viewing at the appropriate time after contract award. The provided documentation must not leave the CBP facilities, be photocopied, nor be pen copied verbatim while viewing. All internal CBP network processes and procedures are based on widely accepted industry standard best practices and those customized to meet specific CBP requirements. These documents include escalation procedures, Standard Operating Procedures (SOP), and troubleshooting techniques.

The following provides guidance that the Contractor shall follow when safeguarding CBP documentation and materials.

· The Contractor shall Operational and safeguard For Official Use Only (FOUO) information in accordance with DHS Directive (MD 11042.1) Safeguarding Sensitive but Unclassified (FOUO) Information" dated January 6, 2005.

· All Contractor personnel is familiar with the CBP Security Policy and Procedures Handbook (HB 1400-05D) Version 7.0, dated November 16, 2017, Volume IV, Chapter 13, Safeguarding Sensitive but Unclassified (FOUO) Information.

· The DHS 4300A Information Technology System Security Program, Sensitive Systems, Version 13.3.

· National Institute of Standards and Technology Special Publication (NIST SP) 800-53 Guide for Information Security Program Assessments.

· System Reporting Form Standards for Security Categorization of Federal Information and Information Systems (FIPS 199).

· Section 508 of the Rehabilitation Act: http://www.section508.gov/

· CBP Information Systems Security Policies and Procedures Handbook.

· DHS Information Technology (IT) Security Policies (NIST800-47).

· Department of Defense Information Assurance C&A Process (DIACAP).

· Systems Engineering Lifecycle (SELC) DHS 102-01-103 Revision 01: DHS 102-01-103 Rev 01

· Systems Engineering Lifecycle (SELC) Framework: https://cbpgov.sharepoint.com/sites/CBP-OA/SitePages/SELC.aspx Performance Requirements The Contractor shall provide the skill set and expertise to achieve the performance objectives and requirements specified below as well as Appendix A.

Additionally, the Contractor shall have certification of the following:

· ISO/IEC 20000 (International Standard for IT Service Management) with ITILv3 Service Delivery Practice Applied

· Corporate IT Security Management (IT-SEC) System Certified ISO/IEC 27001:2013 Standard; NIST 800-53 Compliant Network Infrastructure Monitoring and Operational The Contractor is responsible for providing Network Infrastructure Monitoring and Operational services for the CBP Network(s). Services provided by the Contractor shall include, at a minimum, the services identified in Appendix E, CBP Network Management Catalog of Services. The Contractor is responsible for ensuring that CBP configuration items (CIs) and associated services supported by the CIs are constantly monitored and managed. The Contractor shall ensure the appropriate processes, skills and rules necessary for the effective and efficient Infrastructure Monitoring and Operational are in place. For those areas where there are not appropriate tools, processes, skills, or rules, the Contractor shall make recommendations to the COR and Government Supervisor (GS) for corrective actions necessary to implement a robust Infrastructure Monitoring and Operational capability.

Real-Time Monitoring and Trend Analysis The Contractor is responsible for providing 24x7x365 real-time monitoring, trend analysis and Operational services for the CBP Network(s) infrastructure. Utilizing Government Furnished Equipment (GFE) Network Performance Management Tools and/or real-time reporting engines, the Contractor shall actively monitor the CBP Network infrastructure for service affecting events as they occur. The Contractor shall constantly monitor the health/reliability of the CBP network(s) and search for trend analysis and other predictors of current or potential network issues. The Contractor shall also provide weekly and monthly continuous real-time network bandwidth monitoring.

Event Collection The Contractor is responsible for the collection and signaling of events in the CBP Network infrastructure through the utilization of various GFE Network Performance Management Tools and/or real-time reporting engines. The Contractor shall ensure that appropriate event occurrences from sources in both software and hardware such as network equipment, operating systems, application software and processors that are necessary to accurately depict the current state of the CBP Network are captured and properly displayed. The Contractor shall make recommendations for improving the event capturing capability of GFE Network Performance Management Tools to make event capture more effective.

Event Analysis – Incidents The Contractor is responsible for analyzing network events/alerts to detect network incidents such as failing routers, failed switches, failed application load balancer health checks or other incidents that are impacting the delivery of services to CPB users. If a service affecting incident is detected, the Contractor shall utilize the Incident Management Process to notify the appropriate Points of Contact (POCs) of any/all service affecting incidents. Notifications of service affecting incidents is made within established SOPs.

Events Analysis - Problems The Contractor is responsible for analyzing network events/alerts to detect slow or failing network components, such as failing routers, failing switches or other problems that have impacted or have the potential of impacting the delivery of services to CBP users. The Contractor shall utilize the Problem Management Process to notify the appropriate POCs of any/all service affecting problems that have the potential to impact the delivery of services to CBP users. Notifications of problems is made within established SOPs.

Network Operations and Maintenance Support The Contractor is responsible for providing Operations and Maintenance (O&M) support services for GFE Network Performance Management Tools and/or real-time reporting engines (e.g., NA, ServiceNow (if necessary), Steel Central Products, etc.). The Contractor is responsible for executing the day-to-day routine tasks related to the Operation and Maintenance of Infrastructure Monitoring and Operational Tools. The Contractor shall also be responsible for coordinating and scheduling the performance of infrastructure maintenance activities and for scheduling of any other activities that must be executed during the scheduled time. The Services provided by the Contractor shall include, at a minimum, the services identified in Appendix E CBP Network Management Catalog of services.

Network IT Infrastructure Monitoring and Operational Reporting The Contractor is responsible for regularly reporting the status of Monitoring and Operational efforts. The Contractor shall regularly collect metrics for and report on progress toward meeting Monitoring and Operational performance objectives outlined in the PWS and OIT’s assessment across planned domains such as overall availability, capacity network asset management, and network response.

Network Infrastructure Monitoring and Operational Support The Network Operations Lead is responsible for ensuring that all monitoring and Operational activities are carried out in a timely and reliable way. The Network Operations Lead is to ensure monitoring and Operational activities performed by contract personnel comply with existing CBP Processes and Procedures. The Network Operations Lead will ensure that monitoring personnel are trained in the tools, processes, and procedures required to perform the monitoring and Operational activities outlined in the PWS. The Network Operations Lead will make recommendations as necessary to the Government Supervisor (GS) for changes and enhancements to systems, including documentation, processes, and procedures, as necessary, to improve overall monitoring and Operational capabilities. The Network Operations Lead is responsible for reporting (daily, weekly, monthly) the status of Network(s) and Operational efforts to the GS.

Event Management The Contractor is responsible for providing Event Management Services for the CBP Network(s) Infrastructure. The Contractor is responsible for establishing an Event Management process that facilitates the filtering and categorizing of Events to decide on appropriate actions. The Contractor shall ensure the appropriate processes, skills, and rules necessary for effective and efficient Event Management are in place. For those areas where there are not appropriate tools, processes, skills, or rules, the Contractor shall make recommendations to the COR and GS that CBP could implement to improve Event Management capability.

Maintenance of Event Monitoring Mechanisms and Rules The Contractor shall establish and maintain rules and mechanisms for generating meaningful Events and effective rules for the filtering and correlating of events from Configuration Management (Configuration Items) are approved by the GS. Events are categorized, at a minimum, as informational and can be ignored or as warnings and exception events which require a response. Rules/mechanisms for generating, filtering, and correlating of events is fully documented and submitted to the GS.

Event Filtering and Correlation The Contractor shall present to the GS any events they think are informational or should be filtered for approval. The Contractor shall provide audible and visual alerts for managed Configuration Items.

Event Review and Response Selection The Contractor shall ensure that warnings and exception events for Configuration Items are reviewed, and the meaning of the event is interpreted correctly. The Contractor shall ensure that suitable automated or manual responses to warnings and exception events are implemented in accordance with the Event Management Process and CBP policies and procedures. Notifications of and responses to Events is made within established service agreements.

Event Review and Closure The Contractor shall conduct daily reviews to ensure that all events and warnings presented to network operators for action have been handled appropriately. The Contractor shall also ensure that Event logs are analyzed to identify trends or patterns. The Contractor shall utilize the analysis to suggest corrective actions where necessary to correct trends or patterns.

Event Management Reporting The Contractor is responsible for regularly reporting the status of Event Management efforts. The Contractor shall regularly collect metrics for and report on progress toward meeting Event Management performance objectives outlined in this PWS.

Event Management Support The Contractor shall identify and assign an Event Manager to perform the function of Event Management. The Event Manager is responsible for managing the Event Management process utilized by the Contractor to manage events. The Event Manager will ensure the Event Management process is fully documented and complies with CBP policies, processes, and procedures. The Event Manager will ensure that monitoring personnel are trained in the tools, processes, and procedures required to perform the Event Management activities outlined in the PWS. The Event Manager is responsible for updating the Event Management Process in response to changes to the CBP environment and for making make recommendations for changes and enhancements to systems, including documentation, processes, and procedures, as necessary, to improve the overall Event Management process.

Incident Management Support The Contractor is responsible for providing Incident Management services in support of the CBP Network Incident Management Process define by the GS. The Contractor shall ensure that incident management support personnel have the appropriate skills and training necessary for the effective and efficient handling of Incidents.

Incident Logging and Categorization The Contractor is responsible for providing Incident Logging and Categorization services utilizing GFE furnish tools to facilitate swift and effective Incident resolution. The Contractor is responsible for assigning criticality, impact level, and prioritizing events in accordance with established guidelines (See Appendix C: Network Event Categorizations). The Contractor shall follow documented triage procedures to identify if the incident is related to physical issues or logical issues. Incident Logging and Categorizations is accomplished within timeframes established by the CBP Incident Management Process or other SOPs.

Tier 1 Support Applicable to both NOC and WNOC. The Contractor that provides Tier 1 Support services will coordinate as required by the CBP Incident Management Process, with the CBP Technology Service Desk (TSD), the CBP Major Incident Manager (MIM), Enterprise Operations Center (EOC), the CBP Duty Officer or other required points of contact (POCs) for any/all incidents.

The Contractor is responsible for providing 24x7x365 Tier 1 Support services in support of the CBP Incident Management Process.

Tier 1 Support Services Tier 1 Support technicians is responsible for providing the initial level of support to address events/issues within the CBP IT infrastructure. Tier 1 Support services will provide, at a minimum, initial troubleshooting of events/issues and is responsible for the creating/generating trouble tickets to document/record the incident/event and record all steps taken to resolve the incident. The Contractor-provided Tier 1 Support services is responsible for the resolution of Incidents within established service agreements for Tier 1 support. The Contractor shall work to initiate a fast recovery of IT services by utilizing workarounds as necessary. If a problem cannot be resolved by the Tier 1 support within established SOPs, the Tier 1 support is responsible for escalating the issue to Tier 2 support or other technical support groups as required by the CBP Incident Management process.

Tier 1 Support Services Coordination The Contractor provided Tier 1 Support services will coordinate as required by the CBP Incident Management Process, with the Service Desk, the CBP Incident Manager, or other required points of contact (POCs) for any/all incidents.

Tier 2 Support The Contractor is responsible for providing 24x7x365 Tier 2 Support services in support of the CBP Incident Management Process.

Tier 2 Support Services Tier 2 is more in-depth technical support than is provided by Tier 1 support services. Tier 2 support technicians are responsible for providing help and assistance to Tier 1 personnel for solving technical problems that cannot be resolved by the Tier 1 support personnel. Tier 2 support is responsible for addressing network escalation from Tier 1 support. Tier 2 support services will provide at a minimum, administrative level support, advanced technical troubleshooting/analysis, and remote resolution support. Troubleshooting responsibilities will include but are not limited to:

· Isolation and resolution of power and physical connection issues.

· Isolation and resolution of routing, firewall, load balancer, and device misconfigurations.

· Isolation and resolution of failing or compromised network redundancy.

· Isolation and resolution of network security vulnerabilities and incidents.

· Data and deep packet analysis.

Not applicable to WNOC (Orlando, FL.) LMR and COTHEN support. The Contractor-provided Tier 2 Support services is responsible for the resolution of Incidents within established SOP for Tier 2 support. The Contractor shall work to initiate a fast recovery of IT services by utilizing workarounds as necessary. If a problem cannot be resolved by the Tier 2 support within established SOPs, the Tier 2 support is responsible for escalating the issue to Tier 3 support or other technical support groups as required by the CBP Incident Management process.

Tier 2 and Tier 3 Support Services Coordination The Contractor providing Tier 2 support services will coordinate with CBP Tier 3 Support services in accordance with the CBP Incident Management Process, and the Network Architecture and Engineering Division (NAED). Enterprise Infrastructure Support Team Technical Escalation Procedures with the Service Desk, the CBP Incident Manager, or other required GS approval via email to NAED Tier 3 Support NAEDTier3Support@cbp.dhs.gov to engage for any/all incidents escalated that require CBP Tier 3 Support.

Incident Closure and Evaluation The Contractor shall work to ensure that all incidents are resolved, and communications are forwarded to Stakeholders in accordance with established service agreements. The Contractor shall work with the CBP Incident Manager to conduct regular/monthly incident reviews and evaluations to identify trends and opportunities to improve the Incident Management Process.

Incident Management Telephone Support The Contractor shall provide continuous telephone support for incident escalations from CBP’s Technology Service Desk and on-site support groups. The Contractor shall collect and report Average Wait Time (AWT), Redirect on No Answer (RONA), and Abandon Rate statistics to the Government.

Incident Management Reporting The Contractor shall work with the CBP Incident Manager to ensure that outstanding Incidents are continuously monitored and escalated in accordance with the CBP Incident Management Process and established service agreements. The Contractor is responsible for regularly reporting the status of outstanding Incidents to the Incident Manager. The Contractor shall collect metrics for incidents and report incidents progress weekly.

Incident Management Support Key personnel Incident Manager will be responsible for managing and reporting incident. The Incident Manager will work with the GS in support of incident resolution process. The Incident Manager will ensure that contractor staff is trained in the tools, processes, and procedures required to perform Incident Management services. The Contractor Incident Manager will provide incident process improvement suggestion to the GS. Upon acceptance of suggestions, Incident Management SOPs is updated accordingly by assigned contractor staff.

Problem Management Support The Contractor is responsible for performing the function of Problem Management in support of the CBP Network Problem Management Process. The Contractor shall work with the CBP Problem Manager to provide proactive and predictive trend analysis to identify potential or reoccurring problems, which could cause outcomes such as degraded network performance, system, and site outages, and provide a suitable solution or workaround before an incident or outage can occur or reoccur. The Problem Management personnel have the appropriate skills and training necessary for an effective and efficient Problem Management investigation and resolution.

Predictive Problem Management Support The Contractor shall provide Predictive Problem Management support to review and analyze problems or potential problems to identify trends which could create a degradation in network performance, system, or site outages etc. in order resolve problems and/or provide suitable workarounds before incidents occur or reoccur and predictively.

Problem Categorization and Prioritization The Contractor is responsible for recording and prioritizing Problems in accordance with the CBP Problem Management Process.

Problem Diagnosis and Resolution The Contractor is responsible for diagnosing and identifying the underlying root cause of a network(s) problems. The Contractor is responsible for initiating appropriate resolution actions for network problems for all Network Data, Voice, Video, Cellular, and all other network technologies identified in the scope section (2) of this PWS.

Problem Monitoring The Contractor shall work with the CBP Problem Manager to ensure that outstanding Problems are continuously monitored and escalated in accordance with the CBP Problem Management Process and established SOPs.

Problem Closure and Evaluation The Contractor shall ensure that all successful problem solutions are properly closed and update the ticket. The Contractor shall ensure that problem issues have successfully been resolved prior to being marked as “closed” in the ticket. The Contractor shall ensure the Problem ticket contains a full historical description of the problem solution and that all related to known error tickets are updated. The Contractor shall work with the CBP Problem Manager to conduct regular problem reviews and evaluations to identify trends and opportunities to improve the Problem Management Process.

Problem Management Reporting The Contractor is responsible for the submission of weekly and monthly reports on the status of outstanding Problems, their processing-status, and existing workarounds to the CBP Problem Manager and GS. The Contractor shall work with the CBP Problem Manager to ensure that outstanding Problems are continuously monitored and escalated in accordance with the CBP Problem Management Process and other established service agreements. The Contractor shall collect weekly and monthly metrics report on progress toward meeting Problem Management performance objectives outlined in this PWS.

Network(s) Application Management Support The Contractor is responsible for providing Application Management services in support of the CBP Network Tools/applications throughout their lifecycle. This Application Management service is responsible for all application-related aspects of testing, operating, and improving IT services. These support tasks will include but are not limited to the following:

· Administration, account management, and asset management for all GFE Network Performance Management Tools and/or real-time reporting engines.

· IP address management for the agency to include adding, changing, and removing IP addresses at the request of local sites and in support of new and updated site designs

· Add and remove servers from Load- Balancers pool members

· Load- Balancer VIP and pool member troubleshooting

· Load Balancer health check troubleshooting

· Update SSL VIP certificates on Load Balancer

· Software and security patching and upgrades

· Hardware maintenance of the physical environment for the tool(s)

· Access and account management (e.g., Infoblox, and TACACS)

· Troubleshooting and diagnosing problems The Contractor shall work with CBP GS to ensure the Contractor provided Application Management services are coordinated and planned. The Contractor shall ensure Application Management personnel have the appropriate skills and training necessary for an effective and efficient Application Management capability. The Contractor shall ensure all patches and upgrades are reviewed, properly tested and vetted to the GS for approval before being deployed to the CBP Network environment. The Contractor shall ensure the Configuration Management Database (CMDB) is updated to reflect all approved patches and upgrades implemented by the Contractor.

Request Fulfilment The Contractor shall provide Request Fulfilment services for all authorized/approved CBP Requests.

Service Request Fulfilment The Contractor shall provide Service Requests Fulfilment services for all authorized/approved CBP Service Requests. Service Request fulfilment support will include, at a minimum, support for all services listed in Appendix B of the PWS. The Contractor shall review service requests to ensure that technical requirements are clear, concise, customer requested implementation timelines are feasible and can be accomplished by the Contractor within the requested timeframes. The Contractor shall plan and schedule all tasks required to complete the request and submit those tasks with establish start and completion dates for those tasks. The Contractor shall ensure that tasks are scheduled and completed within expected durations. The Contractor shall follow documented processes and procedures to track and report on the completion of change requests. The Contractor shall ensure the Configuration Management Database (CMDB) is updated to reflect all approved changes implemented by the Contractor.

Change Request Support The Contractor shall provide Change Requests Support services for all authorized/approved CBP Change Requests for the CBP Network environment the Contractor is responsible for managing. Change Request fulfilment support will include test and implementation support activities for authorized changes. The Contractor shall review change requests to ensure that technical requirements are clear and complete, receive concurrence and approval via the GS of plan of action inclusive of identifying outside stakeholder that should be included to ensure customer requested implementation timelines are feasible and can be accomplished by the Contractor within the requested timeframes. The Contractor shall conduct a technical peer review of all configuration changes prior to government board approval for changes on the CBP OIT network infrastructure to ensure the stability of the network infrastructure. The Contractor shall plan and schedule all tasks required to complete the change request and provide estimated start and completion dates for those tasks. The Contractor shall ensure that tasks are scheduled and completed within approved scheduled change/maintenance window. The Contractor shall follow documented processes and procedures to track and report on the completion of change requests. The Contractor shall ensure the Configuration Management Database (CMDB) is updated to reflect all approved changes implemented by the Contractor.

Technical Support The Contractor shall provide Technical Support services for the CBP Network Infrastructure. The Technical Support services will provide technical expertise for the resolutions of Incidents, Problems, assessment, and analysis of the CBP IT enterprise infrastructure.

The Contractor shall identify and assign a Technical Manager who is responsible for managing the Contractor’s Technical Support services. The contractor Technical Manager will ensure the Contractor’s Technology Support Services fully support CBP technology goals and objectives. The Contractor Technical Manager will ensure that Contractor staff providing Technical Support are trained in the tools, processes, and procedures required to perform Technical Support activities outlined in this PWS. The Contractor Technical Manager will make recommendations for changes and enhancements to systems, including documentation, processes, and procedures, as necessary, to improve overall Technical Support. These procedures are submitted to the GS for approval prior to use and execution. Once approved the Contractor shall save the updated procedures to the appropriate SOP repository.

Technology Refresh Planning and Life Cycle Management Support The Contractor shall provide Technology Refresh planning and Life Cycle Management Support services for the CBP Network Infrastructure. The Technology Refresh Planning and Life Cycle Management Support services will include scheduled, continuing assessments of the current CBP Network(s) infrastructure to determine their effectiveness and end of life support and provide written recommendations to the Government for those instances when GFE technology upgrades/enhancements are required. The Contractor shall also review emerging technologies and provide written recommendations for upgrading and/or enhancing technologies to the Government Technology Manager when upgrades or enhancements to existing CBP technologies are warranted.

Network Performance and Analysis Support The Contractor shall provide network performance and trend analysis support services for the CBP Network using Government Furnished Equipment (GFE) tools. The Network Performance and Analysis services will include but are not necessarily limited to, the following services:

· Network trend analysis and reporting

· Continuous real-time network bandwidth monitoring, analysis, and reporting

· Network capacity planning

· Network availability monitoring, analysis, and reporting

· Proactive and Predictive analysis to improve network availability The Contractor shall make recommendations based on monitoring, reporting and analysis of site data and configurations to improve overall site availability and performance. This will include verification, validation of network availability, outage statistics, as well as monitoring the CBP network Quality of Service (QOS) to ensure provisioned services are adequate for specific applications and that these services continue to meet CBP requirements. The Contractor shall submit written results of Network Performance and Analysis activities to the Government Technology Manager.

Network Tools Support The Contractor shall conduct annual assessments of current technologies and provide recommendations to the Government for upgrading and/or enhancing CBP Network management tools. The Technology Assessment services will include scheduled, continuing assessments of current CBP Network management technologies to determine their effectiveness and written recommendations to the GS for those instances when GFE technology upgrades/enhancements are required. The Contractor shall also assist in the onboarding of emerging technologies following the DHS 102-01-103-01 SELC Process. The Contractor shall work with CBP GS to ensure the Contractor provided tool support services are coordinated and planned. The Contractor shall provide predictive and proactive tool management services in support of the CBP OIT Enterprise Infrastructure to include network, applications, and servers. The Contractor shall ensure support staff have the appropriate skills and training necessary for an effective and efficient tool support capability. See Appendix D for current list of GFE tools/technology.

Device / Configuration Acceptance Support When required, the Contractor shall support CBP on-site support teams with the installation of new/updated devices at CBP sites. The Contractor shall provide support to the CBP on-site teams to verify connectivity of the new/updated devices and validate the appropriate configuration of the devices. The Contractor shall also provide administrative level support and troubleshooting as necessary in support of the installation of new/updated devices at CBP sites. The Contractor shall ensure the Configuration Management Database (CMDB) is updated to reflect all new/updated devices.

Technical Support Oversight The Contractor shall identify and assign a Technical Manager who is responsible for managing the Contractor’s Technical Support services. The Contractor Technology Manager will work with the CBP Personnel to ensure support services fully meet the network goals and objectives of CBP. The Contractor Technical Manager will ensure that Contractor staff providing Technical Support are trained in the tools, processes, and procedures required to perform Technical Support activities outlined in the PWS. The Contractor Technical Manager will make recommendations for changes and enhancements to systems, including documentation, processes, and procedures, as necessary, to improve overall Technical Support.

Standard Operating Processes and Procedures Knowledge Management and Performance Reporting Services The Contractor shall provide Knowledge Management and Performance Reporting services for the CBP Network(s) Infrastructure. The Contractor shall assist the ENTSD Service Request Management (SRM) Team in creating and maintaining Standard Operating Processes and Procedures (SOPs) for all activities and responsibilities covered within this contract. The Contractor shall maintain a Knowledge Management System (KMS) library, including reviewing, creating, and updating KMS documents as support needs require. The Contractor shall ensure that all SOP and KMS documentation is compliant with approved templates and are reviewed and approved by GS Information Systems Security Officer (ISSO) Support The Contractor is responsible for providing ISSO support services. The Contractor shall assume and perform ISSO duties as outlined in the DHS 4300A and/or CBP HB 1400-05D in support of the Assessment and Authorization (A&A) process. ISSO duties may include reviewing security solutions and interpreting security policies as they relate to specific security infrastructure, architectures and information systems (IS’s). The ISSO support will include the following support activities:

· Monitor security posture of the system using DHS, CBP policies and Federal Information Security Management Act (FISMA) guide:

· Participate in appropriate actions identified in the DHS-4300A, CBP HB 1400-05D (most current version) and applicable SOPs to certify, accredit and maintain compliance of each IS.

· Notify the Director of Security and/or Information Systems Security Manager (ISSM) when an assigned system requires accreditation or reaccreditation. Contact the CBP Security and Technology Policy (STP) Security Authorization (SA) Team ten months prior to ATO expiration to schedule the reaccreditation kick-off meeting.

· Lead/perform the A&A process/reaccreditation of applicable system in accordance with (IAW) the DHS 4300A, CBP HB 1400-05D and the project work schedule outlined by STP.

· Lead/perform task required to move designated EIOD-ENSB systems into Ongoing Authorization (OA), also known as Continuous Monitoring.

· Assist EIOD ISSM in responding to DHS/CISA/CBP Cyber Data Call responses.

· Conduct self-assessments per DHS/CBP policy of CBP major applications and general support systems, that include vulnerabilities identified at Contractor/consultant facilities.

· IAW with the DHS 4300A, the CBP HB 1400-05D, and program SOP’s the ISSO will acknowledge receipt of Information System Vulnerability Management (ISVM) messages, report compliance with requirements and/or notify the ISSO Branch Chief/Information Systems Security Manager (ISSM) of a POA&M creation for those vulnerabilities that are not remediated IAW with the policies stated above.

· Perform annual reviews/updates to CBP major applications and general support systems as specified by the appropriate DHS/CBP policy.

· Provide policy and security advice to systems designers, implementers and operators.

· Assist in the investigation of security violations and incidents as requested by the Computer Security Incident Response Center (CSIRC).

· Be knowledgeable on current Federal, National, DHS and CBP standards, policies, requirements and procedures.

· Perform reviews of all EIOD-ENSB-initiated Change Requests (CRs), verifying that proposed changes adhere to security standards.

· Provide asset updates to the Vulnerability Assessment Team (VAT) and System Security Plan (SSP), as assets are added, removed, and/or modified.

· Maintain System Documentation:

· Draft, review and submit for Government approval of all information systems security plans and other A&A artifacts as required by DHS 4300A/CBP HB 1400-05D. These artifacts include but are not limited to the development of the following documents:

· ISSO / Alternate Information Systems Security Officer (AISSO) designation letters

· System Owner (SO) letters

· Privacy Threshold Determination (PTA)

· Privacy Impact Assessment (PIA)

· E-Authentication Determination

· Security Controls Testing (Security Test and Evaluation (ST&E)) Plan

· ST&E Plan Test Results

· Authority to Test (ATT) approvals

· Authorization to Operate (ATO) Authorization Letter

· Self-Assessment NIST SP 800-53 Guide for Information Security Program Assessments and System Reporting Form

· Standards for Security Categorization of Federal Information and Information Systems (FIPS 199) Assessment

· Risk Assessment

· System Security Plan

· Contingency Plan

· Contingency Plan Test and Test Results

· ST&E Security Assessment Report

· Plan of Action & Milestones (POA&Ms)

· DHS/CBP waiver artifacts

· DHS/CISA/CBP/OMB audit support

· DHS/CISA/CBP Data Calls support and response

· Cloud “As a Service” support

· FedRAMP sponsorship and support

· Information Security Vulnerability Management (ISVM) review and support

· Weekly review of network scans and log files

· Review and Update System Security Plan annually or when significant system changes occur.

· Review, Update, and Develop Interconnection Security Agreement (ISA) (as applicable).

· Prepare documents and meet requirements for the annual Ongoing Authorization Review Management Board (OARMB).

· Complete/update a NIST SP 800-26 or NIST SP 800-53 review for each major application, LAN(s), or general support system on a yearly basis.

· Prepare weekly summary of vulnerabilities.

· Prepare an end of week summary of assigned system activities to include, but not limited to:

· Current system Compliance highlights

· Current ISVM’s as they relate to assigned systems(s)

· Current POA&M’s being worked DHS Security Policy Requirement All hardware, software, and services provided under this task order must be compliant with DHS 4300A DHS Sensitive System Policy and the DHS 4300A Sensitive Systems Handbook.

Encryption Compliance Requirement

1. FIPS 197 (Advanced Encryption Standard (AES)) 256 algorithm and cryptographic modules that have been validated under FIPS 140-2.

2. National Security Agency (NSA) Type 2 or Type 1 encryption.

3. Public Key Infrastructure (PKI) (see paragraph 5.5.2.1 of the Department of Homeland Security (DHS) IT Security Program Handbook (DHS Management Directive (MD) 4300A). for Sensitive Systems.

4. Awareness of next gen FIPS 203, 204, 205 NIST requirements to support encryption via Quantum computing.

Security Review The Government may elect to conduct periodic reviews to ensure that the security requirements contained in this contract are being implemented and enforced. The Contractor shall afford DHS, including the organization of the DHS Office of the Chief Information Officer, the Office of the Inspector General, authorized Contracting Officer’s Representative (COR), and other government oversight organizations, access to the Contractor’s facilities, installations, operations, documentation, databases and personnel used in the performance of this contract.