OMS_Task_Order_2_PWS_for_O_M_-_Draft_03-07-2018.pdf

DRAFT OPERATIONS, MAINTENANCE, AND SUSTAINMENT

TASK ORDER 02 PERFORMANCE WORK STATEMENT -

OPERATIONS & MAINTENANCE

Source Selection Information – See FAR 2.101 and 3.104-4 1

National Environmental Satellite, Data, and Information Service (NESDIS)

Office of Satellite and Product Operations (OSPO) and

Office of Satellite Ground Services (OSGS)

Draft Operations, Maintenance, and Sustainment (OMS) Task Order 002

Performance Work Statement (PWS)

March 7, 2018

Source Selection Information – See FAR 2.101 and 3.104-4 2

Table of Contents

1.0 Scope 3

2.0 Place of Performance 4

3.0 Requirements 4

3.1 Program Management 4

3.2 Operations 5

3.2.1 Operations Support 5

3.2.2 Operations Training 7

3.2.3 Help Desk Service to Operations 7

3.2.4 Hardware Support to Operations 9

3.2.5 Data Center Infrastructure Management 11

3.2.6 Antenna Operations Support 12

3.2.7 Network and Security Operations 12

3.0 Maintenance 15

3.1 Network & Systems Administration 15

3.2 Capacity Planning 21

3.3 Telecommunications Support 21

3.4 Preventative Maintenance 22

3.5 Antenna Maintenance 26

3.6 Satellite Product Maintenance 28

3.7 Information & Cyber Security Support 28

Source Selection Information – See FAR 2.101 and 3.104-4 3

1.0 Scope

The Office of Satellite and Product Operations (OSPO) and the Office of Satellite Ground Services (OSGS) have shared responsibilities in managing legacy satellite ground systems and product processing systems. OSPO operates and OSGS maintains and sustains these ground systems. This Task Order 02 performance work statement (PWS) specifies the operations and maintenance requirements needed to support the ground systems.

Table 1.1-1 shows Government ground systems that are within the scope of Task Order 02 as well as the type of support required (i.e. operations only or operations and maintenance. These systems may be referred to as “assigned systems” herein.

A detailed list of NESDIS systems is provided in Appendix 1, “NESDIS System Boundary General Correspondence to NESDIS Missions/Functions” within the OMS IDIQ PWS.

Individual system inventories, drawings, descriptions and interfaces are available in the OMS Document Library.

Table 1.1-1 NESDIS Systems to be Supported Under Task Order 02 Operations Maintenance

GOES Ground System

Legacy Geostationary Operational Environmental Satellites Ground System

Y Y

DSCOVR

Antenna/Networking

Deep Space Climate Observatory (Subsystem of GOES)

Y Y

ESPC Environmental Satellite Processing Center Y Y

NJGS NOAA Jason Ground System Y Y

GOES R New Geostationary Operational Environmental Satellites Ground System

Y N

JPSS Joint Polar Satellite System Ground System Y N

POES Ground System

Legacy Polar-orbiting Operational Environmental Satellite Ground System

Y Y

DMSP

Non-NOAA

Defense Meteorological Satellite Program (NOAA Operates this DOD Satellite)

Y Y

MSL Mission Support LAN Y Y

Source Selection Information – See FAR 2.101 and 3.104-4 4

2.0 Place of Performance

The work locations for this task order are listed below:

● NOAA Satellite Operations Facility (NSOF) located in Suitland, MD

● OSGS Program Offices located in Greenbelt, MD and Silver Spring, MD

● NOAA Center for Weather and Climate Prediction (NCWCP) located in College Park, MD

● NASA’s Goddard Space Flight Center located in Greenbelt, MD

● NESDIS Command and Data Acquisition (CDA) Stations located in Wallops, VA;

Fairbanks, AK; and Barrow, AK

● The NESDIS Consolidated Backup Unit (CBU) located in Fairmont, WV

3.0 Requirements

The Contractor shall:

● Provide the required subject matter expertise, personnel, material, equipment, services, and facilities to perform and deliver operations and maintenance support for the mission systems with no interruption of services. Performance requirements are listed in section 4.0.

● Perform operations and maintenance activities at all work sites

● Perform operations and maintenance activities for systems listed in Task Order 02 Table 1.1- 1, as well as the interfaces (as required) to systems listed in Tables 1.1-2 and 1.1-3 of the OMS

IDIQ PWS.

● Provide personnel with professional and technical certifications commensurate with their responsibilities

3.1 Program Management

Program management and transition related requirements supporting Task Order 02 activities are covered under Task Order 01.

Source Selection Information – See FAR 2.101 and 3.104-4 5

3.2 Operations

3.2.1 Operations Support

The Contractor shall provide all processes and resources to implement and deliver products and services supporting the efficient and effective operations of NOAA mission critical systems.

PWS-TO2-OP01: The Contractor shall provide all resources for performing operational activities to include:

-Documentation updates of mission requirements, planning events, system configuration, and operational procedures

-Providing regular and accurate status to the Government -Supporting operational activities for the functionality of ingest, archival, product generation, and dissemination of datasets -Detecting, identifying, and assessing system anomalies -Providing notifications to stakeholders and users (e.g., for outages, etc.)

-Supporting activities and agreements for system components and services -Ensuring a secure operational environment -Establishing, monitoring, reporting, and tracking metrics per section 4.0.

PWS-TO2-OP02 The Contractor shall perform operator support for system components and services to include:

-Monitor and report status and health of system components -Monitor and report system workflow and resource management -Detect, analyze, and resolve system component or service problems and discrepancies -Resolve problems in command and control systems, data, product applications, systems software, and networking systems -Report system problems, discrepancies, repairs, upgrades, and related activities to stakeholders and users -Coordinate with the Government, and implement system updates, optimizations, and upgrades to operational components and services approved by the COR.

Operational components are defined as the hardware, software, communication devices, operating systems, documentation, licenses, and agreements necessary to support operational services.

-Coordinate corrective maintenance -Provide baseline standard operating procedures for all operations activities -Support integration and test, regression testing of system software releases, system upgrades, anomaly investigations, and periodic IT security patches and other operational products

-Support software reviews and testing to verify that modifications provide the intended functionality

-Respond to system issues on a 24-hour, year-round basis -Provide technical assistance in resolving software problems documented in incident reports encountered in operations -Plan responses for critical weather days

Source Selection Information – See FAR 2.101 and 3.104-4 6

-Provide for immediate backup capability for operational systems PWS-TO2-OP03. The Contractor shall coordinate technical issues between the Government and Contractor support teams, and across internal and external stakeholder organizations.

PWS-TO2-OP04. The Contractor shall provide status and reports on system performance to the Government, and fully coordinate on testing, maintenance and release of upgrades, anomaly investigations, and implementing modifications to operational systems.

PWS-TO2-OP05. The Contractor shall provide support to meet system operational availability requirements.

PWS-TO2-OP06. The Contractor shall provide:

-Training, as needed, to staff on functionality and execution of operational procedures (including anomaly, contingency, and recovery)

-Current training material for the operations and maintenance for the program -Contractor shall maintain an training log of all training by employ and report on it status during program review meetings PWS-TO2-OP07. The Contractor shall participate in the meetings shown below, as well as facilitate and record minutes for designated operational meetings at the Government’s request.

Meeting times will be specified by the Government.

-Satellite Operations Control Center (SOCC) ticket review (Monday - Friday) -SOCC stand up -ESPC stand up and ticket review (Monday - Friday) -Contract status (weekly) -ESPC Coordination -Plan of Action and Milestones (POA&M) (weekly) -Executive Review Board (monthly) -Web meeting (monthly) -Help Desk Working Group (bi-weekly)

PWS-TO2-OP08. The Contractor shall document and provide meeting minutes to the Government for these meetings: contract status, CCB, technical working group, Executive Review Board, Web, and Help Desk Working Group within 3 days of the event.

PWS-TO2-OP09. The contractor shall respond to Data Call support requests including Organizational and technical queries. The schedule or the completion of these requests is on a case by case basis.

PWS-TO2-OP10. The Contractor shall respond to Government request for satellite launch and special activity support, to include:

-Providing on-site 24 hours per day x 7 days per week maintenance support for specific launch operations and special events. Typical coverage begins 1-3 days prior to launch and lasts 1-3 days after launch.

-Loading and maintaining affected antenna systems tracking controllers with ephemeris data provided by spacecraft navigators during and after satellite maneuvers.

Source Selection Information – See FAR 2.101 and 3.104-4 7

3.2.2 Operations Training

PWS-TO2-OP11. Contractor shall be able to produce the products in accordance with standard Satellite Analysis Branch (SAB) operating procedures and with minimal oversight after no more than three (3) months of on-the-job training and subsequent certification for each product area.

Government analysts will provide on-the-job training to the Contractor supporting the OSPO SAB to acquire the required skills and expertise to produce real-time, operational volcanic ash, tropical storm, precipitation, oil spill/marine debris and/or fire/smoke products (and other new products as needed) per SAB procedures.

PWS-TO2-OP12. The Contractor shall be able to produce the products in accordance with standard National Ice Center (NIC) operating procedures and with minimal oversight after no more than 3 months of on-the-job training for the snow and ice product or Synthetic Aperture Radar (SAR) imagery acquisition and no more than 18 months of on-the-job training for regional and hemispheric ice charts.

PWS-TO2-OP13.The Contractor shall provide and maintain a current staffing list and update at least monthly with changes (including name, company, hire date and where applicable, departure date). Employee training will also be part of this listing.

3.2.3 Help Desk Service to Operations

PWS-TO2-OP14. The Contractor shall provide help desk support staffing coverage 24 hours per day x 7 days per week x 365 days per year sufficient to handle the daily call, email, and Help Desk ticket volume estimates shown in Table 3.2.3-1:

Table 3.2.3-1. Help Desk Ticket Volume

Daily Average Annual Calls 25 9125 E-mails 20 7300 Help Desk tickets 6 2400

ESPC Helpdesk Ticketing Volume by Product Area

Product Area 2016 (12 months) 2017 Estimated (6 months doubled) Antenna 4 2

Common 161 214

DCS 0 2

DMSP 71 82

DSCOVR 28 38

ESPC 3966 5546

ESPC-NIC 68 54

GOES-NOP 98 110

GOES-RSTU 0 2

IJPS/METOP 1 4

Source Selection Information – See FAR 2.101 and 3.104-4 8

Daily Average Annual

NJGS 56 94

JPSS 6 14

POES 103 168

SARSAT 0 2

SNPP 70 66

PWS-TO2-OP15. The Contractor shall perform help desk and customer support functions to include:

-Customer service and notifications, and identify, recommend, and support automated Help Desk functions.

-Initiate response to Help Desk tickets per the Task Order section 4.0.

These initial contacts are to acknowledge only receipt of the help request and are made on behalf of the Government. The Contractor shall immediately forward each Response to Help Desk concerns ticket to the categorical support person designated by the Government to handle the request.

-Generate and review statistics of system usage -Manage the status log and transaction files -Analyze messages in logs and of any anomalies -Update help pages (Status, News, FAQ, and Warnings) -Collect and report on operational metrics -Facilitate and provide daily logs and tickets for the daily SOCC ticket review

1. Training users and support personnel utilizing Government Help Desk procedures on new processes and to communicate system changes or notifications

2. Ensuring staff proficiency on emergency procedures and conditions required for remaining at work in emergency situations (dismissal or Government closure announcements)

3. Answer calls, assign and track help tickets/incident reports using Government-approved automated systems per Task Order PRS requirements

4. Professionally respond to phone inquiries (to include the individual operator’s name)

5. Coordinating and forwarding user requests to additional support personnel (e.g., Product

Area Lead (PAL)/Government Representative)

6. Monitor Help Desk tickets and ensure those requests that are “Submitted” reach a status of “In Progress”

7. Track problems and coordinate corrective actions to closure per the established procedural process: maintain information in the CRM database, provide status email notifications for all anomalies, outages, and changes to data, products or services

8. Provide user notifications and problem escalation, provide 24x7 notifications to management utilizing established call trees and procedures

9. Log all Help Desk events, generate performance metrics (e.g., number of emails, calls and tickets per day)

10. Generate/ disseminate a daily event log to include actions, all phone calls, email contacts from the previous day by 7:00 am local time the next day

11. Track problems and coordinate corrective actions

Source Selection Information – See FAR 2.101 and 3.104-4 9

12. Maintain (monthly) the Government-furnished CRM database, including users (persons), products, services, satellites, applications, instruments, organizations, and administration, using Government-approved procedures

Systems Analysis Support to Operations

PWS-TO2-OP16. The Contractor shall provide monitoring and job scheduling functions to include:

-Monitoring of IT systems and product applications -Modify job streams, start or stop application systems, rerun jobs, set up and support testing, and monitor production utilizing Government-supplied enterprise scheduler system -Monitor primary and backup IT systems and product applications to provide continuous

24x7 operations utilizing Government Furnished Equipment (GFE) listing (in the Document Library, when available), or Contractor-developed tools

-Monitoring and trending NPP/JPSS products generated by the NESDIS Interface Data Processors (IDPs) utilizing JPSS Ground System Data Quality Monitor (DQM) Tools

PWS-TO2-OP17 The contractor shall perform electronic call tracking and reporting activities to include:

-Provide weekly status reports to the COR (a summary and current status of all maintenance requests)

-Accurately populate all fields in log requests for maintenance, upgrade, de-installation, installation, relocation and preventative maintenance into a Government provided electronic tracking system. The data must be compatible with the Microsoft Excel format. Data fields include:

○ The Date and Time of the request for service

○ Unique tracking number

○ Equipment Type

○ Mission (GOES, POES, DMSP, JASON, etc.)

○ Equipment serial number and DOC inventory barcode number

○ Name of person(s) requesting service

○ Description of services requested

○ Description of actions taken to resolve the request for maintenance

○ Required staff to resolve the maintenance request

○ Time required to resolve the request ( rounded to the nearest 30 minutes)

○ Completion date and time

○ Spare parts used to complete repair

3.2.4 Hardware Support to Operations

PWS-TO2-OP18. The Contractor shall provide systems analysis activities and problem resolution support to operational sites and back up sites. This includes providing all required

Source Selection Information – See FAR 2.101 and 3.104-4 10 personnel for problem resolutions in command and control systems, data, product applications, systems software, and networking systems.

PWS-TO2-OP19. The Contractor shall perform on-site remedial hardware maintenance on the operations system area (e.g., workstation/desktop/hardware support SOCC operation area, satellite operator and engineering workstations/printers, and Ground System Equipment and systems). The required support activities include:

-Correctly identify, diagnose and repair OSPO equipment, -Perform troubleshooting and repair of equipment including replacement of hardware components to maintain operability of OSPO ground system equipment.

PWS-TO2-OP20. The Contractor shall provide support to SOCC operations area satellite operator and engineering workstation by:

-Responding to requests for maintenance within 15 minutes of receiving the request -Replacing hardware used for command, control, and engineering support needing repair or maintenance with Government provided spare workstations or printers within two (2) hours of receiving the request [Defective workstations shall then be repaired and placed in Government Equipment Spares Inventory]

-Complete repairs and replacements of SOCC Operations Area workstations and return the workstations to Government inventory no later than 4:30 p.m. local time on the business day following the date of receipt of the request for maintenance

Repaired workstations and printers must be returned to Government Equipment Spares inventory for use as replacement equipment.

PWS-TO2-OP21. The Contractor shall provide web support services that include:

-Add/maintain web pages to implement new products or capabilities (including new information, additions to old products, and any changes required by new or modified Government standards)

-Collect Web content, create/modify web pages in compliance with Section 2 of the OMS Draft IDIQ PWS

PWS-TO2-OP22. The Contractor shall provide support in Replacement Parts activities: These include:

-Providing replacement parts for repairs (except in emergency situations) -Ensuring replacement parts certification to Original Equipment Manufacturer (OEM) specifications -Ensuring equipment/replacement parts perform the same functions at the same level as the parts or equipment that were replaced -Obtaining written Government (COR) approval prior to ordering replacement parts -Ordering, receipt, and inventory of approved spare and replacement parts -Managing equipment logistics and provide documented inventories (to COR) -Maintaining centralized inventory containing all hardware spare parts, and ensure all on the shelf replacement parts are in working order -Developing, submit, and maintain a Return to Vendor Plan for all replacement parts

Source Selection Information – See FAR 2.101 and 3.104-4 11

PWS-TO2-OP23. The Contractor shall provide support to modifications and upgrades designated by the authorized Government official. These activities include:

-Performing upgrades to mission systems equipment hardware -Performing installation/de-installation of system cabling supporting new installations and relocations of equipment/ systems -Generating, implementing, and documenting organized methods for managing the data center racks and patch cables per best practices (e.g., per ANSI/TIA-942 standard and Building Industry Consulting Service International (BICSI))

PWS-TO2-OP24. The Contractor shall support installation, de-installation and relocation activities. These activities include:

-Performing installations, de-installations, and relocation of mission system equipment and Operations Area Workstations and Printers

-Coordinating all scheduled equipment, installations, de-installations and relocations to ensure minimum downtime and disruption to operational equipment with the authorized Government official

-Performing installations, de-installation, and relocations per OEM specifications

3.2.5 Data Center Infrastructure Management

PWS-TO2-OP25. The Contractor shall support facilities change management support activities.

These activities include:

-CCR completion given any change that has a potential impact on the entire NSOF Admin LAN, NSOF PBX, or cable infrastructure (e.g., Domain Policy changes, adding/removing services or servers on the NSOF Admin LAN, firewall or switch configuration change, PBX modifications, etc.)

PWS-TO2-OP26. The Contractor shall support cable plan management activities to include:

-Providing a minimum of one (1) Building Industry Consulting Services International (BICSI) certified person to be on call at all times

-Building /terminating any necessary IT-related cabling (as directed by the Government).

These include CAT6, coaxial, serial, and fiber optic cables.

-Building/terminating low voltage wiring connections (as directed by the Government), to include serial cables from systems to appliances or antennas.

-Providing cabling management support for all missions, cable connections and cable maintenance

-Providing all cable connections after the operational center plugs into assigned demarcation points (located in one of the six communication rooms in NSOF or at a consolidation point)

-Managing all infrastructure cable connections using a cable database application (iTRACS) containing a graphical interface and output to facilitate the making and tracking of connections

-Coordinating all repairs for problems found on the infrastructure cabling

Source Selection Information – See FAR 2.101 and 3.104-4 12

-Maintaining a cable database (in the NSOF) and the infrastructure cable plant to include reserving patch panel ports in all NSOF communication rooms. Patch panel port reservations must be approved by the Government.

3.2.6 Antenna Operations Support

The Government assets documented in this section require operations support: ViaSat 9.1m Geostationary Satellite Antennas and GOES Variable (GVAR)/GOES ReBroadcast (GRB) Data Distribution and Ingest System. The four 9.1m ViaSat receive-only geostationary satellite antennas and the accompanying ingest and data distribution system are of vital importance to satellite operations. These four antenna systems (located on the roof of the NSOF in Suitland, MD) are used to capture mission critical image data in the GVAR format for the GOES N/O/P satellites and in the GRB format for GOES-R satellites. The data distribution system provides the GVAR/GRB data to the ESPC Advanced Front End Processor Systems (AFEPS). The AFEPS ingest patching system receives and distributes GVAR/GRB signals to required AFEPS equipment.

PWS-TO2-OP27. The Contractor shall support antenna systems. The activities include:

-Loading and maintaining the tracking controller of each system with up-to-date, accurate and validated ephemeris data to ensure proper antenna tracking to a [minimum of 99% operational capability].

-Performing pointing adjustments, and signal peaking of the antennas to maintain data reception.

-Re-pointing antennas to different spacecraft upon direction of authorized Government staff.

-Coordinating 100% of the outages associated with or created by adjusting the antennas or directing them onto different satellites with authorized Government staff.

-Adjusting and repositioning the antennas to different satellites and perform the patching on the AFEPS ingest patch system to ensure GVAR/GRB reception and distribution is maintained the minimum operational capability per the Task Order PRS.

-Performing Antenna alignment and pointing for the four (4) LRIT antennas and the two (2) GEONETCAST antennas in order to maintain an operational capability.

-Performing troubleshooting of problems and perform maintenance for the four (4) LRIT and two (2) GEONETCAST antennas.

-Responding to LRIT and GEONETCAST issues.

-Return the affected out of service antennas to an operational capability

3.2.7 Network and Security Operations

Network Operations deals with monitoring naturally-occurring system events. Security operations focuses on the monitoring of potential “bad actors.” Network and security operations staff work side-by-side to ensure the reliability, availability, and integrity of the operational networks.

Source Selection Information – See FAR 2.101 and 3.104-4 13

PWS-TO2-OP28. The Contractor shall provide resources to support network, application and systems engineering, and security engineering tasks. These tasks include:

-Management of Security tools: ArcSight, Tripwire, Altiris, SolarWinds, ECMO -Supporting IT security scans

PWS-TO2-OP29. The Contractor shall support WAN and LAN support activities. These activities include:

-Maintaining/ updating diagrams of all WAN / LAN data communication connections including: external point to point data communication circuits, internet connections, circuit and/or packet switching, mission voice conferencing system, and dedicated dark fiber connections

-Providing automated alerts from an automated network monitoring system for network (WAN) outages in compliance with any Government-approved service level agreement(s) or federal mandates

-Coordinating with Government-approved communication vendors, customers, partners, and other mission systems to troubleshoot network problems, outages, or any other issues impacting a service level agreement. This includes coordination with leased communication providers to initiate trouble tickets.

-Diagnosing and resolving issues given a network equipment failure or connection outage -Responding to calls from the Shift Supervisor (SO) and ESPC Helpdesk for resolution of an outage or unacceptable SO or ESPC network performance -Providing remote instruction (when not on site) to the Government and onsite support for problem resolutions. This does not absolve the requirement of the Contractor to be physically present when needed.

-Documenting network outage event reports -Maintaining a current documented configuration of OSPO networks (LAN) and location of all components/hardware connected to the OSPO system -Providing network engineering support for all approved change requests associated with the operational networks -Complying with Government-approved configuration practices for WAN network equipment configurations, port assignments, network diagrams, and IP addresses assignments

PWS-TO2-OP30. The Contractor shall support Security Configuration Monitoring activities of security systems of all relevant devices and application, and report security configuration status in relation to the relevant security configuration baselines. The contractor activities include:

-Monitoring of systems using OSPO-approved monitoring tools, SCAP monitoring content, and processes

-Evaluation of identified security configuration settings against relevant OSPO and NASA security configuration baseline

-The electronic transfer of the results of all security configuration monitoring using OSPO standard tools and processes, and at a frequency that meets OSPO requirements, to the OSPO authoritative security risk management repository

Source Selection Information – See FAR 2.101 and 3.104-4 14

-Coordinating with information systems owners, ISSOs and system administrators to ensure awareness and understanding by system personnel of identified deviations from security configuration baselines

PWS-TO2-OP31. The Contractor shall provide security data analysis services, and perform security data analytics services of the IT Security Incidents and Operational Anomalies. These services include:

-Configuring, fine-tuning, and monitoring Security Information Management (SIM) or Security Information and Event Management (SIEM) solutions

-Proactive monitoring of environment using data mining and other advanced data analytic techniques

-Reviewing information in NOAA’s security repositories (such as ArcSight ) to determine patterns, trends, security vulnerabilities

-Summary reporting (e.g. incident summary reports) -Trend analysis -Anomaly detection -Root cause analysis -Monitoring Network Trends -Monitoring system and network device behaviors to establish and maintain a baseline of normal patterns and trends for all networks -Reporting network activities that diverge from normal network patterns or trends -Correlation of data from multiple log sources to validate potential intrusions or assess system risk. Examples of relevant log sources include: centralized log system, patch status reporting system, vulnerability scanning system, firewall logs, full packet capture system. and web content filtering system.

PWS-TO2-OP32. The Contractor shall provide Communications Security (COMSEC) Key Loading and Technical Support. OSPO utilizes various Communication Encryption Devices for satellite command and control and payload processing. The devices are vital to ensure secure and accurate communication with satellites and remote ground stations. The encryption keys of these devices must be updated and in synchronization with remote encryption devices.

Contractor employees supporting this task at the NOAA Satellite Operations Facility (NSOF), 4231 Suitland Road, Suitland, Maryland for Government operations require a security clearance procedure to obtain a SECRET level clearance, and be a U.S.

citizen.The clearance is issued by the DOC Office of Security.

The required activities are included below:

-Perform scheduled weekly, monthly and quarterly loading of COMSEC keying material at the scheduled times per OSPO’s standard COMSEC operating procedures

-Coordinate 100% of the performed loads on the encryption devices with Government satellite operations staff and remote COMSEC technicians to ensure loads do not interrupt or interfere with satellite operations

-Ensure all COMSEC loads performed to an operational capability of no less than 99% as measured by circuit connectivity and availability

Source Selection Information – See FAR 2.101 and 3.104-4 15

-Troubleshooting and diagnosis requests of COMSEC equipment problems (by the COMSEC Account Manager (CAM))

-Repair/replace/ configure, test, verify operations of any COMSEC equipment failure. The Government will provide a replacement unit(s).

-Assist with packaging and shipment of failed unit(s) for depot repair -Proper safeguard of all COMSEC material in Contractor possession in accordance with

COMSEC requirements PWS-TO2-OP33. The Contractor shall provide all support and services for Continuity of Operations (COOP) during OSPO COOP exercises or activations. The Government will also participate in all activities. The level of support at NSOF and the backup site(s) is the same if NSOF is untenable or cannot be used for all or part of normal operations. Upon COOP activation the activities for this task include:

-Deployment to the remote COOP sites -Support of maintenance COOP and disaster recovery plan/ procedures to address the immediate and longer-term needs of the Government -Assessment of vulnerability to significant service interruptions for NOAA missions and programs -Identify/ analyze economic, service, public image, and other implications of extended service disruptions

3.0 Maintenance

High level maintenance functions and requirements are provided in Section 3.3 of the OMS IDIQ PWS. Definitions of maintenance activities is provided in section 1.4 of the OMS IDIQ PWS. Additional requirements are provided below.

3.1 Network & Systems Administration

PWS-TO2-M01. The Contractor shall provide support for all network task support. Network tasks include:

- Perform network management of all Government operational networks and components.

The NSOF facility, remote facilities, CDAs, and CBU facility are the primary locations for this required support. The supported networks are per Appendix 1 of the OMS IDIQ

PWS.

- Provide comprehensive on-site support for network administrators and performance management, at NSOF, Monday through Friday with core business hours (7:00am – 6:00pm local time), excluding Federal holidays

- Provide response support for Help Desk ticket to network outages, issues, and service degradations that cannot await resolution during normal business hours, at the NSOF, or other OSPO network locations, when called or alerted by automated means

- Provide on-site network administrator support for satellite launches and special events requested by the Government, outside of core business hours

Source Selection Information – See FAR 2.101 and 3.104-4 16

- Provide on-site network administration support, perform upgrades and installations, perform troubleshooting, or verify information for documentation, planning and configuration management at remote sites, and other sites with communication paths

- Coordinate with the Information Systems Security Officer (ISSO) on configuring firewall, intrusion detection systems, intrusion prevention systems, routers, switches and all other network-specific equipment to allow authorized and approved traffic. Configuration includes providing alerts of possible compromise, unauthorized use or service interruption.

- Tracking and reporting metrics

PWS-TO2-M02. The Contractor shall provide support for Networks Fault Management. Typical tasks and activities include:

- Diagnosis of problems, conduct of problem resolution given network performance degradation, network equipment failure and connection issues/outage

- Respond to calls for resolution of outage or unacceptable network performance from from Shift Supervisor, ESPC Helpdesk, and Ground System Engineers

- Respond to the Contractor automated network monitoring system alerts on network (WAN/LAN) outages, service degradation, and performance issues according to the established Government-approved service level agreement(s) or federal mandates

- Document event reports on network issues, network performance degradation, network equipment failures, and connection issues/outages

- Participate in mission system anomaly review boards to report on problems and outages and recommend corrective actions

- Coordinate with contract vendors, partners, and other mission systems and stakeholders to troubleshoot network problems, outages, or any other issues impacting a service level agreement. This includes coordinating with contact leased circuit providers to initiate trouble tickets on leased telecommunications circuits.

- Document and maintain a spare parts list, components spare parts, and track this inventory through the equipment life cycle.

- Comply with Government-approved configuration management practices for WAN/LAN network equipment configurations, port assignments, network diagrams, and IP addresses assignments.

- Maintain current (documented) configuration of networks and location of all components/hardware connected to the mission systems.

- Comply with marking requirements for network documentation as “Controlled Unclassified Information” (CUI) and its appropriate sub-categories.

- Review network diagrams and systems documentation for accuracy and correctness after completion of any network configuration installation, movement, addition, or change.

There are approximately 50 network diagrams that are maintained for all applicable ground systems.

- Review and ensure all like components use the same software, firmware, and engineering levels at all sites.

A report in the SolarWinds Orion network monitoring tool will be provided to demonstrate compliance for like components.

Source Selection Information – See FAR 2.101 and 3.104-4 17

- Configure network components per approved baseline configurations; meet Level 2 or OSPO policy for system hardening.

Use reports in SolarWinds Orion network monitoring tool and Tripwire Enterprise to demonstrate success in Center for Internet Security (CIS) level 2 compliance.

- Provide network engineering review, impact assessment, and support for all approved change requests regarding operational networks.

- Review system description documents for impact assessment to operations per the OSPO CM Plan for all new installations.

PWS-TO2-M03. The Contractor shall provide Network Account/Administration Management services. Typical tasks include:

- Adherence to Government-mandated account management policies and procedures for adding, maintaining, monitoring and disabling user accounts on network equipment

- Coordinate with the Government to process account request forms, configuration change requests, and other forms as documented in Government network administration procedures

- Complete account reviews every six (6) months per NIST, DOC and NOAA policy, and per System Security Plan (SSP)

- Administer appropriate access to all system and network devices using secured methods approved by the contract ISSO.

- Coordinate with the Government to create, update, and maintain spare parts inventories, system inventories, rack elevations, and equipment room layouts. These documents formats and data will be approved by the Government.

- Implement network and devices supporting both IPv6 and IPv4.

- Ensure networks support automatic failover to redundant connections.

- Support the use of descriptions to include hostname of connected device, remote port identification for network to network connectivity, and network interface name on workstation/server hardware of connected device into the loaded and enabled device configurations using the network equipment operating system

- Document and manage data center racks and patch cables per ANSI/TIA-942 standard and Building Industry Consulting Service International (BICSI) best practices

- Maintain labeling of network cabling and equipment per ANSI/TIA-606-B labeling standard

- Manage and troubleshoot approved public and private Domain Naming Service (DNS) name spaces owned and operated by the Government. This includes obtaining domain and IP space from the NOAA Network Operations Center (NOC). For public facing networks, managing and assigning the proper IP address to correspond to the correct domain names, and updating DNS names and entries.

- Manage, track, and communicate to appropriate Government personnel all Dynamic Host Configuration Protocol (DHCP) and static IP addresses reservations within the network.

Source Selection Information – See FAR 2.101 and 3.104-4 18

- Maintain accurate records of network port connections and IP assignments for all Government boundary network equipment; identify network port connections with remote hostname, location ID, and Interface ID (port number, Eth0, etc.)

- Provide/ maintain an implementation schedule for any network changes, and applying network operating system updates or patches

- Adhere to Government-provided limitations regarding downtime and SLAs

- Coordinate scheduled network down times with the Government representative and operations personnel.

- Implement/ enforce multi-factor authentication methods for remote users and other Government-identified users.

- Provide remote access (Virtual Private Network (VPN)) services for monitoring, manage methods for remote users and business partners to securely connect to the network and data center computing services over the public Internet.

PWS-TO2-M04. The Contractor shall provide services for Network Performance Management.

The tasks for this include:

- Providing weekly reports on network components to cover: availability, resource utilization, and system performance. The report structure must have logical structure and groupings to represent the status of the operational network. External facing devices will be on a different report than internal only devices.

- Provide weekly, monthly, and yearly network utilization reports for all external data connections and ad hoc analyses requested by the Government.

- Review/analyze system logs daily to identify alerts or potential faults that impact system, and include a daily report to the Government

- Forward critical alerts immediately to the Government representative

- Provide an automated network monitoring system capability to track data throughput, bandwidth utilization, outages, network component CPU and memory utilization, network trends and analysis, alerting, and configuration changes

- Configure the automated network monitoring tool to use groups for alerting and reporting

- Monitor data throughput and bandwidth utilization over the networks, provide recommendations on upgrades and optimization to the Government

- Modify network routes to optimize the network traffic with Quality of Service (QOS) as data flows change within the OSPO environment

- Coordinate problem analyses with the Government and other parties to resolve performance problems PWS-TO2-M05. The Contractor shall provide services for Network Security Management. Tasks include:

- Obtaining written authorization from the respective OSPO system ISSO for the release any information of that system

- Inspecting/ applying the necessary security patches and vulnerability mitigations for system boundary equipment

- Leading, attending, and supporting the OSPO patch vulnerability group (PVG)

Source Selection Information – See FAR 2.101 and 3.104-4 19

- Configure intrusion detection appliances within OSPO networks to gather (daily) information for analysis by OSPO ISSOs

- Monitoring intrusions on OSPO network equipment; reporting all security threats following NOAA Incident Reporting Procedures

- Coordinating with the OSPO ISSO to ensure software, firmware, and engineering revision levels of network components are at minimum levels required by NOAA, NESDIS, and security policy

- Maintain daily awareness of security vulnerabilities, notify the OSPO ISSO of any potential network vulnerabilities

- Gather information by subject matter experts for the OSPO ISSO and other parties evaluating and documenting OSPO systems for System Test and Evaluation (STE), Penetration Test, or like activities

- Implement remedial changes to network components required to meet security compliance mandates, or document the reasons why it cannot, and present alternate means of meeting those mandates

PWS-TO2-M06. The Contractor shall provide support for System Administration and User Support. Activities include:

- Perform systems administration on all servers, workstations, and peripherals located at NSOF facility, remote facilities, CDAs, and CBU facility

- Provide the required systems administration support on-site at the NSOF and NCWCP, Monday through Friday, with core business hours of 7:00am – 6:00pm, excluding Federal holidays

- Perform system administration on current systems, and planned systems including server/workstations/computers, SANs, and peripheral hardware per OSPO Configuration Item (CI) Database by System, OSPO Product Applications and Systems Summary Table Downtime, and Service Level Agreements

- Perform operating system updates, system patches, system audits and system backups and other system administration functions

- Manage user accounts and access rights to system components using a centralized access solution (if one is available on the system)

- Provide system administration support for web hosting services maintained on the operational networks

- Coordinate with the Government to process account request forms, configuration change requests

- Provide system administration support for the consolidation and upgrade of web servers and websites hosted within the system

- Perform system administrative functions, including: support for new data types, new satellites, and product processing changes; support for planned system outages and recovery from unplanned system outages; setting up new test environments; and support for operator monitoring.

- Support installation of operating systems, data and application migration to new servers, decommissioning or installation of servers, load balancing, acceptance testing and integration of new technologies.

- Coordinate scheduled down times with the Government representative and operations lead, adhering to Government limitations regarding downtime and SLAs; coordinate down

Source Selection Information – See FAR 2.101 and 3.104-4 20 time requests of operational systems with the relevant Ground System Engineer, Operations, and the Systems Lead

PWS-TO2-M07. The Contractor shall provide support for System Administration Fault Management. Activities include:

- Provide continuous support diagnosis and resolve problems given a system failure

- Resolve a minimum of 99% of the system outages, or implement an approved procedure/process for system replacement within two (2) hours of notification

- Create / maintain an event report for system outages or anomalies

- Provide system information, artifacts, remedial actions, and other pertinent information to the Government in support of the construction of a Post-Incident Report for major system anomalies with significant impact on primary users

- Review/ analyze system logs daily to identify alerts or potential faults impacting systems;

deliver a daily report to the Government

- Provide /forward critical alerts immediately to the appropriate Government representative

- Utilize alerts from an automated monitoring system to respond to system outages per established Government-approved SLA(s)

- Coordinate with system vendors, customers, partners, and other mission systems to troubleshoot system problems, outages, or any other issues impacting an SLA

- Utilize environmental monitoring tools (temperature, power, etc.) to monitor systems and provide recommendations to the Government when fault management is required.

PWS-TO2-M08. The Contractor shall provide services for System Administration Performance Management. The tasks include:

● Perform day to day system administration on server class systems to support performance tuning and scripting.

● Utilize automated system monitoring tools to predict problems impacting mission system operations

● Provide weekly reports to the Government on individual server/computers, storage area network (SANs), network attached storage (NAS), and peripheral hardware systems including availability, resource utilization, and system performance.

● Coordinate with the Government to optimize application performance by utilizing system options and features that increase efficiency while preserving the performance requirements of other applications sharing the same system resources.

● Provide bi-annual system account reviews. [CDRL]

● Submit system hardening analysis for all new systems before being put into production.

PWS-TO2-M09. The Contractor shall provide support for System Administration Performance Management. The tasks include:

- Maintaining the security of all servers, computers, devices and peripheral hardware connected to OSPO networks

- Perform security management including security patching, vulnerability mitigations, vulnerability scanning, and system hardening

Source Selection Information – See FAR 2.101 and 3.104-4 21

- Comply and adhere with System Security Plans (SSPs) for patching all systems in the configuration managed inventory to include UNIX, Linux, Windows, and desktop applications

- Apply OSPO-approved security hardening standards to network equipment; provide documentation on applicable desktop security rules, including anti-virus configurations and local security policy settings

PWS-TO2-M10. The Contractor shall provide support for IT System Analysis. The tasks include:

- Analysis and reporting on all IT system capabilities to determine operational states and modes, performance trends, and failure mechanisms.

- Conducting trend analysis to correct potential widespread defects such as feedback loops, and report the results

3.2 Capacity Planning

PWS-TO2-M11. The Contractor shall provide support for Capacity Planning. The tasks include:

- Conduct performance or capacity management studies as specified by the Government

- Document capacity Planning and Performance Management procedures that meet processing requirements

- Provide monitoring and reporting of component performance, utilization, and efficiency

- Perform capacity planning to ensure changes to servers/computers or peripherals do not adversely impact system operations

- Monitor and ensure information management practices and existing IT infrastructure keep pace with each other, to maximize return on investment of the organization’s technological and knowledge assets

- Perform component tuning, per Change Management procedures, to maintain optimum performance

- Manage component resources (e.g., devices and traffic) to meet defined availability

- Proactively evaluate, identify, and recommend configurations or changes to configurations that will enhance performance

- Develop /implement improvement plans (coordinated with stakeholders) that meet the service levels specified for the systems

- Monitor /ensure system capacity and storage are sufficient to handle current and future operational data volume and processing requirements and meet the metrics per OSPO Product Application and Systems Summary Document is included in the OMS Document Library.

3.3 Telecommunications Support

PWS-TO2-M12. The Contractor shall provide Telecommunications Support related to the NSOF phone system and the Operations phone system. Voice services support the mission critical operations within NSOF. The tasks for this include:

Source Selection Information – See FAR 2.101 and 3.104-4 22

- Maintain, record, and troubleshoot voice and voice-band services to all organizations. The technical facility will be a Public Branch Exchange (PBX) with a separate cable pair to each port within a breakout box reserved for voice. Services will be based on orders from the organization through the NSOF Admin LAN MAC process.

- Maintain and troubleshoot the voice mail services on the PBX

- Provide, maintain, inventory, and troubleshoot all telephone hardware including the physical phone handset related to Desktop Voice Services

- Respond to calls from SO and OSPO Telecommunications related to the mission voice conferencing system network outages

- Ensure current documented configuration of OSPO networks and components/hardware connected to the OSPO mission voice system

- Provide network engineering support for all approved change requests associated with the mission voice system network

- Diagnose problems and conduct problem resolution given a voice communications system network equipment failure or connection…