Attachment E Seed Task Order Two PWS_0007.pdf

Task Order 002 System Administration Management Performance Work Statement

3/16/2020

140R8120R0005

IMT Enterprise Support Solutions IDIQ

Attachment E

140R8120R0005

Enterprise Support Services IDIQ PWS

Contents

Page

1.0 INTRODUCTION

2.0 OBJECTIVES

3.0 TECHNICAL REQUIREMENTS

4.0 SERVICE REQUIREMENTS

5.0 CLIN CATEGORIES

6.0 TRAVEL

7.0 DOCUMENTS

8.0 DELIVERABLES

9.0 GOVERNMENT FURNISHED EQUIPMENT

10.0 TELEWORK

11.0 HOURS OF OPERATION

12.0 PERIOD AND PLACE OF PERFORMANCE

13.0 ACRONYM LIST

14.0 QUALITY ASSURANCE SURVEILLANCE PLAN

Modified Date: 8/5/2020 11:23 AM Page 3 of 21

Performance Work Statement System Administration Management Services

03/03/2020

1.0 INTRODUCTION

1.1 The Bureau of Reclamation (herein Reclamation) requires Information Technology (IT) enterprise System Administration management support.

2.0 OBJECTIVES

2.1 The primary objective of this contract is to provide the Bureau of Reclamation IT, Enterprise

System Administration, infrastructure operations support for a variety of IT infrastructure services.

2.2 The Reclamation IT Enterprise System Administration develops, installs, maintains, and supports a broad range of systems and technologies.

3.0 TECHNICAL REQUIREMENTS

3.1 The Task Order shall comply with the base requirements in accordance with the IDIQ contract.

3.2 Contractor shall provide support services for the following infrastructure service areas:

3.2.1 Operating System

3.2.2 Virtualization

3.2.3 Database Operations

3.2.4 Operational Security

3.2.5 Web Application Administration

3.2.6 Collaboration Software

3.2.7 Client Configuration

3.3 Each service area is the responsibility of a Government Technical Service Area Lead (TSAL1).

The TSAL is responsible for over site of a particular service area. Contractor shall work closely with the TSAL concerning day to day issues, activities and projects.

1 The TSAL individual is a Federal Information Technology subject matter expert employee that provides technical direction within their respective task area of expertise for activities under this PWS on a day-to-day basis

Modified Date: 8/5/2020 11:23 AM Page 4 of 21

3.4 Operating System Administration Team(s).

3.4.1 Linux. The Linux System Administration Team manages and maintains Reclamation’s

Enterprise Linux servers hosting a variety of commercial off-the-shelf (COTS), internally developed application software, and databases. It is the responsibility of this team to provide reliable, secure, state-of-art IT services in the area of Systems

Administration, including overall support for current and new functions involving enterprise Linux standards and servers and, legacy Linux servers and related activities.

3.4.2 Linux Current operational information:

3.4.2.1 Operating Systems: Red Hat Enterprise Linux 6.x/7.x, CentOS 6.x/7.x

3.4.2.2 Virtualization: VMware vSphere

3.4.2.3 Automated Installation: Spacewalk (or Red Hat

Satellite), Kickstart, Centrify, Puppet

3.4.2.4 Patch Management: Spacewalk, Satellite

3.4.2.5 System Configuration Management and Automation: Puppet, Centrify

3.4.2.6 System Firewalls: Linux IP tables

3.4.2.7 Intrusion Detection Systems: Tripwire Enterprise

3.4.2.8 Storage: SAN and NAS, NFS Server and Client, Fiber Switch

Configuration, LUN Masking, Zoning, Snapshots

3.4.2.9 Monitoring: CheckMK, Splunk

3.4.2.10 Log Management: Syslog/Syslog-ng/Splunk

3.4.2.11 Scripting Languages: bash/ Perl, Ruby

3.4.2.12 Network Services: SSH, HTTP, RSYNC, DNS, RPC, NFS, NTP, SSL

3.4.2.13 Backup & Recovery: VEEAM

3.4.2.14 Risk Management: BigFix, Nexpose

3.4.3 Windows. The Microsoft Windows Administration Team manages and maintains

Reclamation’s Enterprise Windows servers hosting a variety of commercial off-the-shelf

(COTS), internally developed application software, databases, and file and print services. It is the responsibility of this team to provide reliable, secure, state-of-art IT services in the area of Microsoft Windows Administration, including overall support for current and new functions involving Enterprise Windows standards, Enterprise Windows servers belonging to the Department of Interior’s (DOI) Active Directory forest Windows servers and related activities. The team also participates, in conjunction with other teams, in the administration of the DOI Active Directory forest.

3.4.4 Windows Current Operational Environment:

3.4.4.1 Operating Systems: Microsoft Windows Server 2012r2 and later

3.4.4.2 Automated Installations: Microsoft System Center Configuration Manager

(SCCM) Operating System Deployment (OSD); VMware vSphere templates

3.4.4.3 Patch Management: SCCM Software Updates Management; Windows Update

3.4.4.4 System Configuration: SCCM; Group Policy

3.4.4.5 Automated Monitoring: Microsoft System Center Operations Manager

(SCOM); Splunk, IBM Enterprise Manager (IEM) and various other software

3.4.4.6 Scripting: VBScript; PowerShell

3.4.4.7 Backup & Recovery: VEEAM

Modified Date: 8/5/2020 11:23 AM Page 5 of 21

3.4.4.8 Storage Management: Dell Compellent SAN, Dell EqualLogic SAN, Nutanix, EMC

3.4.4.9 Account Management: NetIQ Directory and Resource Administrator (DRA);

Centrify; native Active Directory Utilities

3.4.4.10 Service Ticketing: Avanti Heat

3.4.4.11 Virtualization: VMware vSphere, Horizon

3.4.4.12 Remote Assistance: Bomgar

3.4.4.13 Remote Access: RDP

3.4.4.14 Alerting and Change Control: Quest ChangeAuditor; Microsoft System Center

Operations Manager (SCOM), SharePoint

3.4.4.15 Log Management: Splunk, Event Viewer

3.5 Virtualization. The Virtualization Team manages and maintains Reclamation’s Enterprise

VMware ESXi hosts and VM guests. It is the responsibility of this team to provide reliable, secure, state-of-art IT services in the area of virtualization administration, including overall support for current and new functions involving enterprise virtualization standards, enterprise

VMware physical and virtual servers, and related activities.

3.5.1 Current Operational Environment:

3.5.1.1 Operating Systems: Host – ESXi 6.5 and above, Guest – Windows, Linux, Appliances

3.5.1.2 Specific Enterprise Features: Storage APIs; Distributed Resource Scheduling /

Distributed Power Management; Storage I/O and Network I/O Control; Virtual

Distributed Switches; Storage DRS; Single Root I/O; Auto Deploy/Host

Profiles, NSX

3.5.1.3 Patch Management: VMware vCenter Update Manager, Nutanix Prism

3.5.1.4 System Configuration Management and Automation: VMware Host Profiles

3.5.1.5 Replication: Site Recovery Manager, Nutanix Protected Domains

3.5.1.6 System Firewalls: VMware Firewall Management through vCenter, LUNS, NSX Micro segmentation

3.5.1.7 Backend Storage: Compellent SAN, Nutanix HCI, Exagrid, Data Domain

3.5.1.8 Backend Connectivity: Fiber Switch, Ethernet Switch

3.5.1.9 Monitoring: vCenter Alarms and Alerts; Splunk; Veeam, Prism, CheckMK

3.5.1.10 Log Management: Syslog/Splunk

3.5.1.11 Backup & Recovery: Veeam

3.6 Database Operations. The Database Operations Administration Team installs, configures, manages and maintains Reclamation’s database environments, including overall support for current and new functions involving enterprise database standards, enterprise database systems and related activities.

3.6.1 Current Operational Environment:

3.6.1.1 Operating Systems: Linux 6x and greater, Windows 2012r2 and grater

3.6.1.2 Database Management Systems: Oracle 12c and 19c, SQL

2012/2014/2016/2017 and MYSql

3.6.1.3 Monitoring: Oracle Enterprise Manager, Splunk, IEM, CheckMK

3.6.1.4 Scripting Languages: bash/Perl/TSQL/PLSQL/PowerShell

3.6.1.5 Backup & Recovery: RMAN, VEEAM

Modified Date: 8/5/2020 11:23 AM Page 6 of 21

3.7 Operational Security Administration. The Operational Security Administration Team is responsible for supporting all facets of security from the operations and maintenance perspective.

3.7.1 This team must ensure that all Security Technical Installation Guides (STIGs) and security-related Standard Operating Procedures (SOPs) are implemented and systems are tested regularly for compliance.

3.7.2 This team must work closely with system and network administrators across Reclamation to ensure all server and desktop STIGs and SOPs meet Federal, Department of Interior

(DOI), and Reclamation guidelines and policies.

3.7.3 This team is also responsible for assisting with the configuration of Virtual Private

Networks (VPNs) that exist to allow authorized users to cross the Reclamation security perimeter, ensuring that mail relays function according to sanctioned security guidelines and procedures, and assisting other technical teams to ensure that solutions are implemented in a secure way.

3.7.4 This team will function as the Subject Matter Expert in technical writing for Information

Technology staff throughout Reclamation. Designing documents and instructions that can be easily maintained, updated and understood by Reclamation information technology staff is critical to communications.

3.7.5 Current Operational Environment:

3.7.5.1 Operating Systems: Microsoft Windows Server 2012r2 and later; Microsoft

Windows Client 10 or later, REHL 6 and later, CentOS 6 and later, and other

Linux versions, Mac OSX

3.7.5.2 Patch Management: SCCM, Windows Update, Puppet, Mac update, Spacewalk, BigFix

3.7.5.3 System Configuration: Group Policy, Puppet, Centrify

3.7.5.4 Scripting: PowerShell, Bash, Ruby

3.7.5.5 Account Management: NetIQ Directory and Resource Administrator (DRA);

Active Directory Users and Computers, Centrify, Puppet

3.7.5.6 Service Ticketing: Ivanti HEAT

3.7.5.7 Security Scanning: Tenable Nessus Enterprise Suite, McAfee Vulnerability

Management, IEM, Web inspect, Nexpose, Burp Suite, Spirion

3.7.5.8 Alerting and Change Control: Quest ChangeAuditor, Microsoft System Center

Operations Manager (SCOM), Splunk, Tripwire, SolarWinds, IEM, Puppet

3.8 Web Application Server Administration. Contractor shall be responsible for the installation, configuration, troubleshooting, and maintenance of a variety of Commercial Off-the-Shelf

(COTS) server-based applications and the various web and application-layer servers that are used to host in-house developed applications on both the Windows and Linux/UNIX platforms. These environments consist of various technologies such as clustering, load balancing, Web Services, Enterprise Service Bus, Email List Servers, FTP servers, WebDAV, Proxy Configurations, SPAM filtering and other technologies related to Web Content and Application architectures.

3.9 Contractor shall be responsible for Various Web and Middleware software for Development, Test and Production environments are designed, implemented and supported. Many of the instances of software are collocated on single operating system instances.

3.9.1 Current Operational Environment:

Modified Date: 8/5/2020 11:23 AM Page 7 of 21

3.9.1.1 App Server: Websphere, WebLogic, WildFly, Cold Fusion, Tomcat, ASP.net, Java, PHP, MessageQue, Search Technologies

3.9.1.2 Web Server: Apache, IIS

3.9.1.3 Transport Protocols: sFTP, FTP, SCP, NFS, HTTP, HTTPs, RMI, AJP, SMTP, SNMP, NTP

3.9.1.4 Tools: RPM, Hudson, Perl, HTTPerf, Apache Bench, REGEX, OpenSSL, JConsole, JMonitor

3.9.1.5 Reporting: Splunk, WebTrends, Google Analytics, IEM

3.9.1.6 Content Management: Hudson, Jira, Ant, Svn

3.9.1.7 Traffic Management: F5 LTM, DNS

3.10 Collaboration Software Administration.

3.10.1 The Collaboration Software Team manages and maintains Reclamation’s Corporate and

Denver Office SharePoint system (currently 2013) and will be responsible for designing, developing, implementing, migrating and maintaining various Collaboration platforms for

Reclamation wide use. It is the responsibility of this team to provide reliable, secure, state-of-art IT services in the area of Microsoft SharePoint Services Administration, including overall support for current and new functions involving enterprise standards, enterprise servers and related activities.

3.10.2 Current operational environment:

3.10.2.1 COTS: SharePoint 2010/2013 and above

3.10.2.2 Integration: Microsoft Office Suite 2010/2013

3.10.2.3 Operating System: Windows Server 2012r2 and above

3.10.2.4 Patch Management: SCCM, SharePoint patching, Windows update

3.10.2.5 Development: Informs, ASP.Net, SharePoint Designer, AvePoint DocAve, Adobe Photoshop Elements, Customized Web Parts, Metalogix Site migration manager

3.10.2.6 Reporting/Monitoring: SCOM, Splunk, IEM

3.11 Client Configuration Administration. Reclamation maintains an enterprise-wide Microsoft

Systems System Center Configuration Manager (SCCM) infrastructure to facilitate delivery of software and critical operating system updates, collect hardware and software inventory, and perform other important management tasks. This team creates standard application packages that are deployed using SCCM and maintains the Software Center, to deliver approved applications to a user’s desktop. The team also provides high level Windows desktop support for complicated technical issues that need to be elevated from the Service Desk, and administers a virtual desktop infrastructure (currently VMWare View)

3.11.1 Current operational environment

3.11.1.1 Packaging Technologies: Virtual Applications, Microsoft Installer, Admin

Studio

3.11.1.2 Client Virtualization: VMware View

3.11.1.3 Remote Assistance: Bomgar, Microsoft Remote Assistance, WebEx, LiveMeeting

3.11.1.4 Service Ticketing: Ivanti HEAT

3.11.1.5 Configuration Technology: SCCM 2007/2012, Software Center

3.11.1.6 Tools: Primal Script, Flexera AdminStudio

Modified Date: 8/5/2020 11:23 AM Page 8 of 21

4.0 SERVICE REQUIREMENTS

4.1 Operating System:

4.1.1 Contractor shall create and configure shares, manage NTFS, DFS and share permissions. Troubleshoot user access to files on file servers

4.1.2 Contractor shall create, troubleshoot and maintain new and existing PowerShell scripts for the automation of system administration duties

4.1.3 Contractor shall install and configure print servers and assist support services with installation, configuration and troubleshooting of printers

4.1.4 Contractor shall install, configure and manage terminal server environment, including installation, configuration and troubleshooting of client applications installed on them or published from them

4.1.5 Contractor shall assist Reclamation staff in troubleshooting and maintaining the health of the Active Directory Test and Development labs.

4.2 Virtualization:

4.2.1 Provision Windows and Linux virtual machines.

4.2.2 Provision Storage for virtual machines.

4.2.3 Configure virtual Distributed Switches.

4.2.4 Creating DRS Rules and Groups for hosts and vm's.

4.2.5 Capacity planning of virtual resources.

4.2.6 Use Update Manager to patch and upgrade ESXi hosts.

4.2.7 Update VMtools using PowerShell.

4.2.8 Create and configure VM guests as requested

4.2.9 Implement high availability solutions for Reclamation VMs using various technologies, including but not limited to; VMware Site Recovery Manager (SRM), Compellent Replays, Compellent replication

4.3 Database Operations:

4.3.1 Contractor shall install and configure database software according to Reclamation, DOI and CIS standards.

4.3.2 Contractor shall perform database and system performance and tuning utilizing various tools such as Oracle Enterprise Manager, Microsoft Management Studio, etc.

4.3.3 Contractor shall ensure appropriate backup and recovery plans are in place, monitored and exercised ensuring recoverability of Reclamation database systems.

4.3.4 Contractor shall work closely with development and application hosting teams to deploy solutions for long term scalability of the database environments.

4.3.5 Contractor shall schedule and perform regular database maintenance/patching.

4.3.6 Contractor shall provide support for database client software installs.

4.3.7 Contractor shall monitor and maintain database/system health and security utilizing Oracle

Enterprise Manager, SCOM, Splunk, Microsoft Management Studio and other monitoring and troubleshooting tools.

Modified Date: 8/5/2020 11:23 AM Page 9 of 21

4.4 Operational Security: The Operational security role is critical to the organization in that it must ensure that operation processes are meeting Reclamation’s security responsibilities. This role will work with all of the other service areas in a concerted effort toward overall infrastructure security, particularly as it relates to the Microsoft Windows and Linux environments. The contractor shall provide the following system security support activities unless otherwise directed by the TSAL:

4.4.1 Contractor shall assist Information System Security Officer (ISSO) in updating System

Security Plan (SSP), Contingency Plan, and other Authority and Assessment documentation as directed TSAL.

4.4.2 Contractor shall create and maintain security audit tools (SAT) in conjunction with staff of various service areas.

4.4.3 Contractor shall execute security audit tools against designated systems as required by the

Technical Service Area Lead.

4.4.4 Contractor shall plan, design, review, analyze and report on implementation of FDCC and

USGCB STIG settings across Reclamation and make recommendations for changes in security procedures to security and technical staff.

4.4.4.1 Contractor shall assist technical teams to monitor the FDCC and USGCB implementation and to assess the impact of changes. The contractor shall support the development of documentation for Microsoft Windows 10 and above and Windows Servers to include 2012r2 and above, RHEL 6 and above

Centos 6 and above security policies with Reclamation technical teams.

4.4.5 Manage Nessus, Nexpose, Web inspect, Spirion, Tripwire Servers.

4.4.5.1 Contractor shall be able to administer the software, install updates, troubleshoot malfunctions, and perform software maintenance. Be very familiar with this tool or a similar tool that conforms to the common security vulnerability and exposure (CVSS) standards. Upon request, the contractor will use a

Reclamation standard configuration and scan specified devices. The output will be reviewed for validity and accuracy. The output report will be forwarded to specified security personnel. The contractor must be able to configure and generate accurate outputs of each system. The contractor will verify the output and forward to specified personnel. These reports will be done upon request.

4.4.6 Contractor shall support the monitoring of POAM items in the Cyber Security Assessment and Management (CSAM).

4.4.6.1 Contractor shall participate in the resolution of POAM items relating to infrastructure service areas while coordinating with Technical Service Area

Leads, infrastructure administrators and ISSOs.

4.4.7 Contractor shall perform research services on NIST 880-53; FISMA and DOI related security controls.

4.4.7.1 Contractor shall assist in conduction system security assessments to assure the proper emphasis on Reclamation system security plans. Must have the ability to review conceptual procedures, security program reports, and other software source documentation which might aid in establishing and meeting system security requirements.

4.4.8 Contractor shall support the annual internal control review (ICR) process.

4.4.8.1 Contractor shall construct the gathering of security control information for

Reclamation.

Modified Date: 8/5/2020 11:23 AM Page 10 of 21

4.5 Web Application Server Administration.

4.5.1 Contractor shall deploy all hosted application code to application servers;

4.5.2 Contractor shall configure data sources within application server software and middleware connecting applications and databases; Configure messaging services for message-oriented middleware. Ensure applications are running properly after deployments. Rollback deployment if errors or failure occurs.

4.5.3 Contractor shall install, configure and secure middleware and web software;

4.5.4 Contractor shall configure log rotation for web and application server software; Perform of primary services; Develop and install initialization scripts (init.d) for automatic starting and stopping of processes; configure messaging components. Ensure systems are functioning properly.

4.5.5 Contractor shall install, configure and secure load balancers.

4.5.6 Contractor shall configure backend services for load balancing; Perform failovers for maintenance or incident management. Upgrade and patch load balancers. Ensure credentials and logins are monitored. Install monitors and health checks on load balancers to ensure node and virtual server health. Manage any SSL certificates, review all SSL certificates, and order new ssl certificates 30 days before they expire.

4.5.7 Contractor shall perform general web server administration duties;

4.5.8 Contractor shall Install and configure web server software; Create, Configure, and install

SSL certificates. remove or install unneeded modules. Review and configure security and performance settings

4.5.9 Contractor shall Scripting and automation of procedures:

4.5.10 Contractor shall script and automate deployment procedures for various applications; Script and automate the installation and configuration of web and application server software;

Develop tools to automate and perform administration duties; Perform upgrades to supported and hosted software; Develop system requirements for installations.

4.5.11 Contractor shall document all procedures and processes

4.6 Collaboration Software.

4.6.1 Contractor shall manage SharePoint Topology and Services, shared services and assist with duties focusing on the Central Administration Page of SharePoint

4.6.2 Contractor shall manage and maintain system, application, security and SharePoint, IIS event logs and report on them. Monitoring SharePoint disk space usage through the built-in SharePoint reports for each site collection

4.6.3 Contractor shall regular review cleanup, management and configuration of SharePoint accounts and sites

4.6.4 Contractor shall check on health of SharePoint environment

4.6.5 Contractor shall install and maintain SharePoint technologies, (i.e. SharePoint search

FAST, excel services, etc. including any 3rd party application, plug-ins, features, web parts, templates or other solutions, including but not limited to Bamboo PM

Central, AvePoint DocAve, Metalogix.

4.6.6 Contractor shall monitor SharePoint usage trends

4.6.7 Contractor shall manage upgrades and migrations, global configuration, backup and restores, and data configuration.

Modified Date: 8/5/2020 11:23 AM Page 11 of 21

4.6.8 Contractor shall manage content deployment, user profiles and My Sites.

4.6.9 Contractor shall manage Search, including federated searching of multiple SharePoint environments.

4.6.10 Contractor shall perform patches and upgrades specific to SharePoint that are not deployed using SCCM.

4.6.11 Contractor shall perform Web application and Site Collection administration duties, manage features and solutions for site collection, provides SharePoint site provisioning for site collection, content creation and manages content.

4.7 Reporting and Monitoring.

4.7.1 All Splunk server configurations (web, indexing retention, authentication, etc.)

4.7.2 All Splunk data onboarding operations (inputs, SQL, index-time configurations)

4.7.3 All Splunk data parsing operations (search-time field extractions, event types, tags)

4.7.4 Management of existing apps

4.7.5 Creation of new apps (visual and non-visual) to meet the requirements by the POC

4.7.6 Maintain documentation including what work has been done, what is left to do, and site-specific procedures documenting the Splunk environment.

4.7.7 Create event processing

4.7.8 Manage timestamps

4.7.9 Create indexes for field extractions

4.7.10 Create and manage host values and source types

4.7.11 Parse event segmentation

4.7.12 Manage and ensure proper data fields for file and directory inputs

4.7.13 Manage network, Windows/Linux and any other inputs that may arise (universal forwarders).

4.8 Client Configuration.

4.8.1 Contractor shall operate and maintain SCCM distribution points and assist with enterprise wide SCCM tasks.

4.8.2 Contractor shall develop and maintain SCCM software deployment packages for

Reclamation as requested and according to Reclamation standards. Create and maintain

SCCM packages as presented through the Software Center.

4.8.3 Contractor shall perform maintenance procedures on the Reclamation SCCM environment to ensure continued health of the environment.

4.8.4 Contractor shall troubleshoot and resolve SCCM client health issues. Ensure that all managed workstations have a functional client at all times

4.8.5 Contractor shall participate in Reclamation SCCM Team weekly conference calls.

4.8.6 Contractor shall assist Government staff (SCCM Central Site Admins and AD Domain

Admins) with development and validation of Reclamation standard workstation and server images. Create and maintain SCCM packages as presented through the Software Center.

4.8.7 Contractor shall manage and maintain the virtual desktop infrastructure.

4.8.8 Contractor shall assist service desk personnel with troubleshooting and resolution of complicated desktop issues.

Modified Date: 8/5/2020 11:23 AM Page 12 of 21

4.9 General Administration.

4.9.1 Contractor shall always apply Reclamation system security policies and procedures, STIGs, and Standard Operating Procedures on all new and existing systems.

4.9.2 Contractor shall provide access to existing monitoring services as requested by the

Technical Service Area Lead according to the Service Area System Security Plan.

4.9.3 Contractor shall revoke all system access upon termination of Reclamation employee(s) or contractor(s) within 15 minutes of notification by your Technical Service Area Lead or

COR.

4.9.4 Contractor shall apply system security patches as defined by Reclamation policy according to the service area patching SOP.

4.9.5 Contractor shall verify quarterly validity of active user access to Reclamation systems and services according to Reclamation provided data.

4.9.6 Contractor shall always adhere to the Reclamation change process for modifying the access rules through system firewalls unless approved by the Technical Service Area Lead or

COR.

4.9.7 Contractor shall always provide documentation for all changes to the system software and configuration.

4.9.8 Contractor shall always ensure the accuracy of network and system diagrams subsequent to change.

4.9.9 Contractor shall ensure that performance /event logging is functioning for all new and existing system 100% of the time.

4.9.10 Contractor shall must always report system security incidents in accordance with

Reclamation’s Computer Security Incidence Response procedures within 15 minutes of its subsequent discovery 100 % of the time. See the URL link in Paragraph 12.4.5 for detailed information.

4.9.11 Contractor shall monitor Active Directory Group policy and Puppet for changes related to

Reclamation Security Standards.

4.9.12 Contractor shall documentation: Contractor staff shall provide documentation to the

Government staff on all tasks and projects assigned.

4.9.12.1 This documentation includes but is not limited to, SOPs, flowcharts, wiring diagrams, network diagrams, system design documents, ad hoc reports, Security

Technical Implementation Guides (STIGs), operating and maintenance requirements, progress reports and milestone reports and any other documentation required by the TSAL or COR. Specific documentation will be clarified through technical task clarifications. All documentation, programs and development shall remain the sole property of the Federal Government.

4.9.13 Testing: The contactor shall support and participate in testing efforts of new and current applications whether in development or production to ensure they are operational in the desktop, server, and application environment(s) and do not impact other services.

4.9.14 The contactor shall support and develop documentation for testing and support of in house developed and COTS applications to ensure all aspects of the deployment and operational processes are complete and repeatable. Coordination with the various IT entities within the

IT Services Division program is essential.

4.10 Daily Administration. In support of daily operations, the contractor shall provide:

Modified Date: 8/5/2020 11:23 AM Page 13 of 21

4.10.1 System/Service Monitoring. All outages shall be reported, resolve critical service issues, investigate critical and warning level notifications provided by monitoring services, and log activity.

4.10.2 Backup & Recovery. All backups shall be verified daily and activity shall be logged.

4.10.3 Log Monitoring. All logs shall be monitored. Authentication, OS hardware failure, and administrative activity logs shall be investigated as necessary or as requested by the TSAL or COR. Any and all anomalies shall be reported to TSAL.

4.10.4 Environment Protection. The contractor shall execute data center environment walk-through reviews focusing on checks for hardware failures but should report any noticed anomalies to the Technical Service Area Lead.

4.11 New and Existing System Support. In supporting new and existing systems, the contractor shall:

4.11.1 Evaluate products and services related to each service area and propose suggestions / recommendations of future technologies along with the benefits to Reclamation.

4.11.2 Perform capacity planning related to data growth and system utilization, trend analysis and predict future storage and system resources requirements to minimize resource or space issues

4.11.3 Ensure that a request for change form has been submitted, approved, and scheduled through the RMSS Change Boards or other applicable change board prior to the performing next tasks. Emergency changes should follow the approved change processes as noted at the Change Board site.

4.11.4 Install operating system per build documents or SOPs, (will verify contractor work).

4.11.5 Apply the latest security and recommended patches per vendor recommendations and

Reclamation standards.

4.11.6 Configure operating system parameters according to Reclamation system configuration standards (system configuration and security hardening –SOPs and STIGs)

4.11.7 Install and configure system security scanning software: Currently, but not limited to

Nessus

4.11.8 Install, configure and support system change auditing software: Currently, but not limited to Tripwire, TACACS, SolarWinds, Cisco ACS, Quest Change Auditor, Splunk

Install and configure intrusion detection and prevention software: Currently, but not limited to Splunk ESS, Tripwire, Symantec SIM

4.11.9 Install and configure monitoring server software: Currently, but not limited to Nagios, Splunk, SolarWinds, InterMapper, Ganglia, SCOM

4.11.10 Install and configure monitoring client software: Currently, but not limited to SCOM, NRPE, Splunk, and Syslog-ng.

4.11.11 Install and configure system configuration automation software: Currently, but not limited to SCCM, Puppet, SolarWinds

4.11.12 Install and configure patch management software: Currently, but not limited to SCCM, Windows Update, RHN, Spacewalk, YUM, SolarWinds

4.11.13 Install and configure configuration management database software: Currently, but not limited to Git, (SharePoint), Hudson

4.11.14 Install and configure host-based firewall software: Currently, but not limited to IPtables, Shorewall, Apache reverse proxy with mod_security2, Windows Firewall

Modified Date: 8/5/2020 11:23 AM Page 14 of 21

4.11.15 Document the installation and configuration of systems using Reclamation provided system configuration management tools: Currently, but not limited to SharePoint

4.11.16 Ensure system connectivity to other systems.

4.11.17 Maintain version control of system objects – Configuration Management: Currently, but not limited to SharePoint

4.11.18 The contractor shall ensure all systems comply with approved Reclamation SOPs and

STIGs prior to deploying systems to development, test or production environments. In conjunction with the Federal staff, the contractor shall develop SOPs and STIGs or comply with existing Reclamation documentation.

4.12 Technical Administration. To administer systems, the contractor shall:

4.12.1 Ensure optimal performance of all Reclamation systems. Conduct system performance tuning.

4.12.2 Follow Reclamation’s change management processes, test all changes in a representative test environment before implementation in production, identify and resolve technical issues after implementation of changes identify and provide recommendations for violation resolutions, and log activity.

4.12.3 Ensure that all systems are configured to comply with Reclamation system security policies and procedures.

4.12.4 Ensure that required monitoring and configuration agents and clients are installed and functional on all systems.

4.12.5 Acknowledge and respond to all critical- and warning-level alerts within the various monitoring systems.

4.12.6 Monitor and control access to all Reclamation systems managed by the Enterprise

Operations Division.

4.12.7 Revoke access upon employee or contractor termination. Access to critical systems must be revoked within 15 minutes of notification of termination. Access to other systems must be revoked within one business day.

4.12.8 Create and maintain an up to date diagram for all systems.

4.12.9 Provide hardware installation, implementation, maintenance, and preventive and preemptive troubleshooting.

4.12.10 Contact and assist vendor in the event that hardware and/or software maintenance is required. Maintain vendor contact list.

4.12.11 Perform system hardware upgrades along with system and 3rd party software upgrades.

4.12.12 Perform disk, file system, volume group and logical volume configuration and maintenance.

4.12.13 Respond to emergencies as requested by TSAL Technical Service Area Lead.

4.12.14 Perform operating system network configuration, maintenance and troubleshooting.

4.12.15 Provide technical assistance to other Tier 3 service area teams, Tier 2 teams, and customers.

4.12.16 Provide on-site or remote assistance to regional technical staff regarding the installation, configuration, and troubleshooting of systems.

4.12.17 Write new monitoring checks as needed.

Modified Date: 8/5/2020 11:23 AM Page 15 of 21

4.12.18 Utilize Microsoft Active Directory as the standard authentication mechanism on all systems that support it

4.12.19 Implement high-availability and disaster recovery for production systems as directed by the

Technical Service Area Lead.

4.12.20 Create and maintain a knowledgebase of documentation for all hardware and software troubleshooting.

4.12.21 Work with and provide appropriate information to internal/external auditors.

4.12.22 Work collaboratively as well as independently to support deadlines.

4.12.23 User management and authentication using roles

4.13 Security.

4.13.1 The contractor shall identify, document, and report system security issues according to

Reclamation incident response operating procedures.

4.13.2 The contractor shall provide system risk and impact assessments and risk mitigation strategies, along with participating in risk mitigation activities.

4.13.3 The contractor shall contribute to the implementation and maintenance of DOI Security requirements associated with the Authority and Assessment process.

4.13.4 The contractor shall test and update all security documentation in each service area.

4.13.5 The contractor shall provide technical benchmarks and implement STIGs on all systems for which they are responsible.

4.13.6 Contractor shall analyze and resolve system/application vulnerabilities as they arise and report on a monthly basis.

4.13.7 Contractor shall participate in Internal Control Reviews (ICRs) with the portfolio’s ISSO.

4.13.8 Contractor shall participate in resolution of the Plans of Action and Milestones

(POA&M).

4.13.9 Contractor shall develop and test contingency plans

4.13.10 Contractor shall participate in documenting and testing of contingency plans related to the service area.

4.13.11 Contractor shall ensure STIGs are available and current for each service area

5.0 CLIN CATEGORIES

5.1 The following labor-hour CLINs are required on this Task Order:

CLIN DESCRIPTION QUANTITY

0023 Database Administrator 2 2

0024 Database Administrator 3 1

0028 Data Security Specialist 3 1

0047 Project Manager 2 1

0049 SharePoint Administrator 2

0053 System Administrator 3 2

Modified Date: 8/5/2020 11:23 AM Page 16 of 21

0054 Systems Administrator 4 1

0056 Systems Engineer 2 3

6.0 TRAVEL

6.1 Occasional travel is a requirement for this Task Order to perform tasks in the Regions. All travel must be in accordance with Federal Travel Regulations and only actual expenses will be reimbursed and a not exceed amount will be identified. Travel will be identified as Time-and-

Material CLINS.

7.0 DOCUMENTS

7.1 Standard Operating Procedures (SOPs). Contractor shall maintain and update existing SOP documents and/or manuals to explain various procedures within the Reclamation information systems environment. Contractor shall develop an SOP for any process or procedure that does not have documentation established.

7.2 Contractor shall provide the required documentation to the Government on tasks and projects assigned; specific documentation will be clarified through technical task clarifications. This documentation includes SOPs, flowcharts, wiring diagrams, network diagrams, and system design documents, ad hoc reports, Security Technical Implementation Guides (STIGs), operating and maintenance requirements, progress reports and milestone reports and any other documentation required by the Federal Government.

7.3 The contractor’s invoice shall be in accordance with IDIQ 52.212-4(g) Addendum.

8.0 DELIVERABLES

8.1 The Task Order shall comply with the base requirements in accordance with the IDIQ contract identified in this Task Order for deviation on deliverables.

8.2 The following deliverables as applicable.

PWS Para Brief Description Delivery Schedule

Parent Solicitation Monthly Status Report

10th calendar day

Parent Solicitation Staffing/Transition-In Plan Submit with proposal and update if any personnel changes occur during POP

Parent Solicitation Meeting Minutes

On Going

Parent Solicitation Standard Operating Procedures

On Going

Modified Date: 8/5/2020 11:23 AM Page 17 of 21

PWS Para Brief Description Delivery Schedule

Section 4.0 Briefings and reports pertaining to activities that apply to system security

As Requested

Section 4.0 Briefings and reports for system related installation, configuration, maintenance and support

Section 4.0 Maintain and notify the Government of changes to the on-call rotation schedule

Section 4.0 Daily operations and monitoring to the online reporting site

• System/Service Monitoring

• System Change Monitoring

• Log Monitoring

• Environmental Protection

• System Configuration

Daily

Section 4.0 Change Management

• Provide status using email.

• Ensure that all changes have been verified in a non-production environment.

• Ensure an RFC has been submitted and approved prior to making changes following

Reclamation’s established change process.

Every time a change is required or as directed by the Technical Service Area

Lead

Section 4.0 System Security – contributions to:

• System Security Plan (SSP)

• Authority and Assessment process

• Internal Control Reviews (ICRs)

• Plan of Action and Milestones (POA&M)

• Test contingency plans

As Requested

Section 4.0 Technical documentation:

• System layout

• Security Analysis

• Risk Identification and Mitigation

• STIGs

As Requested

9.0 GOVERNMENT FURNISHED EQUIPMENT

9.1 The Government will provide the following GFE under this Task Order;

9.1.1 Suitable workspace cubicle.

Modified Date: 8/5/2020 11:23 AM Page 18 of 21

9.1.2 Government STIG computer (Accountable Property).

9.1.3 Microsoft Office O365 Suite.

9.1.4 Desk VoIP Telephone/Cisco Jabber.

9.1.5 Photo ID PIV Card (after security adjudication).

9.1.6 Active Directory Account.

9.1.7 Government Email Account.

9.1.8 Mobile Phone (Accountable Property), if applicable depending on position assignment of each contractor personnel. COR has final determination on the issuance of this item.

9.1.9 Multi-functional copier/printer access.

9.1.10 Basic office supplies.

9.1.11 Suitable surface parking near Building 67.

10.0 TELEWORK

10.1 Telework. The Government has determined that the services provided under this Task Order are not eligible for regular scheduled or recurring basis telework. The Government may authorize situational2 telework to Contractor personnel as deemed appropriate on a case-by-case basis authorized by the CO/COR/TSAL and Contractor Project Manager discretion.

11.0 HOURS OF OPERATION

11.1 The Task Order shall comply with the base requirements in accordance with the IDIQ contract for hours of operation identified in this Task Order.

11.2 After-hours support for planned work as well as on-call coverage for unplanned outages will be required under this Task Order frequently.

12.0 PERIOD AND PLACE OF PERFORMANCE

12.1 The period of performance will be October 01, 2020 through September 30, 2021.

12.2 The location for the effort is the Bureau of Reclamation, Denver Federal Center, Building 67 and Building 53 (Data Center), Denver, Colorado, 80225.

13.0 ACRONYM LIST

Acronym Definition Acronym Definition

2 Situational Telework is a case-by-case work arrangement conditional to government advance approval that allows contractor personnel to perform work, during any part of regular scheduled work hours on a particular work day, at an approved alternative worksite as a result of inclement weather, special work assignments, or irregular in nature (i.e., continuity of operations (COOP) in the event of a crisis or national emergency). If granted the contractor shall be responsible to ensure personnel have valid VPN accounts and are connected and responsive to all work requirements with no difference in the level of support, responsiveness and availability while teleworking.

Modified Date: 8/5/2020 11:23 AM Page 19 of 21

CSAM Cyber Security Assessment and Management

CVSS Common Vulnerability Scoring System

DNS Domain Name Servers

DRS Distributed Resource Scheduler

FDCC Federal Desktop Core Configuration

HCI Hyper-converged Infrastructure

ICR Internal Control Review

NAS Network-attached Storage

NFS Network File System

NRPE Nagios Remote Plugin Executor

NTP Network Time Protocol

NTFS New Technology File System.

RDP Remote Desktop Protocol

RMAN Recovery Manager

RPC Remote Procedure Call

SAN Storage Area Network

SAT Security Audit Tools

SCOM System Center Operations Manager

SRM Site Recovery Manager

TACACS Terminal Access Controller Access-Control System

USGCB United States Government Configuration Baseline

YUM Yellowdog Updater

Modified Date: 8/5/2020 11:23 AM Page 20 of 21

14.0 QUALITY ASSURANCE SURVEILLANCE PLAN

Quality Assurance Surveillance Plan

Performance Requirement Standard AQL Surveillance Method Result if AQL not meet

Apply system security patches as defined by Reclamation policy according to the service area patching SOP.

(PWS Section 4.0)

100% of systems patched as defined in Service Area SOP or as directed by the Technical Service Area Lead

100%

Report provided after patching

Document CPARS3 for Adverse Performance Deficiencies.

Ensure that performance /event logging is functioning for all new and existing production systems.

(PWS Section 4.0)

Required systems must have functioning performance/event logging

90%

Random observations. Positive CPARS Input for projects that exceed schedule.

Must always report system security incidents in accordance with Reclamation’s Computer Security Incidence Response procedures

(PWS Section 4.0)

Incidents are reported within 15 minutes of discovery or as directed by your TSAL or COR

100%

Review of monthly security incident report

Document CPARS for Adverse Performance Deficiencies.

All outages shall be reported, resolve critical service issues, investigate critical and warning level notifications provided by monitoring services, and log activity

(PWS Section 4.0)

Outages are reported within 30 minutes of discovery to their TSAL, or COR. 100%

Review logs and other reporting mechanisms, e.g. HEAT, RESC, customers, etc.

Document CPARS for Adverse Performance Deficiencies.

All backups shall be verified including daily, incremental, full, monthly.

(PWS Section 4.0)

Backup & Recovery checked daily 95%

Review monthly. Document CPARS for Adverse Performance Deficiencies.

3 CPARS – Contractor Performance Assessment Report System. A CPAR assesses a Contractor's performance and provides a record, both positive and negative, on a given Contractor during a specific period of time that all Government Contracting Offices may use as past performance reference for pending solicitations, https://www.cpars.gov/.

Modified Date: 8/5/2020 11:23 AM Page 21 of 21

All logs shall be monitored. Authentication, OS hardware failure, and administrative activity logs shall be investigated as necessary or as requested by the TSAL or COR.

(PWS Section 4.0)

Check the logs daily

95%

Review monthly. Document CPARS for Adverse Performance Deficiencies.

Ensure that a “request for change form” has been submitted and approved for all changes and that the Reclamation change process has been followed.

(PWS Section 4.0)

Request for Change forms are submitted, and change process followed 100%

Monthly periodic inspection

Document CPARS for Adverse Performance Deficiencies.

Install, patch and configure systems according to Reclamation system configuration standards

(PWS Section 4.0)

Within the agreed timeframe between the contractor and the TSAL

100%

Monthly periodic inspection

Document CPARS for Adverse Performance Deficiencies.

Document the installation and configuration of systems (PWS Section 4.0)

Within 1 business day after the change is made or as directed by the TSAL

100% File review and random inspections

Document CPARS for Adverse Performance Deficiencies.