COM-F_PWS__18_Dec_2017.docx

FA4452-15-C-0001

(andichm1640)

PERFORMANCE WORK STATEMENT

FOR

Communications, Operation, & Maintenance Functions (COM-F)

Original Date: 31 October 2017 Revision # 8 Date: 12 Dec 2017

Table of Contents

Communications, Operations, & Maintenance Functions (COM-F)

INTRODUCTION

BACKGROUND

The Air Mobility Command (AMC) is responsible to the National Command Authority (NCA) through the Joint Chiefs of Staff (JCS) and the United States Transportation Command (USTRANSCOM) for the planning and execution of global airlift and aerial refueling missions during peace, crisis, and war. AMC is also responsible for the coordination and exercise of airlift and air refueling strategy, doctrine, and plans under the direction of USTRANSCOM. Crisis and wartime support are provided to the unified and specified commands. As an Air Force (AF) Major Command (MAJCOM), AMC is responsible to the Secretary of the Air Force and the Chief of Staff, Headquarters (HQ) United States Air Force (USAF), for the accomplishment of peacetime airlift and air refueling services in a manner that will promote the emergency and wartime capability of assigned forces and ensure the support of all Department of Defense (DoD) components. In addition, AMC is also responsible for aeromedical evacuation operation.

AMC supports the Mobility Air Force mission--"Global Reach." It is difficult, if not impossible, for United States military forces to deploy rapidly or be supported effectively during employment without the assets and expertise provided by AMC. AMC Command and Control (C2), In-Transit Visibility (ITV), and Business Systems (hereafter referred to as “Systems”) processes, procedures, and services must be consistent and reliable in peacetime, during contingencies, and throughout the full spectrum of war. Therefore, the information technology (IT) resources enabling the efficient and effective management of this capability must also provide these services at the highest levels of availability, integrity, and confidentiality.

The 375th Communication Support Squadron (CSPTS) provides command and control, communications, and computer (C4) support to DoD, USTRANSCOM, AMC, and worldwide aerial and water ports. The 375th CSPTS also operates and maintains the Systems supporting the 618th Air & Space Operations Center’s (618th AOC) global mission. The 375th CSPTS, augmented by a contractor providing Communications, Operations, and Maintenance Functions (COM-F) services, is AMC’s answer to meeting the challenges of providing operational support and maintenance of the Systems in a timely, effective, and efficient manner.

The Systems are in a constant state of change as hardware is replaced/upgraded, software versions and patches are fielded, and the entire computing and network environments evolve from a thick/thin client, system centric client server environment to an Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) construct. AMC has begun implementing hardware and software in the enclave environment to support virtualized operations. This process will continue as systems and applications capable of operating in virtualized environments are fielded. The entire operation will evolve over time from a system-centric perspective to an enterprise perspective, with a large proportion of the workload shifting to managing the virtual environment. In addition, all applications are scheduled to be migrated to the USTRANSCOMs instance in the GovCloud. The future products for support in the modernization and cloud remain unknown at this time.

SCOPE

The scope of the contract is for a contractor to provide resources in staffing necessary to meet the performance standards of the performance work statement (PWS), the Systems Availability Thresholds (Appendix 1), systems Original Equipment Manufacturer (OEM) maintenance standards, and the System operational reporting criteria.

The scope of the contract includes systems for which operations and maintenance (O&M) support is provided by 375th CSPTS.

The supported Systems currently include the following:

· Air Carrier Analysis Support (ACAS)

· Advanced Computer Flight Plan (ACFP)

· Consolidated Air Mobility Planning System (CAMPS)

· Dynamic Mission Replanning (DMR)

· Global Air Transportation Execution System (GATES)

· Global Command and Control System (GCCS)

· Global Decision Support System (GDSS)

· Operational Risk Management (AvORM)

· Mobility Air Forces (MAF) C2 Enclaves and Enterprise Services

· Mobility Air Forces (MAF) Automated Flight-Planning Service (MAFPS)

· Mobility Enterprise Information Services Version 4.x.x (MEIS4)

· Trouble Ticket Tracking System (TTTS) (currently “Remedy”)

· USTRANSCOM Distributed Enclave

Specifically, the support services captured in the document encompass all workload associated with the Systems’ production environments unless specifically excluded by the government. The production environment is the software, hardware, and appliances, placed into operation for use by end users. It includes virtual machines and virtual infrastructure. “Production environments” in this context include exercise and training environments but do not include test or development environments (aka C2 Support Enclave). It also includes all sub-systems, modules, and applications within the Systems and all future modifications, modernizations and projects of the Systems.

PURPOSE

The purpose of this contract is to ensure, through the Contractor provided technical and staff support, the operational functionality and readiness of the systems, infrastructure, and services satisfying the COM-F mission needs of the Systems.

COM-F TASKS REQUIREMENTS

Contractor shall notify the Contracting Officer (CO) and the COR immediately of direction received from Government personnel the Contractor considers outside the scope of PWS requirements. The following functions encompass the administrative, documentation, and related functions, which provide support for the program. All tasks and subtasks shall be accomplished within the requirements described in this PWS, and the capabilities described in the Contractor’s Technical and Management Work Plan. The Technical and Management Work Plan will be used to document configuration control, quality control, document storage, and identify all key personnel.

As specific (sub) tasks and technical support arise a plan of action will be developed and agreed to during meetings as the Systems evolve, problems arise, and changes are made. All specific (sub) tasks and technical support will be recorded in Meeting Support Material and presented to the Contracting Officer Representative (COR).

Along with Government developed applications, the software and appliances currently under the purview of the Systems include:

· ArcServe

· Oracle Active Data Guard

· Oracle Applications Software

· Oracle Database Management Software

· Oracle Fusion Middleware Software

· Oracle Golden Gate

· Oracle HTTP Server Software

· Oracle Partitioning

· Oracle Real Application Clusters (RAC)

· Oracle Secure Backup Software

· Oracle Veridata

· Oracle Weblogic Software

· Microsoft Structured Query Language (SQL) Server

· Windows Operating Systems

· Microsoft Active Directory

· Red Hat JBoss Enterprise

· UNIX / Red Hat Enterprise Linux Operating Systems

· UNIX / Solaris Operating Systems

· VMWare

· NetIQ

· SharePoint

· Internet Information Server

· Apache Web Server

· CISCO

· F5

· Citrix

· Altiris

· Symantec & McAfee

· ESRI/ArcGIS

· MongoDB

· PostGres

· IBM ISAM

· IBM DataPower

· Talend

· Bonita

· DROOLS

· RabbitMQ

· ActiveMQ

· MashZone

· HazelCast

· F-Secure Corp. (WRQ Reflection)

· Juniper

· Oracle Audit Vault

· Splunk

· Veritas Netbackup

The support services are provided by individuals having the required expertise and experience on the supported systems. Relevant certifications in Oracle, UNIX, Cisco, Red Hat and Microsoft Operating Systems (OS) are required for personnel directly supporting these technologies in addition to the mandated DoD 8570 certification requirements. The services are provided based on unique system performance and availability requirements of each COM-F program.

PROGRAM MANAGEMENT

The Contractor shall provide administration and management oversight necessary in support of this contract. The Contractor shall designate a point of contact (POC) responsible for the overall management and performance of the PWS. The contractor designee shall be responsible for all requirements of the Contract. The Contractor shall provide the name and contact information of the designee to the COR within 10 business days after Contract award. Any change to this POC shall be submitted to COR within 5 business days prior to change.

Technical and Functional Support The Contractor shall provide the technical and functional activities needed for the program management and project integration/interoperability of this Contact. Contractor shall provide program administration support, to include documentation, Phase-in/Phase-out, Kick Off Meeting, all functional tasks listed throughout Section 2 of this PWS, Data Deliverables in Section 3, Services Summary in Section 4, and Supplemental Requirements in 5.

Contractor shall ensure all documents, reviews, inputs, suggested edits, and recommendations required to complete the following tasks are in alignment with related lateral and higher echelon instructions, guidance, and policies. Contractor shall ensure continuity of performance until completion. The Contractor shall ensure their staff is adequately trained and current on required skills, certifications, and qualifications prior to the beginning of performance and maintained throughout performance of the Contract.

Contract and Task Support Meetings

The Contractor shall support meetings, processes, and events in support of the COM-F Program and provide Meeting Support Material. The Contractor shall participate in working groups teams, which may be ad-hoc, and provide Meeting Support Material as reguided in support of the COM-F Program. The contractor will provide a Monthly Status Report and a meeting agenda in preparation of the Task Management Review (TMR) to COR for review and coordination of the TMR. At the conclusion of the TMR, the Contractor shall provide meeting minutes to the COR. The Contractor shall not conduct any external meetings, conferences, or presentations, whether planned or ad hoc, to discuss any requirements/policies, outside 375th CSPTS without first obtaining the permission of the COR. All Meeting Support Materials shall be submitted to COR for review prior to meetings, conferences, or presentations, and meeting minutes will be submitted to the COR after all meetings, conferences, or presentations.

Client Support Administrator (CSA)/Functional System Administrator (FSA)/Work-Center Environment Administration Support (EAS)

The Contractor shall provide EAS assistance in support of the contract environments in accordance with the 375 Communications Squadron Operating Instruction (currently in draft). The Contractor shall also consider guidance from DoD, AF, and AMC that pertains to and is directive in nature such as System Administration, Networking, and Information Technology (IT).

In order to ensure that Contractor personnel have adequate permissions to provide the EAS, the Contractor shall request access authorization in writing. This request shall contain an explanation of the need for the permissions, the equipment requiring the permissions, the type of permissions required, and the name of the person(s) to include his or her email address, work phone number, and evidence that the individual(s) has the requisite certification in accordance with DoD 8570.01-M, Information Assurance Workforce Improvement Program. The written request is to be delivered to the COR.

Asset Management The Contractor shall provide asset management for Government Furnished Property (GFP) by accomplishing inventory control of the hardware for the Systems located at Scott Air Force Base and Defense Enterprise Computing Center (DECC) St. Louis.

· Inventory control begins when equipment is deemed operational by the government. Once deemed operational, the contractor shall maintain accountability until removed from inventory.

· The contractor shall transfer hardware assets into and out of the production environments.

· The contractor will ensure use of a Standard Form 30 to execute transfers of GFP accountability between existing contracts.

· The contractor shall maintain the equipment inventory and accountability.

· Ensure all assets are tagged, labeled, or marked based on DoD marking standards, MIL Standards 130, when items are subject to serialized item management. All GFP subject to serialized management will be identified in the contract.

· Ensure removal of hardware from production environment, proper disposal and removal from the inventory of hardware assets

Metrics and Reporting Metrics and Reporting shall be established, maintained, and reported on each area of the Services Summary. In addition, the Contractor shall develop, maintain, and report metrics for the purpose of reporting system status, workload, problem resolution, system performance, and contract performance.

The Contractor shall submit a Customer Satisfaction Measurement Plan to the Contracting Officer (CO) The plan must demonstrate the Contractor’s ability to measure customer satisfaction in the performance of this PWS.

OPERATION AND MAINTENANCE (O&M) SUPPORT SERVICES

The Contractor shall provide the support services necessary to operate and maintain the Systems in accordance with the Government provided configuration baseline, operational performance and availability requirements (Appendix 1, System Availability Thresholds), and recommended OEM operation and maintenance procedures. The work described in this PWS shall be performed for all Systems with the exceptions provided in Appendix 2, Services Matrix.

Help Desk

In providing help desk support, the Contractor personnel shall possess working knowledge of the fundamental operations of relevant software, hardware, and networks for the Systems. The contractor personnel also must possess knowledge and experience of call tracking applications and training in customer service. The Contractor shall provide the following 24/7/365 help desk services for the Systems using the published Help Desk Procedures. The current published Help Desk Procedures will be provided to the contractor upon contract performance at which time contractor will maintain Help Desk Procedures.

· Follow help desk procedures relating to system status, reporting and administration

· Stay current with system information, changes and updates

· Update the Automated Call Distributing System (ACD) call routing menu to ensure calls are accurately routed to the appropriate location

· Administer, maintain, modernize and administer automated tools required to sustain help desk operations

· Respond to requests for technical assistance in person, via phone, electronically, and walk-in

· Research questions using available information resources

· Advise user on appropriate action

· Create, change, and delete user accounts per request. This subtask further includes:

· Password issuance & resets

· User authentication provisioning and support

· Establish and modify permissions

· Monitor and respond to requests for new accounts using customer requesting systems (e.g. trouble tickets tracking system, Account Management System (AMS), and GATES Security Service)

· Establish and maintain a log of accounts and account status

· Obtain account access approval when users request non-standard privileges

· Ensure account request documentation is complete and accurate in accordance with the respective program’s account management processes

· Monitor account usage in order to lock and delete accounts based on current security regulation and/or individual program guidance

· Redirect problems to appropriate resources

· Identify and escalate situations requiring urgent attention

· Ensure trouble tickets are documented in the Trouble Ticket Tracking System and protected according to classification level

· Track and route problems and requests and document resolutions

· Track and monitor all customer support requests and troubleshooting and resolution actions in the Trouble Ticket Tracking System

· Provide final trouble ticket close-out by identifying what the actual issue was and how it was it resolved in the Trouble Ticket Tracking System

· Notify appropriate authorities when a trouble ticket becomes Operations Reportable (OR) (e.g. System failures and other situations where System degradation and/or outage is observed or reported) in accordance with Systems’ Operations Reportable (OR) Criteria

· Validate and authenticate all user accounts to ensure only authorized users have account access annually or when required by an applicable Notice to Airman (NOTAM), or according to AMC/A6 guidance,.

System Administration

The Contractor shall provide system administration services consisting of system/servers provisioning, installation, configuration, operation, and maintenance of Systems hardware, software, and related infrastructure. These services consist of performing multiple, highly complex, technical tasks with routine upgrades in order to meet a changing production environment, applications, system designs, configurations, hardware, utilities, and operational conditions. The Contractor shall ensure the Systems hardware, OS, software systems, and related procedures adhere to government approved production configurations; government system availability and reliability standards; and OEM system operation and maintenance procedures. The Contractor shall perform 24/7/365 system administration services for the Systems.

The following is a listing of system administration functions that shall be performed by the Contractor:

· Perform system monitoring including monitor alarms; verify the integrity and availability of all hardware, interfaces, server (including virtual server) resources (e.g., CPU, memory, disk space utilization, disk I/O, etc.), systems, and key processes; review system and application logs; use system-provided dashboards/control panels to perform regular, in-depth monitoring, and verify and document completion of scheduled jobs (e.g. clean-up scripts, backups, and resets)

· Maintain configuration and awareness of virtual environment, including number and type of virtual servers, applications running on those servers, and activate, move, and close applications running on virtual servers.

· Maintain configuration and awareness of physical server environment, including number and type of servers, applications running on those servers, and activate, move and close applications running on physical servers.

· Perform regular security event monitoring to identify any possible intrusions

· Perform backup operations, ensuring all required file systems and system data are successfully backed up to the appropriate media, recovery tapes, disks, drives and utilities are created, and media is recycled and sent off site as necessary. Backup operations shall be verified using two person integrity as a control mechanism to verify successful backups for operation data.

· Perform regular file archival and purging as necessary using the guidance provided by the Systems

· Repair and recover from hardware or software failures, elevating problems to Level 3 support in a timely-manner to avoid further system issues

· Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate Authorized Service Interruption (ASI) scheduling

· Lead troubleshooting teleconferences with users and Level 3 support to resolve open system issues as documented in the Trouble Ticket Tracking System

· Notify appropriate authorities when a trouble ticket becomes Operations Reportable (OR) (e.g. System failures and other situations where System degradation and/or outage is observed or reported) in accordance with Systems’ Operations Reportable (OR) Criteria

· Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply Time Compliance Network Order (TCNO), Information Assurance Vulnerability Management(IAVM), and other downward directed security patches

· Review scans following patch/upgrade application to ensure successful and open trouble tickets for any system security findings identified during the scan

· Configure / add new services as necessary as directed by the systems

· Upgrade and configure System software that supports infrastructure applications according to project or operational needs including new software releases, hotfixes, patches, to production, exercise suites and GDSS training site

· Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the Trouble Ticket Tracking System

· Perform and/or support software and hardware fielding in accordance with software version document releasing instructions provided by AISS or the System developer

· Maintain all procedures associated with sustaining the production environment and document them in Contractor Maintained Procedures

· Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures

· Monitor, maintain, and ensure currency of server certificates

· Provide a monthly report of all server certificates, to include server name, certificate expiration date, and actions needed to obtain new certificates

· Provide performance reporting to support capacity planning

· Perform ongoing performance tuning, hardware upgrades, defragmenting of hard drives, and resource optimization as required

· Configure CPU, memory, virtual servers and images, and disk partitions as required following instructions provided by the systems

· Maintain the Systems environmental and monitoring equipment

· Solve technical issues with all hardware associated with systems’ operation in the production environment including but not limited to: web server, database server, applications server, virtual server, Domain Name Service (DNS), Simple Mail Transfer Protocol (SMTP), appliances, utilities, or other System hardware and software

· Propose software or hardware updates and other systems enhancements to improve systems performance and reliability in the production environment; schedule and coordinate level-appropriate maintenance

· Troubleshoot and resolve any systems service and OS-level issues in the production environment including issues referred by system managers, program offices, the help desk, database managers, network engineers, systems administrators, and other technicians

· Provide advanced troubleshooting skills and the ability to rapidly identify issues and provide restoration of issues

· Keep detailed records/logs of work activities/system performance to support Metrics and Reporting

· Participate in fault isolation, detection, and correction relative to the Systems at host sites and interfaces for both US or oversees locations

· Interpret System practices, diagrams, specifications, drawings, and service orders

· Prepare system inputs, analyze system outputs, and maintain and repair IT systems

· Install, maintain, upgrade, modify, isolate and repair System servers, workstations, thin clients, peripherals and system software/hardware

· Coordinate and implement software code upgrades

· Coordinate and implement installation of new equipment and products. For the purpose of this document, “products” refers to any COTS or GOTS software in support of the Systems.

· Collate and document the necessary information (problem, troubleshooting steps taken, stakeholders coordinated with etc.) for creating requests for higher level technical support

· Ensure full data and system recovery to operational status upon completion of maintenance and/or recovery actions. Status shall be confirmed using two person integrity as a control mechanism.

· Execute after action notification procedures describing maintenance actions to the MAJCOM Communications Coordination Center (MCCC) for ASI and OR events

· Monitor user load balance, for Systems with load balancing capability, and adjust and direct users to alternate sites/enclaves

· Maintain clean servers, related equipment, and Systems areas in accordance with established directives

· Create and maintain scripts for task automation

· Assist in the operational validation of application updates, enhancements and changes for the purpose of assisting in ensuring that assigned applications and related systems function appropriately and meets configuration baseline and database software requirements

· Track and monitor all customer support requests and troubleshooting and resolution actions in the Trouble Ticket Tracking System

· Ensure trouble tickets are documented in the Trouble Ticket Tracking System and protected according to classification level

· Provide final trouble ticket close-out by identifying what the actual issue was and how it was it resolved in the Trouble Ticket Tracking System

· Fully understand the procedures for moving each system operations from one enclave to another enclave and practice the move regularly

· Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems supported in this contract into a Common Computing Environment (CCE) and DOD or commercial cloud environment (such as Infrastructure as a Service (IaaS) and Amazon GovCloud)

Network Administration

The Contractor shall provide network administration services consisting of network/appliance provisioning, installation, configuration, security, operation, and maintenance of Systems hardware, software, and related infrastructure. This includes all related local area and wide area networks or segments, data communications, and associated equipment, software, and appliances within the applicable production environment(s). These services consist of performing multiple, highly complex, technical tasks with a need to routinely upgrade skills in order to meet a changing production environment, applications, system designs, configurations, hardware, utilities, and operational conditions. The Contractor shall ensure the Systems hardware, OS, software systems, and related procedures adhere to government approved production configurations; government system availability and reliability standards; and OEM system operation and maintenance procedures. The Contractor shall perform 24/7/365 network administration services for the Systems.

The following is a listing of system and network administration functions that shall be performed by the Contractor:

· Perform network monitoring including monitoring alarms; verify the integrity and availability of all hardware, appliances, resources, and systems

· Perform regular security event monitoring to identify any possible intrusions

· Repair and recover from hardware or software failures, elevating problems to Level 3 support in a timely-manner to avoid further system issues

· Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate Authorized Service Interruption (ASI) scheduling

· Lead troubleshooting teleconferences with users, and Level 3 support (AISS Engineers), and Level 3 development to resolve open system issues as documented in the Trouble Ticket Tracking System

· Notify appropriate authorities when a trouble ticket becomes Operations Reportable (OR) (e.g. System failures and other situations where System degradation and/or outage is observed or reported) in accordance with Systems’ Operations Reportable (OR) Criteria

· Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply Time Compliance Network Order (TCNO), Information Assurance Vulnerability Management (IAVM), and other downward directed security patches

· Upgrade and configure network software that supports infrastructure applications according to project or operational needs

· Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the Trouble Ticket Tracking System

· Perform and/or support software and hardware fielding in accordance with software version document releasing instructions provided by AISS or the System developer

· Maintain all procedures associated with sustaining the production environment and document them in Contractor Maintained Procedures

· Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures

· Monitor, maintain, and ensure currency of network device/appliance certificates

· Provide performance reporting to support capacity planning

· Perform ongoing performance tuning, hardware upgrades, and resource optimization as required

· Maintain the network environmental and monitoring equipment

· Solve technical issues with network equipment, appliances, utilities, or other network hardware and software

· Propose software or hardware updates and other network enhancements to improve systems performance and reliability; schedule and coordinate level-appropriate maintenance

· Troubleshoot and resolve network service issues, including issues referred by system managers, program offices, the help desk, database managers, network engineers, systems administrators, and other technicians

· Provide advanced troubleshooting skills and the ability to rapidly identify issues and provide restoration of issues

· Provide rapid restoration of network anomalies including assistance with disaster recovery of the network

· Identify, test, diagnose, and analyze troubles in the network

· Keep detailed records/logs of work activities/system performance to support Metrics and Reporting

· Initiate, process, and track requests for firewall configuration changes and firewall configuration waivers when a customer with established System accounts requires network access and assistance,

· Troubleshoot end-to-end network connectivity issues affecting the operations and performance of systems

· Operate and maintain program specific Virtual Private Networks (VPN) and firewalls and other network appliances

· When end-to-end network connectivity and network access issues arise relative to the Systems at host sites, interfaces, or at other locations in the US or overseas, participate in the fault isolation, detection, and correction; coordinate repair efforts; and track status

· Interpret network practices, diagrams, specifications, drawings, and service orders

· Prepare system inputs, analyze system outputs, and maintain and repair various electronic systems

· Install, maintain, upgrade, modify, isolate and repair network devices, appliances, and system software/hardware

· Provide support for the management of the network and performance of duties related to the operation of the network, including change management and problem resolution

· Coordinate and implement software code upgrades in all network equipment

· Coordinate and implement installation of new equipment and products in the network

· Coordinate and implement the removal of network equipment being displaced or replaced

· Coordinate all maintenance actions and schedule System outages with appropriate System Manager

· Collate and document the necessary information (problem, troubleshooting steps taken, stakeholders coordinated with etc.)for creating requests for higher level technical support

· Ensure full availability and security to operational status upon completion of maintenance and/or recovery actions. Status shall be confirmed using two person integrity as a control mechanism.

· Execute after action notification procedures describing maintenance actions to the MCCC for ASI and OR events

· Monitor user load balance for Systems with load balancing capability, and adjust and direct users to alternate sites/enclaves

· Maintain clean network devices, appliances, and Systems areas in accordance with established directives

· Track and monitor all customer support requests and troubleshooting and resolution actions in the Trouble Ticket Tracking System

· Ensure trouble tickets are documented in the Trouble Ticket Tracking System and protected according to classification level

· Provide final trouble ticket close-out by identifying what the actual issue was and how it was resolved in the Trouble Ticket Tracking System

· Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems supported in this contract into a Common Computing Environment (CCE) and DOD or commercial cloud environment (such as Infrastructure as a Service (IaaS) and Amazon GovCloud)

Database Administration

The Contractor shall provide database administration services within the applicable production environment(s).These services consist of performing multiple, highly complex, technical tasks with a need to routinely upgrade skills in order to meet changing database applications, system designs, and production environment conditions. Specific skill-based competencies are required to satisfactorily perform this task, which are the interaction and data transfer between hosts and clients of multiple hardware configurations, preparing and maintaining accurate records, and utilizing the functionality of pertinent software applications. Due to the rapidly evolving nature of IT, the Contractor shall keep pace with system and operational environment changes by ensuring it has the appropriate skill set to effectively perform the database administration task without delay and/or system degradation. The Contractor shall perform 24/7/365 database administration services for the Systems.

The following is a listing of database administration functions that shall be performed by the Contractor:

· Troubleshoot and resolve database system malfunctions to resolve issues by detecting and isolating faults, and taking corrective action to restore to fully operational status

· Upgrade databases for the purpose of ensuring their ability to use new and/or existing databases and related software

· Maintain all procedures associated with sustaining the production environment and document them in Contractor Maintained Procedures

· Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures

· Ensure all issues and outages are properly documented in the Trouble Ticket Tracking System, systems change request repository, configuration management systems, or other government directed location

· Analyze and sustain capacity and performance requirements

· Analyze, consolidate, and tune databases for optimal efficiency

· Monitor systems and platforms for availability and database systems serviceability

· Back-up, cluster, mirror, replicate, and failover assigned databases

· Restore and recover corrupted databases

· Ensure database systems are configured to meet security directives

· Build database servers, install, and configure database software on servers and stations

· Perform database dumps, table maintenance, re-indexing, and database replication maintenance in accordance with industry standards and/or the applicable database maintenance guidance

· Collate the necessary information for creating requests for higher level technical support

· Install patches for applicable database software

· Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply Time Compliance Network Order (TCNO), Information Assurance Vulnerability Management (IAVM), and other downward directed security patches

· Execute and monitor regular and ad hoc job execution in databases

· Consult with software application developers on preparing SQL queries and effective use of features and options of the applicable database software

· Consult with engineers and application developers on existing and potential performance parameters, to include identifying and resolving performance issues

· Resolve problems and meet user needs and system operational and functional requirements

· Assist in the operational validation of application updates, enhancements and changes for the purpose of assisting in ensuring that assigned applications and related systems function appropriately and meets configuration baseline and database software requirements

· Coordinate database activities (e.g. security, upgrading, populating, refreshing, repairing database systems, cataloging, access rights, etc.) for the purpose of ensuring data accuracy and availability

· Monitor compliance with DoD and US Air Force database security and integrity,

· Monitor compliance Personally Identifiable Information (PII) protection standards and procedures IAW DoD Directive 5400.11, Privacy Act Information. Contractor Rule of Conduct is explained in Enclosure 2.

· Using a variety of database support processes, ensure the operation, stability and optimal performance of the production databases and the availability of stored data

· Monitor assigned applications and related systems for the purpose of reducing application downtime and ensuring that assigned applications are available when needed to meet system performance and availability parameters

· Enforce database schemas, tables, procedures, and permissions

· Set up data sharing and disk partitioning according to configuration

· Create and maintain scripts for task automation

· Update operationally required data imports from approved sources and data extracts required to support operational needs, e.g., Digital Aeronautical Flight Information File (DAFIF) and weather data

· Create, delete, and modify database/privileged user accounts and access privileges

· Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the Trouble Ticket Tracking System

· Lead system troubleshooting teleconferences with users, and Level 3 support (AISS Engineers), and Level 3 development to resolve open system issues as documented in the Trouble Ticket Tracking System

· Notify appropriate authorities when a trouble ticket becomes Operations Reportable (OR) (e.g. System failures and other situations where System degradation and/or outage is observed or reported) in accordance with Systems’ Operations Reportable (OR) Criteria

· Monitor all customer support requests relating to database management, once assigned actively troubleshoot and resolve open tickets and report cause and fix action in the Trouble Ticket Tracking System

· Ensure trouble tickets are documented in the Trouble Ticket Tracking System and protected according to classification level

· Ensure final closed trouble ticket accurately reflects what the actual issue was and how it was it resolved

· Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems supported in this contract into a Common Computing Environment (CCE) and DOD or commercial cloud environment (such as Infrastructure as a Service (IaaS) and Amazon GovCloud)

· Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate Authorized Service Interruption (ASI) scheduling Notify appropriate authorities when a trouble ticket becomes Operations Reportable (OR) (e.g. System failures and other situations where System degradation and/or outage is observed or reported) in accordance with Systems’ Operations Reportable (OR) Criteria

· Execute after action notification procedures describing maintenance actions to the MCCC for ASI and OR events

Web Site/Page Support

The Contractor shall provide Web page support by accomplishing the following:

• Create and update Systems’ informational/support web sites/pages. In accomplishing this subtask the Contractor shall:

· Receive requests for Web site/page updates and edit the web page in accordance with the updated request

· Ensure website/page updates meet the requirements of HQ AMC, AF, and DoD web page standards including AFI 33-129, Web Management and Internet Use, and AFI 33-364_AFGM2017-01, Records Disposition – Procedures and Responsibilities

· Post, or forward for posting, and verify updated web page was posted correctly

Configuration Management (CM)

The Contractor shall provide robust CM for the production environment for the Systems by adhering to Government standards and business best practices. To ensure the CM proactively controls system and software configurations the Contractor shall:

· Ensure CM has the latest, and previous baselines as directed by the PMO, configuration baselines of all hardware, software and documents as provided by the Government.

· Provide, manage, maintain, and implement a production baseline Configuration Management Plan (CMP). The CMP shall address management and processes to ensure control of the system configurations and provide accurate tracking and reporting of all baseline hardware, software, virtual images, documentation, and configurations (including virtual images)

· Ensure configuration control is implemented for all items identified in the various Systems’ production configuration baselines and production virtual images. Note: Configuration control of production virtual environments will be much more dynamic as virtual images are created and taken down in response to business demand, outages, and other factors.

· Establish a system for identifying and reporting items that are incorrectly identified in the configuration baseline or missing from the production configuration documents

· Maintain database for tracking all configuration items (CIs) for each of the supported environments. Develop, implement, and maintain plans, processes, procedures, and tools required to provide CM for the Systems production baseline, including the virtual environment, as part of the CM Plan

· Review software and hardware provided for operations and maintenance to ensure all necessary items/documentation are received from the sources provided by the Government, notify the COR of receipt, and annotate any discrepancies

· Establish and maintain a Configuration Management Library (CML) of the current production environment software, hardware, and virtual configuration, technical drawings, and documentation for the baselines operated and maintained by the Contractor

· Maintain database for tracking all configuration items (CIs) for each of the supported environments, ensuring the status of all production baseline CIs are kept current

· Verify that the software loaded on supported systems is the correct version, report noncompliance, and provide status to the COR

· Perform physical configuration audits and take positive, proactive actions to ensure correction of any discrepancies noted

· Support tracking, coordination, and reporting on TCNO, IAVM, and other downward directed security patches

DATA DELIVERABLES

The following items are outputs which fall within the scope of this Contract and are illustrative of the type of work the Government requires. Deliverables required by the Government in accordance with this PWS shall have a Data Item Description (DID). All deliverables shall be reviewed for timeliness, accuracy and format. If a deliverable due date falls on a weekend or holiday, the Contractor shall submit the deliverable on the last work day prior to the due date. In addition, unscheduled project updates shall be provided by Government/COR. Table 1, Data Deliverables, provides a list of the required PWS deliverables.

All products and documentation delivered under this PWS become the property of the Government.

Table 1: Data Deliverables The following deliverables shall be provided IAW Exhibit A-AD:

Monthly Status Report

SERVICES SUMMARY (SS)

The Services Summary table summarizes the most important performance objectives and performance thresholds (specific standards) as identified within the body of the PWS. The absence of any performance objective and threshold from this SS shall not detract from its enforceability nor limit the rights or remedies of the Government under any provision of the Contract The Government Program Manager/COR shall evaluate the Contractor on each SS performance objective on a monthly basis and provide a copy of the evaluation to the Contracting Officer

2.2 et seq, 6.2.6 No more than one valid documented complaint (via e-mail or memo) from the supported program management offices or system managers per System per month in performance of required tasks

Systems are operated and maintained in…