Attch_4_Draft_QASP_COM-F_III_13_Mar_19.docx

FA4452-15-C-0001

(andichm1640)

COM-F III IDIQ PWS

FA4452-18-R-0037

Attachment 1

Communications, Operations, & Maintenance Functions (COM-F) III

IDIQ Performance Work Statement (PWS)

Procuring Office: 763d Specialized Contracting Squadron (763 SCONS)

Requiring Office: HQ AMC Directorate of Operations and Training Division (A6I) 203 West Losey Street Room 3610 Scott Air Force Base, IL 62225

Project Management Office: 375th Communication Support Squadron (CSPTS) 859 Buchannan Street Scott Air Force Base, IL 62225

Original Date 13 Mar 2019 Revision # Table of Contents

2.1.5 Metrics and Report………………………………………………………………11

INTRODUCTION

BACKGROUND

The Air Mobility Command (AMC) is responsible to the National Command Authority (NCA) through the Joint Chiefs of Staff (JCS) and the United States Transportation Command (USTRANSCOM) for the planning and execution of global airlift and aerial refueling missions during peace, crisis, and war. Therefore, AMC is also responsible for the coordination and exercise of airlift and air refueling strategy, doctrine, and plans under the direction of USTRANSCOM. Crisis and wartime support are provided to the unified and specified commands. As an Air Force (AF) Major Command (MAJCOM), AMC is responsible to the Secretary of the Air Force and the Chief of Staff, Headquarters (HQ) United States Air Force (USAF), for the accomplishment of peacetime airlift and air refueling services in a manner that will promote the emergency and wartime capability of assigned forces and ensure the support of all Department of Defense (DoD) components. In addition, AMC is also responsible for aeromedical evacuation operation.

AMC supports the Mobility Air Force mission--"Global Reach." It is difficult, if not impossible, for United States military forces to deploy rapidly or be supported effectively during employment without the assets and expertise provided by AMC. AMC Command and Control (C2), In-Transit Visibility (ITV), and Business Systems (hereafter referred to as “Systems”) processes, procedures, and services must be consistent and reliable in peacetime, during contingencies, and throughout the full spectrum of war. Information technology (IT) resources enables the efficient and effective management of this capability to provide the highest levels of availability, integrity and confidentiality.

The 375th Communication Support Squadron (CSPTS) provides command and control, communications, and computer (C4) support to DoD, USTRANSCOM, AMC, and worldwide aerial and water ports. They also operates and maintains the Systems supporting the 618th Air & Space Operations Center’s (618th AOC) global mission. The 375th CSPTS, augmented by the COM-F contractor is AMC’s answer to providing Communications, Operations, and Maintenance Functions (COM-F) services, is AMC’s answer to meeting the challenges of providing operational support and maintenance of the Systems in a timely, effective, and efficient manner.

The Systems are in a constant state of change as hardware is replaced/upgraded, software versions and patches are fielded, and the entire computing and network environments evolve from a thick/thin client, system centric client server environment to an Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) construct. AMC has begun implementing hardware and software in the enclave environment to support virtualized operations. This process will continue as systems and applications capable of operating in virtualized environments are fielded. The entire operation will evolve over time from a system-centric perspective to an enterprise perspective, with a large proportion of the workload shifting to managing the virtual environment. In addition, all applications are scheduled to be migrated to the USTRANSCOMs instance in the Cloud. The future products for support in the modernization and cloud remain unknown at this time.

SCOPE

The contractor shall provide the resources to meet the performance work statement (PWS), the Systems Availability Thresholds (Appendixes Attachments), systems Original Equipment Manufacturer (OEM) maintenance standards, and the System operational reporting criteria.

This effort includes systems for which operations and maintenance (O&M) support is provided by 375th CSPTS. The contractor will be responsible for all aspects of operations and maintenance of the AMC Command and Control Operational Environment (C2OE) and the Systems that reside in the C2OE. The installation of all equipment and software as well as removal and decommission of hardware is also the COM-F contractor’s responsibility.

Specifically, the support services captured in the document encompass all workload associated with the Systems’ production environments unless specifically excluded by the Government. The production environment is the software, hardware, and appliances, placed into operation for use by end users. It includes virtual machines and virtual infrastructure in both C2OE and Cloud environments. “Production environments” in this context include exercise and training environments but does not include test or development environments (aka C2 Support Enclave C2SE)). It also includes all subsystems, modules, and applications within the Systems and all future modifications, modernizations and projects of the Systems.

PURPOSE

The contractor shall provide technical and staff support, the operational functionality and readiness of the systems, infrastructure, and services satisfying the COM-F mission needs of the Systems.

COM-F TASK REQUIREMENTS

Contractor shall notify the Contracting Officer (CO) and the Contracting Officer Representative (COR) immediately of direction received from Government personnel the contractor considers outside the scope of PWS requirements. The following functions encompass the administrative, documentation, and related functions, which provide support for the program. All tasks and subtasks shall be accomplished within the requirements described in this PWS, and the capabilities described in the contractor’s Technical and Management Work Plan. The Technical and Management Work Plan will be used to document configuration control, quality control, and storage, identify all key personnel, and the customer service satisfaction measurement process. The customer service satisfaction measurement process must identify the contractor’s processes and frequency used to measure customer the performance of this PWS. The contractor shall provide a Monthly Status Report.

As specific subtasks and technical support arise a plan of action will be developed and agreed to during meetings as the Systems evolve, problems arise, and changes are made. All specific subtasks and technical support will be recorded in Meeting Support Material and presented to the COR.

Along with Government developed applications, the software and appliances currently under the purview of the Systems include:

· ArcServe

· Oracle Active Data Guard

· Oracle Applications Software

· Oracle Database Management Software

· Oracle Fusion Middleware Software

· Oracle Golden Gate

· Oracle HTTP Server Software

· Oracle Partitioning

· Oracle Real Application Clusters (RAC)

· Oracle Secure Backup Software

· Oracle Veridata

· Oracle Weblogic Software

· Microsoft Structured Query Language (SQL) Server

· Windows Operating Systems

· Microsoft Active Directory

· Red Hat JBoss Enterprise

· UNIX/Red Hat Enterprise Linux Operating Systems

· UNIX/Solaris Operating Systems

· VMWare

· NetIQ

· SharePoint

· Internet Information Server

· Apache Web Server

· CISCO

· F5

· Citrix

· Altiris

· Symantec & McAfee

· ESRI/ArcGIS

· MongoDB

· PostGres

· IBM ISAM

· IBM DataPower

· Talend

· Bonita

· DROOLS

· RabbitMQ

· ActiveMQ

· MashZone

· HazelCast

· F-Secure Corp. (WRQ Reflection)

· Juniper

· Oracle Audit Vault

· Splunk

· Veritas Netbackup

Relevant certifications in Oracle, UNIX, Cisco, Red Hat and Microsoft Operating Systems (OS) are required for personnel directly supporting these technologies in addition to the mandated DoD 8570.01 Information Assurance Workforce Improvement Program certification requirements. The services are provided based on unique system performance and availability requirements of each COM-F program.

PROGRAM MANAGEMENT

The contractor shall provide administration and management oversight. The contractor shall designate a point of contact (POC) responsible for the overall management and performance of the PWS. The contractor designee shall be responsible for all requirements of the contract. The contractor shall provide the name and contact information of the designee to the COR within 10 business days after contract award. Any change to this POC shall be submitted to COR within five business days prior to change.

Technical and Functional Support The contractor shall provide the technical and functional activities needed for the program management and project integration/interoperability. The contractor shall provide program administration support, to include documentation, Phase-in/Phase-out, Kick Off Meeting, all functional tasks listed throughout Section 2 of this PWS, Data Deliverables in Section 3, Services Summary in Section 4, and Other Pertinent Information or Other Considerations in 5. Contractor shall ensure all documents, reviews, inputs, suggested edits, and recommendations required to complete the following tasks are in alignment with related lateral and higher echelon instructions, guidance, and policies. Contractor shall ensure continuity of performance until completion. The contractor shall ensure their staff is adequately trained and current on required skills, certifications, and qualifications prior to the performance start date and maintained throughout contract performance.

Contract and Task Support Meetings

The contractor shall support meetings, processes, and events and provide Meeting Support Material. The contractor shall participate in working groups/teams, which may be ad-hoc, and provide Meeting Support Material. The contractor will provide a Monthly Status Report and a meeting agenda in preparation of the Task Management Review (TMR) to COR for review and coordination of the TMR. At the conclusion of the TMR, the contractor shall provide meeting minutes to the COR. The contractor shall not conduct any external meetings, conferences, or presentations, whether planned or ad hoc, to discuss any requirements/policies, outside 375th CSPTS without first obtaining the permission of the COR. All Meeting Support Materials shall be submitted to COR for review prior to meetings, conferences, or presentations, and meeting minutes will be submitted to the COR after all meetings, conferences, or presentations.

Client Support Administrator (CSA)/Functional System Administrator (FSA)/Work-Center Environment Administration Support (EAS)

The contractor shall provide EAS assistance in accordance with (IAW) the 375 Communications Squadron Operating Instruction. The contractor shall also consider guidance from DoD, AF, and AMC that pertains to and is directive in nature such as System Administration, Networking, and IT.

In order to ensure that contractor personnel have adequate permissions to provide the EAS, the contractor shall request access authorization in writing. This request shall contain an explanation of the need for the permissions, the equipment requiring the permissions, the type of permissions required, and the name of the person(s) to include his or her email address, work phone number, and evidence that the individual(s) has the requisite certification IAW DoD 8570.01-M, Information Assurance Workforce Improvement Program. The written request is to be delivered to the COR.

Asset Management The contractor shall provide asset management for Government Furnished Property (GFP) by accomplishing inventory control of the hardware for the Systems located at Scott Air Force Base and Defense Enterprise Computing Center (DECC) St. Louis. The contractor shall provide Inventory control, which begins when the Government deems equipment operational. Once deemed operational, the contractor shall maintain accountability until removed from inventory.

· The contractor shall transfer hardware assets into and out of the production environments.

· The contractor shall ensure use of a Standard Form 30 to execute transfers of GFP accountability between existing contracts.

· The contractor shall properly sanitize, uninstall and turn in decommissioned equipment in the operational environment (C2OE) to Defense Logistics Agency Disposition Services (DLADS) per the government’s request and IAW AFMAN 17-1203 Information Technology Asset Management (ITAM).

· The contractor shall maintain the Equipment Inventory Record and accountability.

· The contractor shall ensure all assets are tagged, labeled, or marked based on DoD marking standards, MIL Standards 130, when items are subject to serialized item management. All GFP subject to serialized management will be identified in the contract.

· The contractor shall ensure removal of hardware from production environment, proper disposal and removal from the inventory of hardware assets.

Metrics and Reporting Metrics and Reporting shall be established, maintained, and reported on each area of the Services Summary and reported in the Monthly Status Report. In addition, the contractor shall develop, maintain, and report metrics for the purpose of system status, workload, problem resolution, system performance, and contract performance in the Monthly Status Report.

O&M SUPPORT SERVICES

The contractor shall provide the support services necessary to operate and maintain the Systems IAW the Government provided configuration baseline, operational performance and availability requirements (Appendixes, System Availability Thresholds and Services Matrix), and recommended OEM operation and maintenance procedures. The work described in this PWS shall be performed for all Systems with the exceptions provided in, Services Matrix.

Help Desk

Help Desk support, shall possess working knowledge of the fundamental operations of relevant software, hardware, and networks for the Systems. Knowledge and experience of call tracking applications and training in customer service. Help Desk service shall be 24/7/365 unless a reduced requirement is identified in a TO. The current Help Desk Procedures will be provided to the contractor who will maintain and document all changes in the Contractor Maintained Procedures. The contractor shall ensure that help desk personnel have the mandated DoD 8570.01 Information Assurance Workforce Improvement Program IAT Level I certification. Help Desk personnel are considered Mission Essential as defined in section 5.10. The contractor shall:

2.2.1.1 Respond to requests for technical assistance in person, via phone, electronically, and walk-in.

2.2.1.2 Follow Help Desk procedures relating to system status, reporting and administration.

2.2.1.3 Update the Automated Call Distributing System (ACD) call routing menu to ensure calls are accurately routed to the appropriate location.

2.2.1.4 Create, change, and delete user accounts. This subtask further includes:

· Password issuance & resets.

· User authentication provisioning and support.

· Establish and modify permissions.

· Monitor and respond to requests for new accounts using customer requesting systems (e.g. Trouble Ticket Tracking System (TTTS), Account Management System (AMS), and GATES Security Service).

· Establish and maintain a log of accounts and account status.

· Obtain account access approval when users request non-standard privileges.

· Ensure account request documentation is complete and accurate IAW the respective program’s account management processes.

· Monitor account usage in order to lock and delete accounts based on current security regulation and/or individual program guidance.

2.2.1.5 Validate and authenticate all user accounts to ensure only authorized users have account access annually or when required by an applicable Notice to Airman (NOTAM), or according to AMC/A6 guidance.

2.2.1.6 Advise user on appropriate action.

· Redirect problems to appropriate resources.

· Identify and escalate situations requiring urgent attention.

· Ensure trouble tickets are documented in the TTTS and protected according to classification level.

· Track and route problems, requests, and document resolutions.

· Track and monitor all customer support requests and troubleshooting and resolution actions in the TTTS.

· Provide final trouble ticket closeout by identifying what the actual issue was and how it was it resolved in the TTTS.

· Notify appropriate authorities when a trouble ticket becomes OR (e.g. System failures and other situations where System degradation and/or outage is observed or reported) IAW Systems’ OR Criteria.

2.2.1.7 Research questions using available Information Resources.

2.2.1.8 Stay current with system information, changes and updates.

2.2.1.9 Administer, maintain, and modernize automated tools required to sustain Help Desk operations.

2.2.1.10 Monitor Systems’ operations and respond to faults, alarms, abnormalities, or potential issues.

2.2.1.11 Diagnose and resolve technical hardware and software issues.

2.2.1.12 Administer Help Desk software.

2.2.1.13 Redirect problems to appropriate resource.

2.2.1.14 Identify and escalate situations requiring urgent attention.

2.2.1.15 Track and monitor all customer support requests, troubleshooting and resolution actions.

2.2.1.16 Ensure trouble tickets are documented in the TTTS and protected according to classification level and authenticate all user accounts to ensure only authorized users have account access annually or when required by an applicable Notice to Airman (NOTAM), or according to AMC/A6 guidance.

2.2.1.17 Document and provide the status of trouble tickets when requested by Government entities.

2.2.1.18 Track and route problems, requests, and document resolutions.

2.2.1.19 Prepare activity reports.

System Administration

The contractor shall provide system administration services consisting of system/servers provisioning, installation, configuration, operation, and maintenance of Systems hardware, software, and related infrastructure. These services consist of performing multiple, highly complex, technical tasks with routine upgrades in order to meet a changing production environment, applications, system designs, configurations, hardware, utilities, and operational conditions. The contractor shall ensure the Systems hardware, OS, software systems, and related procedures adhere to Government approved production configurations; Government system availability and reliability standards; and OEM system operation and maintenance procedures. The contractor shall document all procedures in the Contractor Maintained Procedures. The contractor shall perform 24/7/365 system administration services for the Systems unless the TO states less hours are required. The contractor shall ensure that system administration personnel have relevant industry standard certifications in addition to the mandated DoD 8570.01 Information Assurance Workforce Improvement Program IAT Level II certification. System Administration personnel are considered Mission Essential as defined in section 5.10.

The following is a listing of system administration functions that shall be performed by the contractor:

2.2.2.1 Perform system monitoring including monitor alarms; verify the integrity and availability of all hardware, interfaces, server (including virtual server) resources (e.g., Central Processing Unit (CPU), memory, disk space utilization, disk I/O, etc.), systems, and key processes; review system and application logs; use system-provided dashboards/control panels to perform regular, in-depth monitoring, and verify and document completion of scheduled jobs (e.g. clean-up scripts, backups, and resets.)

2.2.2.2 Maintain configuration and awareness of virtual environment, including number and type of virtual servers, applications running on those servers, and activate, move, and close applications running on virtual servers.

2.2.2.3 Maintain configuration and awareness of physical server environment, including number and type of servers, applications running on those servers, and activate, move and close applications running on physical servers.

2.2.2.4 Perform regular security event monitoring to identify any possible intrusions.

2.2.2.5 Perform backup operations, ensuring all required file systems and system data are successfully backed up to the appropriate media, recovery tapes, disks, drives and utilities are created, and media is recycled and sent off site as necessary. Backup operations shall be verified using two-person integrity as a control mechanism to verify successful backups for operation data.

2.2.2.6 Perform regular file archival and purging as necessary using the guidance provided by the Systems.

2.2.2.7 Repair and recover from hardware or software failures, elevating problems to Level 3 support in a timely-manner to avoid further system issues.

2.2.2.8 Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate Authorized Service Interruption (ASI) scheduling.

2.2.2.9 Lead troubleshooting teleconferences with users, Level 3 support, and Level 3 development organizations to resolve open system issues as documented in the TTTS.

2.2.2.10 Notify appropriate authorities when a trouble ticket becomes OR (e.g. System failures and other situations where System degradation and/or outage is observed or reported) IAW Systems’ OR Criteria.

2.2.2.11 Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply Time Compliance Network Order (TCNO), Information Assurance Vulnerability Management (IAVM), and other downward directed security patches.

2.2.2.12 Review scans following patch/upgrade application to ensure successful and open trouble tickets for any system security findings identified during the scan.

2.2.2.13 Configure/add new services as necessary as directed by the systems.

2.2.2.14 Upgrade and configure System software that supports infrastructure applications according to project or operational needs including new software releases, hotfixes, and patches, to production, exercise suites and GDSS training site.

2.2.2.15 Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the TTTS.

2.2.2.16 Perform software and hardware fielding IAW software version document releasing instructions provided by Level 3 support and Level 3 development organizations.

2.2.2.17 Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures and document them in a Contractor Maintained Procedures.

2.2.2.18 Monitor, maintain, and ensure currency of server certificates.

2.2.2.19 Provide a monthly report of all server certificates, to include server name, certificate expiration date, and actions needed to obtain new certificates.

2.2.2.20 Provide performance reporting to support capacity planning.

2.2.2.21 Perform ongoing performance tuning, hardware upgrades, defragmenting of hard drives, and resource optimization as required.

2.2.2.22 Configure CPU, memory, virtual servers and images, and disk partitions as required following instructions provided by the systems.

2.2.2.23 Maintain the Systems environmental and monitoring equipment.

2.2.2.24 Solve technical issues with all hardware associated with systems’ operation in the production environment including but not limited to: web server, database server, applications server, virtual server, Domain Name Service (DNS), Simple Mail Transfer Protocol (SMTP), appliances, utilities, or other System hardware and software.

2.2.2.25 Propose software or hardware updates and other systems enhancements to improve systems performance and reliability in the production environment; schedule and coordinate level-appropriate maintenance.

2.2.2.26 Troubleshoot and resolve any systems service and OS-level issues in the production environment including issues referred by system managers, program offices, the Help Desk, database managers, network engineers, systems administrators, and other technicians.

2.2.2.27 Provide advanced troubleshooting skills and the ability to rapidly identify issues and provide restoration of issues.

2.2.2.28 Keep detailed records/logs of work activities/system performance to support metrics and reporting and include in Monthly Status Report.

2.2.2.29 Participate in fault isolation, detection, and correction relative to the Systems at host sites and interfaces for both US or oversees locations.

2.2.2.30 Log all user access requests in the TTTS.

2.2.2.31 Interpret System practices, diagrams, specifications, drawings, and service orders.

2.2.2.32 Prepare system inputs, analyze system outputs, and maintain and repair IT systems.

2.2.2.33 Interpret System practices, diagrams, specifications, drawings, and service orders.

2.2.2.34 Prepare system inputs, analyze system outputs, and maintain/repair various electronic systems.

2.2.2.35 Install, maintain, upgrade, modify, isolate and repair System servers, workstations, thin clients, peripherals and System software/hardware.

2.2.2.36 Coordinate and implement software code upgrades.

2.2.2.37 Coordinate and implement installation of new equipment and products. For the purpose of this document, “products” refers to any Commercial off the Shelf (COTS) or Government off the Shelf (GOTS) software in support of the Systems.

2.2.2.38 Coordinate maintenance actions and schedule System outages.

2.2.2.39 Collate and document the necessary information (problem, troubleshooting steps taken, stakeholders coordinated with etc.) for creating requests for higher-level technical support.

2.2.2.40 Ensure full data and system recovery to operational status upon completion of maintenance and/or recovery actions. Status shall be confirmed using two person integrity as a control mechanism.

2.2.2.41 Execute after action notification procedures describing maintenance actions to the MAJCOM Communications Coordination Center (MCCC) for ASI and OR events.

2.2.2.42 Monitor user load balance, for Systems with load balancing capability, and adjust and direct users to alternate sites/enclaves.

2.2.2.43 Maintain clean servers, related equipment, and Systems areas IAW established directives.

2.2.2.44 Create and maintain scripts for task automation.

2.2.2.45 Assist in the operational validation of application updates, enhancements and changes for the purpose of assisting in ensuring that assigned applications and related systems function appropriately and meets configuration baseline and database software requirements.

2.2.2.46 Track and monitor all customer support requests and troubleshooting and resolution actions in the TTTS.

2.2.2.47 Ensure trouble tickets are documented in the TTTS and protected according to classification level.

2.2.2.48 Provide final trouble ticket closeout by identifying what the actual issue was and how it was it resolved in the TTTS.

2.2.2.49 Fully understand the procedures for moving each system operations from one enclave to another enclave and practice the move regularly.

2.2.2.50 Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems into a Common Computing Environment (CCE) and DOD or commercial cloud environment (such as IaaS and Amazon Cloud.)

Network Administration

The contractor shall provide network administration services consisting of network/appliance provisioning, installation, configuration, security, operation, and maintenance of Systems hardware, software, and related infrastructure. This includes all related local area and wide area networks or segments, data communications, and associated equipment, software, and appliances within the applicable production environment(s). These services consist of performing multiple, highly complex, technical tasks with a need to routinely upgrade skills in order to meet a changing production environment, applications, system designs, configurations, hardware, utilities, and operational conditions. The contractor shall ensure the Systems hardware, OS, software systems, and related procedures adhere to Government approved production configurations; Government system availability and reliability standards; and OEM system operation and maintenance procedures. The contractor shall document all procedures in the Contractor Maintained Procedures. The contractor shall perform 24/7/365 network administration services for the Systems unless the TO states less hours are required. The contractor shall ensure that network administration personnel have relevant industry standard certifications in addition to the mandated DoD 8570.01 Information Assurance Workforce Improvement Program IAT Level II certification. Network Administration personnel are considered Mission Essential as defined in section 5.10.

The following is a listing of system and network administration functions that shall be performed by the contractor:

2.2.3.1 Perform network monitoring including monitoring alarms; verify the integrity and availability of all hardware, appliances, resources, and systems.

2.2.3.2 Perform regular security event monitoring to identify any possible intrusions.

2.2.3.3 Repair and recover from hardware or software failures, elevating problems to Level 3 support in a timely-manner to avoid further system issues.

2.2.3.4 Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate ASI scheduling.

2.2.3.5 Lead troubleshooting teleconferences with users, Level 3 support, and Level 3 development organizations to resolve open system issues as documented in the TTTS.

2.2.3.6 Notify appropriate authorities when a trouble ticket becomes OR (e.g. System failures and other situations where System degradation and/or outage is observed or reported) IAW Systems’ OR Criteria.

2.2.3.7 Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply TCNO, Information Assurance Vulnerability Management (IAVM), and other downward directed security patches.

2.2.3.8 Upgrade and configure network software that supports infrastructure applications according to project or operational needs.

2.2.3.9 Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the TTTS.

2.2.3.10 Perform software and hardware fielding IAW software version document releasing instructions provided by Level 3 support and Level 3 development organizations.

2.2.3.11 Maintain all procedures associated with sustaining the production environment and document them in Contractor Maintained Procedures.

2.2.3.12 Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures and document them in Contractor Maintained Procedures.

2.2.3.13 Monitor, maintain, and ensure currency of network device/appliance certificates.

2.2.3 14 Provide performance reporting to support capacity planning.

2.2.3.15 Perform ongoing performance tuning, hardware upgrades, and resource optimization as required.

2.2.3.16 Maintain the network environmental and monitoring equipment.

2.2.3.17 Solve technical issues with network equipment, appliances, utilities, or other network hardware and software.

2.2.3.18 Propose software or hardware updates and other network enhancements to improve systems performance and reliability; schedule and coordinate level-appropriate maintenance.

2.2.3.19 Troubleshoot and resolve network service issues, including issues referred by system managers, program offices, the Help Desk, database managers, network engineers, systems administrators, and other technicians.

2.2.3.20 Provide advanced troubleshooting skills and the ability to rapidly identify issues and provide restoration of issues.

2.2.3.21 Provide rapid restoration of network anomalies including assistance with disaster recovery of the network.

2.2.3.22 Identify, test, diagnose, and analyze troubles in the network.

2.2.3.23 Keep detailed records/logs of work activities/system performance to support Metrics and Reporting and include in Month Status Reports.

2.2.3.24 Initiate, process, and track requests for firewall configuration changes and firewall configuration waivers when a customer with established System accounts requires network access and assistance.

2.2.3.25 Troubleshoot end-to-end network connectivity issues affecting the operations and performance of systems.

2.2.3.26 Operate and maintain program specific Virtual Private Networks (VPN) and firewalls and other network appliances.

2.2.3.27 When end-to-end network connectivity and network access issues arise relative to the Systems at host sites, interfaces, or at other locations in the US or overseas, participate in the fault isolation, detection, and correction; coordinate repair efforts; and track status.

2.2.3.28 Log all user network access requests in the TTTS.

2.2.3.29 Interpret network practices, diagrams, specifications, drawings, and service orders.

2.2.3.30 Prepare system inputs, analyze system outputs, and maintain and repair various electronic systems.

2.2.3.31 Install, maintain, upgrade, modify, isolate and repair network devices, appliances, and System software/hardware.

2.2.3.32 Provide support for the management of the network and performance of duties related to the operation of the network, including change management and problem resolution.

2.2.3.33 Coordinate and implement software code upgrades in all network equipment.

2.2.3.34 Coordinate and implement installation of new equipment and products in the network.

2.2.3.35 Coordinate and implement the removal of network equipment being displaced or replaced.

2.2.3.36 Coordinate all maintenance actions and schedule System outages with appropriate System Manager

2.2.3.37 Collate and document the necessary information (problem, troubleshooting steps taken, stakeholders coordinated with etc.) for creating requests for higher-level technical support.

2.2.3.38 Ensure full availability and security to operational status upon completion of maintenance and/or recovery actions. Status shall be confirmed using two person integrity as a control mechanism.

2.2.3.39 Execute after action notification procedures describing maintenance actions to the MCCC for ASI and OR events.

2.2.3.40 Monitor user load balance for Systems with load balancing capability, and adjust and direct users to alternate sites/enclaves.

2.2.3.41 Maintain clean network devices, appliances, and Systems areas IAW established directives.

2.2.3.42 Track and monitor all customer support requests and troubleshooting and resolution actions in the TTTS.

2.2.3.43 Ensure trouble tickets are documented in the TTTS and protected according to classification level.

2.2.3.44 Provide final trouble ticket closeout by identifying what the actual issue was and how it was resolved in the TTTS.

2.2.3.45 Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems into a Common Computing Environment (CCE) and DoD or commercial cloud environment (such as IaaS and Amazon Cloud.)

Database Administration

The contractor shall provide database administration services within the applicable production environment(s). These services consist of performing multiple, highly complex, technical tasks with a need to routinely upgrade skills in order to meet changing database applications, system designs, and production environment conditions. Specific skill-based competencies are required to satisfactorily perform this task, which are the interaction and data transfer between hosts and clients of multiple hardware configurations, preparing and maintaining accurate records, and utilizing the functionality of pertinent software applications. Due to the rapidly evolving nature of IT, the contractor shall keep pace with system and operational environment changes by ensuring it has the appropriate skill set to effectively perform the database administration task without delay and/or system degradation. The contractor shall document all procedures in the Contractor Maintained Procedures. The contractor shall perform 24/7/365 database administration services for the Systems unless the TO states less hours are required. The contractor shall ensure that database administration personnel have relevant industry standard certifications in addition to the mandated DoD 8570.01 Information Assurance Workforce Improvement Program IAT Level II certification. Database Administration personnel are considered Mission Essential as defined in section 5.10.

The following is a listing of database administration functions that shall be performed by the contractor:

2.2.4.1 Troubleshoot and resolve database system malfunctions to resolve issues by detecting and isolating faults, and taking corrective action to restore to operational status.

2.2.4.2 Upgrade databases for the purpose of ensuring their ability to use new and/or existing databases and related software.

2.2.4.3 Maintain all procedures associated with sustaining the production environment and document them in Contractor Maintained Procedures.

2.2.4.4 Maintain a knowledge base of problems and solutions including troubleshooting techniques, fix actions, and applicable procedures and document them in Contractor Maintained Procedures.

2.2.4.5 Ensure all issues and outages are properly documented in the TTTS, systems change request repository, configuration management systems, or other Government directed location.

2.2.4.6 Analyze and sustain capacity and performance requirements.

2.2.4.7 Analyze, consolidate, and tune databases for optimal efficiency.

2.2.4.8 Monitor systems and platforms for availability and database systems serviceability.

2.2.4.9 Back-up, cluster, mirror, replicate, and failover assigned databases.

2.2.4.10 Restore and recover corrupted databases.

2.2.4.11 Ensure database systems are configured to meet security directives.

2.2.4.12 Build database servers, install, and configure database software on servers and stations.

2.2.4.13 Perform database dumps, table maintenance, re-indexing, and database replication maintenance IAW industry standards and/or the applicable database maintenance guidance.

2.2.4.14 Collate the necessary information for creating requests for higher-level technical support.

2.2.4.15 Install patches for applicable database software.

2.2.4.16 Review, install, verify and document all OS patches and upgrades; upgrade administrative tools and utilities; and apply TCNO, IAVM, and other downward directed security patches.

2.2.4.17 Execute and monitor regular and ad hoc job execution in databases.

2.2.4.18 Consult with software application developers on preparing SQL queries and effective use of features and options of the applicable database software.

2.2.4.19 Consult with engineers and application developers on existing and potential performance parameters, to include identifying and resolving performance issues.

2.2.4.20 Resolve problems and meet user needs and system operational and functional requirements.

2.2.4.21 Assist in the operational validation of application updates, enhancements and changes for the purpose of assisting in ensuring that assigned applications and related systems function appropriately and meets configuration baseline and database software requirements.

2.2.4.22 Coordinate database activities (e.g. security, upgrading, populating, refreshing, repairing database systems, cataloging, access rights, etc.) for the purpose of ensuring data accuracy and availability.

2.2.4.23 Monitor compliance with DoD and US Air Force database security and integrity.

2.2.4.24 Monitor compliance with Personally Identifiable Information (PII) protection standards and procedures IAW DoD Directive 5400-11, DoD Privacy Program personnel and DoD contractor Rules of Conduct are explained in Enclosure 2.

2.2.4.25 Using a variety of database support processes, ensure the operation, stability and optimal performance of the production databases and the availability of stored data.

2.2.4.26 Monitor assigned applications and related systems for the purpose of reducing application downtime and ensuring that assigned applications are available when needed to meet system performance and availability parameters.

2.2.4.27 Enforce database schemas, tables, procedures, and permissions.

2.2.4.28 Set up data sharing and disk partitioning according to configuration.

2.2.4.29 Create and maintain scripts for task automation.

2.2.4.30 Update operationally required data imports from approved sources and data extracts required to support operational needs, e.g., Digital Aeronautical Flight Information File (DAFIF) and weather data.

2.2.4.31 Create, delete, and modify database/privileged user accounts and access privileges.

2.2.4.32 Conduct a pre-fielding review of software and hardware fielding documentation and identify any clarifications prior to starting a fielding activity in the TTTS.

2.2.4.33 Perform software and hardware fielding IAW software version document releasing instructions provided by Level 3 support and Level 3 development organizations.

2.2.4.34 Lead system troubleshooting teleconferences with users, and Level 3 support and Level 3 development organizations to resolve open system issues as documented in the TTTS.

2.2.4.35 Monitor all customer support requests relating to database management, once assigned actively troubleshoot and resolve open tickets and report cause and fix action in the TTTS.

2.2.4.36 Ensure trouble tickets are documented in the TTTS and protected according to classification level.

2.2.4.37 Ensure final closed trouble ticket accurately reflects what the actual issue was and how it was resolved.

2.2.4.38 Support the modernization, technology insertion, performance analysis and turning, maintenance, testing, migration and integration of the Systems supported into a Common Computing Environment (CCE) and DoD or commercial cloud environment (such as IaaS and Cloud.)

2.2.4.39 Coordinate and communicate with System Manager and impacted constituencies for activities (e.g. System maintenance and other situations where System degradation and/or outage may occur) to facilitate ASI scheduling.

2.2.4.40 Notify appropriate authorities when a trouble ticket becomes OR (e.g. System failures and other situations where System degradation and/or outage is observed or reported) IAW Systems’ OR Criteria.

2.2.4. 41 Execute after action notification procedures describing maintenance actions to the MCCC for ASI and OR events.

Web Site/Page Support

The contractor shall provide Web page support by accomplishing the following:

2.2.5.1 Create and update Systems’ informational/support web sites/pages. In accomplishing this subtask the contractor shall:

· Receive requests for Web site/page updates and edit the web page IAW the updated request.

· Ensure website/page updates meet the requirements of HQ AMC, AF, and DoD web page standards including Air Force Instruction (AFI) 33-115, Air Force Information Technology Service Management, and AFI 33-364_AFGM2017-01, Records Disposition – Procedures and Responsibilities.

· Post, or forward for posting, and verify updated web page was posted correctly.

Configuration Management (CM)

The contractor shall provide robust CM for the production environment for the Systems by adhering to Government standards and business best practices. To ensure the CM proactively controls system and software configurations the contractor shall:

2.2.6.1 Ensure CM has the latest, and previous baselines as directed by the Program Management Office (PMO), configuration baselines of all hardware, software and documents as provided by the Government.

2.2.6.2 Provide, manage, maintain, and implement a production baseline Configuration Management Plan (CMP). The CMP shall address management and processes to ensure control of the system configurations and provide accurate tracking and reporting of all baseline hardware, software, virtual images, documentation, and configurations (including virtual images.)

2.2.6.3 Ensure configuration control is implemented for all items identified in the various Systems’ production configuration baselines and production virtual images. Note: Configuration control of production virtual environments will be much more dynamic as virtual images are created and taken down in response to business demand, outages, and other factors.

2.2.6.4 Establish a system for identifying and reporting items that are incorrectly identified in the configuration baseline or missing from the production configuration documents.

2.2.6.5 Maintain database for tracking all configuration items (CIs) for each of the supported environments.

2.2.6.6 Develop, implement, and maintain plans, processes, procedures, and tools required to provide CM for the Systems production baseline, including the virtual environment, as part of the CMP and document them in Contractor Maintained Procedures.

2.2.6.7 Review software and hardware provided for operations and maintenance to ensure all necessary items/documentation are received from the sources provided by the Government, notify the COR of receipt, and annotate any discrepancies.

2.2.6.8 Establish and maintain a Configuration Management Library (CML) of the current production environment software, hardware, and virtual configuration, technical drawings, and documentation for the baselines operated and maintained by the contractor.

2.2.6.9 Maintain database for tracking all CIs for each of the supported environments, ensuring the status of all production baseline CIs are kept current.

2.2.6.10 Verify that the software loaded on supported systems is the correct version, report noncompliance, and provide status to the COR.

2.2.6.11 Perform physical configuration audits and take positive, proactive actions to ensure correction of any discrepancies noted.

2.2.6.12 Support tracking, coordination, and reporting on TCNO, IAVM, and other downward directed security patches.

DATA DELIVERABLES

Deliverables required by the Government IAW this PWS shall have a Data Item Description (DID). All deliverables shall be reviewed for timeliness, accuracy and format. If a deliverable due date falls on a weekend or holiday, the contractor shall submit the deliverable on the last business day prior to the due date. In addition, unscheduled project updates shall be provided by Government/COR. Table 1, Data Deliverables, provides a list of the required PWS deliverables.

All products and documentation delivered under this PWS become the property of the Government.

Table 1: Data Deliverables The following deliverables shall be provided IAW Exhibit A-AD: