A07_Attachment_7_-_SPS-A_OM_Standard_Operating_Procedure.pdf

Office of the Deputy Assistant Secretary of the Army (Procurement)

Standard Procurement System – Army (SPS-A) Operations and Maintenance

Standard Operating Procedure (SOP)

Last Update: 30 January 2017

6/22/17 ODASA(P) SPS-A Operations and Maintenance SOP 2

Table of Contents

1 Introduction 2 Roles and Responsibilities 3 System Overview 4 SPS-A Technical Operations and Maintenance

4.1 Daily Maintenance Tasks

4.2 Weekly Maintenance Tasks

4.3 Monthly Tasks

4.4 Ad Hoc Technical Tasks

5 PD² Functional System Administration

5.1 Security Model Maintenance

5.2 Functional System Configuration

5.3 PD² Functional System Operations

6 Communication 7 Summary Appendix A – Consolidated Database Maintenance Schedule

ODASA(P) SPS-A Operations and Maintenance SOP 3

This page intentionally left blank.

ODASA(P) SPS-A Operations and Maintenance SOP 4

1 Introduction This Standard Operating Procedure (SOP) outlines the current operations and maintenance plan and associated schedule for the Standard Procurement System – Army (SPS-A) enclave. The Office of the Deputy Assistant Secretary of the Army (Procurement) (ODASA (P)), in conjunction with the Army Contracting Command (ACC) CIO/G6 organization, currently operates and maintains the SPS- A program. The SPS-A production site is currently hosted at the Army Acquisition, Logistics, and Technology Enterprise Systems and Services (ALTESS) hosting facility, located in Radford, VA.

The ODASA (P) has identified the need for an operations and maintenance plan to control, monitor, and manage application-level operations within the SPS-A enclave. The purpose of this document is to identify and describe a complete operations and maintenance plan for the SPS-A environment.

This plan describes in simple, straightforward terms the processes required to ensure that application operations will continue under all circumstances, and that data integrity is maintained at all times. This document outlines the minimum technical support needed to operate and maintain a stable and secure SPS production environment within the given robust hosting infrastructure.

2 Roles and Responsibilities The following organizations and personnel (where applicable) will be involved in management of the SPS-A and its operations and maintenance activities:

ODASA(P) – Program Sponsor and Oversight ACC CIO/G6 – Program Technical Lead and Information System Security Officer (ISSO) ALTESS – Hosting Environment Oversight Technical System Administrators – Technical Operations and Maintenance Local Site Functional System Administrators – Functional Operations and Maintenance

3 System Overview The SPS-A program provides procurement and requirement functionality to the Army acquisition community. It consists of both physical and virtual server environments supporting the following major components/technologies:

Sybase Adaptive Server Enterprise (RDBMS/Data Layer) SPS webMethods Integration/Broker Server (Integration Layer) PRweb (Web Layer) Citrix XenApp (Presentation Layer) Procurement Desktop – Defense (PD2) (Application Layer) PD2 Application Server Framework (Application/Integration Layer) Citrix Edgesight (Reporting Layer) Citrix Web Interface (Presentation Layer) Citrix NetScaler (Web/Presentation Layer)

ODASA(P) SPS-A Operations and Maintenance SOP 5

Microsoft SQL Server (Reporting/Data Layer) ACBIS BiXtractor (Integration Layer) PD2 Integrity Tool (Application Layer) SPS Legacy Integrations (Integration Layer)

4 SPS-A Technical Operations and Maintenance

4.1 Daily Maintenance Tasks

On a daily basis, the SPS-A Technical System Administration Team will perform the following tasks (manually or via scheduled task/cron job):

Database Backups – Back up the following databases: PD² production databases, PRweb databases, MWS databases, CDR databases, Sybase system databases. Backups are automatically filed to tapes, which are stored both on and offsite, in accordance with the SPS-A Backup and Recovery Plan.

Review Backup Logs – Daily manual review of backup log outputs and automated emails containing high-level backup log data. Remediation plans will be developed and executed in accordance with Sybase and NetBackup best practices, and documented in weekly status reports.

Monitor Application Server Framework Servers – Monitor for EDA and/or document generation failures within the ASF server logs. If any errors, systemic issues, or repetitive document rejections occur, the technical team will work with the appropriate functional system administrators or ALTESS technical administrators, where appropriate, to resolve the errors.

Monitor PD2 Adapter Servers – Monitor PD2 Adapters and associated log files for errors or server irregularities (check 2-3 servers per day on a rotating schedule). If any errors, systemic issues, or repetitive document rejections occur (e.g., GFEBS pre-award validation failures), the technical team will work with the appropriate functional or technical system administrators, where appropriate, to resolve the errors. Please see the SPS-A Architectural Design Document for a list of integrations monitored by the SPS-A Technical System Administration Team.

On a daily basis, ALTESS will perform the following tasks:

Differential Operating System Backups – Differential Operating System-level backups will be completed on a daily basis in accordance with the ALTESS Backup and Recovery Plan.

Perform Disk Scans – Perform regular disk scans on all system disks to locate and repair any damage. These scans are conducted in real-time.

Monitor Server Performance – Monitor CPU usage on all servers and make recommendations for performance improvements as appropriate. This monitoring is conducted in real-time by the ALTESS-deployed Solarwinds monitoring software.

Check Free Disk Space – Check the amount of free disk space remaining on all drives on the database server and make recommendations to add more drives, as appropriate. ALTESS deploys the Solarwinds utility to monitor disk space in real-time, and provide email alters to administrators when ALTESS-defined thresholds are met or exceeded.

ODASA(P) SPS-A Operations and Maintenance SOP 6

4.2 Weekly Maintenance Tasks

On a weekly basis, the SPS-A Technical System Administration Team will perform the following tasks:

Review Error Logs – Review 2-3 Sybase Server error logs. Remediation plans will be developed and executed in accordance with Sybase best practices, and documented in weekly status reports.

Run Update Statistics/Reorg Compact Script – Complete the “Update Statistics/Reorg Compact” task, which updates various indexes in the PD² production database, and compacts to reclaim space and remove row forwarding.

On a weekly basis, ALTESS will perform the following tasks:

Full Operating System Backups – Full Operating System-level backups will be completed on a weekly basis in accordance with the ALTESS Backup and Recovery Plan.

4.3 Monthly Tasks

On a monthly basis, the SPS-A Technical System Administration Team will perform the following tasks:

Perform Database Consistency Checks – Perform database consistency checks (DBCC) including checkstorage and checkcatalog on all databases.

Check Amount of Free Space remaining in Database – Check both the data and log segments of the production database to make sure that there is no danger of running out of space. The SPS-A Admin Team will add more space to the device, when any data segment reaches 88% of capacity. The schedule for this task is elevated to bi-weekly or weekly during the month of September, at the discretion of the SPS-A System Owner.

Validate Connectivity to Sybase Servers – From all VM Admin Desktop instances available, confirm connectivity to Solaris server instances using both the Putty and Cygwin applications.

On a monthly basis, ALTESS will perform the following tasks:

Perform Virus, ACAS, and SCAP Scans – Perform regular scans on all systems:

o Virus Scans – conducted in real-time;

o ACAS Scans – conducted at least monthly;

o SCAP Scans – conducted at least quarterly.

On a monthly basis, the SPS-A Technical System Administration Team and ALTESS will perform the following task:

Review Operation System Logs – Check event viewer and/or other system logs for operating system errors, and troubleshoot as appropriate.

ODASA(P) SPS-A Operations and Maintenance SOP 7

4.4 Ad Hoc Technical Tasks

On an ad hoc basis, the SPS-A Technical System Administration Team will perform the following tasks:

Truncate MWS Document Tables – Run script that will truncate the tables that contain XML document data in the MywebMethods (MWS) database. This script is necessary if the MWS database has a larger volume of data to delete and/or archive than the total size of the associated transaction log.

Update Sybase Passwords every 60 days – In accordance with Army Regulation AR 25-2 and the SPS-A User Access and Account Management SOP, all passwords must be updated every 60 days. To ensure production continuity, when passwords are updated in Sybase, they must also be updated in the PD² Adapter, PD² ASF Server, PRweb, BI Xtractor, the Integrity Tool, and any other application whose function is dependent upon Sybase login account credentials.

Perform SPS Patches, Enhancements, and Upgrades – Apply updates to the SPS product suite for accepted new Service Releases and maintenance releases. Schedules for these updates will be coordinated separately, and in coordination with the ODASA(P) and program stakeholders.

SPS Standard Data Updates – Standard Data updates are posted to the SPS Knowledge Base one or two times per year and must be applied to each PD² DB. The current process involves first applying the newest update to the pilot database (413th – Hawaii) for one day of testing. If testing produces no errors, the update is applied to all enclave production databases.

FAR/ DFARS Clause and Clause Logic Updates – FAR & DFARS clause updates are posted to the SPS Knowledge Base every 2 – 6 weeks and must be applied to each PD² DB. The current process involves first applying the newest update to the pilot database (413th – Hawaii) for one day of testing. If testing produces no errors, the update is applied to all enclave production databases.

PD2 Reference Library Updates – Reference Library updates are posted to the SPS Knowledge Base every 2 – 6 weeks and are applied to the common SPS-A portal after each release.

On an ad hoc basis, ALTESS will perform the following tasks:

Non-PD² Software Upgrades and IAVA Patch Testing and Application – Non-PD² software upgrades and patches will be applied on a monthly basis. The SPS-A Technical System Administration Team will work with the ALTESS to test application compatibility and report outcomes to ODASA (P) and ALTESS. As a best practice, patching will be implemented on a “rolling” basis to ensure maximum functional availability to the SPS-A user community.

ODASA(P) SPS-A Operations and Maintenance SOP 8

Reboot Servers (Physical and Virtual) – Reboot Servers on a regular schedule (see Figure 1 – SPS-A Reboot Schedule).

Figure 1 – SPS-A Reboot Schedule

Server Type Frequency

Citrix web Interface Monthly (during patch cycles only)

Application Server Framework Twice Weekly (Mondays/Thursdays)

PD² Adapter / DTU Servers Three times Weekly (Mondays/Thursdays/Sundays)

Citrix XenApp (PD2) Every 7 days (staggered across server farm)

Sybase RDBMS (Oracle T5 Servers) Monthly (during patch cycles only)

5 PD² Functional System Administration Local site Functional System Administrators (FSAs) play a vital role in maintaining stable and optimized system performance. Specifically, local FSAs will maintain the PD² Security Model and other functional system configurations for each site as local procurement process and personnel changes occur. These tasks include those listed below:

5.1 Security Model Maintenance

FSAs are responsible for updates to the following PD2 System Administration tasks as required:

Classes Groups Approval Types Approval Authority Approval Templates Teams User Types Users PIIN Maintenance Route Templates Warrant Types Procurement Profiles Import, Assign, and Delete a Report Workload Criteria Create preferences

ODASA(P) SPS-A Operations and Maintenance SOP 9

5.2 Functional System Configuration

All non-security, general contract support, and template tasks required to support PD² system configuration. Local site FSAs will be responsible for administering the following System Administration tasks within the PD2 application as it becomes necessary to do so:

Clauses* Clause Rule* Characteristics* Contract Formats* Contract Types* Attachments Checklist Templates Milestone Templates Contract Distribution List (CDL) Templates Procurement Administrative Lead Time (PALT) Contract Line Item Number (CLIN) Templates* Funding Tasks

FPDS-NG

Job Order Tasks Currency Tasks Unit of Issue Buy American Exemption Establish Auto Order parameters Define Local Info Labels Define Automatic Contract Closeout Criteria Analyze Validation Rules Tasks Workload Tasks Establish and Maintain the Set-Aside/Excluded List Maintain License Information Create and Maintain Electronic Signatures in PD² Vendor and Organization Database Management Auto Order (Automatic Ordering) Tasks

*FSAs should not make changes to the default clauses, clause rules, characteristics, contract formats, contract types, or CLIN templates. FSAs can, however, add local, site-created content in these tasks.

ODASA(P) SPS-A Operations and Maintenance SOP 10

5.3 PD² Functional System Operations

Local FSAs are responsible for PD² functional system operations tasks that indirectly support the user, to include:

Creating, modifying, and deleting “Message of the Day” Banners Maintaining Local Clause Logic Adding Local Clauses to the PD² database as required and associating the proper rules with each Clause Revising or deleting Local Clauses as required Updating/maintaining local ad hoc Cognos reports Updating users on functional changes resulting from Service Releases, as well as providing

Release Notes.

6 Communication In accordance with the ODASA (P)-ALTESS SLA, ALTESS will communicate all A-WAN security enhancement and maintenance activities in advance of their occurrence to ODASA (P) and points of contact listed within the SLA. At a minimum, ODASA (P) will communicate this information to the SPS-A System Administration team. If the enhancement implementation or maintenance will, or could potentially result in downtime or performance degradation to the common user community, the SPS-A System Administration team will communicate the impact and timing to the community via the SPS-A listserv (at least one point of contact from every SPS-A site is subscribed to the SPS-A listserv).

At the application level, the same communication mechanisms will apply, and the community will be notified in the event an impact is likely to occur.

7 Summary In order to ensure data integrity, availability, and reliability, it is critical that each of the aforementioned maintenance tasks be completed per the outlined schedule(s). Since the core SPS application is client-server in nature, special attention must be given to the Operating Systems that host its components.

Successful technical and functional maintenance of both the application and its hosting environments will ensure that the ODASA (P) is able to provide a quality solution to support the Army Procurement Community.

ODASA(P) SPS-A Operations and Maintenance SOP 11

Appendix A – Consolidated Database Maintenance Schedule The figure below details the tasks required to be performed on a daily, weekly, semi-monthly, and monthly basis for SPS maintenance and backup operations.

Process Real- Time

Daily Weekly Semi-

Monthly Monthly Ad Hoc Responsible Party

Backups

PD² Production SPS-A Tech

Admins

Sybase System SPS-A Tech

Admins

CDR

SPS-A Tech

Admins

MWS

SPS-A Tech

Admins

Review Database Backup Logs SPS-A Tech

Admins

Backup Val idation SPS-A Tech

Admins Ful l Operating System Backups ALTESS

Di fferentia l Operating System Backups ALTESS

Server Monitoring and Maintenance

Review ASF Server Logs SPS-A Tech

Admins

Review PD2 Adapter Server Logs SPS-A Tech

Admins

Sybase Error Log Review SPS-A Tech

Admins

Val idate Sybase Server Connectivi ty SPS-A Tech

Admins Server Disk Scans ALTESS

Virus Scans ALTESS

ACAS Scans ALTESS

SCAP Scans ALTESS

Review Operating System Logs Both

Database Maintenance

DBCC Checkstorage/Checkcata log SPS-A Tech

Admins

Monitor Database Free Space SPS-A Tech

Admins

Update Statis tics/Reorg Compact SPS-A Tech

Admins

Monitor Server Resources SPS-A Tech

Admins

Truncate MWS Document Tables SPS-A Tech

Admins

Update Sybase Passwords SPS-A Tech

Admins Perform SPS Patches , Enhancements , and Upgrades

SPS-A Tech

Admins

SPS Standard Data Updates SPS-A Tech

Admins FAR/DFARS Clause and Clause Logic Updates

SPS-A Tech

Admins

PD2 Reference Library Updates SPS-A Tech

Admins Non-PD² Software Upgrades and IAVA Patch Testing and Appl ication

ALTESS

Reboot Servers (Phys ica l and Virtua l ) Both

SPS-A Consolidated Operations and Maintenance Schedule