36C10B20Q0218-008.pdf

VBA 1 January 2020

Capacity Plan

Department of Veterans Affairs (VA)

Veteran Benefits Administration (VBA) Performance

Testing

Capacity Planning Report

CLIN3006

Version 2.9 January 10, 2020

VBA i September 2019

Revision History

Date Version Description Author

10/5/17 2.0 Capacity Plan Update FTC

1/16/18 2.1 Capacity Plan Update FTC

4/11/18 2.2 Capacity Plan Update FTC

7/13/18 2.3 Capacity Plan Update FTC

9/10/18 2.4 Capacity Plan Update FTC

1/10/19 2.5 Capacity Plan Update FTC

4/2/19 2.6 Capacity Plan Update FTC

7/9/19 2.7 Capacity Plan Update FTC

9/5/19 2.8 Capacity Plan Update FTC

1/10/20 2.9 Capacity Plan Update FTC

VBA ii January 2020

Table of Contents

1. Introduction

2. Executive Summary

2.1. Transformational Applications

2.2. WebLogic Growth

2.3. Increases in Users

2.4. Conversion from BDN, Claims Growth, and Database Growth

2.5. VETSNET Status

2.6. Summary

2.7. Issues and Recommendations

3. Current Status

3.1. Online

3.2. Batch

4. CPU Utilization and Transaction Load for VBAPROD

4.1. CPU Utilization

4.2. Transaction Load

5. Tuxedo Cluster

6. Monthly Tuxedo Growth

7. Indicators for the Last Twelve Months

7.1. Database CPU Over the Last Twelve Months

7.2. SHARE, LEGACY, COVERS, and CSS Transactions

7.3. MAP-D and RBA Transactions for the Last Twelve Months

7.4. FAS, Awards and VNCOMM

7.5. WINRS Transactions for the Last Twelve Months

7.6. Web Environment

8. Issues and Recommendations

9. Summary

9.1. Tuxedo Cluster Upgrade

9.2. Online Volume

9.3. Web Services

9.4. Batch Volume

9.5. Tuning Opportunities

VBA iii January 2020

9.6. Additional Processor Requirements

9.7. Monitoring Tools

VBA 1 January 2020

1. Introduction This report is prepared in response to Contract VA118-16-D-1006 CLIN1006 Quarterly Capacity

Projection Report. The purpose of this report is to provide an analysis of trends for transaction volumes, and projections on the Corporate Performance for the production environment focusing on VBAPROD. This report will include an analysis of the VETSNET application suites and

Corporate Web applications, CPU utilization, I/O and memory consumption, and general system service utility consumption. The analysis is based on data from 2003 to the present day and has been updated to include data captured in the 4th quarter of FY19.

Data used in this report is collected from the production environment. CPU data is collected directly from VBAPROD. Tuxedo data comes from a website where the Tuxedo Administration team collects production data (http://vbadev.vba.va.gov/~tpadmin/tpadmin_graphs.htm).

2. Executive Summary

The VBA benefit systems consist of Compensation and Pension (C&P), Vocational

Rehabilitation & Employment (VR&E), Education, Loan Guaranty (LGY) and Insurance. Five million veterans receive benefits, and approximately 30,000 veterans are added each month.

Processing is hosted at three locations:

• Austin: Sun-Corporate, IBM (BIRLS), WebLogic

• Hines: Bull – BDN

• Philadelphia – Web

Substantial growth was observed over the past year. The Sun Corporate platform is currently supporting approximately 3.7 million online transactions during peak hour. The trend stalls for

Tuxedo transactions with current year over year growth decreasing by 2%. The decrease was not universal, and some applications grew while others declined. The business drivers for growth are:

• VBMS is a primary driver of growth along with CRM and eBenefits

• Growth in new claims

• Growth of the WebLogic environment

• Additional staff accessing the system including new hires and VHA

• Enhanced functionality of VETSNET

One of the concerns going forward was the test capacity for the BDN and BIRLS. When we ran

150% load test, BIRLS and BDN performed well. The only performance concern at this time is related to CPU utilization for the Tuxedo cluster. It is exceeding utilization levels that will be problematic in a fail over scenario. (See below for a detailed discussion. Tuxedo Cluster)

2.1. Transformational Applications

VBMS and VRM characterize a new class of transformational applications that use web services.

These applications frequently use web services to access the production database through

WebLogic and Tuxedo. Other transactions use web services to access the database directly. The applications that use Tuxedo are generating a high volume of transactions, thereby driving http://vbadev.vba.va.gov/~tpadmin/tpadmin_graphs.htm

VBA 2 January 2020 growth in Tuxedo transactions. At this time, it is clear that VBMS is generating over 50% of the traffic for the corporate database. The current trend indicates growth for applications other than client/server. This trend represents a major shift in architecture and needs to be closely monitored. (Performance testing being performed for VBMS applications is not part of the FTC performance contract.)

2.2. WebLogic Growth

WebLogic is the architecture of choice for new applications. Not all applications using the

Benefits Enterprise Portal (BEP), use Tuxedo middleware services. However, all transactions do access the Corporate Database. The number of transactions executed in this environment continues to increase year over year. July was higher that last year but August is lower.

The largest growth for WebLogic is driven by new applications, primarily VBMS with contributions from VRM (CRM) and eBenefits. These applications use Tuxedo differently than

VETSNET increasing the overall volume significantly. (See Web Environment for details). The

WebLogic environment is a major focus for performance testing. The VBMS applications are out of scope for the FTC performance contract. However, the traffic is simulated for testing purposes.

2.3. Increases in Users

While staff level increases for 2019 are not currently known, the growth in the number of new users was slow in 2019 and that trend is expected to continue for 2020. The backlog for claims that the VBA is struggling to address has seen significant progress. This situation is being addressed primarily through overtime (and more efficient processes) that will not stress the infrastructure currently in place. If a different methodology is utilized, this issue must be revisited.

There are plans to make the SHARE application available to Hospital and Cemetery users via terminal server. Cemetery users will enter “First Notice of Death,” a function currently performed by VBA field staff. Whatever isn’t available through VBMS will be accessed via

SHARE. (COVERS is being phased out through the functionality of VBMS but won’t go away any time soon.) These activities have not created a major increase in transaction load yet. But they do represent a potential source for future growth.

2.4. Conversion from BDN, Claims Growth, and Database Growth

Over 5 million veterans and dependents are currently paid by the Corporate (VBAPROD) system. Historically, these payments increased due to conversion of records from BDN and claims growth. Conversion will not be a load factor going forward. (There is a new project to offload Education benefits from BDN to the Corporate database. The impact is undefined.)

However, there is an organic growth of approximately 30,000 new veteran claims per month.

The addition of active veteran claims causes key tables in the database to grow.

VBA 3 January 2020

Batch programs that sequentially read active claim records incur a substantial penalty with database growth. This is a cause for concern in batch job run times. Online impact tends to be smaller with proper use of indexes, although it does have an impact on database maintenance.

Database growth also occurs because of maintaining user journals in perpetuity. User journals are now approximately 1/3 of the database. Database growth lengthens backup times. New disk drives (Hitachi) have not shown improvement overall.

2.5. VETSNET Status

VETSNET has moved into a maintenance mode. This is the final stage in the support of the full range of C&P processing in the corporate environment. VETSNET clients don’t drive utilization as much on VBAPROD with the shifting to VBMS. The processing of Awards is the most resource intensive online transactions. Previous measurements in the performance environment indicated an increase in CPU utilization of 13% if Awards transactions increase by 400/hr.

Given the capacity of the new database server, this growth will not have a significant impact.

Prior to the establishment of Awards capability on Corporate, the BDN system observed peak hour peak day Awards rates of 2400/hr. Current Corporate peak hour peak day Awards rates are about 3500/hr. Awards rates are expected to grow as increased claims filings and field staff increase to support the increased filings. RBA transactions are, also, heavy users of resources.

The impact of these transactions is less concerning because VBAPROD has sufficient capacity.

Tuxedo usage is up 1% in June 2019 year over year. Over the past several quarters VBMS has driven substantial growth in the environment. There was less than 1% annual increase for FY19.

There is an increased emphasis on SHARE via VBMS and other applications to access BDN data.

Folder Tracking System application is decommissioned. The VHA access will be through

VBMS and SHARE. VHA has the potential to be a very large user community. A better estimate of VHA use is needed going forward since it could impact the system dramatically.

2.6. Summary

A) Applications using Web Services are the largest new source of load for the database.

When one looks at the number of users and compares it to the load generated, there is a disproportionate amount of utilization. This could indicate coding inefficiencies.

B) Tuning opportunities exist for several applications. With system resources at their current levels, this is not an area of emphasis. Given the reliance on new applications like

VBMS and CRM, this effort should be re-evaluated.

C) Online volumes are expected to grow with increased use and enhancement of VBMS, VRM, et al. Although, the pace is undetermined.

D) Batch Loads are expected to increase as more veterans are paid by the VBA and new batch jobs are added. Many of the existing batch jobs were written for an environment that had a significantly lower number of records to process. The performance of these jobs is degrading

VBA 4 January 2020 over time as record count has grown substantially. It is likely that some jobs may have to be rewritten. Batch processing always needs to be monitored closely because one job can cause problems for the entire batch cycle.

E) The primary production server, VBAPROD, was upgraded in July 2013. This is an

Oracle Sun server with 176 CPU’s and 416 GB of memory. This is a massive upgrade relative to the previous server. A tech refresh is being planned for next year.

2.7. Issues and Recommendations

A) WebLogic has grown dramatically using Web Services. The growth is trending higher.

WebLogic is the preferred architecture for new applications. The load for these applications is currently not well defined. Additional capacity has been added throughout the production

WebLogic environment. Tuning and configuration of the WebLogic environment will be a key going forward.

B) The security methodology used in the Corporate environment has changed. Two factor authentication (2FA) has been implemented in several environments including production. This project requires PIV cards for login. The impacts on performance are measurable. Anywhere from a .5-3 second increase in login time. This is deemed as the ‘cost of doing business’ with improved security.

C) The Performance team has developed a performance testing methodology for applications using Web Services. The implementation is complete for VBMS web services. The effort is ongoing for other applications such as CRM. Applications using Web Services are having a major impact on the environment. Coordinated performance testing is not being performed for many of these applications. The most active WebLogic applications are out of scope for the FTC performance contract.

D) CSEM is a security application that has been added to the production environment.

Major CPU spikes have been observed due to increased utilization of this application. Extensive use will create performance issues in production. There is a project to tune this application.

E) Batch processing is growing, and this trend is expected to continue. FAS batch is an area of concern. As the number of records processed increases, some of these jobs fail to run efficiently. Modifications will be required over time. The project to address some of these issues has been completed. The primary focus was FAS080. There are still jobs that require attention. Plans going forward are uncertain. A project to rewrite all FAS batch applications is recommended.

3. Current Status

The current production system is deployed as a seven-board domain on an Oracle Sun M5 server with 176 processors. Two Hitachi units provide disk resources. Tuxedo has a three-node cluster.

Oracle is the primary user of VBAPROD. There are three broad components of workload:

VBA 5 January 2020

• Online essentially operates 24 hours per day.

• Batch runs from 4:00 PM until 6:00 AM Central time typically.

• Background activity from various tools occurs throughout the day. This load appears to be increasing. Given the size of the server, this should not present a problem.

A new server replaced VBAPROD (July 2013). This will provide ample processing power for the next several quarters. Memory resources are not an issue with free memory running well over 200 Gigabytes. The Hitachi systems uniformly provide disk service times in the 5-10ms range, even when backups are running. I/O rates of ~ 2,000/sec are observed during prime online hours. The implementation of RMAN has enabled database backups to finish before the online day starts. I/O rates can be much higher during batch depending on which jobs are running. Disk performance is not generally an issue.

Tuxedo has been moved to a cluster of four servers. These servers are running at a moderate rate of utilization (~35%). The primary objective is to provide a robust, highly available and scalable environment for Tuxedo. These servers will be upgraded as part of the tech refresh. A fourth node has been installed.

3.1. Online

Current online CPU utilization is a little over 28% during the peak hour online day (7:00AM –

6:00PM Central). Utilization will vary seasonally by 5-10%. The hardware upgrade in July

2013 has provided substantial capacity for future growth. Historical data exhibits three peak periods: the second fiscal quarter of the year and from July through October with a peak in

January. This trend continues from year to year. More growth is expected in the online day.

The impact of VBMS and VRM applications has generated significant growth. The new BDN conversion is an unquantified factor going forward. There is a new project to offload the

Education benefits from BDN. Transformational applications (i.e. VMBS, eBenefits), WebLogic, new claims and user growth will be the primary drivers going forward.

Although tuning opportunities exist in VETSNET, there appears to be insufficient development resources available to exploit them in the near term. The cost benefit is questionable. Given the initial impact of VBMS and eBenefits, evaluating coding efficiencies for those applications is a viable option.

3.2. Batch

The batch window is a critical concern as run times continue to increase for FAS batch jobs, which make payments to veterans, being the major concern. The increase in execution is attributable to application design, database growth (journals) and an increased number of records. Increasing number of payments being handled by VETSNET and VBMS will be the prime drivers, along with any new requirements for batch jobs. There is also contention between various batch jobs that have significant impact on jobs such as FAS080. The tuning effort for

FAS080 concluded several months ago. A project to rewrite all FAS batch applications is recommended.

VBA 6 January 2020

Currently, FAS080 performance is typically running in about 5.5 hours. Twelve months ago, it was running in 4.5 hours. The performance of FAS080 has been improved with tuning efforts historically. FAS batch performance has a history of recurring performance issues. A project team has completed its work on tuning.

Performance for EAC has improved and seems to have stabilized.

Adding to the complexity of the batch cycle is the increasing number of jobs competing for the batch window. National Work Queue (NWQ) has added a batch job that is very time sensitive.

The batch window must also support several other activities that impact the batch throughput.

This includes increasing online usage and the data warehouse extracts. Occasional overruns of critical batch jobs into the prime online window are an ongoing concern, although less frequent than in the past. Further complicating batch is the need to concurrently support online access for offices in Alaska, Hawaii and the Philippines. These offices generate moderate overnight volumes for Tuxedo and Oracle database.

Extensive overtime on the weekends is being used to reduce the claims backlog. Second shift may experience uneven performance due to resource competition from batch jobs, although the throughput of the VBAPROD server should help alleviate this problem.

A comprehensive effort to manage the batch window more effectively could be quite productive.

Maximizing Control-M capability to manage the batch window more efficiently would likely improve performance during the batch window.

4. CPU Utilization and Transaction Load for VBAPROD

4.1. CPU Utilization

VBAPROD is the database server for the Corporate database. The Corporate database is approximately 14TB. System CPU utilization for VBAPROD is moderate (~28%) in Q1.

Because of a server added in July 2013, CPU utilization should not be an issue for several quarters.

The graph below shows that CPU utilization will grow minimally over time as new WebLogic and Web Services applications come online, staffing increases and other changes occur.

VBA 7 January 2020

4.2. Transaction Load

Growth year over year has effectively remained flat. (This looks like a seasonal work variation.)

Infrastructure Operations at Hines is planning an environment that will support 4.0 million transactions per hour. Currently, peaks of 3.7M transactions per hour are occurring in production.

There is a backlog of cases at the Regional Offices (RO). RO throughput has increased significantly with the introduction of VBMS and overtime. The major growth is seen in VBMS and SHARE. SHARE is an efficient application for the most part. Given the current growth for

VBAPROD, the system should withstand this additional load. Tuxedo middleware tuning is an ongoing effort to manage the additional load. The Tuxedo hardware has been upgraded as the servers exceed 40%. The logical configuration for the Tuxedo middleware needs to be modified depending on which transactions are growing or shrinking. The primary drivers of new load are:

1) Web Services applications such as VBMS, eBenefits and VRM

2) Increased WebLogic applications

3) New claims

4) Access by the VHA

Historically, the most pronounced effects of additional caseload are expected in Awards, SHARE, RBA and MAP-D. Awards and RBA are CPU intensive due to application design.

More veterans receiving benefits will drive growth in these transactions.

Batch loads will continue to increase as more veterans’ claims are processed by VETSNET and

VBMS. This will increase due to the number of records processed by batch jobs and potentially new batch jobs.

VBA 8 January 2020

5. Tuxedo Cluster

The Tuxedo support team created a clustered environment, so that Tuxedo could move off the

Corporate database server. This configuration satisfies two needs:

1) It prevents contention between the database and Tuxedo middleware.

2) It creates a more robust environment that allows fail over if a single Tuxedo node has issues.

There are currently four servers configured to provide Tuxedo services. If any server fails, the load can be moved to one of the other servers. CPU utilization becomes a critical factor in this configuration. Normally if CPU utilization is less than 70%, it is not considered problematic.

However, in a fail over environment if the CPU is over 40%, it becomes an issue. If Server #1 is running 45% CPU utilization and fails over to Server #2 which is running 45%, there will be degradation in service. The fail over server will be running at 90% CPU utilization. With the addition of a fourth node this should not be a problem.

This configuration requires a different set of threshold metrics because of fail over contingencies.

Once the servers exceed 40%, the environment is at a risk if a failure occurs. The data indicates that VBACRPAPP2 & 3 are over 40% CPU utilization. VBACRPAPP1 is about 24%. Data for the fourth node is incomplete. But it appears to be in the 24% randge.

The fourth node has been implemented. The load balancing is not effective. As you can see from the graph below, the load on one server has shifted dramatically. One server experiences

50% less utilization, the other two about 30%. This may be a load balancing issue with

WebLogic.

Note: Data for the fourth node is recently available. We are working with a short sample size.

VBA 9 January 2020

6. Monthly Tuxedo Growth

There is flat year-to-year. The last few quarters have showed steady increases. January, March, August, and September are historically the most active months. This is a seasonal variation.

The following graph illustrates the number of transactions per month over a one-year period.

VBA 10 January 2020

There are 2 key facts to extract from this graph:

1. The valleys in the graph reflect the seasonal nature of the VBA workload. Fiscal Q2 and Q3 are generally very active quarters. This pattern has been repeated for several years with peaks in March and August. The introduction of new applications may affect this pattern.

2. Growth is flat year to year.

Note: It should be noted that the VBA works in a seasonal environment. January, March, August and September are peak months of activity. If the June utilization is considered, there appears to be no problem. For performance and stability concerns, the environment needs to be configured for peak season.

7. Indicators for the Last Twelve Months

This section will cover key indicators of system use over the last twelve months. The topics covered will be CPU utilization, key applications and some web environments. There were two additional holidays in December. The numbers for December will be impacted accordingly.

(Not all web environments are in scope under the current contract.)

7.1. Database CPU Over the Last Twelve Months

CPU utilization is hovering a little over 28%. There is an anomaly in December where CPU reached 38%. The cause is being investigated.

We have reset this graph to reflect the new configuration for VBAPROD. The scale is so much larger as to be meaningless going forward, hence, the reset. The upgrade occurred in mid-July

2013.

Note: The Performance team does not have access to the production environment.

VBA 11 January 2020

Historically, the busy hour in production has been from 10 - 11AM (Central). For the past few years, peak hour has varied somewhat from this tradition. In this graph, we used data for peak hour during the online day. While the peak hour may be at 10:00 AM. The 9:00AM - 2:00PM timeframe is often within a few percentage points of peak.

The wild card in this equation is batch processing. Currently, the batch window has lower average CPU utilization than the daily online transaction load. The batch load will generate more spikes in utilization. Second shift online users may see uneven performance.

Changing characteristics of batch jobs can cause performance and availability issues. A case in point was FAS37. This job historically processed 1,000 - 2,000 records. Due to a change in processes, the job began processing 72,000 records. This impacted every job running and online users. The Performance team, working with FAS support, reduced the batch run time by several hours. Database CPU utilization will not be an issue for several quarters.

7.2. SHARE, LEGACY, COVERS, and CSS Transactions

The graph below portrays the number of transactions per month over a one-year period for the applications SHARE, LEGACY (BDN and BIRLS), COVERS, WEBHINQ and CSS.

VBA 12 January 2020

SHARE is down 14% year over year. (Part of this decline is due to an extra holiday.) The use of the SHARE application extensively through web applications, such as VBMS and EVSS, has driven substantial growth over the past few years. We were seeing a continuous increase in this trend until this quarter. SHARE is the most heavily used application in the VBAPROD environment. Although SHARE transactions are relatively moderate in resource consumption

(.07 sec service times), the sheer volume contributes significantly to CPU load. (The various periods of transaction decline highlight the seasonal nature of the VBA workload.)

LEGACY refers to the calls to BIRLS and BDN, which are primarily generated because of certain types of SHARE activity. Historically, usage tracks closely with SHARE usage. (This

VBA 13 January 2020 correlation has lessened with VBMS.) Legacy transactions tend to be very efficient. This year

LEGACY shows an increase compared to December 2018 except for COVERS.

COVERS is a lightweight application (.06 sec service times) with historically high usage levels but is down 29% for the year. This drop is a result of VBMS functionality that largely replaces

COVERS. This trend will continue as the VBA uses more electronic forms.

Common Security Services (CSS) usage was up 6% for the year. User logons (authentication) and transactions (authorization) drive CSS utilization. CSS provides authentication and authorization services for users. CSS is very lightweight in terms of per transaction resource consumption (.02 sec service times). The project to replace some of the functionality of CSS with Active Directory and SiteMinder is complete in production. 2FA has not reduced this load to date. A new project to use IAM may have an impact.

WEBHINQ is up 21% for the year. The transaction rate has been declining but for Q1 and Q2 has increased. Q3, Q4, Q1 and Q2/19 have seen a return to the trend of declining usage. It executes about 283,00 transactions per month. This functionality is being replaced with other applications.

7.3. MAP-D and RBA Transactions for the Last Twelve Months

MAP-D transactions increased 16% when compared to the previous year. Static volumes but costly transactions (.21 sec average service time) characterize MAP-D. It is still a large consumer of resources on the online system. The primary consumer is cd_custsvclmserver.

Improvements were made for the August 2009 release that reduced CPU utilization, but additional improvements are possible.

The RBA frontend has been replaced by VBMS (VBMS-R). The Tuxedo and database components for RBA continue to be used. RBA usage is up 38%. The RBA application interface has been superseded by VBMS. The backend components are still active. The following charts depict transaction volumes for MAP-D and RBA over the last twelve months.

VBA 14 January 2020

Both MAP-D and RBA utilize the VETSNET architecture, which generates substantial overhead compared to more traditional coding approaches. The VETSNET architecture features a common database layer that utilizes Oracle OCI (Oracle Call Interface) and attempts to perform buffer management for all VETSNET applications, a function that Oracle handles natively. The architecture also makes extensive use of proprietary code generators to create SQL and DB views. The output of the generators is often highly inefficient and exceedingly difficult to decipher. In the near term, tuning opportunities are limited to optimization of DB indexes and the occasional repair of SQL, which is complicated by the reliance on the code generators.

VETSNET is in sustainment mode.

7.4. FAS, Awards and VNCOMM

FAS and Awards (CPONLINE) constitute the most resource intensive subset of transactions in the system, with average service times of .54 and .30 seconds, respectively. The transactions that

VBA 15 January 2020 generate and authorize Awards transactions have service times of 1.5 and 4 seconds, respectively, and cause substantial system utilization.

VNCOMN is shared set of services for the VETSNET architecture. It mirrors Award utilization usually.

The following charts depict transaction volumes for Awards, VNCOMN and FAS over the last twelve months.

Awards is up 30% over last year. Previous measurements on the Performance system indicated an increase in CPU utilization of 14% for each 400/hr increment in the Awards rate. Given the system upgrade, this is a non-issue. Awards rates are in the 3,800/hour range today. FAS volumes are a relatively small percentage of the overall load and decreased by 15% this year.

VBA 16 January 2020

FAS and Awards utilize the VETSNET architecture and share the same performance and tuning issues previously cited for MAP-D and RBA.

VNCOMN is up 120% year over year. These are common transactions that are shared by the

VETSNET architecture. This requires investigation. It is unclear why we should see this increase.

7.5. WINRS Transactions for the Last Twelve Months

WINRS transaction volumes showed an increase of 3% year over year. Future growth will be driven by veteran use of the applications functions. Most WINRS transactions are not CPU intensive. WINRS does not use the VETSNET architecture.

The following chart depicts transaction volumes for WINRS over the last twelve months.

7.6. Web Environment

NOTE: Given environmental changes, monitoring wlsauthen is less indicative of Web usage. A new methodology has been selected. Web traffic as measured from the WebLogic servers for

VBMS will be used going forward.

Web traffic has increased slightly year over year for July and presumably, down for August. The primary drivers are the Transformational applications such as VBMS, CRM, eBenefits, etc.

WEAMS, WSMS, LGY, CH33 LTS, QAWeb and others are minor contributors to the growing usage. Many users, outside of the traditional body of users, are accessing the production environment via WebLogic using Web Services.

Web applications generate some of the costliest SQL transactions observed in the entire system.

A new class of Transformational applications is generating an extraordinarily high volume of transactions through the WebLogic environment to the Corporate Database. As applications

VBA 17 January 2020 outside of the environment access production, performance testing will be very challenging. It only takes one inefficient SQL query to adversely affect performance for the entire environment.

Note: There was an error in the data collection routine which caused January, February and

December data to be unavailable.

8. Issues and Recommendations

The following critical performance issues challenge the VBA currently:

1. The WebLogic load balancing needs to be reconfigured so that fail over will operate more effectively. (See discussion - Tuxedo Cluster)

2. Coordinated performance testing is a missing component of the current process. There is a requirement for all the applications to be tested simultaneously to provide a comprehensive picture of performance.

3. Performance testing of the WebLogic environment should be a priority.

4. WebLogic is driving dramatic growth in several areas. VBMS, eBenefits and VRM are of particular interest. WebLogic is the preferred architecture for new applications. The load for all these applications is not well defined. However, additional capacity has been added. There are three major categories of applications adding to this load:

• Applications that provide access to external users.

• Transformational Applications that reside outside of the Benefits Performance

Portal (BEP) environment but access the Corporate Database through a dedicated

WebLogic cluster as a gateway to production.

• Applications that reside within the WebLogic environment (BEP).

5. Batch processing is growing, and this trend is expected to continue. This has implications for disk utilization and the timely completion of batch processing, backups, and data warehouse extracts. Given this situation, a project to manage the batch window more efficiently would be useful.

6. CSEM is a security application that has been added to the production environment. The transaction rate is undefined at this point. However, testing in the performance environment has showed this to be an extremely inefficient application. Major CPU

VBA 18 January 2020 spikes have been observed due to increased utilization of this application. Extensive use will create performance issues in production. (There is a project in progress to address tuning CSS.)

Note: Given the CSS/AD project, the future of CSEM is unclear.

9. Summary

9.1. Tuxedo Cluster Upgrade

The Tuxedo cluster upgrade has been completed. WebLogic needs to tune load balancing.

9.2. Online Volume

Online volumes have been flat year over year. Applications using SHARE were the primary drivers for higher transactions counts while others declined or were flat.

9.3. Web Services

Applications using Web Services need to be evaluated for impact on WebLogic and the

Corporate Database. Several of these applications are currently out of scope for the FTC performance contract. The performance team has developed a methodology to simulate the workload for VBMS.

9.4. Batch Volume

Batch loads are expected to increase as more veterans are paid by VBA and new jobs are added.

Monitoring key jobs should be an ongoing effort. Batch tuning would be beneficial.

9.5. Tuning Opportunities

Several applications contain numerous tuning opportunities. WebLogic applications like VBMS, eBenefits and VRM should be the focus going forward. VBMS is driving the majority of the traffic in production (>60%).

9.6. Additional Processor Requirements

The new server for VBAPROD (July 2013) alleviates the need for additional hardware for the foreseeable future for the database server. (A tech refresh is delayed for 2019.)

9.7. Monitoring Tools

Monitoring tools are having little impact on VBAPROD due to the large capacity of the machine.