ITAMS Tickets 1-2-08052011.rtf

ATTACHMENT 7-Priority Tickets

Middleware O&M Support Service Statistics (06/01/10 - 05/31/11)

Date Priority Ticket # Issue Resolution 06/01/10 P1 7148524 Users were getting a 9102 error when attempting to login to EUP EDAS.

The portals, applications and Tivoli were checked and no problems. The SOS on duty concurred and the users verified.

06/03/10 P2 7157407 Tivoli auto-generated message "taskName=eup-eserv-maintainRegistration-TPDS violation(s) was being displayed and users could not login to RUP LA1 or LA2.

The problem cleared itself while we were investigating. There was a problem with testing prior to this morning's implementation of WSCR 7003/F5 Redirect to CCG and WSCR 6956/IPE XML G/W Monitoring in STV. It is, apparently, a known issue that occurs regularly between 5:00am-6:00am when testing is in progress.

06/09/10 P1 7182560 The Mail Utility for EUP is not currently working for users nationwide. Users are not receiving the mail that they should through EUP. There is no error message. This is effecting all users that are trying to get their passwords reset. Effecting people nationwide (500,000).

Somehow group permissions were set to root instead of smmsp. The group permissions for sendmail on ms00394 and ms00395 to smmsp were renamed. Assigned to BSMOSTIR.

06/10/10 P1 7185128 Customers reported that EUP was very slow and they were not able to process transcripts. Also, their application menu did not show up after login. In addition, RUP LA was down.

The file system was full due to /opt/dxserver/logs in backend servers. The problem was resolved by deleting the old log files. The ticket has been closed. An automated process is going to be put in place to prevent this from happening in the future. The ticket was assigned to BSMO-STIR and shared with EOPS-SMTI-MSEMB-MSS 06/14/10 P2 7195849 TDS was not available in Nashville, TN. TDS system was displaying error msg: 2106 -"TDS cannot be accessed at this time.".

The problem cleared itself while we were investigating. Assigned to AD-CS-ESERVICES-ESERVICES.

06/14/10 P1 7198354 RUP-SA was down and all applications ("Where is my refund", online payment agreement etc) were not accessible. web server was down although the other 5 servers in Sterling were working fine. In Chicago all webservers were working. web servers were rebooted but server did not The redirection URL was changed in the IIS configuration on Sterling Web Servers. This resolved the issue. The ticket was assigned to EOPS-SMTI-MSEMB-MSS and shared with ENTSERV-SI-PPMO-PORTALS.

06/15/10 P2 7203770 EUP users getting "500 Internal Server Error" on all applications.

Peoplesoft 1_1 and 2_1 were hung with out of memory errors and hung threads so the JVM's had to be killed and started through the WebSphere GUI to correct the issue.

06/16/10 P2 7209904 The Direct links to RUP SA1 were failing -getting "page cannot be displayed" when accessing IRFOF and MODIEIN in SA1.

Restarted the deployment manager on CCGOAP203 and the JVM's restarted properly. Assigned to BSMO-STIR and shared with EOPS-SMTI-MSEMB-MSS.

06/28/10 P1 7244842 Users were getting "TDS SQL error, statement error 908. SQL652 Deadlock timeout". The thread counts were very high on the DB2.

All PeopleSoft servers were taken down. The thread counts went to zero as expected. As the PeopleSoft servers were brought back on line, however, the thread counts increased and there was row level locking on the DB2. The issue appeared to be with the DB2 database. There were 2 tables that appeared to be locking. The reset on the table space locking after the TDS archive job failed did not work correctly. The DB2 group switched the lock to “any” and then back to row level locking and this resolved the issue. Both the PS boxes and the DB2 database were rebooted. Assigned to AD-CS-ESERVICES-ESERVICES 06/28/10 P1 7244417 Customers were reporting that AMS was responding extremely slowly and preventing them from completing tax case processing (taking 3 minutes or more for the AMS screens to update while they are trying to work tax cases). They were not getting any error messages though. This was impacting approximately 80 collection employees.

The EAIB TCC Integration Servers were unable to recover from the AMDAS connection errors that they began experiencing Friday afternoon. In addition, the F5 failover mechanism was unable to detect the degraded service on the TCC Integration Servers and was therefore unable to fail EAIB over to MCC. Long-term solution options are available to the original AMDAS connection errors. In particular, EAIB has been considering elimination of AMDAS from the communication chain because it was a known weak link. Even more importantly in the short-term, the work on providing a more timely and graceful recovery from these kind of errors has been ongoing and will continue by the EAIB team next year. Assigned to EOPS-SMTI-MSEMBMSS and shared with AD-CS-AMS-IB-SUPPDEL.

07/01/10 P1 7264083 User reported that 1040 submission via MeF had 900 submissions that were not being processed by the EAID broker The problem seems to have been due to the CICS production (interface between NAP and Middleware program) The program was cancelled and “cold boot” of the CICS region was initiated. A cold boot of the AOR region was also performed. Assigned to AD-SP-EFILE-EISCHAPT3 and shared with ENTSERV-SI-IA&E-IDT.

07/11/10 P1 7293657 Applications on RUP SA (IRFOF, OPA, MOD IEIN, FSA, EFP and MWP) were experiencing an unexpected outage. WSCR7153 - MeF 6.1: MQ V6.0.2 FP 9.1.0 Upgrade - EUP & RUP-LA was attempted during the weekend build. However, the transmittals also incorrectly included the RUP-SA application servers. This caused a problem with the ICCE applications, causing the outage and necessitating a back-out of WSCR 7153 - MeF 6.1: MQ V6.0.2 FP 9.1.0 Upgrade were backed out, the JVMs were restarted and all appeared to be working correctly. Assigned to BSMO-STIR.

07/11/10

7293688 After the backout of the software patches in WSCR7153 and the restoration of the prior version of the software, external transmitters were receiving "Backend AMDAS error" in the Modernized e-File, Assurance Testing System (ATS) and Production environments (Chicago only).

An IBM SEV1 ticket was opened to redirect all MeF traffic from Chicago to Sterling so that work could be done on RUP LA1 without impacting users. Troubleshooting diagnosed that MQ was not working properly. The symptoms showed the MQ channels were not connecting to the EDAS servers and Business Object (BO) server.. This caused the MEF submission on PRD and ATS to fail. After the MQ clusters were refreshed, the MQ traffics started to flow. However, there was still an intermittent problem that caused MQ channels to drop sporadically. Further troubleshooting diagnosed that the MQ listener was stopped. After the MQ listener was restarted, the MQ channels started running and stabilized and traffic started flowing between web application server and EDAS, and the BO. Traffic was load-balanced between Chicago and Sterling and all was working correctly. Assigned to AD-SP-EFILE-DEV-MEF6 and shared with EOPS-SMTI-MSEMB-MSS.

07/11/10 P2 7293725 TEC was getting LDAP check failures on backend server. Further analysis showed the server to be running at high CPU and running slow. The server received a request around 20:30pm EST Sunday which started to consume the CPU and completed "pegged" the CPU by 21:30pm. This CPU saturation and state of the server caused resident applications to fail such as eTrust A reboot cleared up the high CPU usage issue but, unfortunately it also cleared out any forensic information that was needed to determine which process consumed the CPU. The investigation will continue to research logs and change control to see if it can be determined what the process was. Cross referenced to DITE ticket #76687. Assigned to BSMO-STIR.

07/12/10 P2 7295890 There was an EUP error affecting anyone who tried to communicate to the Mainframe via AMDAS – “Error code: 1151 (EDA Server and AMDAS Timeout error while waiting for reply from EDA DB Server)”. There was also a possible impact on anyone with NEG TIN roles being unable to navigate within EUP. It was observed that the load average on backend server was around 150, which was very high. The System.Cluster transmit queue on the SAS server was full. When we tried to manually open a channel from SAS we received an error.

The server was rebooted to resolve the high load average problem An MQ SME directed the SA to terminate the MQ cluster repository process so the message on this cluster transmit queue could be cleared. After the clear process was executed, the MQ queue manager was recycled to clear out the remaining outstanding uncommitted threads and the channel issues. After the MQ recycle was completed, the channels started to flow. Assigned to EOPS-SMTI-MSEMBMSS and share with BSMO-STIR.

07/13/10 P2 7304257 Tivoli auto-generated message “Unix_JEDA_Process_Missing [(SYSTEM.SEARCH(UNIXPS.COMMAND," MISSING","/opt/jdk1.5. 0_09/bin/java DJEDA_HOME*”)) ON backend server:KUX ()]” was displayed.

JEDA was started on the MS00394 server and the process started successfully. Assigned to EOPS-SMTI-MSEMB-MSS and shared with EOPS-ECC-SOCC-U1.

07/16/10 P2 7318774 User couldn't install the new E-Services System and requested a P2 ticket so that user account could be setup to access MQ Repaired - Production User ID ("ldb") not assigned to the correct UNIX group ("presrv"). WMQ Security access permissions granted to User ID ("ldb") instead of to group ("presrv"). Assigned to BSMO-STIR.

07/18/10 P2 7320042 Tivoli Auto-generated message " UNIX_Process_EDA_DX1_Missing" was being displayed.

The problem cleared itself while it was being investigated. Assigned to EOPS-ECC-SOCC-U1 and shared with EOPSSMTI-MSEMB-MSS.

07/18/10 P1 7319970 Problems with the SAAS Production Server. The Oracle Application Server needs to be restarted from the command line using the startup scripts Stopped and restarted the SAAS OAS Application.

07/18/10 P2 7320044 Chicago jvm web application server was down, affecting MeF and taxpayer data/input.

JVM was bounced and system restarted.

07/22/10 P2 7341939 After a request was processed in TDS/IVES the history displayed “stats failed TTB error communication failure response code 99”. The Thread counts on 2 of the portal application servers ) JVM’s were well over 300. The Peoplesoft ICPanels were maxed out on PeopleSoft backend servers so the servers could not take any work. There also appeared to be an operational problem on Servers were recycled, the applications were recycled on PeopleSoft backend servers and the start-up script was replaced with a known good one. Everything was processing normally and the ticket was closed. The ticket was assigned to AD-CS-ESERVICES-ESERVICES and shared with BSMOSTIR/ 07/25/10 P1 7347628 Message "IRFOF error system unavailable" was being displayed.

An IRFOF Oracle password was locked. It was unlocked but the program did not gracefully recover so the JVM was restarted. The system came back online and was then processing normally and the ticket was closed. The ticket was assigned to EOPS-ECC-T2UB-T2UAS-U2.

07/26/10

7354117 The customer called requesting that P2 ticket 7236146 be closed and a P1 ticket opened. The original P2 ticket stated that "the development center for the CDE program reported: they tried diagnosing program for a problem that Gulf State users are experiencing....they can get in to CDE via EUP but then get kicked out or get login requests (only in the Gulf State area). This problem started on June 23, 2010”. A workaround was put into place at that time, but the original problem was never fixed. It appears that system upgrades may now prevent the work-around from working The user requested that this ticket be closed. They will be working with IBM and AT&T to pinpoint and resolve issue.. Assigned to BSMO-STIR.

07/27/10 P2 7354970 Many transmitters were reporting that they could not transmit returns or pick up acknowledgments and that they were getting a variety of error messages (Chicago – MeF/RUP LA1 only).

A rolling restart was performed on the MeF JVMs on web application servers. Traffic was redirected to Sterling while the investigation was going on so that the problem would not affect users. An IBM SEV1 ticket (#68900826) was also opened so that IBM could participate in the investigation. A MAC address for the ports on the switch that services those load balancers was on the incorrect VLAN. This was corrected and the problem was resolved. It was also noticed that JVMs 3 and 4 on were started and should not have been so they were shut down. The ticket was assigned to BSMOSTIR and shared with AD-SP-EFILE-DEV-MEF6.

07/29/10 P1 7365666 RUP SA (Self Authentication) - Originally opened for "Where's my refund" 10:15 -processing 75/min, app is reporting 15/min”. This was affecting anyone outside of the IRS (but not inside) and expanded to the RUP SA and RUP LA.

ATT found records missing from the DNS server on the ATT side and made the necessary corrections. Assigned to ENNOM-NMCC and shared with AD-CS-CUSTCOMMSYSSUPSEC.

08/01/10 P2 7373786 A security error was preventing access to eServices.

The permission group of the new ldb app/user was changed to allow it to access MQ properly. This is a temporary workaround put in place to set the group to the previous TTB permission group and it resolved the MQ issues/errors. Work will be completed to push an updated package for permanent/corrected permission groups. Assigned to BSMOSTIR and shared with EOPS-SMTI-MSEMB-MII.

08/02/10

7373874 Customers at ECC-Detroit were unable to access IFS through the SAP Portal, the web browser was in an inconsistent state and the route to the subnet address was unavailable. Message "Cannot connect to the Citrix server. There is no route to the specified subnet address" was being displayed.

IBM rebooted servers due to memory errors being seen in logs and the reboot of the servers resolved the issue. Assigned to BSMO-STIR and shared with BU-CFO-IFS-CC.

08/02/10 P2 7376257 eServices TDS users were experiencing a large number of TTB System Errors leaving transcript requests in Product Retrieval in Progress Status.

Transcripts to be processed were reviewed to determine if there is a data issue that is preventing these from being processed correctly. This will continue to be researched until a fix can be implemented. Assigned to AD-CS-ESERVICESESERVICES and shared with BSMO-STIR.

08/03/10 P1 7384724 The Sterling (RUP) Solaris 10 Veritas cluster was failing and rejecting all MeF Users who need session database access. This was also causing RUP LA2 users to get a 503 error when they tried to login.

The cluster fault was cleared, the cluster was then brought online on primary node and database service was restored. Assigned to BSMO-STIR and shared with AD-SP-EFILEDEV-MEF6.

08/04/10 P1 7389230 The Sterling (RUP) Solaris 10 Veritas cluster was failing and rejecting all MeF Users who need session database access. This was also causing RUP LA2 users to get a 503 error when they tried to login.

The cluster fault was cleared, the cluster was then brought online on primary node and database service was restored. Assigned to BSMO-STIR and shared with AD-SP-EFILEDEV-MEF6.

08/05/10 P1 7395376 RTS was slow for 100 employees. They were getting "technical difficulties", response time was really slow and they were getting knocked off of the system.

Server was rebooted and Websphere was brought back up on the server. Assigned to EOPS-ECC-T2OB-T2TSS-T2AS.

08/06/10 P1 7393460 The RTS realtime system was giving "technical difficulties " The error cleared itself while it was being investigated. Assigned to AD-CS-AMS-AS-CS.

08/06/10 P2 7398823 MeF User Authentication Failed Error. External transmitter could not file any MeF returns because he was receiving an Authentication Failed Error when he logged in. User uploaded a new certificate because the old one expired and was able to log in to MeF to submit any returns.

Incorrect Transmitter uploaded the incorrect certificate. A new certificate was uploaded Assigned to AD-SP-EFILEDEV-MEF6.

08/09/10 P2 7403065 MEF system is giving "404 page cannot be found" error message when fillers access the site.

The URL had been incorrectly defined. MEF-A2A-Remote was updated with the correct IP and port number and the user was able to access. Assigned to BSMO-STIR.

08/09/10

6748801 When the user saves an LOA, an error message (F637 Registration Number Cannot Exceed 15 Digits and erroneous dates and activity letters input on the F637 Application Edit Page cannot be corrected. The ExTRAS Data Administrator needs the ability to correct these errors without having to input a MITS ticket for correction.

The R1 3.2.3 package was pushed to production. Assigned to BSMO-STIR and shared with AD-COMP-ENFORCEXFIRS.

08/09/10 P2 7405181 When customers go to "Get Product" within TDS, they get an error 2103 "The TDS system cannot be accessed at this time."

A new TD0SUBB was implemented but the DB2 bind was not done. To speed the resolution and lessen the impact on users, the transmittal was backed out and resubmitted the following day. The DB2 bind was then done and all was working correctly. Assigned to AD-CP-MF-ONLINEDATA&ACCESS.

08/16/10 P1 7425806 Citrix Services were not Functioning on EUP IFS.

The CITRIX Farm servers were hung with Page/Pool errors and possibly some registry settings being too low. The servers were rebooted and IFS users were able to process without any issues. Assigned to BSMO-STIR and shared with BU-CFO-IFS-CC 08/16/10 P2 7427162 AMS was taking 40 seconds to respond to CIS. There were problems with Network Interfaces on backend server reported on T2Web and TEP.

The interface was corrected and brought back up. The issue was with /etc/hosts. There was a duplicate entry for the IP address. The duplicate entry was removed and the interface was brought back up with the correct IP. In addition, ismain on integration server was stopped and restarted. Assigned to EOPS-ECC-T2UB.

08/16/10 P2 7431692 IBM WEB hosting reported that data Servers could not unmount the Oracle Volume. Some users were getting a 503 error – Service not available.

Common services Databases were rebuilt, the JVMs were recycled and end users tested with no problems. The ticket was assigned to BSMO-STIR.

08/17/10 P2 7431392 EUP users were getting a time-out error when connecting to the EDA server. When the Security admin was adding the roles, he got error code - 1151 (EDA Server and AMDAS Timeout error while waiting for reply from EDA DB server).

The MQ channel initiator was not running. It was started and the AMDAS replies are being delivered to EDA. Assigned to EOPS-SMTI-MSEMB-MSS.

08/17/10 P1 7431516 Users were getting an HTTP 404 error when attempting to login to RUP SA2.

The JVM's on web application server were reporting no application available to handle the request. The JVM was stopped and started and that resolved the issue. Assigned to BSMO-STIR and shared with ENTSERV-SI-PPMOPORTALS.

08/17/10

7433410 Users were reporting that AMS was extremely slow at ATSC and some users were timing out.

A restart of ismain on EAIB backend server was performed. The log on ts00153 was then checked and no errors were observed. The customer reported that the Negative ETIN service no longer times out but the response is still slow. Assigned to EOPS-SMTI-MSEMB-MSS.

08/19/10 P2 7442904 Users are reporting that AMS is slow or freezing.

Deployed a new package implementing code changes to eliminate the slowness in EAIB. Assigned to ENTSERV-SIIA&E-IDT.

08/19/10 P2 7443684 Users in Austin, TX .were reporting TDS problem product type, when selecting Wage & Income Doc. when selecting from W-2, TDS is issuing all of the Wage & income doc's & not just the requested transcripts .IVES users are getting more info than It was discovered that the TTB boxes had been rebooted on 08/15/10 and there was a script that launched the LDB still in there. It was removed and the LDB processes stopped. This resolved the problem. Assigned to AD-CS-ESERVICESESERVICES.

08/19/10 P2 7445115 TDS external customers and IVES coordinators were unable to get detail reports from TDS in their SOR box. They would put the order in but it was never returned to the mailbox.

System was failed over to secondary Qmail server and the reports began to flow. Primary Qmail server has had a new hard-drive installed and appears to be working as well. Assigned to 7445115 and shared with AD-CS-ESERVICESESERVICES.

08/20/10 P2 7447466 Taxpayers could not access IEIN on IRS.GOV website Isolated issue to an AMDAS configuration issue. The server has been taken out of the cluster and was no longer accepting production traffic leaving 2 servers to handle traffic through the weekend. Servers were added back into the cluster and all worked properly. Assigned to AD-CSCUSTCOMM-DATSERSEC and shared with EOPS-ECCT2UB.

08/23/10 P2 7453688 EUP customer reported they could login to TDS but were unable to access P10 (PTIN) and MeF applications.

On 08/22/10 the Legacy PTIN Application was shut down for both RUP & EUP . EUP users will have query access (read only) via Query Manager after 8/29/10. Assigned to ADCS-ESERVICES-ESERVICES.

08/24/10 P1 7457067 ITIN RTS users are getting a "Technical Difficulties" message when the submit anything.

The Websphere application was restarted. Assigned to EOPS-ECC-SOCC-U1.

08/29/10 P1 7474294 EXFIRS server mf00426 was in a hung state after installing OBP firmware update IRTT20630 System ms00426 was rebuilt and the Operating System, VERITAS, and Informatics databases were reloaded. Assigned to EOPS-ECC-T2UB and shared with BSMOSTIR.

08/30/10 P2 7479669 Users not able to access EUP in Andover, MD. Unable to get past the login page, can't change the password and getting "page cannot be displayed" message.

The problem cleared itself while we were investigating. Assigned to EOPS-SMTI-MSEMB-MSS.

08/31/10 P2 7484762 EUP customers were getting error message 404 internal error No problems were found. Had user try again and they were able to access. Assigned to EOPS-SMTI-MSEMB-MSS.

08/31/10

7477287 Users in Ogden, UT were sending email via EUP but addresses never received the email. Error message "failed to convert XML to HTML with error 23000, 68 failed system error". Income verification express service use TDS to send Xscripts to registered agents (IVES third party participants) . Only for requests 2002 and older.

Tuning configuration changes WERE made to CLAS IDRS Connection Recovery to once every 3 seconds (versus current 30 seconds setting applied on Wednesday 9/1/2010) and corresponding CLAS Request Timeout in LDB to 1 second after findings were confirmed in EITE with test of this configuration. LDB Code deployed with XML translation fix after successful test by TAD in EITE. Assigned to AD-CSESERVICES-ESERVICES and shared with EOPS-SMTIMSEMB-MSS.

09/07/10 P1 7500762 Customer stated that they could not perform any functions in ITIN RTS after logging into EUP -the options to perform searches and retrieve W7 were not available - only the navigation bar was displayed.

MQ listener was restarted, the backend JVM was restarted and the portal JVM was restarted for proper sequence of application startup. Assigned to BSMO-STIR and sheared with EOPS-ECC-SOCC-U1.

09/07/10 P1 7501331 User gets server error when trying to login. Not a production issue.

User needed to open a 5081 for access. Assigned to BSMOSTIR.

09/07/10 P1 7501364 Modernized E-file system users were getting an authorization error. External transmitters were not able to transmit tax returns into the modernized E-file system.

Server cache box at mcc was restarted and the cache was cleared on the gateway on both portals. Assigned to AD-SPEFILE-DEV-MEF6.

09/07/10 P1 7500042 Customer could login to EUP, but when attempting to access IFS through EUP Citrix Cluster the system was not responding One of the Citrix Web servers was taken out of rotation. After taking the servers from rotation, all of the Citrix Metaframe servers were rebooted to clear the cache and network settings. Assigned to BSMO-STIR.

09/08/10 P2 7508637 TDS was running very slow. Users receiving "internal server error" and "page cannot be displayed".

Problem may have been caused by an Endeavor Package implemented last Friday, 9/03/10. May have had Trace Turned On. Assigned to EOPS-SMTI-MSEMB-MSS and shared with AD-CS-ESERVICES-ESERVICES.

09/10/10 P2 7509366 User said the "EUP is not working" when trying to do transcripts.

The problem cleared itself while we were investigating. Assigned to EOPS-SMTI-MSEMB-MSS.

09/09/10 P2 7518299 TDS users getting "TDS system has encountered an unrecoverable error" @ AUSC.

A patch was implemented into production to correct a problem with connectivity between the LDB servers and the SACS system. The various servers and thread counts (PeopleSoft, LDB, JVMs) were monitored with no problems reported. Assigned to AD-CS-ESERVICES-ESERVICES.

09/13/10 P1 7524775 Customer receiving TDS error" Internal sever error 404 not found".

A patch was implemented into production on 9/13 to correct a problem with connectivity between the LDB servers and the SACS system. The various servers and thread counts (PeopleSoft, LDB, JVMs) were monitored through the night with no problems reported. Assigned to AD-CSESERVICES-ESERVICES and shared with EOPS-ECCSOCC-U4.

09/13/10

7525476 RUP SA1 IRFOF was having intermittent outages and RUP SA Sterling server, IRFOF DSRT Cluster unable to start through task.

The SRVE0017W message appears a very large number of times in the SystemOut log of each of the irfof cluster members on SA1#203, but not on any of the other five WAS'. The temporary measure taken was to stop all four of those cluster members and allow the other eight in SA1 (and the twelve in SA2) to process traffic. Investigation continues and a report on the research will be issued. Assigned to BSMO-STIR.

09/14/10 P1 7527799 EUP IFS not accessible. SAP portal was not responding - producing response "connection is progress".

CITRIX servers needed to be checked and restarted. Assigned to BSMO-STIR.

09/16/10 P1 7541507 TDS data was stacking up and not getting to the participant's mailbox. SOR very slow.

The issues causing the ITAMS was related to the Labor Day shutdown. We had the SA start another instance of the SOR to handle the increased volume of messages some time ago. The monitor script that verifies the SOR application is running needs to be updated to start 2 instances instead of 1. Assigned to BSMO-STIR.

09/20/10 P2 7551522 TDS mail recipients reported that they were not receiving transcripts through the TDS application. Users stated that she could look in history and it see that the mail was read, but recipients state that they hadn't received it.

Our investigation showed that the system was working correctly, so there was nothing to be fixed. The customers stated that they lost transcripts during Sept. 13-15. We can help the customer to search for anything that proves that the transcripts were actually lost upon their request to do so. Assigned to EOPS-SMTI-MSEMB-MSS.

09/21/10 P1 7552763 The IFS Citrix Cluster was not responding and the Webpage for IFS was not being displayed.

All Citrix servers but data servers were rebooted. CR is awaiting approval/funding for the current servers running Windows 2000 and CITRIX 4 to be replaced with upgraded HW running Windows 2003 or 2008. These new servers will also be running a newer version of CITRIX (either 5 or 6). Also, the Alteons have been approved and have been received. They will be upgraded to F5 and are currently being worked in development. Assigned to BSMO-STIR.

09/21/10 P2 7554539 An eServices TDS customer is missing TDS transcripts from Sept 13 - 15 time frame and needs them to be restored.

We cannot recover the lost transcripts requested by customer. The SOR history is only kept for three days after the mail has been marked "READ". The customer will resubmit the transcripts for processing. Assigned to EOPS-SMTI-MSEMB-MSS.

09/23/10 P2 7565725 EUP users are getting error code 101 and then timing out.

The inet process on web application server had to be recycled. Assigned to BSMO-STIR and shared with ENTSERV-SI-PPMO-PORTALS.

09/23/10 P2 7565711 Andover RUP external users reported that RUP was timing out when they were trying to get to the registration page The problem cleared itself while we were investigating. Assigned to BSMO-STIR and shared with ENTSERV-SIPPMO-PORTALS.

09/23/10

7566257 ITIN users were missing links on the Global Navigation bar. They only see Home, Reference, and Close.

JVM and MQ Listener were restarted. Assigned to EOPSECC-SOCC-U1 and shared with BSMO-STIR.

09/24/10 P2 7570048 Customer was receiving TDS error" Internal sever error 404 not found".

PeopleSoft servers were recycled and a patch was installed.

09/26/10 P2 7572397 Outage: RUP SA - Chicago and Sterling FSAD only. When users tried to transfer data back to FAFSA they were getting a Session Time out.

Resolution code definition: INST SW. IRS FSA-D Application Issue. The IRS completed pushing a new build to production which contains a change in the properties. We tested the "Transfer Back" and the "Return to FAFSA" feature on FSAD and successfully returned to FOTW. Assigned to BSMOSTIR and shared with AD-CS-CUSTCOMM-INTAPPDEV.

09/27/10 P2 7575096 Customer reports they are unable to access applications using EUP. EUP running extremely slow. The users get a message that the item is no longer available and tells them to return to the previous active application Originally assigned to EOPS-SMTI-MSEMB-MSS. Our investigation showed that there did not appear to be any problems with EUP. We were able to login and access applications quickly. After looking at the screen shots we felt that the problem was with Efile. The slowness was caused by a communications problem between the mainframe and one of the PeopleSoft servers. The issue was resolved by stopping and restarting a MQ channel on the mainframe. Assigned to AD-CS-ESERVICES-ESERVICES and shared with BSMO-STIR.

10/04/10 P1 7598925 TDS was experiencing the following problems: running slow; internal serer errors; Deadlock errors; timing out; an SQL error execution unsuccessful checklog.

The archive job that runs on Sunday at 6:30am modified the DB2 database to table or page locking to perform an archive of the data and switched the locking back to row level locking once it completed successfully. However, the table\page locking was still in effect, causing database contention on the DB2. The applications on the PeopleSoft servers were stopped long enough to switch the table\page locking back to row level locking which resolved the contention\SQL errors on the DB2 database. In addition, one of the seven disks on the LDB server became disconnected from the operating system. The disk had to be re-seated to be recognized by the operating system. The server was being restarted and will be monitored to insure it is operating properly. Assigned to AD-CS-ESERVICES-ESERVICES.

10/12/10 P2 7623994 Although EUP was working, numerous customers could not access IFS after the Columbus Day power outage.

Recycled the active directory servers. and then web servers were reboot along with a data server. Assigned to EOPSSMTI-MSEMB-MSS.

10/12/10

7624391 Customer could login to EUP, but when attempting to access any applications they were getting an internal server error -"server encountered internal error or misconfiguration and can't complete your request"..

MEF SAN storage had corrupted volumes and had to be recovered by a vendor (this prevented the file system from coming online when the database servers were rebooted), the Cache Management servers had to be brought back online, 29 of the 32 gateways in the portals had to be recycled and two listeners on the Broker server had to be enabled.. Assigned to EOPS-ECC-T2UB.

10/12/10 P1 7624513 MeF external transmitters could not transmit production returns into the MeF system.

The cache was cleared, the listener was turned off and the XML gateways were rebooted. Assigned to EOPS-LSSIESCPB-T2SM and shared with BSMO-STIR 10/18/10 P2 7646786 Department of Education Return URLs were incorrect and users/testers could not return to the FAFSA.

New URLs and code successfully deployed into DSRT PROD for FSA-D. Assigned to BSMO-STIR.

10/18/10 P1 7647966 Users nationwide were unable to log into TDS and were getting a 404 error.

Recycled WebSphere JVMs on web application servers.

10/20/10 P1 7660493 TDS was down in Austin Texas.

The JVMs were bounced on 1 app server and 1 PeopleSoft server was recycled. The problem was resolved when maintenance was performed on the servers. Assigned to ADCS-ESERVICES-ESERVICE and shared with BSMO-STIR.

10/21/10 P1 7661615 IFS users were not able to connect to the CITRIX server.

Outage was related to a CR being implemented and implementation was thought to be non-intrusive. The work was completed and server was rebooted - users were then able to access the application. Assigned to BSMO-STIR and shared with BU-CFO-IFS-CC.

10/26/10 P1 7678512 Users were unable to access the EUP application, - entered SEID and password and were taken back to the login screen. Users could not do any research when communicating with taxpayers.

DSA databases were rebuilt for internal users. Assigned to BSMO-STIR and shared with EOPS-ECC-T2UB.

10/26/10 P2 7680922 IBM web hosting reported that IBM had no access to Martinsburg servers whatsoever.

Power cable on media converter chassis in CR2 was disconnected. Reconnected power cable and connections were restored. Assigned to EN-NOSS-MARTBRG.

10/27/10 P2 7683419 Symantec Anti-Virus Console was not reporting.

Closed with No Trouble Found. Alerts were delivered properly, but only EICAR had been received. Concern from CSIRC was that actual threat alerts were not being received. However, all Parent Servers working properly. Assigned to EOPS-SMTI-MSEMB-EMS.

10/29/10

7691025 Two separate RUP LA issues. RUP LA2 users were experiencing slowness and getting an error message when logging out. In addition, RUP LA1 users could not access RUP LA1 and were getting "Page cannot be displayed" message. Initially, the Construction Page could not be displayed but then the Webagent couldn't communicate with the SiteMinder Policy server and users For the RUP LA2 issues, JVM web application server was restarted and the issues were resolved. For the RUP LA1 issues, IBM restarted the webservers and the construction page could then be displayed. After the RUP policy servers were restarted one by one on MTC machines, the communication problem was resolved, and users could login. Assigned to BSMO-STIR.

11/03/10 P2 7707520 AMS was very slow creating Cases @ 900 Chelmsford St, MA ) servers were slow).

The problem cleared itself while it was being investigated. Research of logs will continue even though response times have returned to normal. Assigned to EOPS-ECC-T2UB and shared with AD-CS-AMS-IB-SUPPDEL.

11/08/10 P1 7720775 Users were able to login to EUP, but were reporting that they were getting an “Internal Server Error” when attempting to access applications from their EUP menu.

The Oracle Password had expired and was reset for the Oracle user. It was then discovered that the LDAP certificate had expired when attempting to administer JVMs. We had to: Disable Global Security; Update the JDBC Objects; Set Authentication Password; Recycle WebSphere Infrastructure; Recycle Application JVM's; Generate new Certificate for the LDAP user in WebSphere. Assigned to EOPS-ECC-T2UB and shared with BSMO-STIR.

11/09/10 P1 7726035 User received the following error while attempting to update A2A certificate in automated enrollment in the EUP: Error code: 401 (We are unable to process your request. Please contact the IRS Help Desk 866 255 0654 with the error code on the a new certificate was generated and loaded on VDEXML97 and loaded to the certificate store on mtc0078 and mtc0079. Restarted the JEDA process on both servers. This was not a production issue. Assigned to EOPS-SMTI-MSEMB-MSS and shared with BSMO-STIR.

11/10/10 P2 7733672 Users were receiving intermittent errors from MeF MEF JVMs were restarted on all SWAS servers in both Portals. Assigned to AD-SP-EFILE-DEV-MEF6.

11/17/10 P2 7753172 EUP was very slow at the ehelp desk in Austin, Texas.

After viewing the EUP JVM logs it was found that numerous jolt errors were occurring on the PeopleSoft boxes, primarily on PS05. Wintel SA restarted the PS05 Server and it was then looking good. Assigned to EOPS-SMTI-MSEMB-MSS and shared with AD-CS-ESERVICES-ESERVICES.

11/17/10 P2 7752218 PTIN tutorials have not been removed from the e-services tutorial available via the EUP and RUP as requested.

The tutorials were updated in both the EUP and LA1/LA2 portals. Assigned to BSMO-STIR

11/19/10

7760317 User reported that the ITIN (RTS) Real time system was down in Austin , TX.

The Websphere application on web application servers1 was restarted and the ITIN JVMs in the EUP Portal were restarted. After this both web application servers were up, but not working well (W7 submissions were still getting the same "Technical Difficulty" error) and the ITIN backend log was showing a database error when committing (duplicate key exception). Dropped and recreated sequence ITIN.ITIN_SEQ and SEED was set to coincide with first ITIN number in 989-81 series so as to bypass all the 989-80's which were incorrectly duplicated in the W-7 table. This allowed new work to be input, but some cleanup will be required to fix the work input today. The ticket was assigned to AD-CS-IF-IA and shared with EOPS-ECC-DMB-DAST2UDA..

12/02/10 P2 7796408 Users in Austin, SC reported that ITIN RTS was running very slowly. While we were investigating access became completely unavailable.

Both front and backend servers were restarted but that did not solve the problem. The problem was caused by a lack of memory on backend server which caused the service group to fail on backend servers. The service group was brought up, but noted a concurrency violation and faulted the listener resource. It appeared to backend server that backend server was still running the listener, probably because backend server was unable to shutdown properly due to the memory issues. The issue was resolved by clearing the faults: "hagrp -clear portalsg", shutting backend server down completely, to the ok prompt, clearing the autodisabled flag for backend server, onlining the service group on backend server and booting backend server back up (the VCS and Tivoli endpoint also had to be manually started). Assigned to BSMO-STIR.

12/03/10 P2 7801464 Tester received error code 401 when attempting to replace a certificate in Automated Enrollment (AQT environment).

The DITE network team had to add an additional from the server to the xml gateway to resolve the problem. Assigned to EOPS-SMTI-MSEMB-MSS.

12/06/10 P1 7808500 In Release 6.2 Build Drop 4 Test Environment - EITE Domain 498 & 499, the VIN data was not populated in ETEC/VIN database. This was not a production The ETEC JVMs were restarted. Assigned to BSMO-STIR.

12/08/10 P2 7817418 RUP TDS was very slow and with some getting “internal server error”, getting only partial screens and/or being kicked out of the application. There were no problems with EUP. About 60 users were affected.

One of the three LDB servers that process the TDS requests was not working correctly. The server was not taking TDS messages from the mainframe and processing them. The service was stopped and restarted on the server and transcripts started to successfully process through the system. Assigned to AD-CS-ESERVICES-ESERVICES.

12/09/10

7824101 MeF users are receiving the message "Content Expected" when trying to login to the MeF system. This is starting to affect External users such as State Agencies and Corporations.

Per SME, the Policy Assertions code was sent to Xan Ostro for both WSE 3.0 and WCF to remove Timestamp, Created, and Nonce fields from the WS-Security SOAP header in the login request. Assigned to BSMO-STIR.

12/15/10 P2 7843084 There were several different user problems. MEF system user getting HTTP 401 error (access denied) when trying to transmit returns to the IRS external users. One user was deleted in the LDAP; one user's password had expired thru eservices; another user had a certificate issue.

The individual issues were corrected and the users were able to login. However, even after having the SMAPS cleared a problem with the "change password" function remained. This is a known issue that is being worked thru ITAMS ticket P4 7842522. Assigned to BSMO-STIR and shared with BU-MA-ITSFO-A1-MARTBRG-EUP/RUP.

12/17/10 P2 7850412 States are not able to pick up returns that the state taxpayers are submitting due to virus detection.

Tech was able to disable virus check for forums.. Assigned to BSMO-STIR and shared with AD-SP-EFILE-DEV-MEF6.

12/19/10 P1 7853738 Users nationwide for TPPS systems are not able to log into MEF.

RPR had to update their remote trust store with the newly installed (12/17) certificates.. Assigned to BSMO-STIR.

12/20/10 P2 7853819 Tivoli generated this ticket showing "Unix_Process_EDA_DX2_Missing".

The portals, applications, logs and Tivoli were all checked and no trouble was found. The SOS on duty concurred so the ticket was closed with NTF.

12/20/10 P2 7853818 Tivoli generated this ticket showing "UNIX_Process_EDA_DX1_Missing".

The portals, applications, logs and Tivoli were all checked and no trouble was found. The SOS on duty concurred so the ticket was closed with NTF.

12/20/10 P1 7854846 External users were unable to access the RUP website. This is affecting non-IRS tax professionals (not taxpayers).

The Alteon SSL accelerators now present the Entrust L1C issuer as part of the chain for all LA and SA certificate. Assigned to BSMO-STIR.

12/20/10 P2 7857086 The auto processing of thousands of eSERVICES /TDS transcripts was failing to move inventory to “delivery successful” status.

There was a reinstall last Sunday and the Tivoli task from the IRTT was left in place which added a duplicate line that was causing multiple messages to be sent. The duplicate line was removed during the SRT and the duplicate messages have stopped. Assigned to BSMO-STIR and shared with EOPS-ECC-SOCC-U1.

12/22/10 P1 7864057 E-Help Desk reporting sub-applications within EUP extremely slow nationwide. Experienced Integration Broker Failures and slowness on PeopleSoft backend server.

The Wintel SA's recycled PS02 & 03 due to increased ICPanels. This resolved the issues and the customer agreed the ticket could be closed. Assigned to AD-CS-ESERVICESESERVICES.

12/22/10 P1 7865034 CSIRC requested a P1 ticket be opened for a Cross Site Scripting (XSS) vulnerability problem that has been identified on the Restricted User Portal (RUP) and is directly associated with the on board Sidewinder software.

SiteMinder SME was able to correct the software. Assigned to EOPS-SMTI-MSEMB-MSS and shared with BSMO-STIR.

12/22/10

7862681 Critical ITM_ED6_EDAS_ETRUST_SNMP_STATS0 4 event was received by a TEC server.

Closed with no trouble found. Checked all the DSA's and everything was working fine. Assigned to EOPS-SMTIMSEMB-MSS and shared with EOPS-ECC-SOCC-U1.

12/27/10 P2 7868961 Critical ITM_ED6_EDAS_ETRUST_SNMP_STATS0 4 event was received by a TEC server on MS00394.

Restarted dxserver on server. Assigned to EOPS-SMTIMSEMB-MSS and shared with EOPS-ECC-SOCC-U1.

12/27/10 P2 7869412 IRFOF - Needed to have a new Ear File loaded in the portal in box 203 in Chicago and Sterling and needed to have the configuration file corrected for DSRT testing.

Corrections were loaded, tested and verified. Assigned to BSMO-STIR.

01/02/11 P1 7881520 There is a problem in the EAR file causing web app "Making Work Pay " not to work for taxpayers.

Software was corrected on V DD IRFOF110SUN-TS-PROD2011001. Assigned to AD-CS-CUSTCOMM-INTAPPDEV and shared with BSMO-STIR.

01/02/11 P2 7881514 The cache management server, did not recognize Entrust proxy certificate.

The LOCAL_LISTENER parameter needed to be set for MEFD1. This parameter was unset during upgrade to 11g. Assigned to BSMO-STIR.

01/03/11 P1 7882407 Disclosure Authorization internal/external error message:SDLN generation failed because essential data is missing-error 21000,228 o RUP side(ext.);21000,228 EUP side (internal).

The mapping of the Disclosure Authorization service center codes (each state to the correct computing center) needs to be performed by the business each year. The codes for 2011 were not updated by the business and is causing the error codes in the DA application. The business has updated the codes via the DA Menu on the EUP. Assigned to 7882407.

01/03/11 P2 7886871 TDS requests were getting hung in pending requests There were 3 large files in the data directory on the Depush server that prevented Qmail from writing to the data directory. The files had to be moved from the data directory to a reject directory so that Qmail could write to the data directory and the JAVA Mail Interceptor could pick the files up. Once the files were moved transcripts were processed through the system. The Qmail, JAVA Mail Interceptor and PeopleSoft developers are looking into a solution to resolve the problem. Assigned to BSMO-STIR and shared with ADCS-ESERVICES-ESERVICES.

01/05/11

7894930 MeF users could not change password using the change password web service.

EUP was missing a user directory entry for this area of the User Directory tree. This has now been added and passwords can be changed. The error occurred because of a change in the location of the RUP A2A system account in the eTrust directory. There was an MeF project request to split off the A2A account to its own separate database. The MeF application was then configured to know about the location change in order to support the login activity for those accounts. Even though the application was made aware of the changes the password function, shared services was not. When the password change function failed for this user it was brought to light that the password function could not find the user who was trying to change their password. The dirstory pointer was created in order for the password function to find the user and complete the password change. Assigned to BSMO-STIR and shared with AD-SP-EFILEDEV-MEF6.

01/05/11 P1 7901202

6.2 Build drop 4 Test Environment - EITE Domains 498/499.There is not sufficient space on 498/499 to create the GM and other interface files.

This was a non-production DITE issue. 135G of additional swap space was added to MEF DITE servers. Assigned to BSMO-STIR.

01/07/11 P2 7910589 User reported that EUP “TDS system was not running portal user”. Applications were slow or not responding. System resources were being drained by several field requested report jobs that were stuck in a processing state for several hours.

System resources were being drained by several field requested report jobs got stuck in a processing state for several hours. Once the jobs were stopped the field reported the system regained functionality. Assigned to ADCS-ESERVICES-ESERVICES and shared with BSMO-STIR.

01/07/11 P2 7913367 TDS requests in Pending Delivery Notification was at 45,251 and growing. Also, about 10 minutes prior to this the "messages in queue" was over 22,000.

The data server qmail interceptor was recycled and "messages in queue" and TDS Pending Delivery Notifications began to go down. Assigned to AD-CSESERVICES-ESERVICES and shared with EOPS-ECCSOCC-U1.

01/11/11 P2 7921590 EAIB JVMs were slow, impacting AMS customers, and a restart of the JVMs was necessary.

EAIB JVMs were restarted. Assigned to EOPS-ECC-T2UB.

01/11/11 P2 7922647 User said that "ITIN RTS system is down" and getting an error message that "ITIN IS EXPERIENCING TECHNICAL All the indexes on the system were rebuilt. Assigned to EOPS-ECC-DMB-DAS-T2UDA and shared with AD-CS-IFIA.

01/12/11 P1 7925954 Customer reported IFS system is down in Beckley FC.

This was an application problem and should not have been a P1. A correction was made to the application and Subsequent Transports are importing into IFS. Assigned to BU-CFO-IFS-PRODSUPT and shared with BU-CFO-IFSCC.

01/14/11

7937196 E-Trust production server-ms00394 down, affecting EUP/RUP Authentication.

Had to have data restore and databases copied. Assigned to EOPS-ECC-T2USB-UTSS-UTSU4 and shared with EOPSSMTI-MSEMB-MSS.

01/15/11 P2 7940692 The A2A login PW and CERT authentication continue to be a problem.

This was a non-production DITE issue. Backups have been stopped,. Assigned to BSMO-STIR and shared with AD-SPEFILE-EIS-CHAPT3.

01/18/11 P2 7944509 BULK TIN Matching requests were failing to go to the SOR mailbox.

Updated SOR application code used in the XML processing of the messages to allow for receiving the message in multiple chunks instead of just one chunk. Deployed through the DITE environments and into Production. Assigned to BSMO-STIR and shared with AD-CS-ESERVICESESERVICES.

01/22/11 P1 7964735 IFS Processing W2s problem - users were unable to connect thru IFS portal as 'EUP' session was timing out after 3 minutes IBM restarted citrix web servers. Assigned to BSMO-STIR and shared with EOPS-ECC-T2UB-T2UAS-U4 01/23/11 P2 7964818 MeF external transmitters were receiving "backend AMDAS errors in both the production and test environments".

The vendor (Oracle) determined that there was a disk drive problem. The Martinsburg SA's mirrored the one operational disk with the backup environment and turned the system to the application team for testing. The application team tested out there procedures and were satisfied with the work. Since we were baselined and operational, the Oracle SE was told that the disk could be replaced on Monday morning. Assigned to EOPS-ECC-T2UB-T2UTS-U4 and shared with BSMO-STIR.

01/24/11 P2 7967551 The BOE AE Exceptions report does not contain data. The following error was received at least 3 times in EITE AE on 1/24/11: Error code: 102 (We are unable to process your request.