Appendix_H_-_NIMS_Production_Support_Management_Plan_v1_2013.04.25.pdf

Food and Drug Administration

Computational Science Center (CSC) Tool

PRODUCTION SUPPORT MANAGEMENT PLAN

Document Version Number: 1.0

Document Version Date: 4/25/2013

VERSION HISTORY

Version Number

Implemented By Revision Date Approved By Approval Date

Description of and Reason for Change

1.0 Justin Scott 4/25/2013 ‐ ‐ Generic PSMP

CSC Tools Production Support Management Plan

Version 1.0 Page ii

Contents

1.0 INTRODUCTION

2.0 OVERVIEW

3.0 ROLES & RESPONSIBILITIES

3.1 Production Support Team Roles

3.1.1 Production Support Analysts

3.1.2 Production Support Manager

3.2 Production Support Team Members

3.3 Triage Schedule

4.0 COMMUNICATION REGULATIONS (EMAIL & PHONE)

4.1 Email Protocol

4.1.1 Email Folders

4.1.2 Incoming Messages

4.1.3 Outgoing Messages

4.2 Phone Protocol [Under Construction]

5.0 HP SERVICE MANAGER (HPSM)

5.1 Service Desk Interactions

5.2 Assignment Groups

5.3 Severity, Priority, & Resolution Timeframes (SLAs)

5.4 Updating Titles (Brief Descriptions)

5.5 Updating Ticket Statuses

5.6 Updating Journals

5.7 Closing Tickets

6.0 PROCESSING INCIDENTS

6.1 Incidents Reported to ERIC

6.2 Incidents Reported to CDER‐xxxx Help

6.3 Acknowledging Receipt of Incident

6.4 Triaging Incidents

6.4.1 Training Requests

6.4.2 Change Requests & Defects

6.4.3 Data-Related Requests

6.5 Resolving Incidents

7.0 TICKET ADMINISTRATION

7.1 Creating a Ticket

7.2 After Receiving Ticket

7.3 Sending Messages

7.4 After Receiving Message

Version 1.0 Page iii

7.5 Investigations

7.6 Closing Ticket

8.0 METRICS & TRENDING

8.1 HP Service Manager Metrics

8.2 Email & HP Service Manager Metrics

8.3 CSC Service Level Objectives (SLOs)

9.0 PRODUCTION SUPPORT RESOURCES

10.0 GLOSSARY

Tables Table 1: Production Support Roles and Responsibilities

Table 2: Production Support Team Members

Table 3: Production Support Triage Schedule

Table 4: Messages From External Sources

Table 5: Messages From Internal Sources

Table 6: HPSM Assignment Group and PST Email Distribution List

Table 7: SLA Response and Resolution Times

Table 8: HP Service Manager Ticket Statuses

Table 9: HP Service Manager Fields

Table 10: HP Service Manager Metrics

Table 11: Email and HP Service Manager Metrics

Table 12: Production Support Resources

Table 13: Glossary of Terms

Version 1.0 Page 1

1.0 INTRODUCTION

This document outlines the Production Support policies, processes, and procedures that will be used by the Computational Science Center (CSC), the Office of Business Informatics (OBI), and the contracted support through [Vendor] for managing, maintaining, and supporting the [Computational Science Center Tool]. This Production Support Management Plan will cover roles and responsibilities, communication protocol, an overview of the tracking tool, processes for incident resolution, administrative duties, as well as production support metrics and trending.

2.0 OVERVIEW

The goal of [Computational Science Center Tool] production support is to achieve the highest level of customer satisfaction through the best customer service experience we can possibly provide our end‐ users. We aim to provide the most thorough and timely resolution in the most efficient manner possible. [Computational Science Center Tool] will be supported [Day of Week] through [Day of Week] from [Time] to [Time] Eastern Time, according to Section [Section ID] of the tool’ Statement of Work (SOW) associated with contract [Contract Number].

The production support team will utilize the HP Service Manager (HPSM) software that is used by the OITSS Call Center to track trouble calls and to report its resolutions. End‐users who encounter a problem can contact the tool‐specific email account directly or can submit an incident ticket to the Employee Resource and Information Center (ERIC), who would then be responsible for Tier 1 support.

Regardless of which avenue the end‐users choose to get support through, the [Computational Science Center Tool] Production Support Team (PST) will provide Tier 2 and Tier 3 support for their respective tool.

3.0 ROLES & RESPONSIBILITIES

All production support team members are expected to thoroughly document all relevant information, communications, and activities performed to resolve end‐users’ issues and inquiries. When needed, this documentation should include status and results of communicating with external teams to resolve an end‐user’s incident. Examples of communicating with external teams may include:

Referring the end‐user to the CDER‐CSC‐Training (CDER‐CSC‐Training@fda.hhs.gov) team to inquire about training questions/needs.

Referring the end‐user to the CDER eData (eData@fda.hhs.gov) team to resolve data questions for NDA applications.

The resolutions from these communications will still need to be captured and documented in HP Service Manager by the Production Support Team. With very few exceptions, all Production Support Team members will have to log all support activities (and possibly those of others) for reporting and trending purposes.

3.1 Production Support Team Roles

Reported issues and inquiries, referred to in this document as “incidents,” are assigned to their respective production support team member based solely on the nature of the incident. Incidents

Version 1.0 Page 2 should be assigned to specific production support analysts based on the criteria outlined in the Production Support Roles and Responsibilities table below:

Table 1: Production Support Roles and Responsibilities

Role Supported By Responsibilities

Office of Business Informatics (OBI) Roles

SCST Team Leader OBI Approving authority and accountable for OBI employees and resources.

MAED Project Team

Production Support Manager

OBI

Develop all production support processes and documentation.

Subject Matter Expert on all support process‐related questions (versus tool‐related).

Run production support status reports, check metrics, and look for trends.

Responsible for improving process if deficiencies are determined.

Business Lead CSC

Has final sign‐off on all decisions regarding the business process and user needs.

Subject Matter Expert for the application and its data.

Tier 3/Tier 4 support for incidents that could not be answered through other channels.

Project Manager OBI

Ensure completion of project’s production support tasks.

Assist in reviewing production support processes and documentation.

Vendor Production Support Team

Vendor Program Manager

[Vendor] Accountable for Vendor employees and resources.

Vendor Project Manager [Vendor]

Contract requires vendor(s) to manage daily production support tasks

Oversee vendor’s production support team.

Escalate Priority 1/Severity 1 incidents.

Tool Support Analyst [Vendor]

Provide answers to basic issues and inquiries.

Triage and escalate incidents that are beyond skillsets.

“Production Support Analyst” is a broad term that also encompasses Technical Support & Application Support analysts.

Version 1.0 Page 3

Role Supported By Responsibilities

Technical Support Analyst

[Vendor]

Subject Matter Expert for all matters that involve the tool’s computing platform.

Also falls under “Production Support Analyst”

Application Support Analyst

[Vendor]

Subject Matter Expert for all matters that involve the tool’s hardware and network.

Also falls under “Production Support Analyst”

External Support Team(s)

CDER‐CSC‐Training (email account)

CSC

Lead for tool training development, scheduling, and delivery.

Resource for more complicated training‐related inquiries.

eData Support Team OBI Subject Matter Expert and possible resource for data‐ related incidents.

Business Analyst (BA) [Vendor] Assume responsibility for resolving any Change Requests and Defects that are sent to the generic help email account in error.

ERIC Representative ERIC

Provide Tier 1 support for incidents submitted to the OITSS Call Center.

Provide Tier 4 support for incidents submitted to generic help email account which cannot be resolved by the [Computational Science Center Tool] Production Support Team or an associated External Support Team.

3.1.1 Production Support Analysts

The term “Production Support Analyst” is an overarching term that includes the tool support analysts, technical support analysts, and the application support analysts. Production support analysts are the only team members who will be directly assigned to resolving incidents. All other team members are not directly involved with incident resolution but may be involved in other associated processes.

[Computational Science Center Tool] production support analysts interact with end‐users to provide them with information to address their [Computational Science Center Tool]‐related incidents.

Production support analysts should gather all necessary information to fully understand and diagnose the incident. Secondly, they should ensure that what was reported is valid. Different situations may call for different actions from the production support analysts. Some may require an attempt to solve the incident or propose some solutions. Others may just involve gathering information on the problem before passing it along to someone else to resolve for more technical incidents. Each production support analyst should make every effort to ensure the end‐user’s problem is addressed to the best of his/her ability.

Version 1.0 Page 4

3.1.2 Production Support Manager

The Production Support Manager is responsible for establishing the policies and procedures for production support. After establishing this protocol, it is up to the Production Support Manager to monitor the process to ensure that it is being performed properly and that the desired outcomes are being reached. The Production Support Manager is also responsible for generating status reports, analyzing metrics, looking for trends, and entering questions into a Frequently Asked Question (FAQ) database.

o Status Reports – Communicate any highlights and incidents reported within a specified timeframe. Reporting basis is TBD.

o Metrics & Trending – Analyze metrics for incidents submitted in a given period and look for trends.

o FAQ Database Population – Routinely enter all questions submitted to the generic help email account into the [Computational Science Center Tool] FAQ Database. These questions are then used to determine which questions are best suited for a [Computational Science Center Tool] FAQ List.

o Production Support Audits –End‐to‐end review of whether the production support processes are being adhered to properly. Auditing frequency is TBD.

o CDER‐MAED Help Clean‐Up – Clean‐out the generic help inbox and subfolders once per quarter (see section ‘4.1.1 Email Folders’) after making sure that the established production support protocol is being followed.

o Process Analysis & Revisions – The effectiveness of the production support model will need to be reviewed after the model has been implemented and sufficient time has passed.

Ensure that any deficiencies are addressed and that the process is enhanced as a result of any updates.

3.2 Production Support Team Members

The following table specifies who acts in each role on the MAED Production Support Team. Also listed is the contact information for each team member:

Table 2: Production Support Team Members

Role Team Member Email Phone

Office of Business Informatics (OBI)

TBD TBD TBD TBD

[Computational Science Center Tool] Project Team

TBD TBD TBD TBD

Vendor Production Support Team

TBD TBD TBD TBD

External Support Team(s)

TBD TBD TBD TBD

Version 1.0 Page 5

3.3 Triage Schedule

The CSC has established a generic email account for [Computational Science Center Tool], which is listed in the Microsoft Outlook Global Address List as “CDER‐xxxx Help,” with an email address of CDER‐ xxxxhelp@fda.hhs.gov (the alias ‘cderxxxxhelp@fda.hhs.gov’ is also enabled for this account). This email account serves as the main access point for end‐user to find help regarding the [Computational Science Center Tool] application. There are intentions of standing up a phone line directly to the help desk as well. This email account is monitored by the [Computational Science Center Tool] Production Support Team for incidents during the hours outlined in the following table:

Table 3: Production Support Triage Schedule

Day of the Week Hours of Responsibility Primary Triager Secondary (Back‐Up) Monday TBD TBD TBD

Tuesday TBD TBD TBD

Wednesday TBD TBD TBD

Thursday TBD TBD TBD

Friday TBD TBD TBD

4.0 COMMUNICATION REGULATIONS (EMAIL & PHONE)

The CDER‐xxxx Help email account is the preferred method of contact for end‐users seeking assistance with [Computational Science Center Tool]. ERIC does not have the capability to address [Computational Science Center Tool]‐related incidents. ERIC representatives’ ability to provide assistance will be limited to what ERIC has been supplied in a Frequently Asked Questions (FAQ) document (i.e., “How do I get access to the tool?”) provided by the Production Support Manager. Any [Computational Science Center Tool] incidents submitted to ERIC will likely push out the timeline on resolving the incident as ERIC will be acting in a middle‐man capacity only. The process for handling incidents differs depending on where the incident was initially routed.

4.1 Email Protocol

Communication for all [Computational Science Center Tool] production support activities will be managed through the CDER‐xxxx Help email account. All incoming and outgoing correspondences should be directed through the CDER‐xxxx Help email account. It provides a single point of reference for both end‐users as well as the production support team members. All communications to and from the CDER‐xxxx Help email account will subsequently be entered in HPSM, as will be discussed later, for tracking, reporting, and trending purposes. Some emails sent to the CDER‐xxxx Help email may have been directed there in error. Some may have been intended for the CDER eData Team (eData@fda.hhs.gov), while others might have been better addressed by the CDER‐CSC‐Training (CDER‐ CSC‐Training@fda.hhs.gov) email account. Regardless, these too must be entered in HPSM. The CSC wants to be made aware of the misdirected messages and will develop a strategy to help mitigate future instances.

It is recommended to develop a coverage schedule to determine who is primarily and secondarily responsible for monitoring the CDER‐xxxx Help email account, as well as the HPSM inbox. Having a single individual managing support at any given time is highly urged. This is aimed at clearly defining

Version 1.0 Page 6 who is responsible for addressing incidents as well as minimizing the likelihood of duplicated work among multiple production support analysts. Based on past experience, we have determined that this is a legitimate problem that has contributed to the timeliness and quality of service provided. The procedures described below are outlined under the assumption that a single production support analyst is managing the CDER‐xxxx Help email account.

4.1.1 Email Folders

There will be two subfolders under the CDER‐xxxx Help Inbox: ‘Works In Progress and ‘Resolved’. They are each reserved for incidents that are in each stage of resolution. Both the incoming and outgoing messages should be moved to the appropriate folder depending on the status of the incident reported in the email.

Incidents that are new or under investigation should have all emails related to the incident moved into the ‘Work in Progress’ email folder.

Once an incident has been resolved and there is no longer any activity to be performed on an email it should be moved into the ‘Resolved’ email folder.

The only emails that should be in the ‘Inbox’ should be new emails that have not yet been copied into HP Service Manager or that are in the process of being entered into HPSM.

All correspondences from resolved incidents will be deleted out of the inbox and subfolders quarterly by the Production Support Manager. The Production Support Manager will delete correspondences for each quarter near the 15th of the following month (e.g., delete January, February, and March’s resolved incidents on April 15th). Messages from ongoing issues will not be deleted as they may need to be referenced. This will help keep the CDER‐xxxx Help email account from becoming overcrowded and unmanageable. All communications should already be stored in HP Service Manager where they can be referenced after they have been deleted from the CDER‐xxxx Help account.

4.1.2 Incoming Messages

A message arrives in the CDER‐xxxx Help inbox.

Upon initially retrieving the message, all pertinent information from the content of the email (which includes the Timestamp Header) should be copied into HP Service Manager. Several fields in HPSM will need to be populated as determined by the email.

o Initial emails will require a new entry in HPSM.

o Subsequent emails should amend existing entries in HPSM.

After all of the necessary information from the email has been entered in HPSM, the email should be marked ‘Read’ in Outlook and transferred into the appropriate subfolder. This will denote that the email has successfully been transferred to HPSM for investigation.

If more information should become available in a subsequent email then the information in HPSM should be updated accordingly.

Table 4: Messages From External Sources

Unread Message has arrived and has not been copied into HPSM or Responded to.

Read Message has arrived and [is being / has been] copied into HPSM but not responded to.

Check Mark Message has arrived and been copied into HPSM and has been responded to.

Version 1.0 Page 7

4.1.3 Outgoing Messages

All messages received by the CDER‐xxxx Help email account should have responses sent from the CDER‐xxxx Help email account. Attach your personal signature in the message so it is clear who sent the email. This will limit the risk of end‐users responding to your individual email account directly and help ensure that messages are addressed in a timely manner should you not be around to respond to your personal email account.

All messages sent from the CDER‐xxxx Help account should Carbon Copy (CC) the CDER‐xxxx Help email account. When copying the CDER‐xxxx Help account correctly, all administrative functions can be performed in the Inbox since it should contain all messages that were sent.

Once the message or response has been sent to the end‐user and the copy has arrived, the procedures for Incoming Messages can be followed:

o The message/response arrives in the CDER‐xxxx Help inbox.

o Upon initially retrieving the message, all pertinent information from the content of the email (which includes the Timestamp Header) should be copied into HP Service Manager.

Several fields in HPSM will need to be populated as determined by the email.

o After all of the necessary information from the email has been entered in HPSM, the email should be marked ‘Read’ in Outlook. This will denote that the email has successfully been transferred to HPSM for investigation.

o If more information should become available in a subsequent email then the information in HPSM should be updated accordingly.

Once the response has been sent to the end‐user, the initial email and the response should have the “Check Mark” selected from the ‘Follow Up Flag’ field in Outlook. This will identify that both the initial email and the response have been completed and fully copied into HPSM.

o The emails should only be marked as ‘Read’ without the “Check Mark” selected from the time the email is copied into HPSM until the response has been sent and also copied into HPSM.

Table 5: Messages From Internal Sources

Unread Message has arrived and has not been copied into HPSM.

Read Message has arrived and [is being / has been] copied into HPSM

Check Mark Message has arrived and has been copied into HPSM.

4.2 Phone Protocol [Under Construction]

The CSC has plans in the works to establish a phone line to supplement the support provided through the CDER‐xxxx Help email account. Once phone support has been established, this document will be revised to include the protocol for phone support.

5.0 HP SERVICE MANAGER (HPSM)

The Employee Resource and Information Center (ERIC) currently uses HP Service Manager (HPSM) web client to track all incidents they receive. We, too, plan to utilize HPSM as the main tracking and reporting system for CSC tool‐related incidents. Certain training and change control incidents are handled through alternate processes. Each incident reported will be tracked in HPSM as its own entry;

this entry is called a “ticket.” As tickets are created in HPSM, they are given their own unique identifying

Version 1.0 Page 8 number called an “Incident ID” using the naming convention “IM#######.” Each Ticket is capable of capturing substantial amounts of information across numerous standardized fields.

While we encourage our end‐users to contact our production support team directly via the CDER‐xxxx Help email account, we must also consider that current employees are accustomed to using the ERIC support process. FDA employees already report computer problems (among other things) to ERIC;

reporting an incident for a CSC tool would be no different in their mind. Should our end‐users contact ERIC directly in lieu of our production support team directly, utilizing HPSM will allow ERIC technicians to forward the ticket to our production support teams with minimal effort.

HP Service Manager (HPSM) is a tool that clearly captures the following information:

o Each time the ticket itself is modified or updated, HP Service Manager notes the update on the Activity Log, a log that notes the person, time, and section of the ticket that is updated.

o Journal information where the assignee enters (free text) any updates or findings related to the ticket (i.e., Communication With Customer, Routine Update, Analysis/Research, etc.)

o End‐user information such as their organization, phone number, and office location.

o The severity (# of end‐users affected) and priority (based on work‐stoppage) of the incident. This allows easy prioritization of tickets.

o Service Level Agreements o Which area of the tool is affected by the incident in the ticket using standardized language. This is captured in the title after the initial production support analyst analyzes the ticket.

5.1 Service Desk Interactions

As previously mentioned, end‐users may opt to contact ERIC instead of the tool helpdesk directly.

Should they choose this route, ERIC will initially try to resolve the incident on their own. Each of these attempts to resolve an incident is tracked on a record called an “Interaction.” Interaction records are also given their own unique identifying number using the format “SD#######”. An Interaction is the pre‐cursor to a ticket. If ERIC technicians cannot resolve the incident themselves, then the Interaction will be escalated by creating a ticket and transferring it to the [Computational Science Center Tool] Production Support Team.

5.2 Assignment Groups

HP Service Center is structured in a way that each tool essentially has its own “folder” where all of their tickets are stored. These folders are called ‘Assignment Groups,’ however they are sometimes also referred to as ‘Queues.’ Each CSC tool’s assignment group is supported by the respective production support team. HP Service Manager does allow ‘read‐only’ access to all assignment groups for nearly all HPSM accounts. However, only members of each CSC tool’s production support team should access their respective assignment group. A distribution list has been created in Outlook for the [Computational Science Center Tool] Production Support Team to receive HP Service Notifications through (CDER‐xxxx‐PST). HPSM is configured to send a notification to the CDER‐xxxx‐PST distribution

Version 1.0 Page 9 list in Outlook each time a ticket is assigned to the [Computational Science Center Tool] assignment group. A notification is sent to a specific team member when a ticket has been assigned to that individual in HPSM. The table below outlines which team members have HP Service Manager accounts established as well as which members are a part of the CDER‐xxxx‐PST distribution list.

Table 6: HPSM Assignment Group and PST Email Distribution List

Team Member HPSM Account / Email Role

Assignment Group: CDER‐APPS_xxxx

TBD TBD TBD

Distribution List: CDER‐xxxx‐PST

TBD TBD TBD

5.3 Severity, Priority, & Resolution Timeframes (SLAs)

Each ticket has a certain severity and priority value, which can be categorized as 1, 2, 3, or 4, with 1 being the highest severity and 4 being the lowest. In addition to severity levels, there is also a value assigned for priority, Critical (1), High (2), Medium (3), & Low (4).

The MAED Production Support Team shall respond to all outstanding end‐user incidents within the established response time for the corresponding severity level. The table below defines default Severity Level Agreements (SLA) for HP Service Manager. Because the [Computational Science Center Tool] support contract may outline SLAs that differ from those specified within HP Service Manager, the SLAs specified in the contract shall take precedence over the default HP Service Manager SLAs. If this is the case, then the notifications automatically generated by HPSM should be disregarded.

Table 7: SLA Response and Resolution Times

Severity Code

Impact Response Time

Resolution Time

(Urgent)

Global Failure; work halted for many end‐users on a floor, building or entire Center.

Examples: The tool has shut down. Multiple end‐users in building cannot long onto Local Area Network (LAN) or cannot access email.

HPSM: 15

minutes

HPSM: 4 hours

(High)

Single complete failure; work halted for a single end‐user.

Examples: An end‐user is unable to access the tool and is unable to do their work. Individual end‐user cannot access LAN; end‐user’s computer will not boot.

HPSM: 2 hours HPSM: 8 hours

(Medium)

Single problem for an end‐user; end‐user still able to work.

Examples: The uploading of data takes a longer time than expected but still is processed by the tool. The end‐user’s spell check will not run.

HPSM: 8 hours HPSM: 2 days

4 Service Requests & procedural questions. HPSM: 2 days HPSM: 7 days

Version 1.0 Page 10

5.4 Updating Titles (Brief Descriptions)

[Computational Science Center Tool] production support analysts will update the title of the ticket, called the “Brief Description,” so it clearly defines what was reported. Ticket titles should include the general area or category that the incident is related to as well as a brief summary of the specific incident. The ticket title should conform to the following format:

CATEGORY (All Caps) – Brief Summary

(e.g., TRAINING – How to Select Patient Subsets) (e.g., ACCOUNT – New Account Request)

When the ticket is initially created or received, the “Brief Description” will need to be updated; however, if any new findings are made that clarify the incident and change what it was previously identified as, changes to the “Brief Description” should conform to the standard format. The categories will have to account for all tickets routed to the queues. So even if we won’t resolve a Data Load question, for example, if a ticket gets routed to us that is for such a request, then we should have a clear description within the naming convention to account for this.

At this point, the categories for each tool still need to be determined. They will need to be at a high enough level that multiple incidents fall underneath any particular category. Once the list is created it will likely be a running list as new incidents continue to get reported. Some examples of categories would be Account, Error Message, Performance, Training, etc. The CSC is in the process of determining categories, subcategories, & problem types to incorporate into HPSM.

5.5 Updating Ticket Statuses

During the process of resolving a ticket, the production support analyst who is working on a ticket can change the Status of it if they are waiting for a response from the end‐user, other teams, or if the resolution of the ticket depends on assistance from a third party. The following are the different ticket statuses in HP Service Manager and their descriptions:

Table 8: HP Service Manager Ticket Statuses

Status Definition SLA Impact

Statuses Used By xxxx Production Support

Open

Tickets opened in HP Service Manager are initially given an ‘Open’ status. Tickets should not remain in this status for very long. Any update to the ticket whatsoever will reset the status to something else.

Clock is Counting

Transferred Any tickets transferred into the tool’s queue from somewhere else will have a status of ‘Transferred.’

Clock is Counting

(Low) Examples: If an end‐user submits a suggestion or a request for a “nice to have” feature. Relocate a PC; install additional Random Access Memory (RAM); install software upgrade.

Version 1.0 Page 11

Work in Progress

(WIP)

This is the default status for tickets that have been updated. This is the proper status for incidents being actively worked on by a production support analyst.

Clock is Counting

Pending Customer Production support analysts should use this status when they are waiting for additional information from the end‐user.

Stops Clock

Pending Other

This status is to be used when a production support analyst is waiting for additional information or assistance from any party other than the end‐user.

Stops Clock

Closed Once a ticket is closed out after it has been resolved, HP Service Manager will automatically update the ticket status to ‘Closed.’

n/a

Unused Statuses

Accepted n/a n/a

Pending Change n/a n/a

Pending Vendor n/a n/a

Referred n/a n/a

Rejected n/a n/a

Replaced Problem n/a n/a

Resolved n/a n/a

5.6 Updating Journals

There is a “Journal Log” free text field that allows you to enter any important information pertaining to the ticket. Under the Journal Update tab in a ticket, there are 14 update types available for selection.

The update type depends on the nature of the journal entry. Some examples include:

o Customer Contact Initiated – Production Support uses a standard response when initiating contact with the end‐user. This initial contact states that we are aware of the end‐user’s request for assistance and will investigate the matter. The initial contact is also used as an opportunity to request more information from the end‐user if more details are needed to investigate the problem.3 o Customer Interaction – With the exception of the initial end‐user contact, the ‘Customer Interaction’ update type should be used to log all communication with the end‐user. Any emails with the end‐user should be copied and pasted into the ticket journal (with the header – timestamp & to/from/cc), and any calls should be noted with the date/time and a brief description of what was discussed.

o Routine Update – Used when a general note needs to be added to a ticket or a ticket title needs to be updated.

o Research/Investigation – Utilized to note any important information or investigation notes that could lead to a possible solution.

Version 1.0 Page 12

It is important that all relevant investigation findings and correspondences (email, phone calls, or in person) with the end‐users should be entered in the Journal Log in HP Service Manager. The ticket should be regularly updated along the course to resolution.

5.7 Closing Tickets

Once a ticket has a resolution, it may be closed. Please note that the following are acceptable resolutions:

o A resolution has been determined and implemented in the production environment and the end‐user has verified that the incident is resolved.

o An end‐user specifically states in an email that they no longer require assistance and that their ticket may be closed.

o If, after the production support analyst has contacted the end‐user three times over a five business‐day period without receiving any reply, the ticket can be closed due to non‐ response.

o A new release of the tool has been deployed into Production and the incident has been successfully resolved and verified as such. The end‐user has been notified that their issue was successfully resolved but no response from the end‐user is necessary as the resolution should have already been confirmed through independent verification by a production support analyst. Production support analysts are responsible for notifying end‐users and closing tickets after each release for tickets associated with a defect/change request implemented in the latest release.

To close a ticket, click the ‘Close’ button at the top of the ticket. Enter a ‘Resolution’ entry and be sure to click the ‘Save’ button when you are done. It is important to note that whatever information is entered into the ‘Resolution’ field will be sent to the end‐user in an email. Therefore, no information that compromises the integrity of the production support team should be entered in this field.

6.0 PROCESSING INCIDENTS

The processing of an incident varies depending on whether it was forwarded from ERIC or reported directly to the CDER‐xxxx Help email account. The following process flows outline the procedure for properly processing incidents. Actions and decisions in gray boxes are performed by parties outside of the production support team. Actions and decisions in white boxes are completed by the [Computational Science Center Tool] Production Support Team. In places where the flow is broken, different colored boxes denote where the flow later continues.

6.1 Incidents Reported to ERIC

Tier 1 support will be provided by ERIC if the end‐user reports their incident to the IT Call Center rather than the CDER‐xxxx Help email account. In this scenario, ERIC representatives will attempt to diagnose and resolve the incident themselves. If they are unable to resolve the incident on their own, then they will escalate the incident and create a ticket. Once created, the ERIC representative will transfer the ticket to the [Computational Science Center Tool] assignment group, CDER‐APPS_xxxx, where it will be up to the production support team to provide Tier 2 & Tier 3 support.

Version 1.0 Page 13

Very rarely will ERIC be able to successfully address incidents for CSC tools. In the majority of cases, ERIC representatives will transfer the ticket to the [Computational Science Center Tool] assignment group in HP Service Manager. Once transferred, the Production Support Team will have to make sure that the ticket was properly routed to the [Computational Science Center Tool] assignment group. In the event that it was assigned to the [Computational Science Center Tool] assignment group in error, the production support analyst will need to determine if the ticket is related to any CSC tool. If it is not related to a CSC tool, then the ticket should be re‐assigned to the OITSS‐MIS‐ASSIGNED_INCIDENTS assignment group. ERIC will then make sure that the ticket is routed to the appropriate party to ensure that the ticket gets resolved.

Once it has been determined that the ticket was properly assigned to the [Computational Science Center Tool] assignment group, the production support analyst should review the ticket to determine the nature of the incident. The ticket should then be updated to correctly reflect what was reported.

6.2 Incidents Reported to CDER‐xxxx Help

In cases where the end‐user submits their incident to the CDER‐xxxx Help email account, the production support team will be responsible for Tier 1, Tier 2, & Tier 3 support. The production support team will need to copy all incidents reported to the CDER‐xxxx Help email account into HP Service Manager. All information currently available should be copied into HP Service Manager when the ticket is created.

The email protocol for the CDER‐xxxx Help email account should be observed during this process.

Version 1.0 Page 14

It is very unlikely for end‐users to report incidents to the CDER‐xxxx Help email account that are totally unrelated to [Computational Science Center Tool]. However, should this occur, the end‐user should be directed to contact ERIC or directed to the proper channel if it is known.

6.3 Acknowledging Receipt of Incident

The end‐user should be informed that we have received his/her incident and have begun working to resolve it. This notification should be sent as soon as the production support analyst has determined that the reported incident is related to [Computational Science Center Tool] but before any work has begun to resolve the incident. The [Computational Science Center Tool] Project Team has developed several ‘Acknowledgement Templates’ for commonly reported incidents. The production support analyst should check to see if one of these templates can be used for the acknowledgement to the end‐ user. If a template does not exist for the reported incident, the production support analyst will need to draft an acknowledgement from scratch. Depending on the incident, supplementary information can be incorporated into the acknowledgement email. In some cases, the resolution itself can also be communicated in the acknowledgement email. Once the incident has been acknowledged, the production support analyst can being working to resolve the incident.

6.4 Triaging Incidents

The incident should be triaged to the appropriate party as soon as there is sufficient information to determine whose responsibility the incident falls under. The sooner the incident is handed off the sooner it can be resolved. Some incidents that require assistance from external support teams will accidentally be sent to the CDER‐xxxx Help email account by end‐users. When this happens, the end‐ user should be notified that all future requests of this nature should be routed to the appropriate channel as the CDER‐xxxx Help email account is not the correct resource for all incidents related to the [Computational Science Center Tool]. Not everyone involved in supporting [Computational Science Center Tool] uses HP Service Manager. For these cases, the Production Support Team will have to forward the incident to the external support team via email requesting their assistance. All of the pertinent information for resolving the incident should be disclosed in the hand‐off. This will prevent the production support team from acting as a liaison between the ticket in HP Service Manager and the party that’s been delegated to resolve the incident.

6.4.1 Training Requests

The first question posed when triaging a ticket is whether or not it is related to training. This is because certain training inquiries are addressed by the CDER‐CSC Training Team. As previously mentioned, the sooner the inquiry is handed off, the sooner it can be resolved. When an end‐user inquires about the

Version 1.0 Page 15

[Computational Science Center Tool] training, the inquiry should then be forwarded to the CDER‐CSC‐ Training email account for assistance. Once the incident has been handed off, the ticket can be closed.

6.4.2 Change Requests & Defects

A Change Control Process has been established for [Computational Science Center Tool] that is independent of the Production Support Process. However, it is likely that end‐users will submit enhancement requests along with reports of defects to the CDER‐xxxx Help email account. In anticipation of this, the Production Support Process has been developed to accommodate such requests. Any Change Requests (CR) or Defects that have been routed to the CDER‐xxxx Help account in error will be passed off and handled through the Change Control Process. These requests should be forwarded to the appropriate channel and the ticket should be closed.

If the production support analyst determines the ticket is reporting a Change Request or a Defect, the determination will have to be made whether it is a Defect or simply a Change Request. In both scenarios, the production support analyst will forward the incident to the Business Analyst. Change Requests and Defects are processed differently by the Business Analyst. The two are tracked in entirely different tools by [the Vendor] through an internally housed process. After the ticket has been submitted into the [Vendor’s] Change Control Process, the end‐user needs to be informed that the incident he/she reported will be handled through an alternate process from this point forward. Once this has been communicated to the end‐user, the ticket can be closed.

6.4.3 Data-Related Requests

The third and final determination on whether the ticket can be filtered off to an external support group is whether the incident is related to the data, either registering or the content. This does not apply to all tools. For those tools that do not apply, questions related to this filter can be skipped; the incident reported is one that must be resolved by the production support team. Data‐related requests will

Version 1.0 Page 16 involve help from a third‐party support group, the CDER eData Team. The eData Team assists end‐users having difficulty getting data registered/uploaded as well as answers questions about the content of the data itself. In cases where the end‐user is requesting that data be registered/uploaded, the end‐user should be sent the link to the eData Team’s request form. Once the end‐user has filled out the request form, the eData Team will work with the end‐user and the data to remedy the situation. In all other scenarios, the request should be forwarded to the eData via email (eData@fda.hhs.gov) and the ticket should be closed.

6.5 Resolving Incidents

At this stage in the incident resolution process, the ticket should be assigned to an individual production support analyst to resolve. This production support analyst will begin investigating the incident more deeply to get a better understanding of the incident. After investigating further, the production support analyst will again need to determine if the ticket was properly routed to the [Computational Science Center Tool] assignment group. In the event that it was assigned to the [Computational Science Center Tool] assignment group in error, the production support analyst will need to determine if the ticket is related to any CSC tool. If it is not related to a CSC tool, then the ticket should be re‐assigned to the OITSS‐MIS‐ASSIGNED_INCIDENTS assignment group. ERIC will then make sure that the ticket is routed to the appropriate party to ensure that the ticket gets resolved.

Once it has been determined that the ticket can be resolved by the [Computational Science Center Tool] Production Support Team, the production support analyst should make sure that he/she fully understands the incident so that the proper resolution can be determined. After the proper resolution has been determined for the incident, all that’s left to do is work through the steps to implement the resolution. After the incident has been resolved, the production support analyst needs to communicate the resolution to the end‐user. The production support analyst should review the ticket one last time to

Version 1.0 Page 17 make sure that all of the information regarding the incident has been captured correctly in HP Service Manager. Finally, the ticket can be closed.

7.0 TICKET ADMINISTRATION

Ticket Administration is the term used to describe all of the functions associated with a ticket beyond the actual incident itself. Administrative functions will need to be to be performed on tickets in every stage of resolution. In the table below is a summary of the fields in HPSM that are relevant to the production support process.

Table 9: HP Service Manager Fields

Area Field Type Description Required

Customer Information

Service Recipient Fill Form End‐User’s Name Yes

Requested By Fill Form Used only if end‐user is requesting assistance on someone else’s behalf

No

Badge Number Fill Form End User’s Badge Number Yes

Notify Customer By Dropdown Preferred method of contact for end‐‐ user

Yes

Location Fill Form Geographic position of End‐User Yes

Room & Floor Free Text Room & Floor No

Center Fill Form FDA Center No

Organization Free Text Detailed Organization Path No

Email Free Text Email Address No

Phone Free Text Phone Number No

Barcode Free Text FDA Property Number Yes

Incident Categorization

Affected Service Dropdown Broad classification of incident Yes

Category Dropdown General classification of incident Yes

Subcategory Dropdown Moderate classification of incident Yes

Product Type Dropdown Narrow classification of incident Yes

Problem Type Dropdown Descriptive classification of incident Yes

Status Dropdown Current standing of incident Yes

Priority Dropdown Degree of Disruption Yes

Severity Dropdown Extent of Incident Yes

Incident Assignment

Assignment Group Dropdown The team assigned to the incident Yes

Assignee Dropdown The individual within the team assigned to the incident

Yes

Acknowledge Time Timestamp The time the incident is addressed by the individual assignee

No

Incident Description

Brief Description Free Text Short synopsis serving as the title for the incident

Yes

Description Free Text Original summary of the incident.

More thorough and descriptive is

Yes

Version 1.0 Page 18 desirable

Attachments File Upload Upload files related to record (2MB size limit)

No

Update Activities

New Update Type Dropdown Kind of update being posted.

Upon Update

New Update Free Text Thorough and descriptive update to content of ticket

Upon Update

Journal Updates Display Un‐editable field displaying original description and all updates.

Resolution Free Text An update entry that is required when closing a ticket

Eventually

The administrative duties associated with each field will vary based on which stage of resolution the incident is in. The majority of the administrative updates are performed upon initially receiving a ticket from ERIC or while creating a ticket manually. The volume of administrative functions for a ticket levels out after the deluge that occurs when initially handling the ticket. Outlined below are the typical administrative functions that are necessary during or after each milestone of incident resolution.

7.1 Creating a Ticket

Creating a ticket from an incident reported to the help desk inherently has more administrative work associated with it than any other milestone. The tasks specified below should be completed when logging a new ticket. The fields that will always be consistently populated have the appropriate selection specified below.

o Customer Information: Service Recipient, Requested By (if applicable), Badge Number, Notify Customer By, Location, Room & Floor, Center, Organization, Email, Phone, & Barcode= “Not Available”.

o This can be auto‐populated by entering the email address of the end‐user into the Service Recipient field and clicking the ‘Fill Field’ button. This will populate all of the end‐user information automatically.

o Incident Categorization: Affected Service= “IT”, Category= “IT”, Subcategory= “FDA Custom Applications”, Product Type, Problem Type, & Priority.

o Incident Assignment: Assignment Group= “CDER‐APPS_xxxx”, Assignee (based on team roles and responsibilities).

o Incident Description: Brief Description (adhering to naming standardization), Description, & Attachments (if applicable).

7.2 After Receiving Ticket

The required fields will all be populated for tickets transferred to the tool’s assignment group. The main focus for tickets that have already been triaged by ERIC is to ensure the correct information is listed on the ticket. Therefore the journal should be read and the ticket combed over. Based on the content of the journal and how the fields are currently populated, the following may require updating:

Version 1.0 Page 19 o Customer Information: Service Recipient, Requested By (if applicable), Badge Number, Notify Customer By, Location, Room & Floor, Center, Organization, Email, Phone, & Barcode= “Not Available”.

o This can be auto‐populated by entering the email address of the end‐user into the Service Recipient field and clicking the ‘Fill Field’ button. This will populate all of the end‐user information automatically.

o Incident Categorization: Affected Service= “IT”, Category= “IT”, Subcategory= “FDA Custom Applications”, Product Type, Problem Type, Priority, & Status.

o Incident Assignment: Assignment Group= “CDER‐APPS_xxxx”, Assignee (based on team roles and responsibilities), Acknowledgement Time.

o The Acknowledgement Time should be set when the assignee has been selected and they have begun working on the ticket.

o Incident Description: Brief Description (adhering to naming standardization), Description, & Attachments (if applicable).

o Update Activities: New Update Type= “Routine Update”, New Update= “Ticket Administration”.

7.3 Sending Messages

This is the most sensitive part of ticket administration, as it potentially involves the end‐user. When sending messages to the end‐user, or any other party for that matter, only the original description and critical updates should be included. All internal discussion and notes from the investigation should not be communicated to the end‐user.

Very little content should be updated as a result of sending a message; no more information is coming in than is already available. The only ticket updates that should result from a message being sent are as follows:

o Incident Categorization: Status.

o Incident Description: Attachments (if applicable).

o Update Activities: New Update Type, New Update (copy and paste content of message).

7.4 After Receiving Message

Receiving messages requires moderate levels of updates to tickets. They can vary greatly depending on the content. At the very least they will include updates that mirror that of sending messages. The following content may need to be manipulated:

o Customer Information: Service Recipient, Requested By (if applicable), Notify Customer By, Location, Room & Floor, Center, Organization, Email, Phone.

o Incident Categorization: Product Type, Problem Type, Priority, & Status.

o Incident Assignment: Assignment Group= “CDER‐APPS_xxxx”, Assignee (based on team roles and responsibilities), Acknowledgement Time.

Version 1.0 Page 20 o The Acknowledgement Time should be set when the assignee has been selected and they have begun working on the ticket.

o Incident Description: Brief Description (adhering to naming standardization) & Attachments (if applicable).

o Update Activities: New Update Type, New Update (copy and paste the content of the message at a minimum).

7.5 Investigations

Investigations can yield a wide…