EUS_Attachment_Eleven_(11)_-_CSB_Change_Management-Procedure.pdf

CSB specific Change Management Procedures June 11, 2014

Version 1.0 Page 1 of 8

Communications Services Branch (CSB) specific Change Management Procedures

Standard Change Procedure

All CSB production and IT West changes shall be entered in the USPTO ITSM Change Management system (https://itsm.uspto.gov/arsys/shared/login.jsp). The following activities shall be completed prior to the CAB meeting that will review the change.

• If a network change affects multiple switches and could impact the Enterprise, it shall be entered in Rochade, to provide visibility and clarity. This would mean in addition to a CRQ, attendance at the Thursday, 9 a.m. Rochade meeting is required prior to performing the work. If the work is related to a project, an Operational Readiness Review (ORR) meeting is also required.

• Security Impact Assessment (SIA) if hardware or software will be changed.

• Updated network drawings if hardware will be added, removed or re-cabled. The updated network drawings shall be posted to the CSB Sharepoint site (http://usptoocio/OIEO/ISD/csb/Network%20Drawings/Forms/AllItems.aspx)

• Confirm with EMS team that any new devices associated with the change are being monitored by EMS.

• Deployment, testing and rollback procedures

• Submit C3 maintenance mode request (https://ems-web.uspto.gov/ems_app/maintenance_requests.jsp)

• Arrange for AIS testing that is applicable to the change.

• All CRQ shall have individual tasks for implementing the change, maintenance mode, rollback and testing (if required).

• All Enterprise changes, upgrades, etc. shall be classified as “High Impact” even though they may be low risk”

• Once the change has been started, the CRQ must either marked as “Successful” or

“Unsuccessful” in ITSM (once the change has been started, the CRQ cannot be marked as “Postponed” or “Canceled” in ITSM

• All changes that require an outage must be performed during non-production hours o Monday-Friday: 12 a.m. to 5:30 a.m.

o Weekend and holidays: 10 p.m. to 5:30 a.m.

• Most changes that do not require an outage must be performed during non-production hours. Some exceptions are the following:

o Demand maintenance to resolve an outage o Low Risk and limited scope (with Task Manager approval)

• After the change has been completed, the government POC must be on-site the following business day to address any issues

• An IT Support Announcement will be required for most changes that require an outage. If you are unsure if an IT Support Announcement is required, you can ask at the weekly CAB meeting or contact the IT Support Coordinator at ITSupportCoordinator@USPTO.GOV.

All CSB production changes shall be briefed by the change implementer at the weekly Change Advisory Board (CAB) meeting (Tuesdays at 1 p.m. in the MDE 3A44 Conference https://itsm.uspto.gov/arsys/shared/login.jsp http://usptoocio/OIEO/ISD/csb/Network%20Drawings/Forms/AllItems.aspx https://ems-web.uspto.gov/ems_app/maintenance_requests.jsp

Version 1.0 Page 2 of 8

Room or Conference Bridge 571 270-3000 - meeting ID 25449 - password 123). The change implementer should verify their change is on the CAB list of CRQs to review (http://usptoocio/OIEO/PVTD/CR/default.aspx).

Before performing any work, the change implementer shall verify their Change Record (CRQ) is approved and scheduled in ITSM for the planned date.

Before performing any work on production systems, the change implementer shall verify the CAB has approved and scheduled the change for the planned date (http://usptoocio/OIEO/PVTD/CR/Lists/Announcements/AllItems.aspx).

Deployment Procedure

• Contact C3 via e-mail or phone prior to starting work to confirm the affected devices are in maintenance mode.

• 30 minutes prior to the change start time, the Change Implementer shall request C3 capture of EMS and Sitescope status.

• After the deployment is complete, the Change Implementer shall request C3 capture of EMS and Sitescope resolve any discrepancies.

• The Change Implementer shall execute test procedure.

• After any discrepancies have been resolved, the Change Implementer shall ask C3 to remove affected devices from maintenance mode.

After a CRQ task is complete, the Change Implementer shall send a status e-mail to CAB@uspto.gov, NetworkOperationsServices@uspto.gov, CIOCommandCenter@uspto.gov and CSB-FED@uspto.gov. If ITSM is available, update the status of the associate ITSM tasks.

Routine Change Procedure

CSB is in the process of moving routine Change Request Templates to Service Requests.

Examples include the following:

Task Description

ISE exception/port authentication Group printers installation or permitting devices to the ISE profile

IP address assignment IP address requests DNS Host Name assignment - Create static DNS entry Reclaim IP Address Decommission hosts Provide cable label Printing labels Network cable/drop test Investigating network drop and patch cables VQIP updates Host activation/de-activation SOHO router provisioning SOHO installations Switch port activation and description changes

Host activations/de-activations

Add DHCP reservations Static IP address for end users http://usptoocio/OIEO/PVTD/CR/default.aspx http://usptoocio/OIEO/PVTD/CR/Lists/Announcements/AllItems.aspx mailto:CAB@uspto.gov mailto:NetworkOperationsServices@uspto.gov mailto:CIOCommandCenter@uspto.gov mailto:CSB-FED@uspto.gov

Version 1.0 Page 3 of 8

Network drop requests Port activations/de-activations for end users Enable/disable SNMP/Syslog/Debug Network troubleshooting

Service Requests are not reviewed by the CAB but they required approval by Long Lai or Jeff Smith.

Emergency and Quiet Time Change Procedure

The Change Implementer shall provide the CSB task manager the following:

• Change risk

• Impact assessment including any required outages

• CRQ number if available

The CSB task manager shall request required approvals Jeff Smith (CSB) Terri Schenk (ISD) Robert Cobert (OIEO)

Quiet Time does NOT mean that all work stops for 2-4 weeks. Work that does not affect a business area that is in Quiet Time can be scheduled like a Standard Change. Work that affects a business area that is in Quiet Time must be approved by OIEO management and the Business Area Management via the associated IT Liaison.

Note: If the emergency change is approved, the approval will be sent to CAB and Thao Nguyen

(PRMD).

NCM Tasks

• All Cisco NCM tasks that are submitted by Network Engineering shall be approved by a

Network Operations NCM approver.

• All Cisco NCM tasks that are submitted by Network Operations shall be approved by a

Network Engineering NCM approver.

• Emergency changes can be approved via e-mail or verbally by Long Lai (Primary), Tim

Murphy (Secondary) or Jeff Smith (Tertiary).

Version 1.0 Page 4 of 8

Rescheduled CRQs

• If a CRQ must be rescheduled, the change submitter shall update the corresponding

ITSM tasks prior to the next CAB meeting.

• Any NCM tasks associated with the rescheduled CRQ must be canceled and resubmitted

(NCM approval process applies to rescheduled CRQ).

• Before performing any work on rescheduled CRQs, the change implementer shall verify the CAB has reapproved and rescheduled the change for the planned date (http://usptoocio/OIEO/PVTD/CR/Lists/Announcements/AllItems.aspx).

Problems after a CSB Change If there are any AIS problems after a CSB change, the Change Implementer shall brief the CAB with the lessons learned at the next CAB meeting after the change.

Floor Access switch configuration changes – Deployment Sequence Note: deployment should be staggered in Days where applicable. (Minimum pace below, could be slower)

Week 1 Day1

1. IT West

a. Cr-itw-acc-a

b. Cr-itw-acc-b

c. Cr-itw-acc-c

Monitor for 5 days Week 2 Day 1

2. OCIO Beta

a. Cr-mdw-acc-05n

b. Cr-mdw-acc-05s

Monitor for 5 days Week 3 Day 1

3. OCIO

a. Cr-mdw-acc-03n

b. Cr-mdw-acc-03s

c. Cr-mdw-acc-03e

d. Cr-mdw-acc-04n

e. Cr-mdw-acc-04s

f. Cr-mdw-acc-06n

g. Cr-mdw-acc-06s

h. Cr-mde-acc-03n

i. Cr-mde-acc-03s

Week 3 Monitor for 1-2 days

Day 4

Version 1.0 Page 5 of 8

4. Complete MDE and MDW

Day 5

5. RSQ

Day 6

6. CTH, ETH, CPL, TL, CC

Day 7

7. KNX,

Day 8

8. RAN

Day 9

9. REM

Day 10

10. JEF

4.11.

Note: for low risk configuration changes, 5-10 could be combined.

Version 1.0 Page 6 of 8

C3 Procedure for Maintenance Mode Requests as of 6/3/2014

Please use the link below to place Devices/AIS's into Maintenance Mode:

https://ems-web.uspto.gov/ems_app/maintenance_requests.jsp

Sign in with your AD Credentials. Then Click "Click Here to Create a New Request." Below is an Example of where the link takes you to:

Then fill out the Enterprise Management Maintenance Form.

1. Start by filling out the End Date/Time and then fill in Start Date/Time

2. Please be sure to indicate if this is an Emergency request and if the devices are in production.

3. Fill in the CRQ# and Task Number.

4. Then in the Alert Suppression area list the name of the AIS/Devices that need to be in MM.

(Please ensure that you place the entire FQDN device name and AIS- with applications.)

5. Last Click the Update Button and the request will be sent to C3 for scheduling. C3 will schedule your request once the CRQ has been approved.

Version 1.0 Page 7 of 8

Below is an example of a Blank Enterprise Maintenance Mode Form:

Below is an example of a request filled out properly:

Version 1.0 Page 8 of 8

The Enterprise Security Analyst (ESA) team appreciates your joint cooperation on this request. If additional assistance is required, please contact the Enterprise Security Analyst (ESA) team.