Attachment D RTLS Maximo Production Operations Manual.pdf

Intelligent InSites Real Time Location System (RTLS) – Maximo Interface

Production Operations Manual

October 2021

Version 1.11

Department of Veterans Affairs

Intelligent InSites RTLS Maximo Production Operations Manual i October 2021

Revision History Date Version Description Author October 2021 1.11 Version update, correct log file reference Liberty

April 2021 1.10 Version Update, reference RTLS Install Guide for installation instruction

Liberty

September 2020 1.09 Version Update Liberty

June 2020 1.08 Updated:

• Updated software versions

• Updated version numbers

Liberty

October 2019 1.07 Updated:

• Updated software versions

• Updated section cross-references

• Removed reference to Mule

ESB v3.5

• Updated properties

• Updated version numbers

Liberty

August 2019 1.06 Updated:

• SOARD team name to EAWM

• System start-up message

• System shut-down command

• New sites configuration information

• Application error logging details

• Properties (created imbedded spreadsheet)

• Batch Subscription process parameters

Added:

• Description of EAWM team's system monitoring

Liberty

November 2017 1.05 • Removed all sections that were labeled N/A (per VA instruction)

• Updated port numbers in properties file description

DXC

Production Operations Manual ii October 2021

October 2017 1.04 Updated:

• References to quartz scheduler

• Prerequisite software

DXC

February 2016 1.03 Added Appendix C and updated RTLS Properties.

HPES

November 2015 1.02 Subsequent Submission to VA HPES

September 2015 1.01 Subsequent Submission to VA HPES

September 2015 1.0 Initial Submission to VA HPES

Production Operations Manual iii October 2021

Table of Contents

1. System Business and Operational Description

1.1. Purpose

1.2. Scope

1.3. Audience

1.4. References

1.5. Operational Priority and Service Level

1.6. Logical System Description

1.7. Physical System Description

1.8. Software Description

1.8.1. Background Processes

1.8.2. Job Schedules

1.8.3. Dependent Systems

2. Routine Operations

2.1. Administrative Procedures

2.1.1. System Start-up

2.1.2. System Shut-down

2.1.3. Back-up & Restore

2.1.4. Defining a New Site for the Interface

2.1.5. Understanding the Intelligent InSites Event Queues

2.1.6. Adjusting the Periodicity of a Process

2.1.7. Configuring Special Character Processing

2.2. Security / Identity Management

2.3. User Notifications

2.4. System Monitoring, Reporting & Tools

2.4.1. Availability Monitoring

2.5. Routine Updates, Extracts and Purges

2.6. Scheduled Maintenance

2.7. Capacity Planning

3. Exception Handling

3.1. Routine Errors

3.2. Significant Errors

3.2.1. Application Error Log

3.2.2. Application Error Codes and Descriptions

3.2.3. Infrastructure Errors

3.3. Dependent System(s)

3.4. Troubleshooting

3.5. System Recovery

Production Operations Manual iv October 2021

4. Contingency Planning

4.1. Continuity of Operations

4.1.1. Documentation List

A.1. Start and Verify the ESB Application A.2. Subsequent Installation A.3. Configuring SSL/TLS A.4. RTLS ESB Application Component Back-out/Uninstall

Production Operations Manual v October 2021

Table of Figures Figure 1: Components of Maximo RTLS Interface Figure 2: High-Level View of the Interface and Interactions Figure 3: Batch Subscription Process Figure 4: Accessing the Administer Mode on the VA RTLS Toolbar Figure 5: The VA RTLS Toolbar indicating the Show All tab and location of the Maximo Reports Figure 6: Example of a Location Transaction Failure

Production Operations Manual vi October 2021

List of Tables Table 1: Definitions of Cron Fields Table 2: Special Characters Allowed in the Cron Expression Table 3: Characters Not Supported in Version 4.4 Table 4: Sample Message Corresponding to the Steps to Add a Location in Maximo Table 5: Creating, Logging, and Queuing a Reject Table 6: Sample Messages Corresponding to Decommissioning a Location in Maximo Table 7: Message Logged When a Location That Exists Has Been Decommissioned Table 8: Sample Message Corresponding to Updating Assets in Maximo Table 9: Sample Messages Corresponding to Logging Events in RTLS Table 10: Sample Messages Corresponding to Creating an Asset in RTLS Table 11: Sample Messages Corresponding to Moving an Asset in RTLS Table 12: Standard HTTP Status Codes Table 13: Troubleshooting

Production Operations Manual 1 October 2021

1. System Business and Operational Description The Department of Veterans Affairs (VA) is in the process of deploying a Real Time Location System (RTLS) solution nationally to VA Medical Centers (VAMC) and other VA facilities. The VA national RTLS solution is a system of systems employing multiple Commercial Off-The- Shelf (COTS) RTLS systems. One of the primary applications for RTLS is Asset Tracking. The Intelligent InSites RTLS solution provides the ability to track assets that have been tagged as they move throughout a facility and provide valuable insights into workflow, performance, and other metrics associated with asset tracking.

The Enterprise Asset and Work Management (EAWM) project is a multi-year effort to replace VA’s existing asset and service management systems with a single web-based, integrated, enterprise-level system. It leverages a federated, partnership-based enterprise management model to help ensure the needs of Veterans are met. EAWM addresses VA’s critical need for a modernized and integrated asset and service management capability. EAWM is deploying the International Business Machines (IBM) Maximo Enterprise Asset Management COTS solution.

The EAWM Project and RTLS Program have several intersection points. EAWM will eventually become the official system of record for assets and locations (i.e., space definitions). RTLS will become the official system of record for the location of assets. The two systems must communicate and be integrated at specific points via an interface for both systems to have current and accurate information. This integration must also appear seamless to VA end users.

1.1. Purpose

The purpose of this manual is to provide the necessary technical information about the structure and function of the Maximo-RTLS interface to aid technical support staff with the configuration, maintenance, and troubleshooting of the individual interface components that comprise the software solution.

The terms asset and equipment are used interchangeably throughout the manual.

1.2. Scope

This manual describes how to configure, maintain, and troubleshoot the Maximo-RTLS interface. Information about the design including the data elements exchanged between the two systems is covered in the Maximo-RTLS Interface Control Document (ICD). The manual was originally based on VA’s ProPath template which described a full system and support structure.

The Maximo-RTLS interface is an integration between two existing COTS software applications.

To improve navigation and reduce the size of this manual, any heading level 3 or lower heading level from the template where the proposed content is not applicable have been removed. All the original heading 1 and 2 levels from the template remain intact. In addition, any text that was part of the original template and not applicable to the Maximo-RTLS interface has been removed.

1.3. Audience

The audience for this manual is system administrators. This manual provides content related to:

Production Operations Manual 2 October 2021

• ActiveMQ and Mule Enterprise Service Bus (ESB) - For system administrators with a strong working knowledge of Linux, distributed systems, Java Enterprise Edition (JEE), and virtual environments.

• Configuring Secure Sockets Layer (SSL)/Transport Layer Security (TLS) - For systems administrators with a strong working knowledge of both Linux systems administration and enterprise security.

1.4. References

• [Maximo-RTLS ICD] Maximo-Real Time Location Systems (RTLS) Interface Control

Document

• [Maximo-RTLS User Guide] Maximo-Real Time Location Systems (RTLS) User Guide

• [RTLS SDD] Real Time Location System (RTLS) System Design Document (SDD)

• [RTLS EDA] Real Time Location System (RTLS) National Enterprise Data Architecture

• [Cutover Plan] AEMS/MERS-RTLS to Maximo-RTLS Cutover Plan

• [RTLS Asset Tracking Application Usage Document] Real Time Location System

(RTLS) Asset Tracking Application Usage Document

1.5. Operational Priority and Service Level

The scope of the solution addressed by this document is a small subset of the larger VA RTLS system of systems, namely the software interface necessary for RTLS integration with Maximo.

The Maximo-RTLS interface does not have an SLA or a documented understanding of service level needs/expectations.

If there is a disruption of service, then steps can be taken to rectify the problem as described in Section 2.1.1 System Start-up.

1.6. Logical System Description

The Maximo-RTLS interface is a software solution orchestrated by a series of Enterprise Service Bus (ESB) flows. Within the ESB flows, Simple Object Access Protocol (SOAP) and RESTful web service API calls are employed to facilitate communication. Communication with RTLS is accomplished using event queues and Application Programming Interface (API) calls.

Communication with Maximo is managed with a series of SOAP web service methods which are part of the Maximo Integration Framework/Maximo Enterprise Adapter (MIF/MEA). The Maximo-RTLS interface is designed to implement a data synchronization process. The ESB orchestrates read and update operations between RTLS and Maximo to keep asset location data, asset details, and space (i.e., location) data synchronized.

The following figure represents the components of the Maximo-RTLS interface.

Production Operations Manual 3 October 2021

Figure 1: Components of Maximo RTLS Interface

• InSites – the Intelligent InSites API posts and receives messages and persists updates to the Intelligent InSites data store.

• Enterprise Service Bus – Message transformation occurs within the ESB flows. The flows manage the business entities, attributes, and relationships to enable messaging and data exchange for the RTLS-Maximo Interface.

• Maximo – The Maximo Integration Framework/Maximo Enterprise Adapter (MIF/MEA) manages the communication to and from Maximo with a series of SOAP web service methods.

1.7. Physical System Description

The following diagram provides a high-level view of the physical system components.

Production Operations Manual 4 October 2021

Figure 2: High-Level View of the Interface and Interactions

1.8. Software Description

The Maximo-RTLS interface is a software solution orchestrated by a series of ESB flows. The ESB selected for the Enterprise RTLS solution is an open source solution called Mule ESB.

Mule ESB is a Java-based ESB and integration platform for facilitating message routing and data transformation.

A Mule ESB flow is a way to manage or orchestrate services. Flows provide a flexible method to build integrations by choosing building blocks from a standard list of defined components. By separating configuration information from the underlying code using building blocks, building, deploying, and maintaining the interface is easier to manage.

Within the ESB flows, SOAP and RESTful web service API calls are employed to facilitate communication. Communication with RTLS is accomplished using event queues and API calls as described in the Intelligent InSites RESTful web service API Guide. Communication with Maximo is managed with a series of SOAP web service methods which are part of the MIF/MEA. The Maximo-RTLS interface is designed to implement a data synchronization process. Mule ESB orchestrates read and update operations between RTLS and Maximo to keep asset location data and asset details data synchronized. This synchronization is controlled through both scheduled jobs (i.e., batch) which run at a configurable time frequency and outbound message queues on the Maximo and Intelligent InSites systems (i.e., near real time updates).

Additionally, the interface includes some space management functionality. New spaces detected in Maximo generate a corresponding new space to be created in RTLS. Decommissioning of existing Maximo spaces generate an update to inactivate the corresponding space in RTLS to keep space information synchronized.

The following processes comprise the complete feature set of the interface:

Production Operations Manual 5 October 2021

• Equipment Details / New Equipment / Equipment Types

• Equipment Inventory

• Space (Location) Management

For more detailed information about the processes, see the Maximo-RTLS ICD.

1.8.1. Background Processes

The Maximo-RTLS interface suite is comprised of several interface processes that synchronize the data exchange between Maximo and RTLS. The frequency of all processes is established through a single national configuration file. For more information on how to adjust the periodicity of the processes, see Section 2.1.5 Understanding the Intelligent InSites Event Queues. The processes are described below:

• Populate New Asset Details – A process to query Maximo for additional data elements once a new asset is commissioned in RTLS. The additional data elements, once retrieved from Maximo, populate the RTLS asset record.

• Moved Equipment Inventory - Passive and Active tags – A process to update equipment item locations in Maximo based on equipment movement. Maximo is updated with the new location and a timestamp.

• Static Active Tagged Equipment Inventory – A process that queries RTLS for actively tagged equipment items that have not changed locations and uses the last time the equipment item was “sensed by RTLS” to update Maximo with a new timestamp.

• Static Passive Tagged Equipment Inventory – A process that queries RTLS for passively tagged equipment items that have not changed locations and uses the last time the equipment item was sensed by RTLS to update Maximo with a new timestamp. The last time the item was sensed by RTLS is determined by checking an Intelligent InSites custom attribute updated by the RTLS passive tag scanner.

• New/Decommission Spaces – The interface has a single service endpoint for Maximo to push location updates. When a new space is added to Maximo, the interface creates a corresponding location in a designated area within RTLS for manual remediation. When a location is decommissioned in Maximo the interface “inactivates” the location in

RTLS.

1.8.2. Job Schedules

The interface does not rely on scheduled jobs but rather on message stream processing. Specific stream polling intervals are centrally configured for the entire Veterans Health Administration (VHA) in one configuration file. For information on adjusting the stream polling intervals, see section 2.1.6 Adjusting the Periodicity of a Process.

1.8.3. Dependent Systems

The Maximo-RTLS interface connects Maximo to RTLS to permit data to be exchanged between the two systems. The interface code is dependent on the Maximo and RTLS systems being operational and available via the internal VA network.

Production Operations Manual 6 October 2021

2. Routine Operations

2.1. Administrative Procedures

2.1.1. System Start-up

The Maximo-RTLS interface requires the Intelligent InSites and Maximo platforms be operational before the interface is initiated. Follow the steps below to start the interface:

1. Ensure Maximo is up and running by logging into the Maximo user interface.

2. Ensure RTLS is up and running by logging into the Intelligent InSites user interface.

3. Start Mule application under ${MULE_HOME}/bin/mule directory by issuing the following command line command:

./mule start

4. To verify Mule started successfully, check mule_ee.log for the following message:

+ Mule is up and kicking (every 5000ms) +

* - - + DOMAIN + - - * - - + STATUS + - - *

* RtlsDomain-1.3 * DEPLOYED *

* default * DEPLOYED *

* - - + APPLICATION + - - * - - + DOMAIN + - - *

- - + STATUS + - - *

* maximo-1.3.x * RtlsDomain-1.3 *

DEPLOYED *

2.1.2. System Shut-down

The Maximo-RTLS interface is brought to a non-operational state by taking the Mule application offline via the ./mule stop command. Any updates in transit remain in the current persistent queue and resume upon restart. This ensures no data is lost.

Warning: The Maximo-RTLS interface is a national deployment. Taking the RTLS-Maximo Mule application offline will disable activity for every facility connected to the interface.

2.1.3. Back-up & Restore

The interface connects Maximo to RTLS to permit data to be exchanged between the two systems. The interface code does not create, delete, or store data. Therefore, there are no back-up, restore, or rollback procedures for the interface code.

Production Operations Manual 7 October 2021

2.1.4. Defining a New Site for the Interface

The Maximo-RTLS interface is scalable. As new RTLS-enabled sites are ready to deploy the Maximo-RTLS interface, the network connection information is defined in the maximoConfig6.dat file. To establish the new connection, a Reload Locator Service is called.

This allows the connection to be established without having to stop and re-start the Maximo- RTLS application.

The format of the maximoConfig6.dat file is as follows:

VISNxx;Primary station; <sites> ;<time zone>;<Report user>;<hostname>|<port>|<userid>|<password>

Where:

• VISNxx = VISN identifier (CMOP is represented with the string “CMOP” followed by the CMOP numeric identifier)

• Primary station = the primary station that is associated with a VistA instance

• sites = a comma delimited string of stations/sub-stations hosted within the primary station’s VistA instance

• time zone = the time zone designation. Valid values are:

o America/New_York for Eastern o America/Chicago for Central o America/Denver for Mountain o America/Los_Angeles for Pacific

• Report User = a comma delimited string of Intelligent InSites users who will receive reports.

• hostname = the Fully Qualified Domain Name (FQDN) of the Intelligent InSites server

• port = port

• userid = the userid for the Maximo-RTLS application to communicate with Intelligent

InSites

• password = password

Note: The connection to Intelligent InSites is encrypted. The following is an unencrypted example:

VISN23;636;437,438,636,636A4,636A6,636A8;America/New_York;joesmoe;visn23server.med.

va.gov|443|user|password

The following is an encrypted example:

VISN23;636;437,438,636,636A4,636A6,636A8;America/New_York;joesmoe;ENC(rnjTiPVpM5g+f vB0/vdGLibYBtA6kPLUzckiJdiXiwQ3RP3fb5+wZPTsBnnRozQctnOZh+V4EPUB+1R+LEwnjpuwddakzgAb

Follow the steps below to define a new site for the Maximo-RTLS Interface:

1. Edit the {MULE_HOME}/maximoConfig6.dat file properties as follows:

a. If the VISN doesn’t exist, add a new line to represent the VISN and add the hostname, port, userid and password.

b. Add the site.

Production Operations Manual 8 October 2021

2. Save the file.

3. Run the Reload Locator Service: https://{server hostname}/locatorReload

4. The interface will encrypt the file on start up and make a backup of the file.

5. Move the backup file to a secure location.

2.1.5. Understanding the Intelligent InSites Event Queues

The RTLS-Maximo interface uses the Intelligent InSites Event Queue feature. Events in Intelligent InSites can be captured by setting up event queues where each queue controls the type of event being captured. The RTLS-Maximo interface is configured, by default, to receive accumulated events over a sixty-minute timeframe. This timeframe is adjustable (see Section 2.1.6); however, sixty minutes is the maximum timeframe allowed by Intelligent InSites. The event queue remains active for the duration of the timeframe. In addition, if the event queue receives a polling request to pull data from the queue during the sixty minutes, the event queue timer is reset, and a new timeframe begins. If the queue does not receive a polling request within the established timeframe, the queue is destroyed, and accumulated events are lost. The RTLS- Maximo interface recreates a destroyed queue to provide continuous coverage of all Intelligent InSites events the interface monitors.

2.1.6. Adjusting the Periodicity of a Process

The RTLS-Maximo Interface utilizes event queue polling to initiate the interface processes.

Schedules are established using cron expressions. A cron expression schedules a process to run periodically at a fixed time, date, or interval. Each expression is a string comprised of six fields separated by white space in a pattern listing seconds, minutes, hours, day of the month, months, and day of week, respectively.

The following table provides a definition of each cron field.

Table 1: Definitions of Cron Fields

Field Definition

Seconds 0 – 59

Minutes 0 - 59

Hour 0 – 23 where 0 represents midnight

Day of Month 1 - 31

Month 1 - 12

Day of Week 0-6 where 0 represents Sunday

The following table defines special characters allowed in the cron expression.

Table 2: Special Characters Allowed in the Cron Expression

Special Characters Description

Asterisk (*) The asterisk (*) represents every instance (i.e., every hour, every weekday, every month) of a time period.

Comma (,) Comma-separated values can be used to run more than one instance of a particular command within a time period. For example, Day of Month

Production Operations Manual 9 October 2021 expressed as 2,5,22 represents a job that runs on the 2nd, 5th, and 22nd of each month.

Question Mark (?) Question mark is an alternative to ‘*’. It is used instead of '*' for leaving either day-of-month or day-of-week blank.

Slash (/) The forward slash (/) represents increments of ranges. For example, "*/3" in the hour field is equivalent to "0,3,6,9,12,15,18,21" because "*" specifies 'every hour' but the "/3" means every 3 hours.

Examples:

0 0 16 * * 0 = Every Sunday at 4:00 PM 0 0 23 * * ? = Every day at 11:00 PM 0 0/59 * * * ? = Every 59 minutes

If a change is made to the properties file while the interface is running, the interface needs to be re-started. Changes to the properties file are not picked up dynamically.

Follow the steps below to adjust the periodicity of a process:

1. Read the process property descriptions and recommendations in the property file description (see Section 5.6.2 Populate the Maximo-RTLS Property File).

2. Edit the maximo.properties configuration file.

3. Modify the appropriate property.

4. If the RTLS-Maximo interface application is running, re-start the interface to pick up the change.

Note: The maximo.properties file is stored in the {APPROOT} directory which is created during installation (see Installation and Deployment Guide).

2.1.7. Configuring Special Character Processing

For data flowing from Maximo to RTLS, the Intelligent InSites RESTful API does not support the following special characters in version 4.4:

Table 3: Characters Not Supported in Version 4.4

Symbol Definition

‘ Single quotation

“ Double quotation

& Ampersand

The special character constraint has been resolved in Intelligent InSites version 4.5.2.1 and beyond.

The Maximo-RTLS interface contains a configurable parameter that allows the processing of special characters to be turned on and off. The configuration parameter, or property, is called urlEncoding and supports the following values:

• True – represents straight UrlEnoder.encode (with UTF-8 charset)

Production Operations Manual 10 October 2021

• False – represents stripping the above special characters with the exception of the &. The & is replaced with the word “ and ” (i.e. space “and” space).

Follow the steps below to set the property:

1. Edit maximo.properties configuration file.

2. If the InSites version is 4.5.2.1 or later, set the urlEncoding property value to true.

3. If the InSites version is earlier than 4.5.2.1, set the urlEncoding property value to false.

2.2. Security / Identity Management

National Institutes of Technology (NIST)-validated Federal Information Processing (FIPS 140-2) encryption is provided for the encryption of VA sensitive data in transit. The Maximo and Intelligent InSites systems reside behind VA’s firewall in a private network. The communication protocols used are HTTPS/SSL. Access on the ESB, Intelligent InSites, and Maximo servers is protected by account-level username and password which are encrypted using Triple-DES cryptography (PBEWithMD5AndTripleDES) and then encoded as base64 string values. The passwords do not expire.

Data payloads are validated with:

• XML schema

• Common validation routines as appropriate

To ensure the systems are communicating with each other, fault tolerance and peak load management is built in using queuing of requests on both Maximo and ESB.

User management is outside the scope on the interface. System level RTLS security (e.g., national auditing practices) is addressed in the National RTLS System Design Document (SDD).

2.3. User Notifications

The process and procedures for notifying the user community of scheduled or unscheduled changes to the interface is determined by the VA organization responsible for support.

2.4. System Monitoring, Reporting & Tools

The Mule server is monitored by the Nagios IT infrastructure management and alerting system as part of the overall RTLS solution. Nagios generates alerts for memory, CPU, disk, and network issues and the responsible system administration team responds accordingly.

On the Maximo side of the interface, the IBM’s EAWM team is using AppDynamics to monitor the number of transactions received from the interface and has set email alerts for both low and high transaction volume.

2.4.1. Availability Monitoring

The RTLS-Maximo interface application is monitored by the MMC. For more information about MMC, see https://www.mulesoft.com/.

Production Operations Manual 11 October 2021

2.5. Routine Updates, Extracts and Purges

The interface does not create, delete, or store data, so data extracts, database reorganizations, and purging of data are tasks not applicable to the interface code.

2.6. Scheduled Maintenance

The Maximo-RTLS interface has no regularly scheduled maintenance. Secure Sockets Layer (SSL) certificates are renewed annually.

2.7. Capacity Planning

Nagios is capable of monitoring many aspects of server capacity utilization (e.g., CPU, I/O, memory). Some of the relevant planning metrics are already being monitored, others could be configured if needed to facilitate planning activities.

3. Exception Handling

3.1. Routine Errors

Like most systems, the Maximo-RTLS interface may generate a small set of errors that may be considered routine in the sense that they have minimal impact on the user and do not compromise the operational state of the system. Most of the errors are transient in nature and only require the user to retry an operation. The following subsections describe these errors, their causes, and what, if any, response an operator needs to take.

While the occasional occurrence of these errors may be routine, getting a large number of an individual error over a short period of time is an indication of a more serious problem. In that case the error needs to be treated as an exceptional condition.

3.2. Significant Errors

Significant errors can be defined as errors or conditions that affect system stability, availability, performance, or otherwise make the system unavailable to its user base. The following subsections contain information to aid administrators, operators, and other support personnel in the resolution of errors, conditions, or other issues.

3.2.1. Application Error Log

The RTLS-Maximo interface, and more specifically Mule, uses a Java logging framework called log4j. The log file, maximo.log, is defined in the log4j.properties file. The log4j.properties file can be used to configure different logging levels and to enable and disable logging as needed.

The log file captures information about transactions and exceptions that arise while running the interface. The log file is stored in the {MULE_HOME} directory which is established at installation (see Installation and Deployment Guide). As a best practice to avoid the single log file from becoming too large, periodically the log file is trimmed and a new file with a date stamp and sequential number is created. For example, the following log files are generated as maximo.log becomes too large:

Production Operations Manual 12 October 2021 maximo.log maximo-2019-05-06-1.log maximo-2019-05-06-2.log maximo-2019-05-06-3.log In this example, maximo-2019-05-06-1.log is the oldest log file.

For transactions sent from one system that do not successfully post to the destination system (i.e., from Maximo to RTLS or from RTLS to Maximo), the RTLS administrator has access to a report for viewing the details of the failed transactions. Within the RTLS user interface, the administrator selects the appropriate station number and date to view the report. See Section 7 – Appendix C: RTLS Failed Transaction Report for more information.

3.2.1.1. Example Event Log Entries

The subsequent sections provide examples of typical messages that appear in the logs that process events from either system. Messages often appear in request/response pairs (i.e., one component makes a request and another component responds to the request).

3.2.1.1.1. Location Added in Maximo

When a new location is created in Maximo, a request is sent to the interface to create a corresponding location in a designated new parent location within RTLS.

The following messages are logged:

1. When a request is received from the Maximo web services, the SOAP message is logged.

If the message is not logged, then the connection from Maximo is not configured properly.

2. The SOAP message is transformed into an XML message and logged. The message is put onto the esb.maximo.location.events queue in ActiveMQ.

3. When the message is pulled from the queue, the extracted values are logged.

4. Before the location is added, the values to add are logged.

5. If the location does not exist in RTLS, the location is created, and a message is logged.

The following sample messages correspond to the above steps.

Table 4: Sample Message Corresponding to the Steps to Add a Location in Maximo

# Message

1 INFO 2020-09-03 09:46:58,048 WSLocation: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Body> <ser:PublishVA_LOCOUT_INS xmlns:ser="http://service.web.maximo.rtls/" xmlns:xalan="http://xml.apache.org/xslt"> <VA_LOCOUT_INSSet> <LOCATIONS action="update">

<LOCATION>103-77-LN</LOCATION>

<SITEID>636</SITEID>

<STATUS>OPERATING</STATUS>

</LOCATIONS>

Production Operations Manual 13 October 2021

</VA_LOCOUT_INSSet> </ser:PublishVA_LOCOUT_INS> </soapenv:Body> </soapenv:Envelope>

2 INFO 2020-09-03 09:46:58,472 WSLocation: <?xml version="1.0" encoding="UTF- 8"?><location><actionType>update</actionType><stationNumber>636</stationNumber><roo mNumber>103-77-LN</roomNumber><status>OPERATING</status></location>

3 INFO 2020-09-03 09:46:58,535 LocEvent: [636-update-103-77-LN-OPERATING] maximoLocationId[636%7C103-77-LN]

4 INFO 2020-09-03 09:47:01,728 LocEvent: [636-update-103-77-LN-OPERATING] add parent-location=Bxj9x&location-level=Bxl&name=103-77-LN&external-references.maximo-location-id=636|103-77-LN

5 INFO 2020-09-03 09:47:06,321 LocEvent: [636-update-103-77-LN-OPERATING] add success

If the location already exists or an error occurs, a reject is created, logged and queued.

Table 5: Creating, Logging, and Queuing a Reject

# Message

6 ERROR 2020-09-03 10:02:05,433 LocEvent: [636-update-103-77-LN-OPERATING] Location 636%7C103-77-LN already exists

3.2.1.1.2. Location Decommissioned in Maximo

When a location is decommissioned in Maximo, a request is sent to the interface to decommission the corresponding location within RTLS.

The following messages are logged:

1. When a request is received from the Maximo web services, the SOAP message is logged.

If the message is not logged, then the connection from Maximo is not configured properly.

2. The SOAP message is transformed into an XML message and logged. The message is put onto the esb.maximo.location.events queue in ActiveMQ.

3. When the message is pulled from the queue, the extracted values are logged.

4. If the location exists in RTLS, it is decommissioned (ZZ appended to the name) and a message is logged.

5. If the location exists in RTLS, it is set to ‘inactive’ in RTLS and a message is logged.

The following sample messages correspond to the above steps.

Table 6: Sample Messages Corresponding to Decommissioning a Location in Maximo

# Message

1 INFO 2020-09-03 10:05:55,037 WSLocation: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Body> <ser:PublishVA_LOCOUT_INS xmlns:ser="http://service.web.maximo.rtls/" xmlns:xalan="http://xml.apache.org/xslt">

Production Operations Manual 14 October 2021

<VA_LOCOUT_INSSet> <LOCATIONS action="update">

<LOCATION>103-77-LN</LOCATION>

<SITEID>636</SITEID>

<STATUS>DECOMMISSIONED</STATUS>

</LOCATIONS>

</VA_LOCOUT_INSSet> </ser:PublishVA_LOCOUT_INS> </soapenv:Body> </soapenv:Envelope>

2 INFO 2020-09-03 10:05:55,586 WSLocation: <?xml version="1.0" encoding="UTF- 8"?><location><actionType>update</actionType><stationNumber>636</stationNumber><room Number>103-77-LN</roomNumber><status>DECOMMISSIONED</status></location>

3 INFO 2020-09-03 10:05:55,664 LocEvent: [636-update-103-77-LN-DECOMMISSIONED] maximoLocationId[636%7C103-77-LN]

4 INFO 2020-09-03 10:05:55,695 LocEvent: [636-update-103-77-LN-DECOMMISSIONED] decommission maximoLocationId[636%7C103-77-LN]

5 INFO 2020-09-03 10:05:56,367 LocEvent: [636-update-103-77-LN-DECOMMISSIONED] decommission locationId[Bxn5t]

If the location exists but is already decommissioned, then a message is logged and messages 4 and 5 do not appear in the log.

Table 7: Message Logged When a Location That Exists Has Been Decommissioned

# Message

6 INFO 2020-09-03 10:35:42,917 LocEvent: [636-update-103-77-LN-DECOMMISSIONED] previously decommissioned

3.2.1.1.3. Asset Updated in Maximo

When an asset is updated in Maximo, a request is sent to the interface to update the corresponding asset within RTLS.

The following messages are logged:

1. When a request is received from the Maximo web services, the SOAP message is logged.

If the message is not logged, then the connection from Maximo is not configured properly.

2. The SOAP message is transformed into an XML message and logged. The message is put onto the esb.maximo.asset.update queue in ActiveMQ.

3. When the message is pulled from the queue, some of the extracted values are logged.

4. The assigned location (station) and the equipment service status of the asset are retrieved and logged.

5. The equipment type logic starts and the type of entry and the CSN are logged.

6. The uncategorized type id and name are retrieved by CSN and logged.

Production Operations Manual 15 October 2021

7. The type id and name are retrieved and logged along with the uncategorized id and name.

The uncategorized type takes precedence when determining the type for the asset.

8. The values used to update the asset are logged.

9. A success message is logged.

The following sample messages correspond to the above steps.

Table 8: Sample Message Corresponding to Updating Assets in Maximo

# Message

1 INFO 2020-09-03 10:54:24,384 WSAsset: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://service.web.maximo.rtls/"> <soapenv:Header/> <soapenv:Body> <ser:PublishVA_ASSETOUT_INS uniqueResult="ignore" maxItems="1" rsStart="1" creationDateTime="2017-07-17T12:00:00" baseLanguage="English" transLanguage="English" messageID="1" maximoVersion="1"> <VA_ASSETOUT_INSSet> <ASSET action="ignore">

<MAXINTERRORMSG/>

<ASSETNUM>777</ASSETNUM>

<DESCRIPTION>BED-ELECTRIC</DESCRIPTION>

<INSTALLDATE>2017-02-01T12:00:00</INSTALLDATE>

<PARENT>Parent</PARENT>

<PLUSHMODELNUM>100-REF7622</PLUSHMODELNUM>

<PURCHASEPRICE>475.00</PURCHASEPRICE>

<SERIALNUM>P76908-J78KL192</SERIALNUM>

<SITEID>636</SITEID>

<STATUS>OPERATING</STATUS>

<VA_AEMS_MERS_REFID>636 EE19546</VA_AEMS_MERS_REFID>

<VA_CSN>6530-439478</VA_CSN>

<VA_DISPOSITION_DATE>2017-01-01T12:00:00</VA_DISPOSITION_DATE>

<VA_EIL>CMR</VA_EIL>

<VA_MDNS>U$TEST2</VA_MDNS>

<VA_MDNSDESCRIPTION>BED-ELECTRIC</VA_MDNSDESCRIPTION>

<VA_FMDESCRIPTION>?</VA_FMDESCRIPTION>

<VA_PO>500-P348065</VA_PO>

<VA_RESP_DEPT>BIOMEDICAL</VA_RESP_DEPT>

<VA_RFIDTAG>?</VA_RFIDTAG>

<VA_TYPE_OF_ENTRY>NON-EXPENDABLE EQPT</VA_TYPE_OF_ENTRY>

<VA_EIL_RTLS>

<DESCRIPTION/>

<EILID/>

<SERVICE/>

</VA_EIL_RTLS>

Production Operations Manual 16 October 2021

<COMPANIES>

<COMPANY/>

<NAME>C'AIRE INC</NAME>

<ORGID/>

</COMPANIES>

<SITE>

<VA_STATIONNUM>636</VA_STATIONNUM>

</SITE>

<PERSONGROUP>

<DESCRIPTION/>

<PERSONGROUP/>

</PERSONGROUP>

</ASSET>

</VA_ASSETOUT_INSSet> </ser:PublishVA_ASSETOUT_INS> </soapenv:Body> </soapenv:Envelope>

2 INFO 2020-09-03 10:54:24,855 WSAsset: <?xml version="1.0" encoding="UTF- 8"?><maximo><siteId>636</siteId><barcode>636 EE19546</barcode><eventType>updated</eventType><moveRequired>false</moveRequired> <crc32>NA</crc32><assetDescription>BED-ELECTRIC</assetDescription><assetName>BED- ELECTRIC</assetName><assetNum>777</assetNum><cmr>CMR</cmr><csn>6530- 439478</csn><dispositionDate>2017-01- 01T12:00:00</dispositionDate><manufacturer>C'AIRE INC</manufacturer><model>100- REF7622</model><parent>Parent</parent><placeInService>2017-02- 01T12:00:00</placeInService><purchaseOrder>500- P348065</purchaseOrder><responsibleShop>BIOMEDICAL</responsibleShop><serialNumber >P76908- J78KL192</serialNumber><servicePointer/><stationNumber>636</stationNumber><status>OP ERATING</status><totalAssetValue>475.00</totalAssetValue><typeOfEntry>NON-

EXPENDABLE

EQPT</typeOfEntry><vafmequipcat>?</vafmequipcat><vamdns>U$TEST2</vamdns><vamdn sdescription>BED-ELECTRIC</vamdnsdescription></maximo>

3 INFO 2020-09-03 10:54:24,996 AssetUpdate: [636-updated--636 EE19546] maxAssetId[777] crc32[NA]

4 INFO 2020-09-03 10:54:25,887 AssetUpdate: assignedLocationId[Bxd] serviceStatusId[Bxd] serviceStatusName[Not In Service]

5 INFO 2020-09-03 10:54:25,887 EquipType: barcode[636 EE19546] typeOfEntry[NON- EXPENDABLE EQPT] csn[6530-439478]

6 INFO 2020-09-03 10:54:26,341 EquipType: barcode[636 EE19546] uncategorizedTypeId[Bxlw] uncategorizedTypeName[AEMS MERS UNCATEGORIZED NULL] status[200]

7 INFO 2020-09-03 10:54:26,356 EquipType: barcode[636 EE19546] equipmentTypeId[null] equipmentTypeName[null] uncategorizedTypeId[Bxlw] uncategorizedTypeName[AEMS MERS

UNCATEGORIZED NULL]

8 INFO 2020-09-03 10:54:26,481 AssetUpdate: [636-updated--636 EE19546] post owner=CMR&notes=<br/><b>Station Number: </b>636<br/><b>Asset Description: </b>BED- ELECTRIC <br/><b>VA-MDNS Code: </b>U$TEST2 <br/><b>Service: </b>

Production Operations Manual 17 October 2021

<br/><b>Responsible Dept: </b>BIOMEDICAL<br/><b>Type of Entry: </b>NULL<br/><b>PO:

</b>500-P348065<br/><b>Parent: </b>Parent<br/><b>Maximo Status:

</b>OPERATING<br/><b>Disposition Date: </b>2017-01-01T12:00:00<br/><b>CSN:

</b>6530-439478<br/>&serial-number=P76908-J78KL192&service-status=Bxd&type=Bxlw&assigned-location=Bxd&short-name=AEMS&manufacturer=C'AIRE INC&capital-equipment=&annual-value=&depreciation-date=&placed-in-service=2017-02- 01T12:00:00&external-references.maximo-asset-id=777&attributes.hp.maximo-entry-type=NULL&fixed-asset-number=636 EE19546&secondary-fixed-asset-number=&name=BED- ELECTRIC | 636 EE19546&model=100-REF7622&expiration=&value=475.00

9 INFO 2020-09-03 10:54:28,660 AssetUpdate: [636-updated--636 EE19546] success

3.2.1.1.4. Asset Updated in RTLS

When an asset is updated in RTLS, an event is queued in RTLS on an internal queue. All of the events are pulled by the interface on a periodic basis and placed separately on the esb.maximo.asset.events queue in ActiveMQ. The events are processed by the interface one by one.

The following messages are logged for each event:

1. When the message is pulled from the esb.maximo.asset.events queue, the extracted values are logged. The detail information for the asset is retrieved and placed onto the esb.maximo.asset.details queue.

2. When the message is pulled from the esb.maximo.asset.details queue, the common values for events (new, moved and updated) are logged.

3. Since this is an update event, extended information about the asset is logged.

4. The asset information is transformed and placed onto the esb.maximo.asset.maximo queue.

5. When the message is pulled from the esb.maximo.asset.maximo queue, the values are logged.

6. The asset information is retrieved from Maximo, transformed, and logged. The data is placed onto the esb.maximo.asset.update queue.

7. When the message is pulled from the esb.maximo.asset.update queue, some of the extracted values are logged.

8. The assigned location (station) and the equipment service status of the asset are retrieved and logged.

9. The equipment type logic starts and the type of entry and the CSN are logged.

10. The uncategorized type id and name are retrieved by CSN and logged.

11. The type id and name are retrieved and logged along with the uncategorized id and name.

The uncategorized type takes precedence when determining the type for the asset.

12. The values used to update the asset are logged.

13. A success message is logged.

The following sample messages correspond to the above steps.

Table 9: Sample Messages Corresponding to Logging Events in RTLS

# Message

Production Operations Manual 18 October 2021

1 INFO 2020-09-03 12:32:01,998 QueueAsset: site[636] event[updated] barcode[] asset[BxcCnZ]

2 INFO 2020-09-03 12:32:02,518 AssetDetail: site[636] event[updated] barcode[636 EE19546] asset[BxcCnZ] maxAssetId[777] maxLocId[636|107-BLDG B] station[636] locName[636-BLDG B-107_approx] dateEntered[2017-10-08T13:50:00-04:00] lastScanned[] sensor[oat-passive^null^false]

3 INFO 2020-09-03 12:32:02,640 AssetDetail: assignedLocation[Bxd] fixedAssetNumber[636 EE19546] manufacturer[APPLE] model[100-REF7622] name[BED-ELECTRIC | 636 EE19546] owner[CMR] placedInService[2017-02-01T12:00:00-04:00] serialNumber[P76908-J78KL192] serviceStatus[Bxd] shortName[AEMS] type[Bxlw] value[475.00]

4 INFO 2020-09-03 12:32:02,832 AssetDetail: site[636] event[updated] barcode[636 EE19546] asset[BxcCnZ] crc32[4113810905] valid[true]

5 INFO 2020-09-03 12:32:02,901 AssetMaximo: [636-updated-BxcCnZ-636 EE19546] maxId[777] loc[107-BLDG B] invDate[2017-10-08T13:50:00.000-04:00] tag[oat-passive- 33142a0e7461a7c00000261f] crc[4113810905] check[false] move[false]

6 INFO 2020-09-03 12:32:05,569 AssetMaximo: <?xml version="1.0" encoding="UTF- 8"?><maximo><siteId>636</siteId><eventType>updated</eventType><rtlsAssetId>BxcCnZ</rtl sAssetId><barcode>636 EE19546</barcode><moveRequired>false</moveRequired><crc32>4113810905</crc32><ass etNum>777</assetNum><assetDescription/><assetName/><vafmequipcat/><vaaemsrefid>636 EE19546</vaaemsrefid><vamdns>U$TEST2</vamdns><vamdnsdescription/><purchaseOrder >500- P348065</purchaseOrder><parent>Parent</parent><status>OPERATING</status><csn>6530- 439478</csn><lastInvDate>2019-03-01T12:32:05.171- 10:00</lastInvDate><cmr>CMR</cmr><serialNumber>P76908- J78KL192</serialNumber><totalAssetValue>475.0</totalAssetValue><model>100- REF7622</model><dispositionDate/><placeInService>2014-01-01T00:00:00.000- 10:00</placeInService><responsibleShop/><typeOfEntry>NXE</typeOfEntry><stationNumber> 636</stationNumber><manufacturer>APPLE</manufacturer><servicePointer>ENGINEERING< /servicePointer></maximo>

7 INFO 2020-09-03 12:32:05,675 AssetUpdate: [636-updated-BxcCnZ-636 EE19546] maxAssetId[777] crc32[4113810905]

8 INFO 2020-09-03 12:32:06,438 AssetUpdate: assignedLocationId[Bxd] serviceStatusId[Bxd] serviceStatusName[Not In Service]

9 INFO 2020-09-03 12:32:06,444 EquipType: barcode[636 EE19546] typeOfEntry[NXE] csn[6530-439478]

10 INFO 2020-09-03 12:32:06,824 EquipType: barcode[636 EE19546] uncategorizedTypeId[Bxcml] uncategorizedTypeName[BED SCALE COMBO PAT WEIGH 6530-439478] status[200]

11 INFO 2020-09-03 12:32:06,834 EquipType: barcode[636 EE19546] equipmentTypeId[null] equipmentTypeName[null] uncategorizedTypeId[Bxcml] uncategorizedTypeName[BED SCALE

COMBO PAT WEIGH 6530-439478]

12 INFO 2020-09-03 12:32:06,938 AssetUpdate: [636-updated-BxcCnZ-636 EE19546] post owner=CMR&notes=<br/><b>Station Number: </b>636<br/><b>Asset Description: </b> <br/><b>VA-MDNS Code: </b>U$TEST2 <br/><b>Service: </b>ENGINEERING <br/><b>Responsible Dept: </b> <br/><b>Type of Entry: </b>NXE<br/><b>PO: </b>500- P348065<br/><b>Parent: </b>Parent<br/><b>Maximo Status:

</b>OPERATING<br/><b>Disposition Date: </b><br/><b>CSN: </b>6530-439478<br/>&serial-number=P76908-J78KL192&service-status=Bxd&type=Bxcml&assigned-location=Bxd&short-

Production Operations Manual 19 October 2021 name=BED&manufacturer=APPLE&capital-equipment=&annual-value=&depreciation-date=&placed-in-service=2014-01-01T00:00:00.000-10:00&external-references.maximo-asset-id=777&attributes.hp.maximo-entry-type=NXE&fixed-asset-number=636 EE19546&secondary-fixed-asset-number=&name=636 EE19546&model=100-REF7622&expiration=&value=475.0

13 INFO 2020-09-03 12:32:07,313 AssetUpdate: [636-updated-BxcCnZ-636 EE19546] success

3.2.1.1.5. Asset Created in RTLS

When an asset is created in RTLS, an event is queued in RTLS on an internal queue. All of the events are pulled by the interface on a periodic basis and placed separately on the esb.maximo.asset.events queue in ActiveMQ.The events are processed by the interface one by one. If processing is successful, an update event for the asset is created and placed onto the esb.maximo.asset.events queue.

The following messages are logged for each event:

1. When the message is pulled from the esb.maximo.asset.events queue, the extracted values are logged. The detail information for the asset is retrieved and placed onto the esb.maximo.asset.details queue.

2. When the message is pulled from the esb.maximo.asset.details queue, the common values for events (new, moved and updated) are logged.

3. The asset information is transformed and placed onto the esb.maximo.asset.maximo queue.

4. When the message is pulled from the esb.maximo.asset.maximo queue, the values are logged.

5. If the Maximo date check is turned on, the date entered location is retrieved from Maximo and logged.

6. A success message is logged.

The following sample messages correspond to the above steps.

Table 10: Sample Messages Corresponding to Creating an Asset in RTLS

# Message 1 INFO 2020-09-03 12:55:00,974 QueueAsset: site[636] event[new] barcode[] asset[BxcymQ]

2 INFO 2020-09-03 12:55:01,583 AssetDetail: site[636] event[new] barcode[636 EE11076] asset[BxcymQ] maxAssetId[] maxLocId[] station[] locName[636A8-1-1W103] dateEntered[2020- 06-10T20:15:47.1896487-04:00] lastScanned[2018-08-08T12:00:00.000-05:00] sensor[centrak-rtls^null^false]

3 INFO 2020-09-03 12:55:01,598 AssetDetail: site[636] event[new] barcode[636 EE11076] asset[BxcymQ] crc32[NA] valid[true]

4 INFO 2020-09-03 12:55:01,614 AssetMaximo: [636-new-BxcymQ-636 EE11076] maxId[] loc[] invDate[2020-06-10T20:15:47.189-04:00] tag[centrak-rtls-643852] crc[NA] check[false] move[null]

5 INFO 2020-09-03 12:55:01,678 AssetMaximo: [636-new-BxcymQ-636 EE11076] lastInvDate[2019-06-11T17:19:19.447-04:00] maxInvDate[2019-03-01T14:40:16.089-10:00]

6 INFO 2020-09-03 12:55:01,741 AssetMaximo: [636-new-BxcymQ-636 EE11076] success messageID[1] updatedTime[2020-09-03T12:55:01.710-10:00]

Production Operations Manual 20 October 2021

3.2.1.1.6. Asset Moved in RTLS

When an asset is moved in RTLS, an event is queued in RTLS on an internal queue. All of the events are pulled by the interface on a periodic basis and placed separately on the esb.maximo.asset.events queue in ActiveMQ.The events are processed by the interface one by one.

The following messages are logged for each event:

1. When the message is pulled from the esb.maximo.asset.events queue, the extracted values are logged. The detail information for the asset is retrieved and placed onto the esb.maximo.asset.details queue.

2. When the message is pulled from the esb.maximo.asset.details queue, the common values for events (new, moved and updated) are logged.

3. The asset information is transformed and placed onto the esb.maximo.asset.maximo queue.

4. When the message is pulled from the esb.maximo.asset.maximo queue, the values are logged.

5. If the Maximo date check is turned on, then the date entered location is retrieved from Maximo and logged.

6. A success message is logged.

The following sample messages correspond to the above steps.

Table 11: Sample Messages Corresponding to Moving an Asset in RTLS

# Message

1 INFO 2020-09-03 13:45:09,672 QueueAsset: site[636] event[moved] barcode[636 EE20656] asset[]

2 INFO 2020-09-03 13:45:10,580 AssetDetail: site[636] event[moved] barcode[636 EE20656] asset[BxcCSq] maxAssetId[888] maxLocId[636|B702B-25] station[636] locName[636-25- B702B_approx] dateEntered[2020-06-11T13:00:00-04:00] lastScanned[] sensor[centrak-rtls^null^false]

3 INFO 2020-09-03 13:45:10,580 AssetDetail: site[636] event[moved] barcode[636 EE20656] asset[BxcCSq] crc32[NA] valid[true]

4 INFO 2020-09-03 13:45:10,596 AssetMaximo: [636-moved-BxcCSq-636 EE20656] maxId[888] loc[B702B-25] invDate[2020-06-11T13:00:00.000-04:00] tag[centrak-rtls-690068] crc[NA] check[false] move[null]

5 INFO 2020-09-03 13:45:10,621 AssetMaximo: [636-moved-BxcCSq-636 EE20656] lastInvDate[2019-06-11T17:19:19.447-04:00] maxInvDate[2019-03-01T14:32:08.074-10:00]

6 INFO 2020-09-03 13:45:10,643 AssetMaximo: [636-moved-BxcCSq-636 EE20656] success messageID[2] updatedTime[2020-09-03T13:45:10.627-10:00]

3.2.2. Application Error Codes and Descriptions

The interface uses standard HTTP status codes.

Production Operations Manual 21 October 2021

Table 12: Standard HTTP Status Codes

HTTP Error Code Description

400 Bad Request. This error code could potentially be related to bad data.

404 Not Found. The resource the interface was trying to communicate with was not found.

500 Internal Server Error. One of the systems is not available (i.e., Maximo or Intelligent InSites).

3.2.3. Infrastructure Errors

The maximo.log captures the results of some failures such as a license expiration for Mule.

3.3. Dependent System(s)

The Maximo-RTLS interface is dependent on the Maximo and Intelligent InSites systems being up and operational.

3.4. Troubleshooting

Table 13: Troubleshooting

Issue Possible Cause Solution

Expected asset inventory update was not posted to Maximo.

A failure occurred in the interface’s validation.

View the maximo.log file for additional details.

A location status change in Maximo was not posted to

RTLS.

A failure occurred in the interface’s validation.

View the maximo.log file for additional details.

After a new…