2-8--ADOT_ECD_DLSSU_RFP--NEC_Technical_Support_FINAL_2022-04-18.pdf

An NEC Response to the Arizona Department of Transportation – Enforcement and Compliance Division ECD Drivers License Security Software RFP – BPM004292

Section 8 – Technical Support

Page 8-1

8 Technical Support RFP Reference: APP Portal, Financial Questionnaire, Proposal Requirements for Drivers License Security Software Overall Solution, Technical Support; ADOT ECD DLSS RFP, Scope of Work, Section 6

8.1 Customer Support

NEC takes great pride in its reputation for providing the most dedicated customer support organization in the biometrics industry. Working in partnership with our clients, NEC refines its support service offerings by focusing on key success factors, including proactive structured communication, commitment to excellence, and continuous improvements. NEC follows industry best practices and frameworks in a wide variety of areas—e.g., Project Management Institute (PMI) for project development and management, Scrum Alliance for software development, Information Technology Infrastructure Library (ITIL) processes for incident and change management, and Lean/Six Sigma—throughout a range of business activities and processes. Best practices are ever-changing and where one may not directly cover a given need or situation, NEC may leverage the above or similar guidelines to create original best practice frameworks.

Mission-critical solutions such as NEC's NeoFace® Facial Recognition System (FRS) solution for the ADOT ECD DLSS project require a comprehensive sustainment and warranty solution that meets or exceeds all expectations for service, functionality, and support. NEC understands that downtime is not an option in a high-availability environment. NEC possesses the requisite knowledge and experience to ensure that NeoFace FRS will be maintained in an operational status with minimal interruptions. By combining NEC’s service model NeoFace FRS solution, the amount of training and implementation time for NEC’s help desk personnel will be reduced. NEC will support software licensed to ADOT for use during the term of the contract as it pertains to NEC's NeoFace FRS solution.

NEC’s proposed service plan will include local field support engineers (FSE) throughout the State of Arizona. NEC’s U.S.-based, CJIS-compliant Network Operations Center (NOC) is operated by a dedicated engineering team that performs call center and Remote Monitoring Service (RMS) activities, including service request logging, response, tracking, escalation, resolution, and report management.

* * * B E G I N C O N F I D E N T I A L I N F O R M A T I O N * * *

Key benefits of NEC’s full-time maintenance and support services include:

• Global access to experts in one of NEC’s Biometrics Centers of Excellence.

• Dedicated telephone, remote technical, web portal, data center, and maintenance support services from the

NEC NOC.

• Rapid remote and on-site response times to minimize interruptions to ADOT operations.

• Proactive, 24 x 7 x 365 monitoring, diagnostics, and instant alerts for immediate problem identification and resolution, including:

Well-defined support roles and responsibilities to deliver maximum efficiency while ensuring that all critical DLSS functions are fully covered.

NEC provides comprehensive sustainment services that meet or exceed all expectations for service, functionality, and support.

Page 8-2

Parallel problem management and technical escalation to ensure that issues are promptly and fully resolved by FSEs and levels 3 and 4 support staff, as required.

Emphasis on clear lines of communication to ensure responsiveness to our clients.

Active NEC participation in all DLSS-related meetings to ensure that all open deficiencies have been addressed in a timely manner and that proper feedback is relayed to the end user.

• Expert diagnostic and problem resolution by qualified technicians and engineers with demonstrated field service experience and up-to-date training.

• Customized system monitoring and remote access tools supported by highly trained FSEs who are familiar with all aspects of biometrics technology (products, functionality, workflows, etc.) and can rapidly adopt and support new technology.

• Cost-effective service agreements tailored to your specific needs, ensuring a lower total cost of ownership.

* * * E N D C O N F I D E N T I A L I N F O R M A T I O N * * *

Following acceptance of the NeoFace FRS solution, NEC will transition activities from NEC’s Project Management Office (PMO) to its customer support organization, whose responsibilities are detailed in this document and who will manage the operational support and sustainment of the NeoFace FRS solution. NEC is committed to providing the highest quality customer support to ADOT through the NOC, multi-tiered support levels, web portal services, responsive field service teams, and dedicated management staff.

8.1.1 Multi-Tiered Support Model

NEC’s maintenance and support services for the NeoFace FRS solution are based on a standard four-tiered support model that follows industry best practices for delivering service support and ensures that all NeoFace FRS components will remain in optimal operational condition for the duration of the contract. Following is a brief description of the four tiers of NEC support:

• Level 1 Support – Consisting of NOC personnel, level 1 support staff are assigned to initiate trouble tickets, log related incident information, and engage or handle initial escalation to appropriate next level support personnel.

• Level 2 Support – Consisting of either or both Site Reliability Engineering (SRE) and other subject matter experts (SME), level 2 support staff will collaborate with ADOT support staff to resolve common incidents pertaining to cloud infrastructure and NEC’s solution where specialized competencies are required.

• Level 3 Support – Consisting of implementation and delivery personnel, level 3 support staff, along with related managers, support when normal support processes fail to resolve an incident within the required timeframe.

• Level 4 Support – Consisting of software developers and R&D team members primarily dedicated to new product development, level 4 support staff, including senior management, provide support if levels 1 through 3 cannot resolve an incident.

NEC’s NOC in combination with SRE staff is designed to enable the fastest mean time to repair. NEC will provide the necessary issue resolution personnel who can answer specific account questions and/or perform troubleshooting activities addressing all services provided within the contract. NEC’s issue tracking and ticketing system ensures that the NOC support team can track and report on all reported issues or requests. NEC’s NOC will

Page 8-3 also provide ADOT the ability to review and monitor all incidents/tickets and related metrics and to access various reports for monitored components. NEC’s approach to high-quality and prompt service will be provided in a manner that contributes to the overall success of the ADOT ECD DLSS project.

8.1.2 Proactive Monitoring

NEC’s NOC provides monitoring, detection, and remedial maintenance for errors and issues encountered in the NeoFace FRS solution. The NeoFace FRS solution will be monitored for system events on a 24 x 7 x 365 basis. Alerts generated by monitored systems are automatically sent to the NOC, alerting NEC of potential problems and enabling an unparalleled, proactive approach to service and support.

The proposed NeoFace FRS solution for the ADOT ECD DLSS project has been developed and engineered in the United States by NEC’s Advanced Recognition Systems division. It includes both industry-standard and NEC-proprietary diagnostic tools and resources. Upon receipt of an error notification/alert, NEC personnel will execute a remote, computer-aided failure analysis on system and application software. Additionally, NeoFace FRS error logs will be monitored and analyzed to provide support personnel with component level information regarding the health and status of the system.

Information gathering on all problems is of utmost importance, regardless of their nature (intermittent or consistent) and irrespective of their type (hardware, software, network, or microcode). As problems are detected or reported, the most likely cause will be determined, and steps will be taken toward resolution. Resources including FSEs, management, factory-level advisors, and developers will be used as required to solve problems in a manner causing as little impact as possible to ADOT operations.

8.2 Web Portal Access

* * * B E G I N C O N F I D E N T I A L I N F O R M A T I O N * * *

NEC will provide an online portal for ADOT staff to review incidents and tickets, metrics, and various standard reports for monitored components. Figure 8-1 shows the ticketing process from sign-in to description entry to review of ticket history. Additional reports may be included depending on contracted services. All reports can be viewed on demand. The portal provides a means for ADOT staff to input trouble tickets and change requests, search a knowledge database for known issues, and review real-time and near-real-time information regarding its network.

Page 8-4

Figure 8-1: NEC Customer Support Portal Example Screens

* * * E N D C O N F I D E N T I A L I N F O R M A T I O N * * *

8.3 NOC and Support Workflow

* * * B E G I N C O N F I D E N T I A L I N F O R M A T I O N * * *

NEC’s NOC operates on a 24 x 7 x 365 basis. NEC will provide ADOT a web-based trouble ticketing system and toll-free telephone, voicemail, and email support. Designated users will be assigned individual accounts and trained on how to submit, track, update, and escalate issues as needed.

The NOC's automated call management capabilities will quickly connect ADOT to appropriate NEC support staff. If an assigned team member is unavailable, ADOT staff can leave a voicemail, contact another available team member, or escalate the call to the next level of support.

NEC provides proactive system monitoring, enabling early detection and rapid recognition of potential problems (software, hardware, and security breaches) for the NeoFace FRS solution.

Whenever a deficiency is identified, whether detected automatically via NEC’s proactive monitoring services, reported by NEC support staff or ADOT staff, a trouble ticket will be issued. Trouble ticket resolution will proceed according to NEC’s flow of support, as illustrated in Figure 8-2 and described in the following sections.

NEC’s proactive system monitoring enables early detection and rapid recognition of problems.

Page 8-5

Figure 8-2: NEC NOC Flow of Support

8.3.1 Problem Identification

The flow of support may be initiated automatically by NEC's remote management services (RMS) system, or it may begin when an NEC or ADOT staff member reports a problem. Problems identified by the RMS are detected, whereas problems identified by staff members are discovered.

8.3.1.1 Problems Detected via Remote Monitoring Services

As part of maintenance services, the NeoFace FRS servers will be monitored for both hardware and software efficiencies. This monitoring will provide true 24 x 7 x 365 system support and immediate, automatic system call initiation. NEC’s RMS uses tailored applications to monitor disk storage, job queue data, connectivity, database services, application services, and interface connections. When a problem occurs, an error will be recorded in the Event Log. The Event Monitoring function will then detect the failure and automatically notify the NOC.

Page 8-6

Regardless of whether the problem is related to hardware, software, or network,1 a trained NEC technician will review the problem, log the call into the system via Microsoft’s Team Foundation Server (TFS) application, notify ADOT management of the issue, and immediately assign and/or dispatch the appropriate NEC engineer. As soon as the problem is logged into the system, a trouble ticket will be issued, which can be used by all involved staff members to track the status of the problem. Problems will be prioritized and worked according to the level of severity. For more information, please see Section 3, "Cloud Solution & Maintenance Requirements."

8.3.1.2 Problems Discovered by NEC and ADOT

Both NEC and ADOT staff members, in the course of their daily duties, may discover issues with the NeoFace FRS solution. When a staff member reports a problem, the event will be logged into the NOC system and then managed in the same manner as RMS-detected problems.

Problems may be reported by ADOT staff in one of three ways:

• Web-Based Ticketing – Designated staff will have access to NEC’s web-based trouble ticketing system. NEC’s successfully proven trouble ticketing system will provide a simple method to submit, track, and update issues that require escalation to NEC Customer Support. Authorized staff will each receive an account and training on the ticketing system. Each trouble ticket will include the problem report number, problem description including software versions (if applicable) and affected component(s)/sub-component(s), the time the problem call was received, the priority (severity level) assigned, NEC engineer assigned, the root cause of the problem, the actions taken to resolve the problem and/or prevent its recurrence, a history of all other actions taken including communications between NEC and ADOT, and the time when the problem was corrected and communicated to ADOT.

The web-based ticketing solution will also track product enhancements (change orders). NEC will conduct a preliminary evaluation within 30 days and update the ticket with the preliminary evaluation. NEC will use this information in product enhancement planning.

• Telephone/Voicemail – NEC will provide a toll-free telephone number for ADOT staff to initiate trouble tickets.

This telephone number is managed by an automated system that will quickly connect ADOT with the appropriate Customer Support personnel. For non-critical issues, ADOT may also leave detailed voicemails describing any issues. NEC will respond to voicemails within 24 hours, excluding weekends and holidays.

In the unlikely event that the NOC is unavailable, the ADOT management team will also have direct numbers for the local FSEs and the Support and Data Center Services Manager.

• Email – As an alternative to both web portal and telephone contact methods for non-critical issues, ADOT staff may use email or any other reasonable means to request support from the NOC.

1 For detected network problems outside the NEC scope of maintenance, such as WAN-related issues, the NEC engineer will work with ADOT to assist and resolve such problems.

Page 8-7

8.4 Escalation and Tracking

NEC provides a problem escalation system that enables tracking of all escalated support issues through Microsoft TFS. All escalated calls are assigned a severity/priority level and will either be closed successfully or updated and escalated through the NEC escalation hotline.

Escalation notifications are routed directly to levels 3 and 4 for engineering support. NEC will regularly provide updates to ADOT staff on the status of escalated problems. The ADOT will also be able to monitor the status of escalated issues through NEC’s web portal. The system allows calls to be treated on a priority basis with internal Management and Technical escalation paths for immediate and proper level of attention.

NEC’s escalation path is illustrated in Figure 8-3. See Section 3, "Cloud Solution & Maintenance Requirements," for more detail regarding NEC's remedial maintenance methodology.

Figure 8-3: ADOT ECD DLSS Emergency Call Process and Escalation Path

Through TFS, ADOT staff will be able to generate reports on the status of trouble tickets for tracking and review.

The report will include report number, submitting agency, problem description, time of call, priority, actions taken, and the problem’s cause and resolution. The report log will also indicate when all units were installed for ADOT.

A call log report listing all open, closed, or escalated calls can also be retrieved at any time from the NOC. ADOT staff will be provided a client portal to view the status of support calls.

NEC’s problem escalation system enables tracking, prioritization, and the immediate and proper level of attention required for the operation and sustainment of the NeoFace FRS solution.

Page 8-8

8.5 Customer Support Team

NEC’s dedicated customer support organization will maintain the ADOT ECD DLSS project throughout its contract years. As a solutions company, NEC is offering a structured and proven management team, with comprehensive support and maintenance methodologies. These support and maintenance strategies offer a meaningful and systematic approach to providing reliable performance levels required for the NeoFace FRS solution. To provide this all-inclusive type of operational and customer support, NEC is providing the structure, oversight, and management required for a smooth transition from implementation to quality operational support through the contract lifecycle.

NEC will staff the proposed Customer Support team, as shown in Figure 8-4, with experienced staff members that have knowledge of the current ADOT locations, data, and users. These key members will be dedicated to the successful operations of the NeoFace FRS solution.

Figure 8-4: Customer Support Team

Page 8-9

In addition, NEC’s NOC, located in Rancho Cordova, California, will provide 24 x 7 x 365 remote support for ADOT, ensuring rapid responses while minimizing system interruptions.

The Support and Data Center Services Manager will monitor and track the status of open/outstanding issues and provide a detailed monthly report to NEC’s senior level management. The monthly report includes monthly operational statistics, issue resolution, and the status of fixes that require further research. The Support and Data Center Services Manager will also participate in local meetings.

The Service and Support Manager has direct oversight of the NEC Advanced Recognition Systems customer support organization and serves as the next level of escalation if an issue is not resolved satisfactorily. The Service and Support Manager is also responsible for NEC’s Maintenance contract with ADOT and communicating with the Support and Data Center Services Manager before the annual maintenance contract renewal.

The Customer Support team will provide personnel experienced not only with NEC products, but also with biometrics, interfaces, and operations, thereby ensuring the operations and reliability required for a best-in-class public safety organization.

The following table identifies the roles and responsibilities of the Customer Support team.

Table 8-1: Customer Support Team Roles and Responsibilities

Name/Role Responsibilities

Eugene Le Roux Executive Sponsor

• Provides an interface to NEC’s CEO and senior management.

• Understands ADOT’s business needs.

• Manages and negotiates contract terms and conditions.

Dan Cinnamon Project Director

• Responsible for compliant execution of the contract, possessing due authority and responsibility over all resources required for such execution.

• Provides overall leadership of the project team and serves as the single point of contact for the ADOT Project Director and executive staff.

• Works with the ADOT Project Director and agency representatives to ensure timely and effective responses to information requirements and to resolve all problems.

• Primary technical liaison for the ADOT project team responsible for defining and obtaining implementation of the project.

Jeff Siao Service and Support Manager

• Provides leadership to the Customer Support Team

• Responsible for project operations.

• Primary liaison between the NEC Customer Support team and the ADOT project team with overall responsibility for maintaining operations.

Larry Moran Support and Data Center Services Manager

• Reports to the Service and Support Manager and manages IT solutions development for support services.

• With the Service and Support Manager, assists in planning, coordinating, and accomplishing system delivery and installation.

NEC’s NOC is operated by a dedicated engineering team, ensuring rapid responses while minimizing system downtime.

Page 8-10

Name/Role Responsibilities

• Develops the Operations and Maintenance Support Plan and coordinates updates to detailed plans and procedures to accomplish effective operations and maintenance support.

• Responsible for 24 x 7 data center infrastructure and operations, support call center operations, and biometric application systems monitoring and support. Service areas include IT platforms, data integrity, security, and biometric applications.

• Coordinates security, governance, and compliance programs.

• Secondary technical liaison with the ADOT project team responsible for planning, coordinating, and implementing operations and maintenance support capabilities.

• Participates in preparation and delivery of operations and maintenance support training for system administrators and support staff.

• Oversees and directs the operations and maintenance support activities during the production period.

Local Field Support Engineers (as required)

• Provide secondary technical support.

• Responsible for planning, coordinating, and implementing operations and maintenance support capabilities.

• Participate in preparation and delivery of operations and maintenance support training.

• Provide oversight and direction of the operations and maintenance support activities during the production period.

For more information on NEC’s proposed staffing, see Section 4, "Project Team," and Section 5, "Key Personnel Bio."

* * * E N D C O N F I D E N T I A L I N F O R M A T I O N * * *

8.6 Availability of Updates

NEC is proposing an NEC-hosted system. NEC understands the complexities involved with administering and maintaining geographically dispersed cloud systems. NEC has successfully managed numerous implementations featuring remote workstations that were not only geographically remote, but often had limited bandwidth connections to the central data centers. To address such challenges, NEC has designed its architecture to simplify software deployment and rollout.

8.6.1 System and Software Upgrades

NEC will maintain the system software that is part of the server environment for the DLSS. NEC will maintain all software components, including, but not limited to, OS software, database software, and other software installed in the server environment that is not just our application software. NEC will perform system software updates to keep server software current with new software releases that will also encompass all third-party software updates, patches, bug fixes, etc., and will be delivered in coordination with the ADOT Project Manager. Software updates will include, bug fixes, minor and major version releases including product feature enhancements.

Page 8-11

The DLSS will be sustained and updated without affecting established databases, the application software, or the operating systems. Scheduled and unscheduled software updates released by NEC will be installed during periods of the maintenance window as mutually agreed upon by ADOT and NEC. NEC will make every effort to ensure that the transition will not cause system degradation or downtime. In addition, updates to the electronic PDF user manual resulting from software updates will be made available upon ADOT's request.

NEC will ensure software updates are compatible with the application software, including any application modifications provided by NEC under the contract. Prior to installing any third-party software, or any update thereto, NEC will test the third-party software’s compatibility with the current version of the server software. NEC will ensure that the software is compatible with all required and critical updates to third-party software, including, without limitation, service and compatibility packs and security patches, promptly upon their release.

In coordination with ADOT's Project Manager, NEC will maintain the DLSS’s compatibility with ADOT's client environment by providing service updates and essential upgrades to the solution application software and reciprocal cloud platform infrastructure in accordance with NEC’s Evergreen Program.

8.6.2 NEC Software Release Policy

* * * B E G I N C O N F I D E N T I A L I N F O R M A T I O N * * *

The NeoFace FRS solution hosted on the Microsoft Azure Government℠ cloud platform provides a lower total cost of ownership via an easier and more affordable path for future system updates and upgrades through NEC's Evergreen Program. NEC's Evergreen Program will provide continual updates of application software and cloud infrastructure services throughout the life of the contract, keeping ADOT's operating systems, software versions, databases, and tools current with the latest technology advances. NEC will notify ADOT of the available upgrades, which, if acceptable, can be pushed out to the system in real time. The Evergreen Program is subject to NEC's Software Release Policy, which is described below.

Benefits of NEC's Evergreen Program:

• Continual application of security patches and version updates, ensuring CJIS security compatibility of operating systems, databases, and application servers.

• Updated software documentation and training to ensure technicians and managers are current on new software changes related to each update.

As part of our continued commitment to providing the best and most up-to-date solutions, NEC is offering an Identity as a Service (IDaaS) solution to our customers as part of our cloud deployments. Unlike the historic system upgrade process of periodic full system replacements, NEC's Evergreen Program comes with all our cloud deployments. The Evergreen Program will keep the DLSS sustained and updated with the newest compatible software releases to our core product application software while maintaining ADOT-specific interfaces, workflows, and configurations. The IDaaS Evergreen Program negates the large effort for requirements gathering, data migration, testing, and training typical of full system replacements.

As part of NEC’s Evergreen Program, software updates will include, but not be limited to, enhancements, minor and major version releases, and other improvements and modifications to the software (including application software), subject to NEC's Software Release Policy, delineated below.

Page 8-12

NEC’s software release policy follows an industry-standard release management process. The main activities involved in release management are:

• Establishing a planning policy for the implementation of new versions

• Developing new versions

• Testing new versions in an environment that simulates the live environment as closely as possible

• Validating the new versions

• Implementing new versions in the live environment

• Version control

Every software release version is identified by three numbers. For example, in software release 4.6.1, 4 identifies a major release, 6 identifies a minor release, and 1 identifies a supplemental release.

The following is NEC’s policy for our application software upgrades:

• Major releases also known as New Versions of the application software, include advances in biometric technology (e.g., a newer matching algorithm), technology major releases reflect significant improvements in the biometric matching performance. ADOT is entitled to receive major releases free of charge if an operations and maintenance support contract is in effect.

• Minor releases, also known as Enhancements, augment the utility, efficiency, user interface and experience, and/or functional capability of the software. ADOT is entitled to receive minor releases free of charge if an operations and maintenance support contract is in effect.

• Supplemental releases materially impact the operational or functional performance of the software and include patches and trouble fixes. ADOT is entitled to supplemental releases without expense if an operations and maintenance support contract is in effect.

NEC will update software deliverables when released to correct any deficiency found in the software product. NEC will ensure that software is compatible with all required or critical updates to third-party software, service and compatibility packs, and security patches, promptly upon their release. These updates will be provided at no cost to ADOT for the life of the contract. Any engineering changes released by the equipment and middleware suppliers will be passed to ADOT at no additional cost, as long as the changes do not impact the operations of the DLSS. No service will run on software that is no longer supported by its licensor. Product updates that add new features and functions to the DLSS will be made available through the change order process. NEC will provide corrective maintenance in accordance with the operations and maintenance support contract at no additional cost to ADOT for all software supplied by NEC for the life of the contract. NEC Customer Service and Support management and FSEs will work directly with ADOT staff to ensure little or no downtime while installing releases.

* * * E N D C O N F I D E N T I A L I N F O R M A T I O N * * *