2-3--ADOT_ECD_DLSSU_RFP--NEC_Cloud_Maintenance_FINAL_2022-04-18.pdf

An NEC Response to the Arizona Department of Transportation – Enforcement and Compliance Division ECD Drivers License Security Software RFP – BPM004292 Section 3 – Cloud Solution & Maintenance Requirements

Page 3-1

3 Cloud Solution & Maintenance Requirements RFP References: APP Portal, Financial Questionnaire, Proposal Requirements for Method of Approach, Cloud Solution & Maintenance Requirements; ADOT ECD DLSS RFP, Scope of Work, Sections 4 and 5

In response to the ADOT ECD DLSS RFP Document sections cited above, NEC Corporation of America (NEC) has provided the following cloud solution and maintenance approach narrative for the proposed DLSS solution based on our NeoFace® Reveal technology.

3.1 DevOps

To support cloud solutions, NEC is using our DevOps approach. The DevOps is a combination of two words:

Development and Operations. DevOps is a practice that allows a single team to manage the entire application development life cycle, that is, development, testing, deployment and operations. The DevOps concept is encouraging faster, better, more secure delivery to our customer. As a part of DevOps, we are also incorporating better security infrastructure to accommodate the Secure Software Development Lifecycle (SSDLC). The DevOps approach will ensure the following benefits:

• Better system monitoring, faster response, improved stability, and sustainability

• Improved SLA

• Full transparency and seamless communication

• Improved resource availability

• More automation to reduce or eliminate human errors and improved quality by using continuous integration / continuous deployment (CI/CD) of pipeline

• Improve end-to-end security and vulnerability tracking

• Incorporate tools across all functions to improve collaboration

Figure 3-1: Standard DevOps Process

Page 3-2

3.1.1 Secure Software Development Life Cycle

NEC's Secure Software Development Life Cycle (SSDLC) follows a secure-by-design methodology from product creation through end-of-life, as illustrated by the figure below.

Figure 3-2: Secure SDLC

3.1.2 Continuous Integration and Continuous Deployment

Continuous integration and continuous deployment (CI/CD) is a method to frequently deliver features/apps to customers by introducing automation into the stages of app development for seamlessly integrating, securing, validating, and deploying new code. Using CI/CD methodology, DevOps and site reliability engineering (SRE) teams continuously streamline secure code development, integration, testing, and delivery, which includes deployment, monitoring, etc.

3.1.3 Security Validation Procedures

* * * B E G I N C O N F I D E N T I A L I N F O R M A T I O N * * *

NEC's SSDLC follows a secure-by-design philosophy from product creation through end-of-life. NEC uses industry-accepted standard security processes to ensure a secure software development lifecycle.

Open-source code has existing unknown vulnerabilities. However, we will have monitoring tools that track potential attacks. Any potential attack will be identified, tracked, and analyzed. Security monitoring tools will generate a notification and then be used to mitigate the attack. NEC uses the following security monitoring tools:

• Nagios – network security monitoring tool

• Splunk – network security monitoring tool https://www.redhat.com/en/topics/cloud-native-apps https://www.redhat.com/en/topics/automation https://www.redhat.com/en/topics/cloud-native-apps/why-choose-red-hat-cloud-native https://www.redhat.com/en/topics/integration https://www.redhat.com/en/topics/devops https://www.redhat.com/en/topics/devops/what-is-sre

Page 3-3

• OSSEC – intrusion detection tool that provides real-time analytics for system security

Before production release, NEC will run an end-to-end penetration test as well.

Figure 3-3 shows our full security validation methodology.

Figure 3-3: Security Validation Methodology

3.1.4 Network Operations Center and Site Reliability Engineering To streamline customer support and system monitoring, initial triaging (level 1 issues) will be performed by NEC's Network Operations Center (NOC) 24 x 7 x 365. The activities performed by NOC teams for proactive monitoring and sustainment are as follows:

• Actively monitoring and checking for potential errors.

• Testing new builds and releases in a production test environment to ensure full functionality before deployment to the production environment.

• Reviewing logs and testing code reaction for new builds and releases.

• Performing vulnerability assessment and applying OS patches and updates as applicable.

• Data visualization and ensuring captured data is saved in the logs.

• Monitoring image capture software.

• Tracking end-of-life of all relevant software components and observability data analytics.

If an issue is not resolved by the NOC team or needs higher level review, it will be escalated to the SRE team. The SRE team will resolve levels 2 and 3 issues. Level 4 issues will be escalated to the development and system engineering team.

Page 3-4

Table 3-1: Issue Resolution Teams

Issue Level Team

Level 1 NOC Team

Level 2 and Level 3 SRE Team, Field Support Engineer (FSE)

Level 4 Development and System Engineering Team

3.1.5 Alert Management

NEC-implemented tools will perform cloud infrastructure monitoring and send notifications to the NOC and SRE teams in case of the following event types:

• Network outage

• Database outage

• Storage space

• Application health monitoring

• Automated log analysis

• CPU, memory, network and other OS metrics usage

All alerts will be categorized for further analysis and handling.

When an incident is identified through either the automated monitoring tools or discovered by ADOT or NEC staff, the problem will be logged, ticketed, tracked, escalated, and resolved according to the process described in Section 8, "Technical Support.”

NEC will respond back to ADOT and resolve each reported problem within the required timeframe, as described in the following section. NEC’s representative will work with ADOT’s Project Manager and technical support staff to identify and correct problems while keeping all key personnel informed of the resolution status.

3.2 Problem Severity Levels

RFP Reference: RPF Document, Scope of Work, Section 6.5.3

The severity level for each reported problem will be assigned based on the Service Level Agreement (SLA) definitions provided in the following table and per ADOT ECD DLSS RFP Document, Scope of Work, Section 6.5.3.2, "Severity/Response/Communicate/Resolution Table," including those mutually agreed upon and finalized in the resultant Agreement. NEC will respond to each reported defect within two hours of ADOT'S defect notification.

Table 3-2: Problem Severity and Service Levels

Severity Response Time Communicate Action Plan

Resolution Time Liquidated Damages per Occurrence

1 30 min 240 min 12 hr $350 if resolution time missed, plus additional fee $125 for each

Page 3-5

Severity Response Time Communicate Action Plan

Resolution Time Liquidated Damages per Occurrence additional hour over resolution time.

2 30 min 480 min 24 hr $125 if resolution time missed, plus additional fee $50 for each additional hour over resolution time.

3 120 min 48 hr 336 hr $50 if resolution time missed, plus additional fee $25 for each additional hour over resolution time.

ADOT and NEC resources will agree to and assign the severity level to each problem reported by ADOT to NEC’s Customer Support. NEC will assign the severity level to all problems detected by its own monitoring system, where applicable.

NEC will provide a Monthly Operations Report, which will detail any system activities, such as maintenance, outages, or other critical events throughout the life of the contract.

3.2.1 Liquidated Damages

Credits may accrue for Unscheduled Downtime, including NEC’s failure to meet the Response and Resolution Time requirements (hereinafter “Liquidated Damages”). For purposes of assessing Liquidated Damages, see Table 3-2.

NEC will notify ADOT as soon as NEC is aware of an issue, including providing the length of anticipated downtime plus the description or purpose of unscheduled maintenance.

Without limiting any other rights and remedies available to ADOT, either pursuant to the proposed Service Levels, by law, or in equity, ADOT shall be entitled to Liquidated Damages calculated based on the length of Unscheduled Downtime as provided in Table 3-2.

ADOT reserves the right to assess Liquidated Damages for failure to meet the service level standards in this contract.

These damages will apply on a per service level standard basis if NEC does not comply with stated requirements and fails to meet a service level standard.

Liquidated Damages, in any amounts, are not and shall not be construed as penalties and, when assessed, will be deducted from ADOT’s payment due to NEC. If the payment is insufficient to cover the deficiency, NEC will immediately pay the amount of the insufficiency to ADOT within 15 days following receipt of written notice of the amount due. NEC respectfully requests that Liquidated Damages do not exceed 10% of the maintenance fees paid to NEC in any one Contract Year.

3.2.2 Correction of Deficiencies

When a deficiency is identified, the problem will be logged, ticketed, tracked, escalated, and resolved according to the process described in Section 8, "Technical Support.” Based on the severity/priority assigned to each deficiency

Page 3-6 by ADOT, NEC’s NOC and SRE team will respond to ADOT and resolve each deficiency within the timeframes defined in Table 3-2.

NEC understands that the timeframe for correction of deficiencies begins when NEC is first notified of a problem— whether by the RMS, NEC staff, or ADOT staff—and ends when ADOT determines that the deficiency has been resolved.

3.3 Maintenance and Sustainment Services

This section describes NEC’s approach to satisfying the maintenance and sustainment requirements of the ADOT ECD DLSS RFP. Based on our experience within the biometrics industry, NEC is uniquely suited to provide a successful operational model of this size, complexity, and magnitude. To maintain a high operational standard, all hardware, software, engineering, and development support is now directly handled out of NEC’s Advanced Recognition Systems division headquarters office in Rancho Cordova, California.

NEC’s commitment to quality and excellence will provide ADOT a comprehensive maintenance and sustainment service package designed to keep the DLSS solution in optimal operational condition. As part of the Maintenance Services package, NEC will provide 24 x 7 x 365 support for the DLSS solution as needed and within the response time requirements. All service support calls will be routed through the NEC call center and dispatched to the appropriate NEC NOC team member. For statistical reporting purposes, all service calls will be logged into the call collection system.

NEC’s monitoring tools will conduct proactive and ongoing system health, performance, and security vulnerabilities check as an integral aspect of its Maintenance Services program. The DLSS solution will be monitored using RMS.

Working in conjunction with our preventive and remedial maintenance programs, the RMS service will be an added feature, providing advance alerts to performance and problem issues. For more information on NEC’s RMS, see Section 8, "Technical Support.”

3.3.1 Scheduled Maintenance and Downtime

NEC will perform preventive maintenance that anticipates and prevents problems on a mutually agreeable, scheduled basis. Preventive maintenance is scheduled to have minimal impact on ADOT operations. ADOT will receive a minimum of two weeks' notice before any anticipated scheduled downtime, plus a description of purpose, to include preventive maintenance scheduling. ADOT may select any alternative period of preventive maintenance coverage. In addition, all planned/scheduled downtime will be performed after hours of business, i.e., Monday through Friday 5:00 A.M. to 6:00 P.M. MST, excluding State holidays and/or mandated furlough days. Routine implementation of patches and/or fixes for NEC's solution shall not interrupt the availability of facial recognition software services to the Department during normal business hours.

For the DLSS solution, NEC will provide software inspections, system diagnostics, and verification operation in accordance with NEC’s specifications. The NEC and ADOT Project Managers will agree upon the scheduling of preventive maintenance after implementation is completed. NEC will provide ADOT a checklist of tasks completed, tasks not completed, and reasons for incomplete tasks.

NEC possesses the requisite knowledge and experience to ensure that ADOT's DLSS will be available with minimal interruptions.

Page 3-7

Preventive maintenance will be scheduled in advance based on the established schedule and will occur during working hours agreed to by ADOT or each DL Office.

A typical preventive maintenance schedule is provided in Table 3-3 below.

Table 3-3: Preventive Maintenance Schedule

Subsystem Purpose Frequency

DLSS components Inspection, system diagnostics, and operation verification Semi-Annually

NEC will provide all maintenance services, including installation of software updates and security patches during scheduled downtime (e.g., late evening hours or early morning hours) to avoid high volume times when the system needs to be available to users. Scheduled downtime for performing preventive maintenance or other maintenance services at any site will not exceed two hours unless agreed upon in advance by ADOT. For all varieties of scheduled downtime, NEC will maintain electronic reports of downtime events and provide these reports to ADOT on a monthly basis.

In addition, NEC will schedule production management such as batch processing, job scheduling, automated import/exports, etc. at intervals agreed upon between NEC and ADOT. The production control schedule will be mutually agreed upon between NEC and ADOT and will be oriented around periods when the DLSS solution is expected to have the lightest use, such as during scheduled downtime.

3.3.2 System Backups and Data Management

Adequate backups are essential to ensure that data and systems configuration can be restored with minimal delay in the event of a catastrophic failure. NEC’s backup and data management strategy for the DLSS solution copes seamlessly with any malfunction and ensures continuity in the event of a disaster.

NEC uses a variety of market-leading technologies to provide a comprehensive backup and restore service. In the event of a catastrophic system failure, NEC’s Backup and Data Management strategy is designed (under most occurrences) to recover DLSS operations within a minimal period.

All disk imaging and system configuration backup policies will be published as part of the data documentation and property management processes. NEC will back up system configurations for continuity of operations considerations.

NEC's solution will perform backups on all system records once every 24 hours, 7 days per week, 365 days per year, and will use a snapshot-based approach to ensure there is no impact to system operations.

During the testing phase of the project, NEC will demonstrate the ability to restore the system using the development or test environments to ensure the operability of a backup process.

3.3.3 Remedial Maintenance

NEC’s support services include a comprehensive remedial maintenance program to ensure that ADOT’s system remains operational for the duration of the contract. If the NEC-provided cloud solution or software fails or otherwise demonstrates deficiencies when used as intended, NEC will restore all functions to their originally

NEC’s market-leading technology will provide ADOT comprehensive backup and restoration services.

Page 3-8 implemented state and in accordance with their documented technical specifications. This includes the repair, replacement, or exchange of system components and/or software. This service will be provided 24 x 7 x 365 for DL Offices.

Remote diagnostics and levels 2 through 4 support will be provided to isolate and further diagnose system or subsystem issues. All problems will be coordinated through assigned NEC FSEs. FSEs will regularly update ADOT regarding the status of the DLSS solution and estimated timeframes of problem resolution. See Section 8, "Technical Support," for more information on escalation and coverage.

For remedial maintenance calls, depending on the issue at hand, NEC may use VPN remote access to correct a system or remote workstation issue.

Remote access has historically proven to be extremely effective, significantly reducing downtime that may occur due to travel time. NEC will deploy remote control software with all peripheral workstations to provide immediate support to ADOT users. This software enables FSEs to work interactively with ADOT users and share screens to better understand the reported concern.

* * * E N D C O N F I D E N T I A L I N F O R M A T I O N * * *

3.3.3.1 Notification and Tracking Tools

NEC will use Microsoft Team Foundation Server (TFS), a web-based problem tracking system that integrates tightly into the development, support, and Quality Assurance (QA) internal processes, to automatically generate system emails each time the status of logged defects/issues changes. For example, if a software defect is discovered and logged, the initiating party will receive an email notification when work has begun on the issue, when the item is released for QA, and when the item is released. NEC also uses TFS to resolve trouble ticket issues or reported defects for Configuration and Patch Management.

3.3.3.2 Technical Coordination with Network Support

As part of the support package, NEC will provide ongoing cooperative support to assist with any technical network issues and security configurations that could affect the performance of the DLSS solution. This coordination will be in support of ADOT administrative staff and remote IT department staff. NEC maintains that working together to resolve a system network issue is the quickest way to correct the problem. NEC will also provide periodic reporting of routine activities, problems/issues, and resolution status.

Most service calls can be addressed remotely with tools available to the engineer on hand, thereby limiting downtime.