IM Evaluation Report 9-26-16.doc

Commonwealth of Massachusetts

Department of Transportation

Registry of Motor Vehicles Division and

Executive Office of Energy & Environmental Affairs

Department of Environmental Protection Scoring and Evaluation Report for the Procurement File For Responses to the RFR for the Contractor for the Vehicle Inspection and Maintenance Program

COMMBUYS Bid#: BD-16-1030-0R100-0R001-00000006553

Agency Document Number: MDOTBOSTON-IM Contacts:

Sharon Weber, MassDEP (617-556-1190)

Mary-Jo Griffin, MassDOT/RMV (857-368-7321) September 26, 2016 Table of Contents

31.

Strategic Sourcing Team Decision

32.

SST Procurement Process

32.1.

Strategic Sourcing Team

52.2.

Modifications to the Request for Responses

62.3.

Evaluation Process Memo

62.4.

Receipt of Responses

62.5.

Screening for Response Completeness

72.6.

Subgroup Meetings and Initial Scoring of Technical and Cost Responses

73.

SST Preliminary Scoring of Responses: Criteria 1 – 6

83.1.

Criteria 1: Overall Management Capability, Accountability, and Experience with Inspection and Maintenance Programs

183.2.

Criteria 2: Data Management

293.3.

Criteria 3: Provide Equipment, Equipment Maintenance, and Inspection Station Service

333.4.

Criteria 4: Program Support Elements

473.5.

Criteria 5: Cost and Financial Capability

493.6.

Criteria 6: Supplier Diversity Plan (SDP)

504.

SST Final Scoring of Responses

504.1.

Reference Checks

514.2.

Interviews

534.3.

SST Scoring Adjustments

544.4.

SST Selection of an Apparent Successful Bidder

545.

The SST Scoring Spreadsheet

Attachments:

1. Internal Scoring and Procedure Document

2. Questions asked of each Reference

3. Interview questions asked of each Bidder

4. Overall Point Score for the Responses Spreadsheet

Strategic Sourcing Team Decision

By the deadline date of April 7, 2016, the Massachusetts Department of Transportation/Registry of Motor Vehicles Division (MassDOT/RMV) and the Massachusetts Department of Environmental Protection (MassDEP) (collectively the “Agencies”) Strategic Sourcing Team (SST) received four responses to its Request for Responses (RFR) for a Contractor to implement the Vehicle Inspection and Maintenance (IM) Program (COMMBUYS Bid#: BD-16-1030-0R100-0R001-00000006553). These responses were from Applus Technologies, Inc. (Applus), Opus Inspection, Inc. (Opus), Parsons Environment & Infrastructure Group, Inc. (Parsons), and SGS Testcom, Inc. (SGS) (collectively the “Bidders”).

During the evaluation process, the SST assessed the Bidder’s submitted responses, conducted interviews with Applus, Parsons, and SGS, and obtained statements from Bidders’ references. As a result of this evaluation, the voting members of the SST gave the Bidders the following overall ratings, based on a 10 point scale:

Applus 6.8

Opus 4.2 Parsons 6.1

SGS

5.9 Given that Applus received the highest score, the voting members of the SST designated Applus as the apparent successful Bidder, and voted that the Agencies begin contract negotiations with Applus. In addition, the SST voted that, if an agreement as to the programmatic, fiscal, or administrative terms of the contract could not be reached within a reasonable period of time as determined by the SST, the Agencies would begin negotiations with the next highest scoring Bidder, Parsons.

The voting members of the SST arrived at the scoring results through the process described in the Evaluation Process Memo document, posted on COMMBUYS on March 21, 2016, and the Internal Scoring and Procedure Document and spreadsheet, dated April 7, 2016. In accordance with Step 6 of the Evaluation Process Memo, the SST wrote this memorandum summarizing the Bidder selection process and the reasons for the selection of the apparent successful Bidder. This document details the procedure and rationale used in scoring the Bidders’ responses for the selection of Applus as the apparent successful Bidder.

1. SST Procurement Process

1.1. Strategic Sourcing Team

The Agencies took appropriate precautions to ensure that the SST would be unbiased and fair in its assessment of the Bidders’ submissions. Prior to convening the SST, MassDEP was informed that a former MassDEP Assistant Commissioner had been retained by Parsons as a professional consultant with regard to its RFR response. As a result, MassDEP SST voting members and non-voting subgroup members completed, signed and filed forms for “Disclosure of Appearance of Conflict of Interest” as required under M.G.L. c. 268A, § 23(b)(3).

The voting members of the SST were:

MassDOT/RMV Mary-Jo Griffin, Director of Vehicle Safety and Compliance Services Michelle Lavoie, Director Strategic Planning MassDOT/RMV Sarah Zaphiris, Deputy Registrar of Operations

MassDEP Stephanie Cooper, Chief of Staff

Christine Kirby, Director, Division of Air and Climate Programs Sharon Weber, Deputy Director, Division of Air and Climate Programs The SST established subgroups assigned to evaluate specific technical, managerial and financial criteria. The subgroups were comprised of both voting and non-voting technical staff. The subgroups conducted initial evaluations of Bidders’ responses and recorded their impressions on draft evaluation worksheets, presented their work to SST voting members during scheduled meetings, answered questions, and clarified issues when requested by voting members. The voting members of the SST then made final determinations regarding Bidders’ scores on specific criteria. These determinations were summarized on the attached spreadsheet titled “Overall Point Score for the Responses.” This process is described in greater detail in Section 3 below. Section 3 also includes the SST’s summary comments evaluating Bidders’ responses, drawn from the subgroup draft evaluation worksheets and discussion among the SST members. The members of the subgroups and the criteria they evaluated are listed below.

Subgroup 1: Overall Management Capability, Accountability, and Experience with Inspection and Maintenance Programs

Paul Davis, I&M Program Branch Chief, and Sharon Weber of MassDEP; Mary-Jo Griffin and Terry Hayes, QA Manager, Vehicle Inspection, of MassDOT/RMV.

Subgroup 2: Data Management

Terry Hayes, Mary-Jo Griffin and Mike Devaney, Assistant Director of Vehicle Safety and Compliance Services, of MassDOT/RMV; Paul Lavallee, Senior Business Analyst, MassDOT-IT, Bob Shaw, Technical Operations Lead, MassDOT/RMV ATLAS Program, William Grant, ALARS Modernization, and Rodolfo Chata, Database Administrator, of MassDOT IT; Craig Woleader, Environmental Analyst, and Marc Bennett, Environmental Analyst, of MassDEP.

Subgroup 3: Provide Equipment, Equipment Maintenance, and Inspection Station Service

Craig Woleader, Marc Bennett and Julie Ross, Regional Planner, of MassDEP; Mary-Jo Griffin, Terry Hayes and Mike Devaney of MassDOT/RMV.

Subgroup 4: Program Support Elements

Motorist Assistance Network: Sharon Weber, Paul Davis, and John Flemmi, Environmental Analyst, of MassDEP; Mary-Jo Griffin, Terry Hayes, Paul Lavallee and Mike Devaney of MassDOT/RMV.

Inspector Training: Sharon Weber, Julie Ross and John Flemmi of MassDEP; Mary-Jo Griffin, Terry Hayes, Mike Devaney and Paul Lavallee of MassDOT/RMV.

Vehicle Maintenance Initiative: Sharon Weber and Paul Davis of MassDEP; Mary-Jo Griffin, Terry Hayes, Mike Devaney and Paul Lavallee of MassDOT/RMV.

Motorist Communications: Sharon Weber and Tom Higgins, Regional Planner, of MassDEP; Mary-Jo Griffin, Terry Hayes and Mike Devaney of MassDOT/RMV.

Subgroup 5: Cost and Financial Capability Sue Ann Richardson, Environmental Analyst, Tim Cahill, Senior Financial Analyst, Paul Davis, Craig Woleader and Marion Marcucelli, Environmental Analyst, of MassDEP; Therese Moran, Director of RMV Revenue, of MassDOT/RMV.

Subgroup 6 – Supplier Diversity Plan Mary-Jo Griffin, Michelle Lavoie, and Sarah Zaphiris of MassDOT/RMV.

Modifications to the Request for Responses

The RFR was posted on the Massachusetts (MA) COMMBUYS website on December 17, 2015. The deadline for the first round of questions on the RFR was January 7, 2016. Questions were received from Applus, Gordon Darby, Parsons, SGS, and CARCO Group. Answers to the first round of questions were posted on COMMBUYS on January 22, 2016.

A Bidder’s Conference and Webinar for the RFR was held on January 27, 2016. Representatives of 7 entities attended the conference and representatives of 5 entities registered for the webinar. No questions were asked at the Bidder’s Conference. An optional COMMBUYS Training session was also held for Bidders on January 27, 2016. Representatives from several potential Bidders attended. The Bidder’s Conference and Webinar attendee list was posted on COMMBUYS on February 1, 2016.

The deadline for the second round of questions on the RFR was February 11, 2016. Questions were received from Gordon Darby, Opus, Parsons, and SGS. Answers to the second round of questions were posted on COMMBUYS on February 24, 2016.

In response to questions submitted on the RFR and to clarify aspects of the RFR, one amendment to the RFR was posted on COMMBUYS on March 9, 2016.

1.2. Evaluation Process Memo

On March 21, 2016, the Evaluation Process Memo document was posted on COMMBUYS. This document summarized the process that the members of the SST would utilize to evaluate the Bidder responses and described the criteria to be evaluated and the percentage of weight each criteria would be allocated as part of the total analysis of a Bidder’s response, as listed below at the beginning of sections 3.1 through 3.6. An Internal Scoring and Procedure Document and spreadsheet, dated April 7, 2016, detailed subcriteria and their weighting, as listed below in italics at the beginning of sections 3.1 through 3.6.

1.3. Receipt of Responses

Applus, Opus, Parsons, and SGS posted responses prior to the April 7, 2016, 2 pm deadline. In addition to the technical and cost responses, Bidders submitted the following physical samples:

1.4. Screening for Response Completeness

In accordance with Step 1 of the Evaluation Process Memo, selected SST members screened the RFR Responses using the Bidders Response Checklist found in Attachment K to the RFR to determine whether each Bidder met the threshold requirements for consideration of its bid. Marion Marcucelli of MassDEP and Terry Hayes of MassDOT/RMV examined the four submissions using the RFR Amended Attachment K: Bidder Response Checklist, posted on COMMBUYS on March 9, 2016. This examination included: (1) an initial screening to determine whether the responses met the packaging, format, and page limit requirements of the RFR; (2) a detailed screening to determine whether the responses contained all of the elements required in the RFR; and (3) a review of whether minimum Specifications and Bidder qualifications set forth in the RFR had been met. Based on the review of the two screeners, the SST determined that each Bidder met the threshold requirements for consideration of its bid.

In accordance with Step 2 of the Evaluation Process Memo, the SST determined that all four responses were “complete and in substantial conformance with the RFR” and should be analyzed in greater detail.

1.5. Subgroup Meetings and Initial Scoring of Technical and Cost Responses In accordance with Step 3 of the Evaluation Process Memo, the SST completed a preliminary evaluation of the Bidder’s responses. The purpose of the preliminary evaluation was for the SST to review and rate the responses, on a preliminary basis, based upon the Bidder’s responsiveness to the evaluation criteria. The SST subgroups met separately to analyze their respective sections of the responses and assign initial scores to the responses for each criterion, which were then presented to the SST.

2. SST Preliminary Scoring of Responses: Criteria 1 – 6

In accordance with Step 3 of the Evaluation Process Memo, the SST evaluated the technical responses on their merits before analyzing cost responses, thereby enabling an unbiased scoring of the technical quality of each bid.

The SST convened three times, on May 13, 2016, June 6, 2016, and June 10, 2016, to meet with the subgroups, review the subgroup analysis, and to score, compare and contrast each of the responses with regard to the six criteria. In addition to the voting SST members, appropriate subgroup members attended these meetings, as well as lawyers from both Agencies, in an effort to ensure that the SST assessment process complied with legal guidelines pursuant to 801 CMR 21.00, procurement for goods and services.

At the first two meetings of the SST, the voting members scored each technical response for each of Criteria 1 through 4, and Criteria 6. This scoring is described in detail in Sections 3.1 through 3.4, and 3.6, below. At the third meeting, the cost subgroup presented its evaluation to the SST and the SST completed its scoring of the cost responses for criteria 5 as described in Section 3.5 below.

This section 3 explains the scores assigned by the voting members of the SST to each Bidder for Criteria 1-6. It highlights the differences between the Bidders that the voting members of the SST considered particularly significant.

The scores for the responses are based on the following 10 point scale.

Scoring Quality Standard 0 points - Unacceptable

1 point - Acceptable –

2 points - Acceptable

3 points - Acceptable +

4 points - Good –

5 points - Good

6 points - Good +

7 points - Good ++

8 points - Outstanding –

9 points - Outstanding

10 points - Outstanding +

For Criteria 1-5, each of the respective subgroups met and scored Bidders’ responses based on articulated subcriteria and their associated weighting, in accordance with the Internal Scoring and Procedure Document and initial spreadsheet, dated April 7, 2016. Criteria 6 had no subcriteria. The SST then convened and the subgroups met with the SST voting members to discuss and clarify their initial assessments. After a detailed discussion and re-assessment by the voting members, the SST then performed its own scoring of each subcriterion to determine the total score for the applicable criterion. In scoring the subcriteria, the SST considered (1) the contents of each Bidder response to the response elements and (2) the presentations by the subgroups on the significant strengths and weaknesses of and differences between the responses, including information addressing the questions listed under each criterion in the evaluation document.

Following the meeting between the SST voting members and the subgroups, the resulting qualitative score and weighted score for each criterion and subcriterion was recorded on the attached spreadsheet titled “Overall Point Score for the Responses.”

2.1. Criteria 1: Overall Management Capability, Accountability, and Experience with Inspection and Maintenance Programs

Criteria 1 represented 25% of the total possible score of each response and consisted of six subcriteria. Each of the subcriteria was scored individually, with the scores weighted as follows:

A. The Bidder’s ability to manage and implement the Program, including providing key personnel and staffing plans and demonstrating how its key personnel and subcontractors will be responsible and accountable to the Agencies on critical performance criteria. [Weight = 5.0%] B. The Bidder’s ability to develop and implement management and tracking systems to ensure contract performance. [Weight = 3.4%] C. The Bidder’s depth and breadth of experience in developing and implementing IM programs (including the experience of any proposed subcontractors). [Weight = 4.0%] D. The Bidder’s ability to develop and have testing equipment, Workstations, data systems, and Motorist Assistance Centers operational and on schedule to meet the October 1, 2017 startup date. [Weight = 5.0%] E. The Bidder’s ability and capacity to be flexible and responsive to potential Program changes during the length of the Contract. [Weight = 3.4%] F. The overall quality of the response, including how well the Bidder understands the challenges of managing the Program and how well the Bidder incorporates a systematic approach to quality assurance and quality control, as well as continuous improvement throughout the contract. [Weight = 4.2%] The above subcriteria were scored as described below:

A. The Bidder’s ability to manage and implement the Program, including providing key personnel and staffing plans and demonstrating how its key personnel and subcontractors will be responsible and accountable to the Agencies on critical performance criteria.

Applus Good++

Opus Acceptable+

Parsons Good

SGS

Acceptable

Scoring Summary Comments

Applus – The SST found that Applus has broad experience with the program elements that MA has in its program and Applus has experience with various sized programs. The SST found that Applus’ limited corporate structure and the high executive-level involvement with program implementation (Chief Executive Officer 30%, Director of Operations 35%) was good. Among the proposed staff for the MA program are: the Director of Operations with experience in Federal Motor Carrier Safety Administration (FMCSA) enforcement cases in Texas; a documentation specialist to document all systems, processes and procedures for the program; and scan tool staff that worked on the California (CA) certification. Applus has already implemented a program in Idaho using its CA-certified scan tool. The SST found that Applus proposed that it will provide its own CA-certified scan tool and its own Vehicle Inspection Database (VID).

Opus – The SST rated Opus as Acceptable+ primarily because they found Opus’ proposal lacked clarity regarding staffing levels, dedication of staff to the MA program, and the role of key personnel such as the Program Manager. The SST found that Opus has experience with various sized programs and broad experience with the program elements that MA has in its program. However, because Opus has only recently purchased the scan tool provider (Drew Technologies), there is no history of Opus providing its own scan tool.

Parsons – The SST found that Parsons has broad experience with the program elements that MA has in its program, and that Parsons has experience with various sized programs. The SST determined that Parsons proposed a good management structure with a Vice President as Program Manager who would have responsibility for the project’s success. The SST found that Parsons did not clearly identify the number of personnel who would be assigned to some positions. The SST noted that Parsons did not include any resources during program development and implementation that would be provided by the scan tool provider.

SGS – The SST rated SGS Acceptable primarily because of SGS’ limited experience as an equipment provider and because the required information for SGS’ major subcontractor Worldwide Environmental Products (Worldwide), the equipment supplier, was not included in SGS’ response, nor was the required supplier’s list. The SST found that SGS has experience with larger programs (SGS is responsible for the VID in CA) and SGS’ major subcontractor Worldwide has experience with smaller programs and that this is a good distribution of experience. The SST also recognized that SGS has experience with the program elements of Massachusetts. The SST believed that SGS’ response showed a lack of understanding of the program’s current Vehicle Maintenance Initiative (VMI).

Staffing Plan Comments Applus – The SST found that Applus proposed both an implementation and operations team and provided an organization chart for each. Applus provided a list of 18 key staff and indicated the percent of time that each person would be involved in implementation and operations. The staffing plan included 48 positions that would be involved with the program and their responsibilities. Of the four key personnel, the Program Manager and the Assistant Program Manager would have MA as their primary work location.

Opus – The SST found that while Opus listed 50 technical staff who would be associated with the MA program, all were already allocated to other assignments and the proposal failed to identify the percent of effort each staff person would dedicate to the MA program (10 of these 50 also have roles with the Virginia (VA) program). The proposal identified 18 key personnel, but the SST noted that the staffing plan did not indicate if their role would be for implementation and/or operations. Also, the staffing plan was not clear regarding the MA staff assignments: one person was shown as spending 100% of her time in Arizona (AZ) and 90% of her time in MA. The SST also found contradictions between the organization chart and table regarding to whom staff would report so that it was difficult for the SST to determine the exact role of the Program Manager. The SST also found that a resume of the President of Drew Technologies, the scan tool provider, was included, however no Drew technical staff were identified as working on this project, even though the RFR required identification of personnel.

Parsons – The SST found that Parsons proposed both an implementation and operations team, and provided an organization chart for each. The SST also found that Parsons provided a list of 42 staff involved with developing, operating or managing the program, and provided the percent of time that most positions would be involved. Of the 6 key personnel, 3 would have MA as their primary work location. Parsons identified the person responsible for developing the workstations; however, no technical staff from the scan tool provider was identified, as requested in the RFR.

SGS – The SST found that SGS proposed an implementation and operations team, and provided an organization chart by functional group showing staff assignments and the key personnel to whom staff report. SGS identified 59 persons that would be involved with the implementation and operations of the program, and identified 5 key personnel and the percent of time each would be assigned to MA. Two key personnel (the Program Manager and the Motorist Assistance Center (MAC)/Operations Manager) would have MA as their primary work location. However, the SST found that SGS did not provide any information for its major subcontractor, Worldwide.

Distribution of Effort among Subcontractors:

All Bidders provided information regarding their subcontractors, and the portion subcontractors represent of proposed costs. The following table summarizes each Bidder’s subcontractors:

Training

Total Subcontract (# and % of cost) 37 subcontractors total % unclear 6 subcontractors ~15-20% 34 subcontractors ~12% 3 subcontractors ~21%

B. The Bidder’s ability to develop and implement management and tracking systems to ensure contract performance.

Applus Outstanding–

Opus Acceptable

Parsons Good–

SGS

Good+

Scoring Summary Comments

Applus – The SST found that Applus’ response provided a comprehensive discussion of management and the most detail for a management plan and tracking systems. Applus proposed very good management tracking tools including Project Management Body of Knowledge (PMBOK), Microsoft (MS) Share Point, MS Project, and Elementool’s Change Management Tool software for tracking various program elements and software changes. They also committed to using Department of Defense (DOD)-based Integrated Project Teams to implement the program. The SST noted that Applus’ Quality Management system has been International Organization for Standardization (ISO) 9001 certified since 2007.

Opus – The SST found that Opus’ response provided no clear overall management plan. Opus proposed using the Agile-based software development system. Opus’ discussion centered on tracking changes, and proposed using MS Team Foundation Server (TFS) to manage change orders and enhancements. TFS integrates MS Visual Studio, MS Project and MS Excel, leaving the SST with the opinion that Opus’ change tracking management met the RFR requirements.

Parsons – The SST found that the Parsons response proposed using the core project management plan they have been using in Massachusetts for the past 8 years, but lacked details regarding this management plan, though did indicate they would continue to use Parsons’ DocSpot for document tracking and sharing and MS Project with modifications and enhancements for the new program features. However, the SST noted that the Parsons proposal failed to address tracking software changes which SGS, not Parsons, tracks under the current contract.

SGS – The SST found that the SGS proposal provided good management and tracking tools and provided a good discussion of their management system and approach, noting their successful use of PMBOK in 5 other states. SGS proposed using MS Project, Hotline Expert Automation Tool (HEAT) software, the SGS Management and Reporting Tool (SMART), and JIRA software for tracking various program elements and software defects. However, the SST noted that SGS’ proposal focused more on change tracking (which SGS does for the current MA program) than on their overall management approach.

C.

The Bidder’s depth and breadth of experience in developing and implementing IM programs (including the experience of any proposed subcontractors).

Applus Good++

Opus Good–

Parsons Good

SGS

Good–

Scoring Summary Comments

Applus – The SST found that Applus provided a good history of the lessons learned from the problems they experienced under the previous contract in MA, how it led to the evolution of their firm, and Applus’ improved approach to IM programs. The SST noted that Applus had a broad range of experience, can provide its own VID, and has had a history manufacturing scan tools, manufacturing one of the CA-certified OBD scan tools. The SST noted that Applus’ proposed no middle-man for VID or scan tool, which provides accountability, which has been an issue for the MA program in the past. The SST found that Applus provided very detailed information on litigation, convictions, investigations, judgments and other actions.

Opus – The SST found that Opus provided a history of Opus providing databases, and indicated experience with a variety of the elements proposed for the MA program. The SST noted that most of their experience in the US was with smaller programs, although Opus was recently awarded the New York (NY) program. The SST also noted that while Drew Technologies will provide some scan tool and vehicle diagnostic expertise to Opus, there is no history of Opus’ partnership with Drew Technologies yet. The SST noted that Opus provided only a cursory description of litigation, convictions, investigations, judgments and other actions.

Parsons – The SST found that Parsons supplied its history of providing programs using the contract management approach with subcontractors providing various services. Parsons showed experience with a broad range of services, including providing a VID for some larger programs. The SST noted that Parsons has relied on subcontractors to provide inspection equipment, and that the responsiveness of equipment manufacturers to the prime contractor has been an issue for the MA program in the past. The SST also noted that Parsons provided only a cursory description of litigation, convictions, investigations, judgments and other actions.

SGS – The SST found that SGS showed it has a limited history with IM programs, although most of SGS’ experience is with larger programs. SGS had substantial experience providing VIDs and had experience providing equipment in NY for over 10,000 inspection stations. The SST noted that the SGS proposal relied on a subcontractor to provide the OBD scan tool. The SST is concerned about this reliance because responsiveness of equipment manufacturers to the prime contractor has been an issue in the MA program in the past. The SST also noted that the SGS proposal showed SGS has limited experience providing repairer-related services or training and did not understand the current VMI program. The SST found, however, that Worldwide has substantial experience providing inspection equipment, including a CA-certified scan tool, and some experience as a prime contractor with smaller programs.

Relevant Experience Comments Applus – The SST found that Applus indicated Applus had experience with 12 IM programs, and provided a discussion of 10 of those. The SST was not concerned that information was not provided for the programs outside the United States (US), in Canada (Ontario) and Ireland. The SST noted that Applus had substantial experience in all elements of the MA program, including video monitoring, and had been awarded contract extensions and renewals. The SST also noted that Applus acquired AutoLogic, a scan tool firm, in 2006, now makes its own CA-certified OBD scan tool, and now provides its own VID. The SST also noted that Applus highlighted that it is a member of the International Motor Vehicle Inspection Committee (CITA) and can incorporate CITA’s standards into their solutions.

Opus – The SST found that Opus indicated Opus had experience with 21 IM programs, and that Opus clearly presented all requested information for the 5 programs that are similar in scope to the MA program. The SST noted that the recent acquisition of Drew Technologies will enable Opus to provide its own CA-certified scan tool, in addition to its own VID, since the Opus proposal indicated that Drew Technologies will provide its CA-certified OBD tool, repair diagnostics, and advanced vehicle communications expertise to Opus.

Parsons – The SST found that Parsons provided a discussion of 12 other IM programs with which it has had experience, including that Parsons had broad experience covering all elements of the MA program and had been awarded contract extensions and contract renewals. The SST noted that Parsons now provides its own VID but would need to rely on subcontractors to provide CA-certified scan tools.

SGS – The SST found that the SGS proposal indicated that SGS has experience with 5 other IM programs, and that Worldwide, their major subcontractor, has experience with 6 other IM programs. The SST noted that together, the two companies have broad IM experience, with the exception of waivers and other MAC-related services and VMI programs. The SST noted that SGS provides its own VID but would need to rely on Worldwide to provide the scan tool.

Specific Qualifications (defaults; litigation; investigations; debarment; data breach, certificate of good standing) Comments Applus – The SST found that Applus’ response indicated that there are no defaults, criminal convictions or investigations, debarments or data breaches. The SST noted that Applus provided details regarding 5 civil cases, of which 4 were dismissed by the court. The fifth involved a patent infringement where Applus filed a counter claim; the court dismissed the original claim and the parties agreed to dismiss the counter claim prior to a court decision. Applus holds a certificate of good standing from MA Secretary of the Commonwealth.

Opus – The SST found that Opus’ response indicated that there were no issues other than some minor normal business-related claims pertaining to small claims, insurance matters, and employee disputes. The SST noted that no further details were provided. The SST noted that Opus holds a certificate of good standing from MA Secretary of the Commonwealth.

Parsons – The SST found that Parsons’ response indicated that there are no defaults, criminal convictions or investigations, debarments or data breaches. The response noted that given the scope of Parsons’ operations there are a number of civil lawsuits pending in the normal course of business at any given time but these will have no adverse impact on Parsons’ financial condition or its operations. No further details were provided. The SST noted that Parsons holds a certificate of good standing from MA Secretary of the Commonwealth.

SGS – The SST found that SGS’ response indicated that there was no litigation, defaults, investigation, debarments or data breaches for SGS or Worldwide. SGS holds a certificate of good standing from MA Secretary of the Commonwealth. Worldwide holds a Certificate of Status from the California Secretary of State.

D.

The Bidder’s ability to develop and have testing equipment, Workstations, data systems, and Motorist Assistance Centers operational and on schedule to meet the October 1, 2017 startup date.

Applus Good+

Opus Acceptable

Parsons Good+

SGS

Good

Scoring Summary Comments

Applus – The SST found Applus’ proposed thorough schedules for specification and acceptance testing plan (ATP) development, the idea of a rollout website, the workstation delivery schedule, the training curriculum, the orientation training schedule, the MAC deployment schedule, the repairer training effort, and the covert auditing schedule. The SST expressed some concern that advertising for inspector training would be too early. The SST noted that Applus did not provide a schedule for alternative workstation providers.

Opus – The SST felt that Opus’ proposal did not provide enough time for the software specifications or for the software and hardware ATPs that would be needed to successfully start the program. The SST also found that delivery of the Workstations five months prior to the beginning of the program was much too early. Otherwise many of the proposed dates and milestones for other segments of the program seemed reasonable, including the data migration schedule, industry communications schedule, MAC and VMI rollout schedules, and covert auditing schedule. Opus did not provide a schedule for alternative workstation providers.

Parsons – The SST found Parsons proposed thorough schedules for specification and ATP development, including an as-built spec, and workstation delivery recognizing the extra time required to install cameras. The SST also found Parsons’ orientation training window, MAC deployment, repairer training effort, and covert auditing schedule useful. The SST expressed some concern that the advertising for inspector training was too early.

SGS – The SST found the proposed schedules for the specification and ATP development, workstation delivery, communication development schedule, orientation training, MAC deployment, and covert auditing schedule were good. The SST expressed concerns that the advertising for inspector training was too early, and that the VMI training development period was too short for the level of sophistication required in those classes.

E.

The Bidder’s ability and capacity to be flexible and responsive to potential Program changes during the length of the Contract.

Applus Good++

Opus Acceptable–

Parsons Good–

SGS

Good+

Scoring Summary Comments

Applus – The SST felt that Applus provided the most comprehensive and extensive list of potential issues along with a good summary of the solution for each. The Applus proposal indicated that potential problems and mitigation approaches are included as part of their management plan, and that the list of potential problems would be updated by Applus and discussed at project management meetings as an ongoing aspect of their management process upon implementation.

Opus – The SST noted that Opus failed to directly identify any key problems that could be anticipated in implementing the program. As a result, the SST also noted that the Opus proposal did not specify any approaches that Opus would use to resolve key problems. As a result of these shortcomings, the SST felt an Acceptable- was the appropriate score for this subcriterion.

Parsons – The SST noted that the Parsons response identified only one potential risk: the installation of cameras in the inspection bays. The SST noted that Parsons proposed a mitigation measure for this risk through the use of a MA Minority and Woman-Owned Business Enterprise (MBE/WBE)-certified subcontractor specializing in security system installations to aid in the development of installation plans for cameras, but the SST also noted that the Parsons response failed to identify such firm as a subcontractor in its response. Parsons points to their success implementing programs in other locations as examples of their successful risk management, including program transitions from other contractors, as being the reason there were no additional risks.

SGS – The SST believed that SGS presented a well thought out list of risks, with a good evaluation of each risk, the potential of each risk, and proposed solutions. The SST took note of SGS’ plan to update the list during implementation.

F.

The overall quality of the response, including how well the Bidder understands the challenges of managing the Program and how well the Bidder incorporates a systematic approach to quality assurance and quality control, as well as continuous improvement throughout the contract.

Applus Good++

Opus Acceptable+

Parsons Good+

SGS

Good++

Scoring Summary Comments

Applus – The SST found Applus’ flow chart excellent because it showed a detailed understanding of the inspection process and even suggested several improvements to the inspection process, such as an odometer check and an improvement to the opacity test. The SST found that Applus also provided clear discussions of the flow chart, inspection station connectivity requirements, motorcycle station requirements, and the obstacles that would result from the use of 3rd party workstation vendors. However, the SST determined the Applus flow chart should not earn a higher score because it incorrectly put the safety test before the emissions test, the response provided no mention of alternatives to Automated Clearing House (ACH) payments (necessary for government agencies and municipalities), and there was no discussion regarding stations that fail to pay.

Opus – The SST gave Opus a low score on this subcriterion because it included mistakes and errors in the test logic presented in their flow charts, specifically, the safety test being performed before the emissions test; the failure to allow the safety test to be done on out of state vehicles, which will be required for some; the lack of the required gross vehicle weight rating (GVWR) cut-off for diesel opacity in the opacity test logic; and the lack of the test interrupt process from the OBD test logic. The SST found problems with Opus’ discussion of requirements for Inspection Station connection to the VID, and the connection discussion was not easy to follow. The SST found no mention of experience with multiple vendors in other programs; and a good discussion of 3rd party vendor obstacles.

Parsons – The SST determined that Parsons provided a detailed flow chart but found some decision point information was lacking, such as GVWR for test type, and some new car scan pass/fail information was incorrect. The SST found that Parsons proposed a detailed station participation agreement including requirements for motorcycle stations. The SST felt Parsons has experience with 3rd party workstation vendors in Georgia (GA) and Pennsylvania (PA) and provided a very good comprehensive discussion of obstacles that result from using 3rd party vendors. However, the SST determined the Parsons flow chart should not earn a higher score because it suggested exemptions from cameras might be necessary in certain cases and it lacked some details on which standards were to be met by Parsons equipment and which by the 3rd party vendor’s equipment.

SGS – The SST found that SGS provided a detailed flow chart and a detailed station participation agreement. However, the SST noted that some decision-point information was lacking, such as GVWR for test type, and some information, such as kit cars and new vehicle scans, was missing. The SST also felt SGS provided good explanatory text and additional detail for the inspection interrupt and MAC referral logic. The SST felt SGS showed good experience with 3rd party workstation vendors in CA where they run the VID and the SST determined that SGS provided a very comprehensive discussion of obstacles that result from using 3rd party equipment vendors. However, the SST found that SGS provided little discussion of requirements for motorcycle stations and determined SGS’ response proposed improperly charging motorcycle stations for test authorizations.

Inspection Station Network Comments Applus – The SST felt Applus provided a very good outline of their approach to the network, including orientation seminars and webinars, a web page dedicated to roll out and enrollment, staff visits to stations to determine best locations for cameras, reminders to stations to enroll, and help desk availability. Applus provided network connectivity standards for inspection stations and also for motorcycle stations. The SST also felt that Applus did a good job addressing the 3rd party workstation provision, noting that Applus could support it but that it would result in an increase to the per inspection fee due to ATP of other vendors, modifications to rollout, training of help desk staff, modifications to orientation training, etc. This included Applus also noting that having four vendors in the GA program had quadrupled efforts for software changes, ATPs, service, and help desk responsiveness, and also created problems with consistent test results if a vehicle visits multiple stations with different equipment.

Opus – The SST felt that Opus’ discussion of connectivity was not easy to follow. The SST scored the Opus proposal lower because its discussion included dial-up internet connections while the RFR specified either Ethernet or wireless network card for connecting. It was unclear to the SST if stations can handle the required network address translation to a private subnet to be able to connect to Opus’ VID, and the requirement for Workstations to operate in ‘kiosk mode’ was not explained. While the SST found Opus’ proposed online enrollment for inspection stations, including the ordering of equipment and supplies, was helpful, it was not clear what advantage there would be for tracking and filling orders out of state at its center in Connecticut (CT). The SST scored Opus’ proposal lower because Opus did not mention multiple vendors experience in other programs. The SST noted that the Opus proposal did provide some discussion of obstacles with 3rd party vendors that was requested, including: a discussion of 3rd party vendor requirements, such as designing software to Opus’ specifications; enrollment of stations with 3rd party workstations would require a visit from a technician to ensure correct security and communication protocols; and concerns with 3rd party vendors meeting the implementation schedule.

Parsons – The SST noted that Parsons proposed using station participation agreements to manage the network, and met RFR requirements for stations to have high-speed internet. The SST also noted that Parsons proposed an option for stations to recycle their existing workstations for a fee. The SST scored the Parsons proposal lower because it proposed exemptions for camera requirements, which was not allowed for in the RFR. The SST noted that Parsons has experience with 3rd party vendors in the GA and PA programs and provided a comprehensive discussion of obstacles including implementation and ATP, delays in VID and software deployments, inspector training with more than one workstation, and workstation quality control (QC) analysis. Parsons also noted that 3rd party vendors would increase costs associated with testing and possible liquidated damages, which would need to be reimbursed by the 3rd party vendors.

SGS – The SST noted that SGS proposed building on its network experience in the current program. SGS proposed using a management and reporting tool to manage the network, requiring high-speed internet or a waiver from the Agencies if it is not available (the RFR specifically required only high-speed internet); and public stations would also be required to post hours of operations. They proposed a video drive on the workstation that would be used to buffer video content from the workstation to the VID with all video files uploaded to the VID by the end of the day; the workstation would not shut down until all the files have uploaded. The SST noted, however, that there was only minimal discussion of motorcycle station requirements. SGS has experience in CA where it provides the VID for multiple vendors. SGS would require that a field service representative be present when the 3rd party workstation was installed (there would be a service fee to the 3rd party vendor), and that service and supplies would have to be provided by the 3rd party vendor. There would be no remote access to 3rd party workstation so an SGS Field Service Representative (FSR) would need to be present for repairs, and there would be complications with testing, security, accessibility, scheduling, and connections. SGS improperly proposed requiring motorcycle stations to pay fees to access the network. The SST noted that SGS proposed a comprehensive station participation agreement.

2.2. Criteria 2: Data Management

Criteria 2 represented 25% of the total possible score of each response and consisted of three subcriteria, with some of the subcriteria further divided into categories. Each of the subcriteria was scored individually, with the scores weighted as follows:

A. The Bidder’s ability to develop and maintain reliable data systems that properly, reliably, and quickly upload data for the Agencies to use. [Weight = 19%] Subcriteria A was further divided into twelve categories with the 19% weight divided up as follows. The descriptions for items 1 to 12 below were drawn from RFR Section 3.2.1.5:

1.

3% Proposed Database System 2.

3% Reliability and Response Time 3.

2% Automated Licensing and Registration System (ALARS) Interactions 4.

1% Potential Causes of Data Transmission Problems 5.

2% Inspection Software 6.

1.5% Data Transfer, Storage and Tracking 7.

1.5% Data Security and Quality Assurance (QA) 8.

3% Database Access and Reporting 9.

0.75% Problem Tracking and Help Desk Software 10.

0.5% Migration of Inspection Data from Previous Program 11.

0.25% Location of Data Processing Facilities 12.

0.5% Database and Inspection Software ATP

B. The Bidder’s ability to revise data systems as needs change, such as MassDOT/RMV’s modernization of its ALARS system. [Weight = 2.0%] C. The Bidder’s ability to address improper and fraudulent inspections and vehicle tampering, and the Bidder’s ability to provide the Agencies with recommendations on how to revise inspection procedures in software coding to improve the detection and management of fraudulent practices and tampered vehicles. [Weight = 4.0%] The above subcriteria were scored as follows:

A.

The Bidder’s ability to develop and maintain reliable data systems that properly, reliably, and quickly upload data for the Agencies to use.

1. Proposed Database Systems – This overview must include a logical data model for all data in the data warehouse category. Bidders shall include an overview of the System architecture and major components of the Database, including all redundancies and backup systems. A complete, detailed data dictionary is not required for this response. A brief description of the methodology and tools used to create the overview and a key explaining any diagram must be included.

Applus Good++

Opus Good

Parsons Good++

SGS

Good+

Scoring Summary Comments

Overall, the SST found that all Bidders had met and exceeded RFR requirements for their proposed database systems. Applus, Parsons and SGS proposed private clouds, similar to what has been used for the last 2 inspection programs. Opus proposed a public cloud, Amazon, which is reliable and has a very high uptime. Though either system would work, the SST concluded the private cloud has proven to be secure and the Bidder would have more control over the server.

Applus indicated the Agencies’ staff will be able to use any type of modern computing device to access the VID, including a desktop PC, laptop, tablet or smartphone, which the SST concluded was a very desirable feature. Applus proposed Internap and CoreSitem datacenters in Atlanta and Chicago, which are ISO 9001 certified for quality, but there was no mention of the Service Organizational Control (SOC) report SOC2 or ISO datacenter ratings.

Opus proposal met Federal Risk and Authorization Management Program (FedRAMP) standards. However, the SST had data security and control concerns with utilizing a public cloud, and the potential for data corruption. Opus proposed no other security certificates.

Parsons complies with FedRAMP. Both the Richardson and Culpepper data centers hold Statements on Standards for Attestation Engagements (SSAE) 16 (SOC1) and SSAE 16 (SOC2), and are ISO/International Electrotechnical Commission (IEC) 27001 certified. The SST found Parsons to have excellent security measures and certifications for the database system.

The SST concluded the current SGS VID structure works well. SGS stated that SGS’ SMART could be accessed from the Agency offices, home networks and mobile networks, which the SST found useful. SGS recommended 2 certification choices, but left it up to the Agencies to choose. The SST expected SGS to recommend one, not require the Agencies to choose.

2. Reliability and Response Time – Explain how the Database will meet or exceed the reliability and response time specifications for the anticipated volume of inspections.