RFP_23NC0412_Attachment_14_Evaluation_and_Scoring_Criteria.pdf

23NC0412 ATTACHMENT 14 – EVALUATION AND SCORING CRITERIA

Name of Bidder: ________________________________________

Name of Evaluator: ________________________________________

Total Points Available:

Technical Evaluation: 360 points

Cost Evaluation: 160 points

Total: 520 points

Scoring Definitions:

GOOD Proposer’s response provides sufficient details and/or examples demonstrating the ability to meet all of the requirement(s).

FAIR

Proposer’s overall response provides details and/or examples, but does not demonstrate the ability to meet all the requirement(s). The bidder may have addressed particular requirements with insufficient details, or not at all, but the overall response demonstrates that the bidder possesses the ability to meet the requirement(s).

POOR

Proposer’s response provides insufficient details and/or examples demonstrating the ability to meet the majority of requirement(s).

FAIL Proposer’s response is omitted completely, or does not address or relate to the requirement(s).

1. DESIRED QUALIFICATIONS (60 POINTS)

FAIL POOR FAIR GOOD

1.1 Project Team Description and Organizational Chart (10 points): The Proposer provides a detailed project organization chart and description of its team organizations and key personnel and clearly shows the specific role that each team member will have in performing the project functions.

0-2 3-5 6-8 9-10

1.2 Project Team Accountability (5 points):

The Proposer’s project organizational chart describes the administrative mechanisms by which each team member will be held accountable for performance.

0-1 2 3-4 5

1.3 Team Organization and Governance (10 points): The proposer describes the project team’s experience and competency in the following areas:

• Experience and expertise in: developing and delivering tools and resources to support staff regulatory oversight, stakeholder engagement, reporting mechanisms, data visualization, secure information transmittal, and project management;

• Managing employee turnover rate;

• Managing subcontractors, if applicable;

• Managing demand to scale operations up/down;

• Managing employee bandwidth; and

• Redundancy Plan (Plan to ensure continuity of service through turnover of personnel).

1.4 Project Administration Experience (15 points): Proposer’s team demonstrates understanding, expertise, capacity, and experience in successfully managing complex administration, data management, and responding to client or participant needs in a timely and competent manner. The team shall include at least one in house, senior, and experienced overall Project Manager. Proposer’s team must demonstrate ability to manage a team including multiple subcontractors, if applicable. Experience shall reflect the Proposer’s effectiveness in managing time and resources across multiple projects, setting work plan schedule, and workflow tracking.

0-3 4-8 8-12 12-15

1.5 Relevant Experience (20 points): The Proposer’s Project Team should demonstrate expertise, capacity, and experience by providing examples and 0-5 6-10 11-15 16-20 describing their role in: developing and delivering tools and resources to support staff regulatory oversight, stakeholder engagement, improving reporting mechanisms, data visualization, secure information transmittal, and project management. The following related skills will be assessed:

A. Supporting staff optimization of portfolio oversight B. Program reporting assessment, analysis, and recommendations for process improvement:

C. Development of performance reporting templates;

D. Data visualization;

E. Engagement with stakeholders on various energy policy matters F. Study recommendations and subject matter expertise in energy efficiency;

G. Developing and implementing a catalog and indexing approach and format for collected digital records;

H. Collecting and conveying collected information securely (this should also include the ability to redact sensitive information automatically);

I. Tracking progress and analyzing progress towards required and interim goals and metrics.

Maximum Points Possible: 60

TOTAL SCORE:

2. Work Samples (15 points)

FAIL POOR FAIR GOOD

2.1 SAMPLE REPORT #1 (5 points): Proposer’s sample work should be an excerpt of a past guidance document and associated tool intended to facilitate assessment of compliance with regulatory requirements. The sample work product should be appropriate for a public, non-technical audience.

0-1 2 3-4 5

2.2 SAMPLE REPORT #1 (5 points): Proposer’s sample work product is of high quality and shows a clean and simple to use layout and design of an electronic form.

0-1 2 3-4 5

2.3 SAMPLE REPORT #2 (5 points): Proposer’s sample work product should be a demonstration of data visualization via a dashboard or other means to facilitate assessment of progress. Ideally, the visualization should be dynamic and allow the user to manipulate what and how data is shown. The sample sufficiently and

0-1 2 3-4 5 articulately communicates results and outcomes. The proposer’s role in the diagram should be evident.

Maximum Points Possible: 15 TOTAL SCORE:

3. Overall Approach (60 points)

FAIL POOR FAIR GOOD

3.1 Proposer sufficiently details the approach to execute project oversight and the plan to coordinate with subcontractors (if applicable) to ensure success of the work.

(10 Points)

3.2 Proposer sufficiently and clearly identifies specific interdependency and interaction points with the Commission and between deliverables. It is not sufficient to only acknowledge the interaction points. The proposer must clearly articulate how the interaction points will be managed by the Proposer and address how risks for meeting Commission specific deadlines and Commission needs will be mitigated. (20 Points)

0-5 6-10 11-15 16-20

3.3 Proposer sufficiently details the approach to address all identified deliverable submission requirements for all deliverables under this RFP. (10 Points)

3.4 Proposer’s narrative and approach to each deliverable sufficiently demonstrate knowledge of Commission processes, Commission proceedings and familiarity with the energy efficiency market. (10 Points)

3.5 Proposer’s narrative demonstrate the proposer’s approach for dealing with uncertainties due to evolving needs such as: changes in policy objectives, Commission directions in implementing D. 23-06-055 or other related proceedings, or coordination with other stakeholders, including utilities. (10 Points)

MAXIMUM POINTS POSSIBLE: 60 TOTAL SCORE:

4. Tasks and Deliverables (190 Points)

FAIL POOR FAIR GOOD

Task 4: Coordination Meetings with CPUC Staff

4.1 Proposer narrative demonstrates their ability to effectively coordinate, attend, and facilitate project coordination group and other meetings with CPUC Staff. (5 Points)

0-2 3 4 5

Task 5: Stakeholder Workshops & Webinars

4.2 Proposer narrative demonstrates their ability to effectively host public stakeholder engagement activities, such as half to full day in-person all party meetings or workshops, two-hour webinars, conference calls, or in-person meetings of various durations, and includes pricing options for each type of activity. The proposer also demonstrates experience in hosting public workshops and/or webinars and successfully engaging stakeholders. (20 Points)

0-5 6-10 11-15 16-20

Task 6: Presentations

4.3 Proposer sufficiently details a logical and effective approach to prepare presentation materials on a variety of different topics. (10 points) 0-2 3-5 6-8 9-10

Task 7: Resources and Tools for Review of PA Filings and Other Submissions

4.4 Proposer sufficiently details a logical and effective approach and demonstrates competence to develop resources to support staff review of energy efficiency portfolio administrator filings and other submissions. Proposer demonstrates understanding of all task deliverables, including creating templates and checklists to develop and review portfolio applications, annual and quarterly reports, and compliance filings. (10 Points)

0-2 3-5 6-8 9-10

4.5 Proposer demonstrates their ability to visualize information via Reviewing information that is converted into charts, graphs, animations, and other graphics.

Proposer shows experience with Tableau, PowerBI, or equivalent programs. (20 points).

0-5 6-10 11-15 16-20

4.6 Proposer sufficiently details a logical and effective approach to develop public dashboards conveying portfolio administrator performance and progress. (10 points)

0-2 3-5 6-8 9-10

4.7 Proposer sufficiently details a logical and effective approach to support enhanced reporting of portfolio administrator performance to streamline existing processes and develop new ones. Proposer’s narrative demonstrates understanding of this task and desired reporting enhancements, such as identifying options for inputs, assumptions, analytical frameworks, data sources, thresholds, and outcomes for enhanced energy efficiency performance assessment. Proposer’s narrative also demonstrates experience with reporting processes, procedures, templates, guidance, checklists, or other resources to improve functionality, usability, manageability, and/or display of information. (10 points)

4.8 Proposer’s narrative demonstrates sufficiently details a logical and effective approach to performing database queries using software such as PostgreSQL or SQL Server, or other software capable of handling large datasets, such as SAS or R. (5 points)

0-2 3 4 5

Task 8: Data Transfer and Documentation

4.9 Proposer’s narrative includes a clear plan for using a secure online platform and an indexing or cataloging approach to support records retention. The Proposer also has information in their narrative detailing how they will retain and destroy records.

(20 Points)

0-5 6-10 11-15 16-20

Task 9: Support Staff Performance Monitoring: Indicators, Metrics, and Goals

4.10 Proposer sufficiently details a logical and effective approach to review and make recommendations on the utility of Energy Efficiency Portfolio Sector and Segment Indicators and Metrics to inform outcome-based oversight by ED and stakeholders.

(10 points)

0-2 3-5 6-8 9-10

4.11 Proposer sufficiently details a logical and effective approach to conduct analysis to identify noteworthy trends, correlations, and associations in the reported information and prepare this information in public reports. (10 points)

0-2 3-5 6-8 9-10

4.12 Proposer sufficiently details a logical and effective approach to support market support and equity goal analysis. Proposer’s approach includes coordinating with portfolio administrators on options for how goals should be structured, adopted, and implemented, reviewing proposed goal constructs and processes, and developing and evaluating goal quantification assumptions and methods. (10 points)

0-2 3-5 6-8 9-10

Task 10: Study of EE Branch Use of Cost-Effectiveness

4.13 Proposer sufficiently details a logical and effective approach to conduct an analysis of energy efficiency cost effectiveness methods, tools, and activities. The proposer’s narrative also includes a clear plan to make recommendations to Energy Division staff regarding potential changes to cost effectiveness policy, implementation, and tools. (10 points)

0-2 3-5 6-8 9-10

Task 11: Monitoring High Impact and High-Risk Programs

4.14 Proposer sufficiently details a logical and effective approach to develop tools and processes to identify high impact and high-risk programs, projects, and delivery approaches and will develop recommendations for staff regarding monitoring strategies. Proposer’s narrative includes an approach to identify HI/HR programs, review program details, assess performance and feasibility of success (10 points)

4.15 Proposer sufficiently details a logical and effective approach to identify and analyze mechanisms IOUs use for third party contract development and management that contribute to greater or lesser program and portfolio implementation risk as well as benefits and innovation. (10 points)

0-2 3-5 6-8 9-10

Task 12: Training

4.16 Proposer sufficiently details a logical and effective approach to provide training to CPUC Staff on all tools, materials, and data visualization elements designed for Tableau or PowerBI (or equivalent) that are developed throughout the course of this contract. (5 points)

0-2 3 4 5

4.17 Proposer sufficiently details a logical and effective approach to provide training to external stakeholders. (5 points)

0-2 3 4 5

Task 13: Subject Matter Expert Services

4.18 Proposer’s narrative demonstrates their ability to act as a subject matter expert on matters related to energy efficiency. (5 points)

0-2 3 4 5

Task 15: Study Recommendations

4.19 Proposer sufficiently details a logical and effective approach to identify research needs and providing expert guidance on the development of various studies’ (identified in D.23-06-055 and elsewhere) scopes of work and content, to ensure that relevant studies can contribute to Commission oversight activities. (5 points)

0-2 3 4 5

MAXIMUM POINTS POSSIBLE: 190 TOTAL SCORE:

5. Work Plans (Task 1) (35 points)

FAIL POOR FAIR GOOD

5.1 Work Plans provide a step-by-step account of how the Proposer plans to complete all work outlined in the RFP. The Work Plans clearly delineate in a logical and realistic manner the main activities and resources required to meet the Commission mandated schedule and all deliverable requirements for each sector. (15 Points)

0-3 4-8 8-12 12-15

5.2 Work Plans are well-organized, detailed, and comprehensive, describing the tasks that will lead to the completion of all work in the Contract, including activities, tasks, methodology, resources, timeline, personnel, budget, and coordination with commission staff, utilities, and stakeholders. (10 Points)

5.3 Work Plans identify the team member(s) responsible for each activity and any associated needs from or impacts on CPUC staff and/or other Contractor(s). Tasks are appropriately staffed at proper levels of authority and proper number of personnel. (10 Points)

MAXIMUM POINTS POSSIBLE: 35 TOTAL SCORE:

Combined Technical Evaluation and References Score

Maximum Points Possible: 360 TOTAL SCORE: