RFP_23NC0465_Attachment_14_-_Evaluation_and_Scoring_Criteria.pdf

23NC0465 ATTACHMENT 14 – EVALUATION AND SCORING CRITERIA

Name of Contractor: ________________________________________

Name of Evaluator: ________________________________________

Total Points Available:

Technical Evaluation 435 points

Scoring Definitions:

GOOD Proposer’s response provides sufficient details and/or examples demonstrating the ability to meet all of the requirement(s).

FAIR

Proposer’s overall response provides details and/or examples, but does not demonstrate the ability to meet all the requirement(s). The bidder may have addressed particular requirements with insufficient details, or not at all, but the overall response demonstrates that the bidder possesses the ability to meet the requirement(s).

POOR

Proposer’s response provides insufficient details and/or examples demonstrating the ability to meet the majority of requirement(s).

FAIL Proposer’s response is omitted completely, or does not address or relate to the requirement(s).

1. DESIRED QUALIFICATIONS (40 POINTS)

FAIL POOR FAIR GOOD

1.1 General Qualifications (10 points):

1. The successful Proposer should demonstrate experience in successfully managing complex projects involving multidisciplinary teams comprised of:

a. Multiple subcontractors, if applicable to the proposed Project Team,

b. At least one senior experienced overall project manager,

c. At least one manager for each sector and measure group, and

d. In-house staff.

2. The successful Proposer should demonstrate that they can and will effectively manage time and resources across multiple projects, set work schedules, and organize workflow.

0-2 3-5 6-8 9-10

1.2 Stakeholder Communications (5 points):

1. The successful Proposer should demonstrate experience in leading stakeholder workshops and engaging with stakeholders.

2. The successful Proposer should demonstrate experience working with energy stakeholders in a public regulatory process.

3. The successful Proposer should demonstrate experience in collecting and communicating data through various media, including graphs, charts and tables.

4. The successful Proposer should demonstrate success in presenting and communicating evaluation results, recommendations, and findings to related and/or non-technical audience.

0-1 2 3-4 5

1.3 Regulatory Knowledge (10 points): The successful Proposer should demonstrate expertise and understanding of California’s building and appliance codes, California regulations, federal regulations, energy efficiency policies, and industry best practices regarding energy efficiency evaluation.

0-2 3-5 6-8 9-10

1.4 Evaluation Skills and Experience (15 points):

1. Evaluation Experience – The successful Proposer should demonstrate experience in the following energy programs evaluation related skills:

a. Conducting in-depth program evaluation studies in a regulatory context;

b. Designing and implementing sampling plans to inform statistically significant study results and findings.

c. Developing and implementing complex qualitative and quantitative study designs;

0-3 4-8 8-12 12-15

d. Developing and conducting surveys through a variety of channels, e.g., telephone, internet, or physical surveys to address specific questions about a program;

e. Designing and conducting Randomized Control Group studies to determine program influence and impacts;

f. Collecting and organizing heterogeneous data from an array of sources;

g. Conducting analysis of quantitative and qualitative energy program data

(including energy savings estimates);

h. Performing program evaluations and energy policy analysis that produces actionable recommendations for program decision-makers;

i. Developing and providing specific feedback on energy efficiency methodologies for evaluation projects;

j. Conducting energy and demand savings estimation;

k. Measuring load/impact profiles, free ridership, measures effective and remaining useful life; and

l. Developing net-to-gross values, gross realization rates, measures costs, estimation, and evaluation.

2. Research Experience – The successful Proposer should demonstrate experience in conducting technical research projects on a variety of energy program topics including expertise in the multi-disciplined research areas of engineering, finance, econometrics, behavioral and market research related to energy programs.

Maximum Points Possible: 40

TOTAL SCORE:

2. RESEARCH SPECIFIC EXPERTISE AND SKILLS (100 points)

FAIL POOR FAIR GOOD

2.1 Market Support (30 points): The successful Proposer should demonstrate an understanding of unique aspects of energy efficiency programs that meet the market support segment definition. The successful Proposer should have demonstrated experience with the evaluation methods and cutting-edge approaches to study the variety of programs that make up this segment and help support the long-term success of the energy efficiency market. Program examples may include: finance, workforce education and training, marketing, and emerging technologies.

0-7 8-15 16-23 24-30

2.2 Equity (30 points): 0-7 8-15 16-23 24-30

1. The successful Proposer should demonstrate an understanding of unique aspects of energy efficiency programs that meet the equity segment definition, including how those programs:

a. Address disparities in access to energy efficiency programs

b. Promote non-energy benefits

c. Reduce energy related GHG and criteria pollutants

d. Provide workforce opportunities

2. The successful Proposer should have expert knowledge of Environmental and Social Justice (ESJ) issues, issues facing the hard-to-reach and underserved populations, as well as issues facing disadvantaged communities. The successful Proposer will have experience with the various study methods and research designs to study programs that make up this segment.

2.3 Codes and Standards (20 points):

1. The successful Proposer should demonstrate familiarity with current industry building and appliance standards (Such as the California Title 20 and Title 24 standards), as well as current Federal appliance, equipment energy standards, and an understanding of the energy code development processes at the state and federal levels.

2. The successful Proposer should have demonstrated experience with the evaluation methods and cutting-edge approaches to study these programs.

0-5 6-10 11-15 16-20

2.4 Regional Energy Networks, Community Choice Aggregators, and Local Government Partnerships (20 points): The successful Proposer should demonstrate an understanding of unique aspects of energy efficiency programs offered to and through local governments, i.e., cities, counties, special districts, Community Choice Aggregators, and Regional Energy Networks.

0-5 6-10 11-15 16-20

Maximum Points Possible: 100 TOTAL SCORE:

3. SAMPLE REPORTS (30 points)

FAIL POOR FAIR GOOD

3.1 Sample Report 1:

1. The sample work product is an excerpt of a past evaluation report, accompanied with a brief (one page or shorter) explanation that clearly explains how the work reflected in the report is similar to what is requested in this RFP.

2. Proposer’s sample work product is of high quality and shows a clean and visually appealing graphic layout and design.

0-3 4-8 8-12 12-15

3. Proposer’s sample work product sufficiently and articulately communicate results and outcomes of the report. The sample work product is appropriate for a public, non-technical audience.

(15 Points)

3.2 Sample Report 2:

1. The sample work product is an excerpt of a past evaluation report, accompanied with a brief (one page or shorter) explanation that clearly explains how the work reflected in the report is similar to what is requested in this RFP.

2. Proposer’s sample work product is of high quality and shows a clean and visually appealing graphic layout and design.

3. Proposer’s sample work product sufficiently and articulately communicate results and outcomes of the report. The sample work product is appropriate for a public, non-technical audience.

(15 Points)

0-3 4-8 8-12 12-15

Maximum Points Possible: 30 TOTAL SCORE:

4. OVERALL APPROACH (30 points)

FAIL POOR FAIR GOOD

4.1 The Proposer sufficiently details the approach to execute project oversight and the plan to coordinate with subcontractors (if applicable) to ensure success of the work. (5 Points)

0-1 2 3-4 5

4.2 The Proposer sufficiently and clearly identifies specific interdependency and interaction points with the Commission and between deliverables. It is not sufficient to only acknowledge the interaction points. The proposer must clearly articulate how the interaction points will be managed by the Proposer and address how risks for meeting Commission specific deadlines and Commission needs will be mitigated. (5 Points)

4.3 The Proposer sufficiently details the approach to address all identified deliverable submission requirements for all deliverables under this RFP. (5 Points)

4.4 The Proposer’s narrative and approach to each deliverable sufficiently demonstrates knowledge of energy efficiency policies, Commission processes, Commission proceedings, industry best practices regarding energy efficiency evaluation, and California’s energy regulations. As an overall approach, the proposer shall sufficiently demonstrate their understanding of how Commission rulings affect industries and other evaluations. (10 Points)

4.5 The Proposer’s narrative demonstrates the proposer’s approach for dealing with uncertainties due to evolving needs such as: changes in policy objectives, Commission directions in new energy efficiency decisions, decisions in other related proceedings, or coordination with other stakeholders, including utilities. (5 Points)

Maximum Points Possible: 30 TOTAL SCORE:

5. TASKS AND DELIVERABLES (205 Points)

FAIL POOR FAIR GOOD

Deliverable 7 (DT-01): Data Collection, Transmission, and Storage Plan

The Proposer sufficiently details a plan that:

1. Describes the approach and timeline for data collection throughout the life of the contract.

2. Describes the approach for providing CPUC staff with descriptive statistics, data requests, data collection forms and surveys, quality check methodologies, training materials, and documentation of all data sets and calculations.

3. Clearly describes the approach for securely retaining and transmitting all data collected for long-term archiving in a format readily accessible to CPUC staff.

(5 points)

0-2 3 4 5

Deliverable 9 (MS-01): Market Support Real-Time Evaluations

The Proposer sufficiently details an approach to effectively design and conduct four to six evaluations per year that provide feedback and recommendations on in-flight Market Support programs that are in the early to mid-implementation stage, with timelines for these evaluations taking less than a year and could target monthly, quarterly, or semi-annual reports. The types of evaluations pursued could include short term process evaluations or embedded evaluations.

1. The Proposer clearly defines the number of programs under each sector that will be analyzed, the rationale for selecting their recommended programs for analysis, and an effective method(s) for information gathering.

2. The Proposer sufficiently details an innovative and effective approach to conduct qualitative and quantitative analyses, including how it will provide

0-5 6-10 11-15 16-20 actionable program design recommendations to improve the programs. (20 points)

Deliverable 10 (MS-02): Market Support Annual Evaluations

The Proposer sufficiently details an approach to effectively design and conduct six to twelve evaluations of Market Support programs on an annual basis and how those evaluations adhere to the California Energy Efficiency Evaluation Protocols.

Evaluation types include impact evaluations, process evaluations, and market studies.

1. The Proposer clearly defines the number of programs under each sector that will be analyzed, the rationale for selecting their recommended programs for analysis, and an effective method(s) for information gathering.

2. The Proposer sufficiently details an innovative and effective approach to conduct qualitative and quantitative analyses on current IOU energy efficiency programs, including how it will provide actionable program design recommendations to improve the programs and measure the effectiveness of its implementation.

3. The Proposer provides multiple methodologies that meet applicable requirements of the CPUC evaluation protocols. The methodologies shall align with the data collection methodology described in the Proposer’s approach for this Deliverable to ensure consistency throughout the evaluation period. (30 points)

0-7 8-15 16-23 24-30

Deliverable 11 (MS-03): Market Support Portfolio-Level Evaluations

The Proposer sufficiently details an approach for conducting a portfolio-level study of the Market Support segment once during the contract period, which shall analyze segment performance and spending levels, and provide recommendations – both for the overall portfolio as well as individual program administrators. (15 points)

0-3 4-8 8-12 12-15

Deliverable 12 (EQ-01): Equity Real-Time Evaluations

The Proposer sufficiently details an approach to effectively design and conduct four to six evaluations per year that provide feedback and recommendations on in-flight Equity programs that are in the early to mid-implementation stage, with timelines for these evaluations taking less than a year and could target monthly, quarterly, or semi-annual reports. The types of evaluations pursued could include short term process evaluations or embedded evaluations.

0-5 6-10 11-15 16-20

1. The Proposer clearly defines the number of programs under each sector that will be analyzed, the rationale for selecting their recommended programs for analysis, and an effective method(s) for information gathering.

2. The Proposer sufficiently details an innovative and effective approach to conduct qualitative and quantitative analyses, including how it will provide actionable program design recommendations to improve the programs. (20 points)

Deliverable 13 (EQ-02): Equity Annual Evaluations

The Proposer sufficiently details an approach to effectively design and conduct six to twelve evaluations of Equity programs on an annual basis and how those evaluations adhere to the California Energy Efficiency Evaluation Protocols. Evaluation types include impact evaluations, process evaluations, and market studies.

1. The Proposer clearly defines the number of programs under each sector that will be analyzed, the rationale for selecting their recommended programs for analysis, and an effective method(s) for information gathering.

2. The Proposer sufficiently details an innovative and effective approach to conduct qualitative and quantitative analyses on current IOU energy efficiency programs, including how it will provide actionable program design recommendations to improve the programs and measure the effectiveness of its implementation.

3. The Proposer provides multiple methodologies that meet applicable requirements of the CPUC evaluation protocols. The methodologies shall align with the data collection methodology described in the Proposer’s approach for this Deliverable to ensure consistency throughout the evaluation period.

(30 points)

0-7 8-15 16-23 24-30

Deliverable 14 (EQ-03): Equity Portfolio-Level Evaluations

The Proposer sufficiently details an approach for conducting a portfolio-level study of the Equity segment every two years, which shall analyze segment performance and spending levels, and provide recommendations – both for the overall portfolio as well as individual program administrators. (15 points)

0-3 4-8 8-12 12-15

Deliverable 15 (C&S-01): Codes and Standards Impact and Process Evaluation

The Proposer sufficiently details a plan to conduct one evaluation per year of Codes and Standards programs and sub-programs, and effectively evaluate compliance with 0-5 6-10 11-15 16-20 the Federal standards and State building standards per Title 20 (appliance standards) and Title 24 (energy efficiency building codes), established by the California Energy Commission. The approach should also outline a combination of both impact and process evaluation methods to recommend improvements to the C&S programs and sub-programs. If relevant, the approach shall detail how it complies with the Codes and Standards and Compliance Enhancement Evaluation Protocol, from the California Energy Efficiency Evaluation Protocols. (20 points)

Deliverable 16 (GOV-01): Assessment of Community Choice Aggregators and Local Government Partnerships

The Proposer sufficiently details an effective and innovative approach to evaluate energy efficiency programs administered by current and any future Community Choice Aggregators (CCAs), and if directed by energy division staff, Local Government Partnerships (LGPs). This approach shall include details for analyzing the benefits of non-resource programs, and also consider the impact of all CCA and LGP programs on the overall portfolio, including a determination of any impacts the CCA and LGP programs have on participation in, or effectiveness of, programs administered through the IOUs. (20 points)

0-5 6-10 11-15 16-20

Deliverable 17 (GOV-02): Impact/Process Evaluations for Regional Energy Networks

The Proposer sufficiently details an effective and innovative approach to evaluate energy efficiency programs administered through Regional Energy Networks (RENs).

The Proposer also details its approach to making recommendations regarding program performance over the course of the evaluation period. (20 points)

0-5 6-10 11-15 16-20

Deliverable 18 (PD-01): Additional Portfolio and Market Level Studies

The Proposer sufficiently details its approach for conducting up to 3 portfolio and/or market level studies annually relating to the Market Support and Equity segments. The Proposer shall detail how it will provide at least three approaches/price options (e.g., secondary research vs. primary research) for a given evaluation/study. (5 points)

0-2 3 4 5

Deliverable 19 (PD-02): Additional Policy Analysis Studies

The Proposer sufficiently details its approach for conducting up to 3 studies annually relating to programs that are implemented in response to legislative mandates or Commission directives. At a minimum, the Proposer shall detail how it will provide at least three approaches/price options (e.g., secondary research vs. primary research) for a given study. (5 points)

0-2 3 4 5

Maximum Points Possible: 205 TOTAL SCORE:

6. WORK PLAN (30 points)

FAIL POOR FAIR GOOD

6.1 The Proposer’s Work Plan provides a step-by-step account of how the Proposer plans to complete all work outlined in the RFP. The Work Plan clearly delineates in a logical and realistic manner the main activities and resources required to meet the Commission mandated schedule and all deliverable requirements for each sector. (10 Points)

6.2 The Proposer’s Work Plan is well-organized, detailed, and comprehensive, describing the tasks that will lead to the completion of all work in the Contract, including activities, tasks, methodology, resources, timeline, personnel, budget, and coordination with commission staff, utilities, and stakeholders. (10 Points)

6.3 The Proposer’s Work Plan identifies the team member(s) responsible for each activity and any associated needs from or impacts on CPUC staff and/or other Contractor(s). Tasks are appropriately staffed at proper levels of authority and proper number of personnel. (10 Points)

Maximum Points Possible: 30 TOTAL SCORE:

COMBINED SCORE

Maximum Points Possible: 435 COMBINED TOTAL SCORE: