Attachment 01 - DRAFT Statement of Objectives NSF Engines.docx

STATEMENT OF OBJECTIVES (SOO)

Project: NSF Regional Innovation Engines Program Evaluation (RIEPE)

1.0 Purpose:

NSF is seeking an external evaluation team to work closely with NSF program directors on assessing the impact of the NSF Regional Innovation Engines program (NSF Engines) and the extent to which the program is achieving its intended goals. This comprehensive evaluation of the program will inform programmatic decisions, increase our understanding of how regional innovation ecosystems are created, and determine the overall success of the program.

2.0 Scope:

The Contractor, in collaboration with NSF Program Directors, will be responsible for conducting a mixed-methods evaluation of the portfolio of NSF Engine projects. It will be critical for the contracting team to identify macro-level indicators to determine regional economic development and social impacts and real-time metrics related to the activities of each Engine Awardee. They shall also consider how the evaluation could capture the qualitative impact of an Engine’s activities on the region while documenting the state or evolution of the regional governance and networks within each Engine. To determine the long-term impacts of the funded Engines, the Contractor shall conduct a comparative analysis to assess the extent to which NSF Engines were able to catalyze and/or accelerate regional innovation ecosystems across the Nation.

The NSF Engines program is making long-term investments at an unprecedented scale and time frame — up to 10 years and $160 million per Awardee — to establish inclusive regional innovation ecosystems capable of evolving and sustaining themselves for decades to come. Presently, there are nine engines spanning 16 states. The regions of impact for each NSF Engine may evolve and it’s anticipated that the number of NSF Engines may increase. NSF can only anticipate specific evaluation needs (e.g. metrics, data analytics, reporting requirements) over relatively short time horizons (approximately two years). The NSF Engines program evaluation needs will evolve as NSF Engine Awardees progress and the program matures. Comment by Todaro, Alyssa: Information should be tailored here to adequately portray the scale to the prospective offerors.

Is this requirement for both Type 1 and Type 2 engines? Comment by Han, Xueying Shirley: This only applies to Type 2 awards. We were specific to only say NSF Engines which refer to Type 2 awards. NSF Engines Development Awards refer to Type 1 awards. Comment by Todaro, Alyssa: Understood, thank you! Comment by Han, Xueying Shirley: The NSF website isn’t updated that we now only have 9 Engine teams spanning 16 States—does it matter if this is different? Comment by Todaro, Alyssa: It doesn’t matter. We will put what’s most accurate in the SOO, but we may receive questions from industry regarding the inconsistency.

This evaluation will assess the causal pathways that link the drivers of the NSF Engines program to the anticipated programmatic goals; serving as a critical tool for TIP leaderships and NSF Engines Program Directors to manage the program. Outcomes of the study will also be valuable to each Engine Awardee as they conduct award-level evaluations. Evaluation findings will inform teams of when and how they need to adjust plans or course-correct over the 10-year funding period.

The Contractor will be expected to:

1. Work closely with NSF to integrate the evaluation and data collection efforts already underway with the evaluation conducted by the Contractor and NSF.

2. Coordinate with the evaluators hired by NSF Engine Awardees to ensure evaluation data collection across all Engine teams is conducted in a consistent manner and the data can be aggregated and assessed across the entire portfolio of awards.

3. Utilize advanced data analytics methodologies encompassing structural equation modeling, hierarchical modeling, Bayesian analyses, and more, with a proficiency in conducting analyses within platforms like R, RMarkdown, Python, SQL, Tableau, and NVivo.

4. Apply econometrics and psychometrics, perform data imputation, and calculate non-response bias.

5. Conduct semi-structured interviews, focus groups, case studies, and qualitative analysis (including training for and calculating interrater reliability).

6. Craft reusable code to streamline processes, adhering strictly to FAIR principles for data management, and utilizing version control systems, notably Git.

7. Utilize analytical tools to aid in the measurement and monitoring of innovation ecosystems and regional economic development, such as Qualtrics, IMPLAN, or REMI models.

8. Apply for access to, and work with, public as well as restricted-use federally held data such as the American Community Survey from the Census Bureau, and the Current Employment Statistics Survey from the Bureau of Labor Statistics.

9. Analyze and integrate data from private databases including Lightcast, Pitchbook, LinkedIn, and similar sources.

10. Stay abreast of the latest literature and studies, leveraging the latest academic research, as appropriate, to ensure that NSF Engines program evaluation framework remains robust, relevant, and informed by cutting-edge developments in place-based innovation.

3.0 Period and Place of Performance: Comment by Han, Xueying: We understand it may not be feasible to make a 10-yr award. We have left this in for situational awareness on how we are thinking about this but we are okay with having this removed for the RFI going out—happy to default to what DACS recommend. Comment by Todaro, Alyssa: A 16-year contract isn’t feasible. Further, given the dynamic nature of NSF Engines, shorter terms will allow for the ability to change course as the program evolves (e.g., how many engines will continue past the first two years, how many new engines will need evaluation, etc.) Comment by Han, Xueying Shirley: Would you recommend us removing this section entirely or changing it to something like 1 year base with 4 year option and then recompile in year 5? Comment by Todaro, Alyssa: I’ve added language for the purposes of moving the RFI along. However, the IGCE will help us understand a better plan/structure.

NSF anticipates an initial one-year base period with four one-year option periods.

The primary place of performance is the Contractor’s site. Travel is anticipated in support of this requirement. The Contractor will work closely with NSF and the evaluators from each awarded Engine and will be expected to work in a mix of virtual, hybrid, and in-person settings. Comment by Todaro, Alyssa: Will it be at the Contractor’s site or at NSF?

4.0 Background:

The mission of the NSF Engines program is to accelerate the development of sustainable, inclusive, and geographically diverse regional innovation ecosystems that advance key technologies and address pressing societal, national, or geostrategic challenges. The program aims to catalyze robust partnerships rooted in scientific and technological innovation to positively impact regional economies. Each NSF Engine supports the development of diverse regional coalitions to engage in use-inspired research and development, translation of innovation to society, and workforce development — with the goal of growing and sustaining regional innovation ecosystems throughout the U.S.

The program has seven articulated goals:

1. Establish self-sustaining innovation ecosystems

2. Establish nationally recognized regional ecosystems for key industries

3. Broaden participation in innovation ecosystems

4. Advance technologies relevant to national competitiveness

5. Catalyze regions with nascent innovation ecosystems

6. Increase job creation

7. Increase economic growth

Programmatic Evaluation Design for the NSF Engines Program

Evaluation has been integrated into the NSF Engines program from the program’s launch. Each Engine Awardee is expected to include an evaluation team that will assess their Engine’s progress towards its stated goals and help inform the Engine’s leadership team on actions such as prioritizing activities and developing/modifying processes, policies, and practices when necessary.

NSF will also be overseeing a program-level evaluation that focuses on (1) the extent to which the program is able to create sustainable, inclusive innovation ecosystems at geographically diverse regions across the United States; (2) whether the program has established pathways to build sustainable, inclusive, innovation ecosystems, and (3) whether regions with an Engine have experienced greater innovation in emerging technology fields, economic development when compared to similar regions without an Engine investment. The program-level evaluation will be informed by the progress of individual Engine Awardees, macro-level indicators related to relevant industries, socio-economic metrics, and metrics on social capital/networking. It will also include qualitative data about the Engine Awardees to provide more detailed context on the impact the Engines have in their regions. Comment by Todaro, Alyssa: Is this separate from the Contractor’s evaluation or is this referring to the work that the Contractor will do? Comment by Han, Xueying Shirley: This is referring to the work that the Contractor will do in conjunction with guidance from NSF. This is what’s different from this program evaluation compared to traditional ones. Under traditional evaluations, the Contractor typically comes in after the fact and will do these things independently with NSF providing access to certain resources, information, or data or to explain the structure of the program. With this evaluation, we are still determining what’s needed at each stage, which data sources we’re trying to access and how that should be integrated into the evaluation, etc. Instead of the Contractor being the main driving force of the program evaluation, NSF will be. I’m not sure if that’s made clear in our SOO—we need a contractor that is willing to work with us and pivot with us as we expand the evaluation framework. A lot of the work will be done by us and the Contractor will be expected to take everything and do independent analyses for their reports but they have to work with us, they are not working independently as most program evaluations go.

The Contractor will work closely with NSF Program Directors to implement an evaluation plan to determine the extent to which the NSF Engines program has met its goals and the impact these Engine investments have had in their respective regions. NSF has an overall structure for the evaluation of the NSF Engines program; however, there is opportunity for evaluators to shape this evaluation particularly as it relates to analyses used to assess the establishment of innovation ecosystems and the economic impact Engines are making in their respective regions.

5.0 Performance Objectives

Deliver Impact Evaluations of the NSF Engines Program: The contracted evaluation team will be expected to deliver a series of impact evaluations of the NSF Engines program to determine the extent to which the program has met its stated goals. The evaluations should use quantitative and qualitative data to build a narrative about the program’s short-, medium-, and long-term impacts. We anticipate the following impact evaluations:

1. Phase I Impact Evaluation—short-term impacts of NSF Engines following the completion of Phase I of the program (i.e., the first two years after award). This impact evaluation should assess the program’s efficacy in enhancing awarded teams’ understanding of regional assets and strengths; in bolstering governance and management structures within the awarded Engine teams; in fostering new partnerships; and in facilitating the development of strategic initiatives tailored to address specific societal, national, or geostrategic challenges, thereby providing valuable insights into its immediate impact and potential areas for refinement.

2. Phase II Impact Evaluation—medium-term impacts of NSF Engines following completion of Phase II of the program (i.e., the first five years of award). The Contractor will analyze and define the medium-term impacts expected to manifest within 5 years of the program, providing insights into the program’s effectiveness in fostering innovation, collaboration, and sustainable growth within the targeted regions.

3. Phase III Impact Evaluation—this evaluation should assess whether NSF Engines was able to meet its programmatic goals based on teams that completed all three phases of the award. The Contractor is expected discern the programmatic milestones and transformative shifts among the Engine teams over the 10-year period and provide critical insights into the effectiveness of the program in meetings its goals.

4. Post-Program Completion Impact Evaluation (5 Years)—assess continued progress and advancements made by Engine teams and identify changes that have taken place since program conclusion. This impact evaluation will determine the extent to which the program has sustained its momentum and efficacy in meeting programmatic goals beyond its initial implementation phase.

Integrate NSF-Collected Data into the Impact Evaluation. The Contractor shall work closely with NSF to incorporate NSF-collected data into their proposed evaluation. The intention is to reduce burden on Engine awardees and prevent redundancy across evaluation efforts. There is no expectation that the Contractor use all the data collected by NSF, and the Contractor should be familiar with navigating the OMB clearance process for any data collection efforts they wish to conduct that falls outside of the work already being done by NSF.

Provide Information/Guidance to Engine Awardees. The Contractor shall use data collected as part of the program evaluation to provide each Engine Awardee with regular evaluation outcome reports specific to the Awardee’s region. They shall provide feedback on each Engine Awardee team’s individual evaluation plans and serve as a resource for helping Engine evaluators provide specific data required for the program-level evaluation.

6.0 Operating Constraints Comment by Todaro, Alyssa: Are there any known operating constraints?

I’ve pulled some which I saw recurring in sample SOWs Comment by Han, Xueying Shirley: Maybe one more which is that the Contractor must be eligible to receive clearance from NSF to access data from NSF systems. And that all work produced, including but not limited to any dashboards that are created, must meet NSF system requirements.

Expectations for Handling of Sensitive, Confidential, and Proprietary Information

NSF's privacy program includes policies, plans, training, and technical safeguards to protect sensitive information, which includes Personally Identifiable Information (PII). NSF recognizes the importance of protecting sensitive information and implements policy, guidance, and best practices to safeguard information from inappropriate access, use, or disclosure. The contractor must indicate in their proposal their approach to meeting agency expectations for secure confidential, proprietary, and personally identifiable information (PII).

Additionally, all data produced from this contract shall be considered property of the NSF. No release of any data in any form is permitted until the COR issues an official release authorization to the contractor. The contractor is expected to comply with internal written procedures to ensure that data and or any products produced because of this contract are not released without the expressed consent and authorization of NSF. All materials and copies of materials developed while conducting this project, are to be forwarded to the HSI program COR or disposed of in accordance with instructions from the COR upon completion of work.

Information Technology Requirements

The contractor shall meet Federal and NSF IT security and privacy requirements and may be tasked to conduct/support NSF in completing security/privacy related activities including, but not limited to, system Privacy Threshold Analysis and Privacy Impact Assessment.

Evaluation Guidelines

The evaluation must meet NSF’s guidelines related to evaluation as detailed in NSF’s Evaluation Policy: https://new.nsf.gov/od/oia/eac/evaluation-policy-april-2023