EAPS SOW final 2021.03.02.pdf

Page i Controlled Unclassified

National Science Foundation

SCOPE OF WORK

EVALUATION AND ASSESSMENT PROGRAM SERVICES

March 2021

This document is for Unclassified/for Official Use Only

Acquisition Sensitive

Evaluation and Assessment Program Services

Page i Controlled Unclassified

Table of Contents

1.0 Background

1.1 NSF

1.2 Organizational Background

2.0 Scope of Services

2.1 Staffing

3.0 Current Environment

3.1 General Requirements

3.2 Current System Environment

FOUO – Acquisition Sensitive and Confidential

1.0 Background

1.1 NSF

The National Science Foundation (NSF) is an independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense..." To achieve its mission, NSF supports:

• Fundamental science and engineering research. Most NSF funding supports basic research across all science and engineering disciplines and their interdisciplinary connections, including basic research about science, technology, engineering and mathematics (STEM) education.

• People. NSF helps train current and future scientists through a range of human capital development programs supporting students, post-docs and faculty through scholarships, fellowships, research experiences, early career awards, and so on.

• Strategic and cross-cutting themes and strategies. NSF investments in research and people often address areas of national priority such as artificial intelligence and strategies such as public-private partnerships.

With an annual budget of $8.075B (FY 2010), NSF is the funding source for approximately 25 percent of all federally supported basic research conducted by America's colleges and universities. In many fields such as mathematics, computer science and the social sciences, NSF is the major source of federal backing.

NSF is led by a director who oversees NSF staff and management responsible for program creation and administration, merit review, planning, budget and day-to-day operations. A 24-member National Science Board (NSB) comprised of eminent individuals meets six times a year to establish the overall policies of the foundation. The director and all Board members serve six-year terms. Each of them, as well as the NSF deputy director, is appointed by the President of the United States and confirmed by the U.S. Senate.

At present, NSF has a total workforce of about 2,100 at its Alexandria, VA, headquarters, including approximately 1,400 career employees, 200 scientists from research institutions on temporary duty, 450 contract workers and the staff of the NSB office and the Office of the Inspector General.

For more information about NSF please visit our website at http://www.nsf.gov/.

1.2 Organizational Background

The Contractor(s) will provide support to the Evaluation and Assessment Capability (EAC) section within the Office of Integrative Activities (OIA) in the Office of the Director (OD).

EAC provides agency-wide leadership and coordination in evidence-building activities, including program evaluation; expert support and resources for data collection, integration, management, analysis, and reporting; and leadership and resources to improve the National Science Foundation’s evaluation capacity. The urgent and rapidly evolving evidence needs of the Agency as it implements program and foundation-wide learning agendas create the requirement for this vehicle. EAC is responsible for Title 1 and supports Titles 2 and 3 of the Evidence Act.

EAC also serves to enhance NSF's ability to conduct evidence-based planning and policymaking. EAC promotes a culture of continuous learning and improvement and serves as a catalyst and clearinghouse of programmatic data, insights, and problem-solving techniques to support and inform leadership decisions. One of EAC's primary services is facilitating the collection, analysis, interpretation, dissemination, and use of quantitative and qualitative findings from studies, such as internal or external evaluations.

EAC’s work is deeply collaborative to ensure that evaluation and program improvement activities meet the needs of stakeholders, both internal (e.g., directorates and offices) and external (e.g., the scientific community and other Federal agencies). EAC also seeks to harness existing NSF program data by creating systematic data and knowledge management systems and increasing ease of access to resources, tools, and measures. Additionally, EAC strives to leverage insights throughout NSF, at other Federal agencies, and to the public by disseminating methods and findings, as well as facilitation of processes to engage stakeholders in evidence use in unique and innovative formats.

To attain these goals, EAC works with staff across Agency—namely, NSF directorates and program offices, an EAC steering committee, an EAC Coordination Group, evaluation and data analytics experts in each directorate, and the Directorate for Social, Behavioral & Economic Sciences, National Center for Science and Engineering Statistics (NCSES)—to design studies, develop or review evaluation frameworks and plans, review questionnaires and other data collection instruments, collect quantitative or qualitative data, analyze data or review data analyses conducted, prepare data files, create or review interim and final deliverables (such as reports) and solicit feedback from experts and stakeholders on evaluation and other findings, disseminate findings (including presenting findings to various audiences or organizing report releases), and develop standards and expectations for evidence-building activities (including evaluations).

2.0 Scope of Services

The contractor must provide expert support for evaluation and assessment activities in support of EAC's mission and related to complying with the Evidence Act, which includes developing learning agendas and implementing selected rapid cycle studies specified in the learning agendas or requested by Agency stakeholders. These services will typically be independent, self-contained, and repetitive, such as developing an evaluation framework, conducting literature reviews, and landscape studies, collecting qualitative and quantitative data, analyzing data, and/or disseminating findings. Such activities may include, but are not limited to:

• Evidence Act Deliverables

Facilitating yearly updates of the Agency-wide learning agenda, development of annual evaluation plans, and execution of studies included in the Agency-wide learning agenda or in support of the Agency-wide capacity assessment. This will include providing subject matter and technical expertise, facilitating stakeholder engagement, building capacity of stakeholders, providing communications support (editing, visualizations, and so on), and conducting activities listed under subsequent bullets.

• Research and Evaluation

Designing and conducting a wide range of studies, including landscape studies, systematic literature reviews, descriptive and exploratory studies, program evaluations (process, implementation, outcome, and impact evaluations), policy studies, cost-benefit analyses, performance measurement, and so on. This may include, but is not limited to:

o Designing monitoring and evaluation frameworks and plans to guide strategic development and execution of studies in support of learning priorities (including facilitating the development of evaluation questions, theories of change, and logic models to guide those frameworks and plans) o Designing data collections (including robust sampling approaches with a solid rationale and appropriate data collection approaches) o Drafting questionnaires and other data collection instruments (including researching constructs and measures, preparing drafts for review, drafting packages to obtain OMB clearance, and pre-testing protocols prior to making final revisions) o Developing data analysis plans, with sufficient detail to enable NSF to provide feedback and ensure that planned work aligns with stakeholder expectations and benefits from stakeholder insights (on programs or NSF administrative data) o Collecting quantitative and qualitative data (including primary and secondary data and linking/matching to rely on existing data) o Analyzing data and providing specialized statistical support using multiple methods (including modeling, bibliometric analysis, portfolio analysis, and network analysis) and different designs (including matching techniques to develop well-matched comparison groups and conducting randomized control trials) o Preparing a wide range of high-quality deliverables (reports, briefs, one-pagers, flyers, presentations, and so on) and soliciting feedback from experts and stakeholders on these deliverables; deliverables will include data files listed under Dissemination

• Monitoring Data Systems

Developing, refining, and using data systems to support ongoing monitoring and future evaluations of programs and strategies. This includes developing dashboards to facilitate access to information.

• Data Analytics

Develop, manage, and conduct data analytics to support evidence building activities. This may include, but is not limited to:

o Dashboards —includes the ability to publish formal, Web-based and/or mobile reports with intuitive interactive displays of information, including dials, gauges, sliders, check boxes and traffic lights. These displays indicate the state of the performance metric compared with a goal or target value.

o Ad hoc query —the capability, which enables end-users to ask their own questions of the data using natural language, without relying on IT to create a report. In particular, the Contractor must have a robust semantic layer to allow end-users to navigate available internal and external data sources.

o Search-based Data Analytics — Provide a natural language search index to both structured and unstructured data sources and map them into a classification structure of dimensions and measures (often, but not necessarily leveraging the Data Analytics semantic layer) that end-users can easily navigate and explore using a “elastic” search (Google-like) interface. This capability extends beyond keyword searching of Data Analytics platform content and metadata.

o Mobile Data Analytics — Provide the capability that delivers report and dashboard content to mobile devices (e.g., smartphones and tablets) in a publishing and/or interactive (i.e., bidirectional) mode, and takes advantage of the interaction mode of the device (i.e., tapping, swiping and so on) and other capabilities not commonly available on desktops and laptops, such as location awareness.

o Augmented analytics – Provide the use of automation, machine learning and natural-language processing to query the data; speed up the time to prepare data; automatically find patterns in the data, and communicate the analytical findings to a broader set of users using natural-language generation and other interfaces.

o Online analytical processing (OLAP) — Provide end-users to analyze data with extremely fast query and calculation performance, enabling a style of analysis known as "slicing and dicing.”

o Interactive visualization — Support end-users' ability to display numerous aspects of the data more efficiently by using interactive pictures and charts, instead of rows and columns.

o Predictive modeling and data mining — Provide end-users support to classify categorical variables and to estimate continuous variables using advanced mathematical techniques.

Provide integration of models easily into Data Analytics reports, dashboards and analysis, and business processes.

o Scorecards — Provide support to take the metrics displayed in a dashboard a step further by applying them to a strategy map that aligns performance indicators with a strategic objective.

o Immersive analytics. Assist EAC in identifying value areas where augmented and virtual reality can be applied for enhanced analytics and data visualization.

o Data storytelling. Assist EAC in developing processes and capabilities that can help personnel to better communicate and data and information that portrays strategic value.

• Dissemination

Activities may include:

o Disseminating findings (including reformatting of reports for inclusion in the EAC product gallery) to various audiences by organizing releases to appropriate audiences, organizing events (such as conferences, symposia, workshops), and other activities o Creating data inventories, delivering data files for restricted (internal or licensed) and unrestricted (public) use, and preparing documentation files with detailed information to facilitate use (including codebooks or data dictionaries)

• Technical Assistance and Facilitation

Activities may include:

o Providing technical assistance for a variety of tasks, such as engaging stakeholders in developing learning questions, providing technical experts to conduct quality assurance reviews, and developing standards for NSF evidence-building activities o Facilitating stakeholder discussions to obtain input, feedback, and/or build consensus on a variety of topics (such as a study or system design and implications of findings) o Building stakeholder capacity to participate in or use research and evaluation studies/findings (such as training on evaluation for NSF staff who are preparing for or overseeing an evaluation)

• General program services

Providing a wide range of cross-cutting program services related to supporting EAC's mission and the activities required under the Evidence Act. On occasion, the Contractor may be directed to conduct research or engage in activities to generate findings that can be used to inform a decision regarding whether to conduct a study or the approach to pursue, to supplement an ongoing study conducted under another award, or to conduct quality assurance reviews of work conducted internally at NSF or externally by another contractor.

Examples include inquiry and facilitation to ensure common understanding of terms and operationalize them, feasibility studies to test whether a program is ready for a formal evaluation, and exploratory reviews to identify administrative data that may be used in an evaluation.

• Special projects

The contractor shall be responsive to additional ad hoc requests for analyses, briefs, presentations, and other learning agenda and evaluation support services that may arise over the life of the contract. The contractor shall provide support for ad-hoc requests to meet the information needs of important constituents such as federal agency heads, Congress, the National Science Board, policymakers (such as those in OMB and the Office of Science and Technology Policy), or NSF senior management.

2.1 Staffing

The Contractor must provide leaders and experts capable of proactively identifying, conducting, and communicating findings from evaluation and other evidence-building activities, including support activities such as facilitation and stakeholder capacity-building. As such, the Contractor is encouraged to:

• Provide an experienced team that can fulfill the range of activities included here, including subject matter (as outlined in section 1.1) and methodological, facilitation, and communications expertise (as outlined in section 2.0).

• Assign appropriate and flexible staffing plans that are tailored or adapted to needs and help maintain continuity through a mix of experienced senior-, mid-, and junior-level staff. As technologies change, NSF expects that the Contractor will adjust resources and provide ad hoc expertise for projects and other short-term assignments, as well as reduce head-count as appropriate.

• Establish continuous support and develop contingency plans to avoid work disruptions or threats to quality due to staff changes. This includes being proactive in establishing and maintaining operational guides that document the NSF organization, responsibilities, and processes that can be used to train resources, transfer knowledge, and create the basis for knowledge management.

• Replace, reduce, modify, and/or add staffing resources as needed in a timely, professional and consistent manner, with approval from NSF in the case of key personnel.

3.0 Current Environment

3.1 General Requirements

The Contractor shall perform all the management and administrative tasks to satisfy the requirement of this SOW while adhering to all NSF and Federal IT policies.

3.2 Current System Environment

EAC currently maintains the following data and information repositories:

• EAC Tools – This is a repository of data analytic tools that have been developed by NSF staff. EAC Tools is available to all NSF staff.

• EAC website available to the public (https://www.nsf.gov/od/oia/eac/) – This website provides information about EAC to the public.

• EAC Portal intranet site available to EAC staff – This intranet site contains current and historical information on EAC