SOW REU.pdf

October 2020

Statement of Work

I. Project Title

Comparative Evaluation of Undergraduate Research Opportunities on Student Interest in Research Careers in Computer & Information Science & Engineering (CISE) Fields

II. Period of Performance

Eighteen-month base period and two (2) twelve-month option periods.

III. Overview of the Project

Computer and Information Science and Engineering has unique, longstanding challenges attracting and retaining diverse students in academia and research careers. The total number of CISE PhDs increased by 20% between 2011-2019, however, the relative participation of underrepresented groups has remained flat during this time. Women have consistently represented approximately 20% of PhDs since 2011, and American Indians/Alaskan Natives, African Americans, Hispanics/Latinos, and Hawaiian Pacific Islanders have collectively comprised only 3.5% of CISE PhDs and have experienced no significant increase since 20111.

The purpose of this project is to evaluate the longitudinal impact of research experiences (that is, REU Sites, internships, other independent research experiences) for CISE undergraduate students on several important career outcomes, such as student interests in research careers and their enrollment in CISE graduate programs. Findings from this evaluation will be used to inform the CISE REU program as well as other programs in the Education Workforce Cluster aimed at diversifying the CISE community.

Outcomes of this work include: 1) collection and reporting of comparative data about CISE undergraduate research experiences, and 2) a better understanding of student attitudes and pathways to research careers in computing with a particular emphasis on underrepresented groups.

NSF REU Program: The REU program seeks to expand student participation in all kinds of research--both disciplinary and interdisciplinary—encompassing efforts by individual investigators, groups, centers, national facilities, and others. It draws on the integration of research and education to attract a diverse pool of talented students into careers in science and engineering, including teaching and education research related to science and engineering, and to help ensure that these students receive the best education possible.

1 U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), 2011-2019, survey data. Retrieved from https://nces.ed.gov/ipeds/use-the-data on Sept 2020.

The NSF REU program is comprised of REU Sites and REU Supplements. REU Sites are based on independent proposals to initiate and conduct projects that engage a number of students in research, and REU Supplements are included as a component of proposals for new or renewal NSF grants or cooperative agreements or may be requested for ongoing NSF-funded research projects.

Since 2009, some CISE REUs have utilized the CISE REU Toolkit (Award# 1936534) that provides evaluation resources and services to CISE REU Site PIs, facilitate students’ applications to CISE REU Sites, and track alumni of participating sites. In 2021, CISE REU Sites will be able to use the NSF-wide REU application and data system – Education and Training Application (ETAP) – to manage REU Site applications and obtain sufficient data to enable the longitudinal tracking of students. This project will build upon the CISE REU Toolkit and ETAP.

IV. Scope of Work and Deliverables

The contractor is responsible for the design, implementation, and analysis of the comparative evaluation of CISE undergraduate research experiences to facilitate NSF’s ability to make evidence-based decisions about its human capital development programs.

The work has two major components:

1. To collect demographic, attitudes, and outcomes data for undergraduates in CISE fields;

and

2. To determine the impact of different types of research experiences—namely, those provided through REU Sites, internships, and independent research experiences— compared to a representative of students who do not participate on any of these experiences on students’ decisions to pursue research careers in computing.

Research questions include but are not limited to:

• Does participating in research experiences in REU Sites, REU Supplements, internships, and/or other independent research experiences impact student attitudes and pathways to CISE careers and other research experiences? These might include participating in additional research experiences, seeking and finding a research job that is related to CISE, pursuing further education in CISE, such as targeted technical training, master’s or Ph.D. If so, how?

• To what extent are the goals of the REU Sites program ((1) recruitment and retention of undergraduate students in science and engineering fields and (2) increasing diversity in the nation’s talent pool in these areas) being met by the CISE REU Sites?

• In what ways does participation in the REU program (through a Site or supplement) impact recruitment and retention of students who are underrepresented in computing?

Retention is broadly construed to include retention in the field, including retention in CISE education and employment.

Data collection should include but is not limited to:

• Longitudinal comparative data on the (1) demographic and educational background characteristics, (2) career progression (educational attainment and employment outcomes), and (3) attitudes towards computing careers of participants in various research experiences (REU Sites, REU supplements, and other independent research experiences) and student who do not apply or participate in these types of research experiences (the comparison group(s)).

• Data collection should leverage existing data sources whenever possible to reduce respondent burden (for example, the Data Buddies Survey includes data on student experiences and outcomes across the undergraduate CISE community)

• Data collection should not include retrospective surveys to assess student participation and attitudes.

The proposal submitted in response to this statement of work will address the following tasks.

• Task 1. Evaluation design, planning and management

• Task 2. Stakeholder engagement

• Task 3. Implementation of evaluation

• Task 4. Data analysis and reporting

• Task 5. Ad hoc Requests

IV. Requirements and Deliverables

Contractor requirements:

• Demonstrate experience with, and deep understanding of, the CISE community.

• Demonstrate understanding the various pathways to research careers in computing.

• Demonstrate previous experience evaluating CISE undergraduate pathways into and through CISE-related careers. Data collection should include:

o Student demographics o Student support structure o Students’ methods for paying for college o Student career interests and plans o Students’ sense of satisfaction with the computing program in which they are enrolled

• Demonstrate familiarity with evaluating programs designed to study broadening participation in computing.

• Demonstrate expertise in analyzing and reporting disaggregated data to address intersectional concerns unique to CISE fields.

• Demonstrate expertise in evaluating undergraduate STEM education or research programs.

• Adhere to the Privacy Act of 1974 and the NSF Act of 1950 as amended, to protect the confidentiality of data about individuals.

• Have the knowledge and skills to comply with the Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation and the NSF Evaluation Policy.

• Have the knowledge and skills to comply with the American Evaluation Association’s Guiding Principles for Evaluators and the Development Assistance Committee of the Organization for Economic Cooperation and Development’s Evaluation Quality Standards for Development Evaluation.

• Have the staff and skills to work with a variety of NSF internal and external stakeholders under the direction of the Contracting Officer’s Representative (COR) and to develop mechanisms and processes to facilitate communication, coordination, and decision-making.

• Must collaborate with the contractor of the NSF Education and Training Application (ETAP) for data collection on CISE REU Sites and Site applicants/participants.

Deliverables2

• Kickoff Meeting. A 2-hour virtual kickoff meeting, plus a one-hour virtual meeting with the Contracting Officer’s Representative (COR), shall be held within 5 business days of award and include a presentation of the approach and overall work plan and timeline.

• Transition In/Work Plan. Revisions to the proposed approach, which serves as the analysis and management plan for the study, will be due within a month of the kickoff meeting.

• Program and project management services include contract administration and compliance, audit, back-office related activities (e.g., invoicing), account management, program management, and other recurring routine functional tasks to plan, control and ensure technical objectives are delivered on schedule and within defined budgets.

2 The briefs, reports, and presentations shall demonstrate effective visual communication. For examples and guidelines see:

https://www.kauffman.org/evaluation/evaluation-reporting-guide/ https://www.nsf.gov/od/oia/eac/ https://www.nsf.gov/od/oia/eac/

• Instruments, protocols, detailed analysis plans3 with tables and graphics4, and other related draft and final sub-item deliverable products required to complete deliverables, will be determined in the work plan and delivered to NSF as they are completed with proper documentation to facilitate current and future use.

• All components of the evaluation plan that will require OMB clearance in accordance with the Paperwork Reduction Act will be identified.

• Briefings, presentations, reports, and a dissemination plan to share findings with the CISE community will be determined in the work plan.

• All data and associated data dictionaries collected during the duration of the contract will be provided to NSF. This includes raw data and data files used in analysis. These shall be accompanied by documentation (including information about how the data were collected, sample designed, response rates achieved, non-response bias analysis conducted, and so on) to facilitate use.

• The contractor will provide annual reports with descriptive statistics for:

• Each CISE REU PI, providing detailed information about their REU students compared to other REU students from similar REU sites.

• CISE REU POs, detailing the participation and outcomes for each site and an aggregate analysis by year and over time. This report will include similar analysis for the comparison groups, namely, students who participated in other research experiences or no research experience.

• The public, including aggregate analyses of undergraduate research experiences in CISE disciplines (overall, not by Site or PI).

• The contractor will provide a final report, executive summary, and presentation reflecting the outcomes of the impact evaluation (internal to NSF).

• NSF will review the final submission of deliverables listed, and may request clarifications, revisions, or additional information, which shall be due no later than 12 weeks prior to the end of the contract.

V. Place of Performance

The place of performance shall be the contractor’s site(s).

3 Analysis plans are required prior to conducting any analysis. These shall be based on mock or old data for illustration purposes.

4 Figures and tables shall provide detail and graphic depiction(s) of results/ conclusions with the source(s) of data and relevant methodological notes so they may be read as standalone deliverables and reproduced. In addition, when appropriate, tables and figures should provide point estimates and other information required for statistical use.

VI. Special Requirements

Meetings

The contractor shall work with designated CISE program staff (as well as other stakeholders), who will provide guidance and background information as needed. In addition, CISE will provide the COR to manage the contract. The names of agency staff assigned to the roles noted above will be inserted in the resulting contract by the NSF Division of Acquisition and Cooperative Support (DACS).

The contractor shall participate in the following meetings:

• Kickoff meeting to present and receive feedback on the design, workplan, and timeline

• Initially monthly and later quarterly (at a minimum) with the COR for the contract throughout the period of performance to discuss progress

• Other meetings, if proposed by the contractor or NSF, related to engagement with other

NSF internal audiences for planning may be required

Meetings should be included in the timeline.

At least three calendar days prior to each meeting, the contractor shall provide an agenda for the meeting to the COR. The agenda shall minimally describe the topics for discussion (including primary activities undertaken since the previous meeting, challenges encountered and solutions identified, and outstanding issues that need to be addressed) and identify items requiring action and/or decisions. Within one week of each meeting, the contractor shall provide the COR with the minutes of the given meeting. These minutes should describe each topic discussed, conclusions reached, action items identified, and individuals responsible and deadlines for completing each action item.

Ad hoc Requests

The contractor shall include a contingency for ad hoc data collection, analyses, or reporting requests intended to meet the evolving needs of the project sponsors and stakeholders each year of this project (up to 10 such requests). These funds shall be budgeted in a separate task to facilitate monitoring of expenses.

VII. Contractor Key Personnel

Key personnel for this engagement include the following:

• Project Director – Senior Evaluator or Senior Researcher

• Research Associate

• Research Assistant

• Additional personnel will be at the discretion of the contractor, but the qualifications of the personnel must be included in the proposal.

VIII. Government Furnished Property

There will be no government provided property provided. The contractor shall provide all management, tools, supplies, equipment, and labor necessary to complete this SOW.

IX. Work plan, Timeline and Milestones

The work plan, timeline, and milestones required to complete this study must be included in the proposal.