EPSCoR_Evaluation_SOW_Final.pdf

Attachment A – Statement of Work

National Science Foundation

EVALUATION OF THE EXPERIMENTAL PROGRAM to STIMULATE

COMPETITIVE RESEARCH (EPSCoR)

Attachment A

Statement of Work

National Science Foundation

Division of Acquisition and Cooperative Support

4201 Wilson Boulevard, Suite 475

Arlington, VA 22230

Statement of Work

I. Project Title

Evaluation of the Experimental Program to Stimulate Competitive Research (EPSCoR).

II. Overview of the EPSCoR Program

Program Synopsis

EPSCoR is designed to fulfill the mandate of the National Science Foundation (NSF) to promote scientific progress nationwide. A jurisdiction1 is eligible to participate in EPSCoR programs if its level of NSF research support is equal to or less than 0.75 percent of the total NSF research and related activities budget for the most recent three year period. Through this program, NSF establishes partnerships with government, higher education and industry that are designed to effect sustainable improvements in a jurisdiction’s research infrastructure, Research and

Development (R&D) capacity, hence its research competitiveness.

The mission of EPSCoR is to advance excellence in science and engineering research and education in order to achieve sustainable increases in research, education and training capacity and research competitiveness that will enable EPSCoR jurisdictions to have increased engagement in areas supported by NSF.

EPSCoR goals are to:

Catalyze the development of research capabilities and the creation of new knowledge that expands jurisdictions’ contributions to scientific discovery, innovation and knowledge-based prosperity;

Establish sustainable Science, Technology, Engineering and Mathematics (STEM) education, training and professional development pathways that advance jurisdiction-identified research areas and workforce development;

Broaden direct participation of diverse individuals, institutions and organizations in the project’s science and engineering research and education initiatives;

Effect sustainable engagement of project participants and partners, the jurisdiction, the national research community and the general public through data-sharing, communication, outreach and dissemination;

Impact research, education and economic development beyond the project at academic, government and private sector levels.

1 Jurisdictions include states, territories and commonwealths (i.e., Puerto Rico) of the United States.

There are four distinct components of the EPSCoR program. The Research Infrastructure

Improvement Program and Co-funding are the primary focus of this evaluation. Workshops and

Conferences together with Outreach Travel, are indirect support strategies that promote knowledge exchange and networking.

Research Infrastructure Improvement Program (RII). The RII program uses several different investment strategies to support lasting improvements in a jurisdiction’s academic research infrastructure to increase research competitiveness. These include, but are not limited to, the following four tracks:

o RII Track-1. Awards provide up to $20 million total for 5 years to support physical, human and cyber infrastructure improvements in research areas selected by the jurisdiction’s EPSCoR steering committee as having the best potential to improve future research competitiveness.2 o RII Track-2 FEC (Focused EPSCoR Collaborations). Builds interjurisdictional collaborative teams of EPSCoR investigators in scientific focus areas consistent with NSF priorities. Projects are investigator-driven and must include researchers from at least two RII-eligible jurisdictions. The development of early career faculty is a critical component of sustainable STEM capacity and research competitiveness. The award amount depends on the number of eligible jurisdictions participating in the project. The awards range up to $1.5 million per year for up to four years.3 o RII Track-3 Building Diverse Communities. Seeks to broaden the participation of underrepresented groups in STEM fields supported by NSF—underrepresented minorities, women, persons with disabilities and those in underserved rural regions of the country. The awards are up to $750,000 for up to five years to support the strategic goal of broadening participation to improve future research competitiveness of EPSCoR jurisdictions.4 o RII Track-4 EPSCoR Research Fellows. Provides opportunities for non-tenured investigators to further develop their individual research potential through extended collaborative visits to the nation’s premier private, governmental, or academic research centers. Through these visits, the EPSCoR Research Fellows will be able to learn new techniques, benefit from access to unique equipment and facilities, and shift their research toward transformative new directions. The experience gained through the fellowship is intended to provide a foundation for research collaborations that span the recipient’s entire career. These benefits to the Fellows are also expected to, in turn, enhance the research capacity of their institutions and EPSCoR jurisdictions.

2 http://www.nsf.gov/pubs/2016/nsf16557/nsf16557.pdf 3 http://www.nsf.gov/pubs/2016/nsf16511/nsf16511.pdf, https://www.nsf.gov/pubs/2017/nsf17503/nsf17503.htm 4 http://www.nsf.gov/pubs/2013/nsf13553/nsf13553.pdf http://www.nsf.gov/pubs/2016/nsf16511/nsf16511.pdf https://www.nsf.gov/pubs/2017/nsf17503/nsf17503.htm

Co-funding: Leverages EPSCoR investment with that from other NSF Directorates and

Offices. It facilitates participation of EPSCoR scientists and engineers in Foundation-wide programs and initiatives.

Workshops & Conferences. EPSCoR welcomes unsolicited proposals from EPSCoR jurisdictions for workshops involving the EPSCoR community. These workshops focus on innovative ways to address multi-jurisdictional efforts on themes of regional to national importance with relevance to EPSCoR’s goals/objectives and NSF’s mission.5

Outreach Travel. EPSCoR enables NSF staff from all Directorates and Offices to work with the EPSCoR research and education communities regarding NSF opportunities, priorities, programs and policies.

III. Overview of the Evaluation of the EPSCoR Program

Evaluation Purpose

This evaluation will examine the legislative objective of increasing the research competitiveness of jurisdictions receiving EPSCoR funding. It is informed by the findings and recommendations from the EPSCoR retrospective evaluation completed by the Science and Technology Policy

Institute (STPI) in 2012.6

The next step in the evaluation of EPSCoR, is prospective and longitudinal. Although the historical data for each EPSCoR jurisdiction from inception to 2012 from the STPI report will provide valuable background, this study will emphasize new data from 2013 to the present. The purpose of this forward-facing evaluation is two-fold:

(1) To develop a flexible framework to explore, define and measure research competitiveness in relation to the unique jurisdictional contexts of each EPSCoR awardee; and

(2) To collect and use evidence of jurisdictional progress toward research competitiveness over time for strategic program improvement.

An understanding of how and why the unique context and strategic choices made by the EPSCoR jurisdictions contribute to their progress towards increased research competitiveness, individually and in aggregate, will inform the use of these findings to refine existing and tailor future EPSCoR program level activities.

Evaluation Context

The previous EPSCoR evaluation indicated that future program evaluation efforts should focus on the research competitiveness of investigators, participating institutions and EPSCoR jurisdictions as defined by at least three parameters—more proposals, higher proposal success

5 http://www.nsf.gov/pubs/2012/nsf12588/nsf12588.pdf 6 https://www.ida.org/~/media/Corporate/Files/Publications/STPIPubs/2015/P-5221.ashx rates and larger awards.7 In addition to contributions to the intellectual merit review criterion, NSF is interested in exploring how a long-term investment program like EPSCoR contributes to the second merit review criterion, broader impacts.8 Broader impacts include:

(1) Contribution of the research to advancement of the field;

(2) Contribution to a diverse, competitive workforce;

(3) The integration of research and education;

(4) Enhanced infrastructure for research and education;

(5) Dissemination of results; and

(6) Potential and realized benefits for society (in particular economic development)

The evaluation must define a common set of program-level process and outcome indicators that will enable the construction of a consistent picture of research competitiveness strategies and progress toward results across all jurisdictions in order to create status snapshots of research competitiveness now and to monitor progress over time.

Design Considerations

Six design elements provide a foundation for the requirements for the three tasks covered by this statement of work. The contractor shall consider these foundational elements, however they are not intended to preclude contractor creativity. The contractor is encouraged to propose alternatives that are likely to be more effective strategies to frame and conduct the specified program-level evaluation.

(1) Cluster, Multi-site, Multi-level

(2) Stakeholder Engagement

(3) Utilization-focus

(4) Meta-evaluation/Independent Evaluation Oversight Group

(5) Use of Administrative Data

(6) Formative and Summative

(1) Cluster, Multi-site, Multi-level Context: NSF is interested in a variety of approaches ranging from multi-site cluster9 evaluation designs to single subject designs to address the challenge of contextual, process and outcome variation across jurisdictions and levels (individuals and institutions) with respect to research competitiveness.

(2) Stakeholder Engagement: The contractor shall understand and address the priorities and concerns of the intended users, the NSF EPSCoR leadership and program officers, and those primary stakeholders who will be making programmatic decisions on the basis of the

7 Research competitiveness for this study is in terms of NSF funding only: however, analysis of other sources of

State and/or Federal R &D funding may be considered as contextual variables and/or as supporting performance measures, for which any claim of EPSCoR contribution is outside the scope of this inquiry.

8 http://www.nsf.gov/od/oia/special/broaderimpacts/ 9 See W.K. Kellogg Foundation Evaluation Handbook available online at: http://www.wkkf.org/resource-directory/resource/2008/04/designing-initiative-evaluation-a-systems-orientated-framework-for-evaluating-social-change-efforts http://www.wkkf.org/resource-directory/resource/2008/04/designing-initiative-evaluation-a-systems-orientated-framework-for-evaluating-social-change-efforts http://www.wkkf.org/resource-directory/resource/2008/04/designing-initiative-evaluation-a-systems-orientated-framework-for-evaluating-social-change-efforts http://www.wkkf.org/resource-directory/resource/2008/04/designing-initiative-evaluation-a-systems-orientated-framework-for-evaluating-social-change-efforts findings of the evaluation. The contractor shall work closely with an Evaluation Technical

Team (ETT) comprised of 3-5 representatives from the NSF Office of Integrative Activities

(OIA)—the co-sponsors of this evaluation, the EPSCoR and Evaluation and Assessment

Capability (EAC) Sections. The ETT will assure the contractor has access to NSF program and evaluation expertise throughout the entire evaluation process. The ETT will play a key role in:

a. Determining what will be considered credible evidence of outcomes during planning;

b. Reviewing and commenting on all evaluation deliverables prior to the Contracting

Officer’s Representative (COR) approval and acceptance;

c. Discussing preliminary findings; and

d. Using the findings to prioritize actions throughout the evaluative inquiry process.

(3) Utilization-focus: NSF will work closely with the contractor to employ practices that deepen stakeholder understanding of evaluation and commitment to use. It is important to include key NSF EPSCoR program staff in the creation of a shared vision for the evaluative inquiry process and its outcomes. The contractor is encouraged to design the evaluation with the premise that all aspects of evaluative inquiry, from planning through implementation, will have an impact on use for the purpose of program improvement.

(4) Meta-evaluation/Independent Evaluation Oversight Group: This program evaluation will include an expert panel, or comparable process to conduct an independent, external review of the quality/utility of the evaluation process (e.g., framework, plan, implementation and final report). The composition of this panel will be informed and approved by ETT. Members will be subject to explicit, enforced propriety and confidentiality standards for the duration of the evaluation addressed in Section IX. The panel is advisory only; final decisions will be made by EPSCoR and EAC.

(5) Use of Administrative Data: NSF strongly encourages the maximum use of extant EPSCoR data that minimizes the burden of additional data collection. The contractor may utilize other data sources such as data available from Research.gov, existing national surveys, bibliometric services, research publications and presentations in the public domain.

(6) Formative and Summative: This evaluation study will prospectively examine EPSCoR processes and outcomes related to research competitiveness, covering the period FY2013

(October 1, 2012) through the end of FY2017 (September 30, 2017). This request is for both types of evaluation methods/techniques in order to provide EPSCoR with an ongoing feedback mechanism for program modification (formative) as well as the real-time metrics to assess progress toward research competitiveness (summative).

Data Sources

In conducting a rigorous, longitudinal evaluation of the EPSCoR program, the contractor shall use existing project-level proposals as well as annual monitoring information and reports;

however, the contractor has the option to include additional secondary data to construct common measures of aggregate, clustered and individual jurisdiction attributes, activities and outcomes.

Investment in common research competitiveness-related process and outcome measures will streamline the access to and use of evaluation data for on-going systematic and ad-hoc program level learning about research competitiveness.10

The use of National Center for Science and Engineering Statistics (NCSES) research and development surveys, such as the Higher Education Research and Development Survey, 11

Survey of Federal Science and Engineering Support12 and the Survey of State Government

Research and Development,13 may provide valuable jurisdiction context relevant to research competitiveness. Other datasets from the Bureau of Labor Statistics and the Department of

Education may also provide useful contextual information. The contractor shall consider the sufficiency and appropriateness, as well as potential sources of bias for all data sources used.

The contractor should keep in mind that a flexible, mixed methods approach appropriate to the complex and adaptive nature of EPSCoR is sought.

IV. Scope of Work

The technical proposal submitted in response to the tasks outlined in the Scope of Work section should be organized and presented as an evaluation plan14.

Evaluation Tasks

This evaluation will collect and examine evidence that describes the contribution of EPSCoR program investments (RII all tracks, co-funding, workshops and outreach) to research competitiveness across jurisdictions. There are three tasks to be performed:

(1) Evaluation Initiation Phase (3 months duration, post-award);

(2) Evaluation Execution Phase (24 months duration, post-award); and

(3) Final Reporting Phase (3 months).

Task 1. Evaluation Initiation Phase

The contractor shall develop a program-level evaluation framework, focused on research competitiveness that includes common measures15 to construct a consistent, aggregate picture of the range of activities and progress toward outcomes across jurisdictions over time. This framework will identify new ways to integrate jurisdiction level heterogeneous data into common and/or composite process and outcome indicators that are sensitive to the high degree of

10 Common measures may support the development of performance sub-groups which reduce the degree of heterogeneity across jurisdictions and/or enable the discovery and enriched understanding of the variety of trajectories toward research competitiveness observed across EPSCoR jurisdictions.

11 http://www.nsf.gov/statistics/srvyherd/ 12 http://www.nsf.gov/statistics/srvyfedsupport/ 13 http://www.nsf.gov/statistics/srvystaterd/ 14 http://www.wmich.edu/sites/default/files/attachments/u350/2014/plans_operations1.pdf 15 http://www.wmich.edu/sites/default/files/attachments/u350/2014/Indicator_checklist.pdf http://www.wmich.edu/sites/default/files/attachments/u350/2014/plans_operations1.pdf implementation variability and the range of unique contextual factors inherent across the

EPSCoR portfolio.

The initial evaluation framework submitted in response to this SOW must address the evaluation questions as specified, below, as well as:

(1) The criteria for what constitutes evidence;

(2) The processes and methods for obtaining evidence; and

(3) How evidence can be analyzed, synthesized and used to determine both the direction and degree of progress toward desired outcomes/impact in complex, adaptive situations.

After a Kick-Off Meeting and receiving additional information from NSF, the evaluation framework may need revisions to secure NSF approval and acceptance.16 The revised evaluation framework will be used to further explicate the evaluation design and a more detailed plan. The framework must be approved and accepted by ETT.

The evaluation framework is the first step in designing an ongoing mechanism for supporting evaluation of the contributions toward research competiveness achieved through the EPSCoR program. As articulated in this Statement of Work, the framework will primarily rely on using existing, common EPSCoR jurisdiction level data collection and warehouse capacities, and administrative data, with the addition of limited new data collection if necessary. The framework must be based on sound evaluation principles and use a rigorous mixed methods design, including both quantitative and qualitative approaches, as appropriate, to address the evaluation questions posed by the EPSCoR program.

Evaluation Framework Questions

(1) Contextual Variability. To what extent and in what ways does the research competitiveness context currently vary across EPSCoR jurisdictions? What factors best describe the common characteristics that typify this contextual variability? Are there any clusters/groups of jurisdictions with common contextual features that can be identified across the program?

(2) Strategy Variability. To what extent and in what ways do the science and engineering research base and mechanisms currently deployed for improvement vary across jurisdictions? What common characteristics typify the range of implementation variability? Are there any clusters/groups of jurisdictions with common implementation strategies that can be identified across the program?

(3) Outcome Variability. To what extent and in what ways does the variability in context and strategy across EPSCoR jurisdictions influence the identification of relevant indicators of research competitiveness? What jurisdictional, institutional and other characteristics

16 For any evaluation task deliverable, the contractor shall work in close consultation with the ETT to secure approval followed by formal acceptance by the COR.

typify the range of variability observed in research competitiveness definitions and performance? Are there any clusters/groups of jurisdictions with common context and/or strategy characteristics that can be used to understand variability in research competitiveness?

(4) Effectiveness. What differences and similarities exist with respect to implementation strategies and levels of research competitiveness, as defined for this study, for EPSCoR jurisdictions? Are there specific strategies, or combinations of strategies, with evidence of stronger influence or contribution toward research competitiveness than others (see

Table 1 below for examples of metrics)17 For example, how do EPSCoR and non-

EPSCoR institutions in similar Carnegie higher education institutional classification categories currently compare with respect to research competiveness as defined for the study? What career pathways have been developed? To what extent are these career pathways diverse and inclusive, especially for early career researchers?

(5) Institutionalization. What on-going evaluation processes, practices and structures, in particular those related to stakeholder engagement, data collection and analysis, are feasible to support and sustain the current and future implementation of a longitudinal program level evaluation with common measures and a consistent, yet flexible, analytic approach18?

(6) Improvement. What insights can be drawn from the evidence compiled to address questions 1 through 5 that can be used to inform programmatic strategic directions?

Levels/Units of Analysis

The analyses shall be across all EPSCoR jurisdictions and will include analyses of individual, and, to the extent possible, clusters or groups of jurisdictions with similar performance or other characteristics, based on feasibility determination during the design stage. The within-jurisdiction analyses shall be conducted at the individual researcher and institutional (sub-analyses by Carnegie or other classification) levels; this analysis is dependent upon any constraints presented by the quality and availability of extant data. In the event of unanticipated data access or quality issues, the contractor shall develop feasible alternatives within the scope of work. Ideally, the contractor shall identify methods that allow for reasonable, interpretable depiction of each jurisdiction’s own research competiveness trajectory (performance baseline and annual progress made over time).

Minimum Data Requirements (Metrics)

The evaluation plan shall, at a minimum, include measures and/or indicators to include, but not limited to, those shown in Table 1.

17 The contractor is encouraged to suggest additional/alternative metrics and/or indicators of research competitiveness. The evaluation is not limited to the data requirements provided in Table 1.

18 Recognizing that the process would need to be flexible and responsive to Congressional and other requests.

Table 1. Minimum Research Competitiveness Data Requirements

Relevant Jurisdiction Process and Outcome

Indicators

Data Source(s)

Context, institutional characteristics and EPSCoR award history (all components—RII in all tracks; Co-funding and Workshops/Outreach)

EPSCoR Jurisdiction Proposals and Annual Reports

Number and type of EPSCoR award activities within each component, amount of funding allocated

EPSCoR Jurisdiction Proposals and Annual Reports

Number of EPSCoR and non-EPSCoR NSF research grant and/or award proposal submissions

NSF publically available data

Size, number and type of NSF research grants and/or awards (EPSCoR and non-EPSCoR as well as including Research Experiences for Undergraduates

(REU), Graduate Research Fellowship Program

(GRFP), Early Career and other NSF education and human capital programs)

NSF publically available data

Research dissemination and productivity19 Bibliometric services such as Web of Science or

Scopus, Patent data

Start-ups / commercialization

Contribution to broader impacts20 EPSCoR Jurisdiction Proposals and Annual Reports, Department of Education Integrated Postsecondary

Education Data System data

Subtask 1.1–Revise Evaluation Framework and Plan

The Contracting Officer’s Representative (COR) and Evaluation Technical Team (ETT), in close consultation with the contractor, will review and comment on the evaluation framework and preliminary evaluation design considerations. The feedback and recommendations from this process shall be used by the contractor to finalize the evaluation framework and plan.

The contractor shall secure information about the program, its stakeholders and its evaluation history to inform the development of a relevant research competitiveness framework. This information gathering may include activities such as: facilitated discussions with NSF program staff and other internal stakeholders; meetings, interviews, or focus groups with external stakeholders (e.g., principal investigators and co-principal investigators, past NSF reviewers);

19 May include other productivity measures such as service on NSF Merit Review panels, relevant professional collaborations and networking depending on NSF access policies and the final evaluation plan approved by the EAC and EPSCoR.

20 http://www.nsf.gov/od/oia/special/broaderimpacts/ or http://www.nsf.gov/od/oia/publications/Broader_Impacts.pdf http://www.nsf.gov/od/oia/special/broaderimpacts/ http://www.nsf.gov/od/oia/publications/Broader_Impacts.pdf reviews of past research and evaluation findings; analysis of legislation and other documents describing the program such as program solicitations and literature reviews.

To that end, the contractor shall meet with the COR, and ETT as needed, no more than bi-weekly, during Evaluation Initiation Phase to discuss issues and provide an update on project status. Specific items to be covered are: revisions to the evaluation framework, study objectives, study tasks, data collection strategies and instrumentation. In addition, depending on the approach, issues related to relevant institutional contacts (as needed), the Office of Management and Budget (OMB) clearance process and adjustments to the project schedule may need to be addressed. When appropriate, as directed by the ETT and COR, the contractor shall seek limited, low burden feedback on the framework and evaluation plan from the EPSCoR program officers not participating on the ETT.

The contractor’s final evaluation framework and plan must be vetted with the Meta-evaluation/Independent Evaluation Oversight Group (IEOG). The ETT shall consider the IEOG suggestions and recommend those changes deemed appropriate.

The contractor’s final evaluation framework and plan shall provide a clear linkage between the evaluation questions, the evidence sought and the methods to be used. The contractor shall provide an explanation of the conclusiveness the study is likely to support and any trade-offs made in developing the plan. The contractor shall, in addition, document all assumptions, procedures, and modes of analysis selected to demonstrate the extent to which competing explanations can be addressed.

Subtask 1.2–Develop Evaluation Plan Briefing Memo

The contractor shall produce a summary document for the evaluation plan that provides an overview of this evaluation. The contractor shall work closely with the ETT to develop the format, outline and style template for the memo. The final deliverable shall be 508 compliant.

This briefing memo shall:

(1) Describe the EPSCoR program and its research competitiveness objectives; including the program’s logic model or other conceptual illustration;

(2) Present a matrix (or an alternative summary display) of the final approved research questions, the methods used to collect evidence, data sources, the measures and indicators relevant to each question;

(3) Summarize the overall evaluation design and analytic approach; providing the rationale for each strategy; and

(4) Summarize the data collection, analysis and reporting schedule.

Task 2. Evaluation Execution Phase

Execution refers to data collection, analysis, visualization, interpretation, and definitions and metrics of research competitiveness. The contractor shall execute the EPSCoR evaluation at the program-level to create an aggregate picture of the patterns and trends across a range of activities, progress toward research competitiveness and broader impacts of interest to NSF across EPSCoR jurisdictions. The evaluation shall explore the ways and extent to which common and unique research competitiveness mechanisms, pathways, factors and conditions are present across EPSCoR jurisdictions. The evaluation shall describe changes in individual, institutional and jurisdictional research competiveness and related contributions to broader impacts.

The revised and approved evaluation plan developed in Task 1 guides the three evaluation implementation subtasks required to prepare a response to Task 2. Additionally, during the implementation phase, the contractor shall meet not less than quarterly with the full ETT as indicated by the COR. Bi-weekly meetings shall be held with the COR (typically via phone).

Subtask 2.1–Secure OMB Clearance(s) for All New Data Collection

NSF encourages contractors to make full use of extant and administrative data to minimize the burden of additional data collection. However, should additional data be required to implement the design as planned, the contractor shall develop and/or adapt, pilot and validate all new data collection methods or protocols necessary to conduct the study as designed. Data collection and analysis should be designed to gather the most relevant information.

The contractor shall minimize response time burden and demonstrate the degree of validity and reliability pertinent to data use, as appropriate. The contractor shall assist NSF with securing all

OMB clearances required. The contractor shall develop Information Collection Request draft packages for clearance that meet the requirements in the Paperwork Reduction Act and OMB and NSF guidelines.

If the survey or interview of EPSCoR Principal Investigators (PIs) or other stakeholders is determined appropriate to the goals of the project, any sampling strategies and/or respondent selection criteria used by the contractor shall be determined by sound methodology, discussed with the ETT and recommended by the COR. The contractor shall develop and revise instrumentation required to conduct the evaluation as designed; including preparation of the

OMB package for cognitive testing protocols prerequisite to instrument finalization and the

OMB package for instrument use.

Subtask 2.2–Gather Evidence (Data Collection, Analysis, Documentation and Transition)

The contractor, with assistance from the ETT, shall identify required information to conduct the study. The contractor shall not have direct access to the RII Track-1 EPSCoR Reporting Core database nor the RII Track-2 Data Outcome Portal. EPSCoR will provide the contractor with these data.

The contractor shall develop, design and refine as needed, any data analyses, interpretations, and displays of new and extant data to address the evaluation questions listed in the approved evaluation plan. An assessment of data availability, completeness and comparability will be undertaken. The contractor shall provide an analysis plan that:

(1) Describes how the contractor will analyze the data (e.g., the statistical or other methods to be used);

(2) Identifies constructs;

(3) Provides justification; and

(4) Provides example tables, charts and other visualizations for how data will be displayed.

In addition, access to extant and/or new data shall be described in a data collection operations plan (including specification of data quality assurances, security precautions and the schedule for data collection) that is reviewed by the ETT, recommended by the COR and included as part of the evaluation plan document.

The contractor shall provide all documentation related to data collection and analysis such as, but not limited to, data dictionaries, code books, and database structure/contents required for NSF to transfer and use all data at the conclusion of the project. All copies of materials developed during the conduct of this evaluation, including for example, but not limited to, surveys, computer programming codes, files, web files are to be transmitted to the COR, a potential future contractor or disposed of in accordance with instructions from the COR upon completion of work. The contractor is responsible for maintaining records for each task and for transmitting those records to the COR or a potential future contractor. Personally identifiable information

(PII) and confidentiality shall be maintained in compliance with all applicable federal and jurisdictional laws.

Subtask 2.3– Provide Timely Reporting of Evaluation Status via Interim Reports

The contractor shall create interim briefing memos to inform timely discussion of preliminary data and findings with the ETT. Opportunities shall be provided by the contractor for the ETT to reflect on the meaning and significance of preliminary evaluation data or findings for just-in-time adjustment of the evaluation process. The contractor shall present previews as data become available, preliminary analyses and draft findings to the ETT according to the schedule in the approved evaluation plan.

In addition, the contractor may be required to prepare ad hoc briefing memos in response to information requests from the COR that emerge from issues raised by the ETT or other NSF stakeholders. These briefing memos shall be based on the evidence compiled from the findings available to date. The contractor may be requested to assist with the development of dissemination strategies and solicit feedback from stakeholders (as described under Design

Considerations). The focus and format for the interim and ad hoc reports will be determined, as the need arises, with the ETT and recommended by the COR.

Task 3. Final Reporting Phase

During the Final Reporting Phase, the contractor shall facilitate conversations with the ETT, to the extent feasible, with additional EPSCoR stakeholders to codify lessons learned about research competitiveness from the program evaluation. These lessons may be included, if recommended by the ETT, in the evaluation report to the extent permitted by NSF policies and statutes.

The contractor shall prepare a draft final report document with a table of contents and appendices including all instrumentation, models and other deliverables related to the development and finalization of the evaluation. In addition, the contractor shall prepare a PowerPoint presentation that provides an overview of the evaluation plan and present the overview to the ETT.

After presentation to the ETT, the contractor shall make requisite edits to presentation materials and the draft report in order to make a presentation to NSF EPSCoR and the external review panel. Prior to completion of the final evaluation report, the evaluation process, findings and conclusions shall be examined in order to establish the strength and credibility of the evidence and to ensure that the key messages from the evaluation are clear and consistent with all findings.

The draft and final evaluation report shall address each evaluation question and present relevant findings. The contractor shall justify all conclusions and recommendations with evidence provided by the evaluation. Recommendations shall be actionable, clearly identifying and reflecting an understanding of EPSCoR context. In addition, the contractor shall note potential barriers/facilitators to follow up and action steps. The contractor shall also be prepared to present the evaluation results to audiences as specified by the ETT and COR. Materials shall be prepared specific to the information needs of internal and external audiences and may include NSF senior management and/or policymakers. The final deliverables shall be 508 compliant.

Post-Award

The following documents shall be used as the starting point for developing the set of specific standards of evaluation process and product quality to be used by the COR to determine the acceptability of services and deliverables provided by the contractor. The COR will engage the

ETT in the review and selection of the specific criteria designated as priorities for guiding this evaluation. The set of final acceptance criteria recommended by the COR shall be presented for adoption by the ETT and discussed with the contractor. The contractor shall work closely with the COR, to the extent requested, to address any specific acceptance requirements and take action(s) as needed.

Evaluation Plans and Operations Checklist https://www.wmich.edu/sites/default/files/attachments/u350/2014/plans_operations1.pdf

Evaluation Report Checklist https://www.wmich.edu/sites/default/files/attachments/u350/2014/evaluation-reports.pdf

Program Evaluations Meta-evaluation Checklist https://www.wmich.edu/sites/default/files/attachments/u350/2014/program_metaeval_short.pdf https://www.wmich.edu/sites/default/files/attachments/u350/2014/plans_operations1.pdf https://www.wmich.edu/sites/default/files/attachments/u350/2014/evaluation-reports.pdf https://www.wmich.edu/sites/default/files/attachments/u350/2014/program_metaeval_short.pdf

V. Special Requirements

Data Ownership, Propriety and Confidentiality Expectations

The importance of confidentiality cannot be overstated, given the broad array of individuals and entities having vested interest in EPSCoR and EPSCoR-like programs. The contractor and members of the Meta-evaluation/Independent Evaluation Oversight Group may not publish, or in any other way, reveal data, information or the results of the study or other analyses or NSF internal conversations of data collected or accessed for the study without prior written approval from NSF EPSCoR.

The contractor shall transfer to NSF EPSCoR all work products, including web surveys and management systems, e-files used to produce the paper version of the surveys, release and transfer domain name for all surveys, materials necessary to perform data collection, bounce back messages from survey email addresses, and any other items associated with and necessary to administer, analyze, and finalize the surveys.

All data collected are the sole property of NSF and shall not be disclosed or distributed to any non-NSF EPSCoR (excludes COR, EAC) sources without prior written approval from NSF

EPSCoR.21 All reports shall be submitted as MS Office compatible hardcopies and e-files. An electronic copy of the final deliverable(s) shall be delivered to NSF COR.

Access to NSF Data Systems and Tools

NSF will provide the contractor with EPSCoR reports, mailing lists, and other available materials (e.g., awardee annual reports, extant surveys of program participants) deemed by the evaluation plan as relevant to the task appropriate to the success in pursuing the goals of this activity. The contractor shall also have public access to NSF databases such as FastLane and

Research.gov. In addition, the contractor shall be granted access to NSF’s internally developed open source text/document clustering tools to expedite the exploration as needed for trends and patterns in EPSCoR documents. The contractor must provide for secure handing of confidential, proprietary, or personally identifiable information (PII).

Monthly Status/Activity Report

The contractor shall complete and submit monthly status reports to the COR for contract duration. The monthly status report shall summarize activities undertaken, challenges and solutions as deemed appropriate, and planned work for the forthcoming month, highlighting any outstanding issues that need to be addressed in the forthcoming month. The monthly status report shall all highlight personnel changes associated with work on the contract.

In addition the monthly status report, the contractor shall also include a monthly expenditure or financial report. The contractor shall submit a monthly expenditure report, submitted as an excel workbook, containing: (1) labor category hours expended, actual vs. planned, individual and

21 The COR may, if allowable and approved according to established NSF policies and statues, request that the contractor be granted permission by EPSCoR to present and/or publish approved excerpts produced under contract.

cumulative total projected to the end of the current period, (2) funds expended during the period, actual vs. planned and a cumulative total projected to the end of the current period.

Meetings

The contractor shall participate in meetings with the COR for the EPSCoR evaluation throughout the contract period of performance. At least three calendar days prior to each meeting, the contractor shall provide an agenda for the meeting to the COR. The agenda shall minimally describe the topics for discussion and identify those items requiring action and/or decisions.

Following each planning related meeting, the contractor shall provide the COR with the minutes of that meeting within one calendar week. These minutes should describe each issue discussed, conclusions reached as well as the assignment of responsibilities and deadlines for each action item noted.

VI. Contractor Key Personnel Qualifications, Roles & Responsibilities

The table below contain a list of the key personnel, including a description of minimum qualifications, and roles & responsibilities for supporting the EPSCoR evaluation.

Table 2. Contractor Labor Category Qualifications

Contractor

Role

Minimum Qualifications Roles & Responsibilities

Management

Lead

Minimum of a Master’s degree, PhD preferred, and 10-15 years evaluation-project specific management experience of Federal contracts >$1M/year

Lead point of client contact. Final responsibility for all analyses and deliverables.

Technical

Lead

PhD degree required in an evaluation related field and 15 years’ experience directing multi-disciplinary technical teams on Federal contracts >$1M/year

Provides overarching expert evaluation methodological guidance, facilitation, thought leadership and education for the client and project team.

Senior

Evaluation

Consultant

Minimum of a Master’s degree, PhD preferred and 8 years’ experience conducting and managing Federal evaluation projects.

Responsible for day-to-day management and leadership. Coordinates and directs the activities of the team. Including data collection efforts, conducts primary and secondary analyses, develops presentations, databases, spreadsheets, drafts reports and other deliverables.

VII. Agency Staff, Roles and Responsibilities

The contractor shall work with a highly professional and dedicated internal technical team from the Office of Integrative Activities representing a collaboration between EPSCoR and the EAC

Sections. NSF EPSCoR leadership and staff (as well as other stakeholders, as needed) will provide programmatic content expertise and the EAC will contribute evaluation, measurement and assessment expertise. In addition, the EAC will provide the COR to manage the contract.

The names of agency staff assigned to the roles noted above will be inserted in the resulting contract by DACS.

Schedule of Deliverables

Task Deliverable Due Date Submit To

Subtask 1.1 Evaluation Framework and evaluation Plan Revision

90 calendar days after date of award

COR

Subtask 1.1 Final Evaluation Framework and Final Evaluation Plan

Mutual agreement between the COR and contractor

COR

Subtask 1.2 Evaluation Plan Briefing Memo

90 calendar days after date of award

COR

Subtask 2.1 Information Collection Request draft package

90 calendar days after date of award

COR

Subtask 2.2 Documentation relating to data collection and analysis

Included in the Final Evaluation Plan. See Subtask 1.1

COR

Subtask 2.2 Analysis Plan 90 calendar days after date of award

COR

Subtask 2.3 Interim Briefing Memos

Quarterly COR

Subtask 2.3 Ad Hoc Reports and/Presentations

As requested by the

COR

COR

Task 3 Draft final report 90 days prior to the award expiration date

COR

Task 3 PowerPoint presentation (final)

45 days prior to the award expiration date

COR

Task 3 Final Evaluation Report On or before award expiration date

COR

Data ownership, propriety, and confidentiality (Composite database)

Transfer of all data and work products

Final day of award COR

Meetings Meeting Agenda At least 3 calendar days prior to each meeting

COR

Meetings Meeting Minutes No more than 7 calendar days following each meeting

COR

Monthly Status/Activity Report

Monthly By the 15th day of each month

COR

Monthly expenditure report

By the 15th day of each month

COR