Attachment C - MSIPP Evaluation and Monitoring Plan.pdf

DOE/NNSA MSIPP Evaluation Plan (October 1, 2018 – September 30, 2021)

Submitted by: Quality Measures LLC

MSIPP Program Mission: Create and foster a sustainable STEM-pipeline that prepares a diverse workforce of world class talent through strategic partnerships between Minority Serving

Institutions and the DOE/NNSA Enterprise

Evaluation Questions Indicators/Metrics Data Collection Strategies

Comments Source of Data Data Instrument Frequency Responsible Person

Crucial Success Factor #1: Strengthen and expand MSI STEM educational and/or research capacity in DOE/NNSA mission areas

EvalQ#1: To what extent did the MSIPP program activities strengthen and expand MSIs’ capacity and research experiences in NNSA/DOE mission areas of interest?

Educational

• # of new MSIPP-type courses, labs, degree programs, curriculum modifications

• # of enrollments & passing grade averages

• # of Certificates

Metric Reports MSIPP Database Quarterly POC at MSIPP

Institutions

Inform the extent to which academic programs prepared students for MSIPP opportunities (Skills & knowledge) as well as faculty’s ability to expand the

MSIPP curriculum, lab, and other educational experiences for students. Credentialing.

Non-Metric:

Students

POCs @ MSIs &

Lab/Plants

NNSA Leadership

• FGs w/ students

• Surveys w/ students, faculty, and Plant/Labs

• Interviews w/ POCs and

NNSA Leadership

• FGs Annually

• Surveys

Annually

• Interviews Y1 &

Y3

Evaluator & MSIPP

POCs at MSIs and

Labs/Plants

Professional Development (students and faculty):

• # and quality of workshops

• # and quality of externships

Metric Reports MSIPP Database Quarterly POC at MSIPP

Institutions

Workshop Quality: Repeat sessions, participant post-responses, and presenter’s knowledge

Externships Quality:

Preparedness, hire-ability, and alignment of research interests and needs @ DOE/NNSA

Non-Metric:

Faculty, Students

(including GAs), and

Lab/Plant POCs

• FGs w/ students

• Surveys w/ students, faculty, and Plant/Labs,

• Interviews w/ POCs and

NNSA Leadership

• FGs Annually

• Surveys

Annually

• Interviews Y1 &

Y3

Evaluator, MSIPP

POC, and NNSA

Leadership

Dissemination:

• # of publications

• # of symposium & conference presentations and posters/awards

• # of internal gov’t reports

Metric Reports

(Faculty, Student, and Plant/Lab

Authors)

MSIPP Database

Quarterly

POCs at MSIPP

Institutions and

Lab/Plants

Also include awards for best paper, poster presentations and other accolades specific to work produced by MSIPP collaborations

Engagement:

• # of faculty in meaningful collaborative research w/ DOE/NNSA staff, laboratories, plants, and agencies in

MSIPP

• # and quality of co-advising

• # and quality of DOE/NNSA Enterprise speaking engagements or events/activities at MSIPP institutions for faculty, students, and staff.

Metric Reports

Lab/Plants

Collaboration Quality: Level of cooperation and sharing of resources per agreements between MSIs and Labs/Plants.

Co-Advising & Speaking

Engagements Quality: Post evaluations of presenters’ relevance of information and ability to engage audiences.

Student research outcomes.

Non-Metric:

Faculty, Students

(including GAs), and

Lab/Plant POCs

• FGs w/ students

• Surveys w/ students, faculty, & labs/plants

• Interviews w/ POCs &

NNSA Leadership

• FGs Annually

• Workshop Surveys

• Surveys Annually

• Interviews Y1 &

Y3

Evaluator, MSIPP

POC, and NNSA

Leadership

Evaluation Questions Indicators/Metrics Data Collection Strategies

Comments Source of Data Data Instrument Frequency Responsible Person

EvalQ#2: To what extent did the grants create opportunities for

MSIPP students to conduct cutting edge research on MSIPP campuses that enhanced student learning and research opportunities?

Research:

• # and quality of collaborative research projects for students and faculty with

DOE/NNSA resources on MSIPP campuses

• # and quality of undergraduate and graduate research opportunities and experiences at MSIPP campuses

Lab/Plants

Measure student preparedness for lab/plant opptys., presentations from collaborative research, hiring or hire-ability of students, timing of completed research.

Also, growth/loss of faculty credentialing/eligibility (i.e., ABET)

Non-Metric:

Faculty, Students (UG

& Grad), and

Lab/Plant POCs

• FGs w/ students

• Surveys w/ students, faculty, and labs/plant

POCs

• Interviews w/ POCs

FGs Annually

Surveys Annually

Interviews Y1 & Y3

Evaluator & POCs at

MSIPP Institutions and

Lab/Plants

Crucial Success Factor #2: Target collaborations between MSIs and the DOE/NNSA enterprise to increase interactions that provide MSIs direct access to DOE/NNSA STEM resources.

EvalQ#3: To what extent did the

MSIPP collaborations between

MSIs and the DOE/NNSA

Enterprise increase interactions and provide MSIs direct access to

DOE/NNSA STEM resources?

Interactions:

• # of onsite visits of faculty to labs and labs to MSIs

• # of telecons, meetings, and webinars

• Level of involvement and helpfulness

Direct Access:

• Access to lab equipment per lab protocols/process

Lab/Plants

Measure interactions based on metrics and self-reported experiences of faculty/labs & plants via surveys and interviews. Also, contextualize

“access” based on protocols and processes of Labs/Plants and MSIs for equipment use, data-sharing, and research collaborations

Non-Metric:

Faculty, Students (UG

& Grad), and

Lab/Plant POCs

• Surveys w/ students and

POCs @ MSIs and labs/plants

• Interviews w/ POCs and

NNSA Leadership

Surveys Annually

Interviews Y1 & Y3

Evaluator, MSIPP

POC, and NNSA

Leadership

EvalQ#4: To what extent did the

MSIPP program provide internships that aligned w/ students’ academic preparation and professional goals?

Internship Experiences:

• # and quality of internship experiences

(growth in knowledge, skills/abilities)

• # and quality of professional/career development activities/events provided to students outside of day-to-day internship work duties

• # of student publications and conferences presentations (single and joint)

Lab/Plants

Measure knowledge, awareness, behavioral growth via surveys and interviews w/ students, faculty, & Labs/plants site supervisors. Conduct FGs w/ students during industry and

MSI events, and possibly professional conferences where students conduct poster presentations

Non-Metric:

Faculty, Students (UG

& Grad), Lab/Plant

POCs & NNHQ

• FGs with students

• Surveys w/ students and

POCs @ MSIs, labs/plants, & NNHQ

• Interviews w/ POCs

FGs w/ students

Surveys Annually

Interviews Y1 & Y3

Evaluator & POCs at

MSIPP Institutions and

Lab/Plants

Crucial Success Factor #3: Grow the number of minority (MSIPP) students who graduate w/ STEM degrees

EvalQ#5: To what extent did

MSIPP funding enhance the position of MSIs to attract and retain students in STEM disciplines of interest to the DOE/NNSA enterprise?

• # of new, returning, and transfer students at

MSIPP institutions in STEM degree programs (retention/persistence)

• GPA Eligibility and graduation rates of

MSIPP and non-MSIPP students

• # of new, returning, and transfer students at

MSIPP institutions in STEM degree programs (retention/persistence)

• # of students from MSIPP institutions who enter graduate school in STEM degrees

Lab/Plants

Measure academic preparation

(UG & Grad), , hire-ability or intent-to-hire students

(Labs/Plants). Compare MSIPP and non-MSIPP students in STEM with regard to retention/persistence.

Non-Metric:

Faculty and Students

(UG & Grad)

• Surveys w/ students, and

POCs @ MSIs and labs/plants

FGs w/ students

Surveys Annually

Interviews Y1 & Y3

Evaluator and MSIPP

POC

Evaluation Questions Indicators/Metrics Data Collection Strategies

Comments Source of Data Data Instrument Frequency Responsible Person

Crucial Success Factor #4: Grow the number of minority (MSIPP) graduates and post-doctorates hired in DOE/NNSA STEM Workforce

EvalQ#6: To what extent did the

MSIPP program activities result in an increased number of minority graduate students and post-doctorates hired in the

DOE/NNSA enterprise?

• # of graduate student offers, rejections, acceptance of positions in STEM or

DOE/NNSA enterprise

• # of post-doctoral placements from MSIPP

Institutions in STEM careers

• # of post-doctoral placements from MSIPP

Institutions in the DOE/NNSA Enterprise in a STEM position

Lab/Plants

Metrics will inform the number of students entering and remaining within the pipeline in disciplines of interest to

DOE/NNSA. Additional measures—surveys, focus groups, and interviews—will inform the ways in which these activities informed the metrics as well as how and why certain experiences and activities had an impact, and the extent of the impact.

Non-Metric:

Faculty, Students (UG

& Grad), and

Lab/Plant POCs

• FGs w/ students

• Surveys w/ students and

POCs @ MSIs and labs/plants

• Interviews w/ POCs and

NNSA Leadership

FGs Annually

Annual Surveys w/ students, and POCs

@ MSIs and

Labs/Plants

Interviews Y1 & Y3

Evaluator, MSIPP

POC, and NNSA

Leadership

DOE/NNSA MSIPP Evaluation Monitoring Plan (October 1, 2018 – December 15, 2021) | Submitted by: Quality Measures LLC

Evaluation & Monitoring Activities

Year 1 (Oct. 2018 – Sept. 2019) Year 2 (Oct. 2019 – Sept. 2020) Year 3 (Oct. 2020 – Sept. 2021)

2nd Qtr

Jan-Mar ’19

3rd Qtr

Apr–Jun ‘19

4th Qtr.

Jul-Sept ’19

1st Qtr

Oct-Dec ’19

2nd Qtr

Jan-Mar ‘20

3rd Qtr

Apr-Jun ’20

4th Qtr

Jul-Sept ’20

1st Qtr

Oct-Dec ’20

2nd Qtr

Jan-Mar ‘21

3rd Qtr

Apr-Jun ’21

4th Qtr

Jul-Sep ’21

1. Evaluation Framework and Plan Mar. 21

(Finalize)

Apr. 16, 2019

(Share @ Mtg)

Oct. 1

(Review)

Oct. 1

(Review)

Instrument Design and IRB Approval

2. Develop Data-Collection Tool & Protocols Mar 30

3. DOE/IRB Application and Approval* Apr 1

(Application)

Jun 1

(Approval)

Implementation Fidelity

4. Share evaluation plan and data-collection of metrics with MSIPP grantees

April 16-18, April 2020 April 2021

5. Create and Monitor Data-Sharing Agreement with

MSIs (FERPA Best Practices)

Apr 1

(Create)

Jul 1

(Monitor)

Jul 1/Aug 1

(Monitor)

6. Review of MSIPP and Consortium level evaluation activities and challenges

Consortia Annual Meeting Consortia Annual Meeting Consortia Annual Meeting

MSIPP Technical Meeting MSIPP Annual Technical Meeting MSIPP Technical Meeting

Consortia Site Visits Consortia Site Visits Consortia Site Visits

Data-Collection & Analyses (See NOTE Below)

7. Consortia will send Quarterly METRICS to

Evaluator

March 13

(1Qtr/General)

Apr 30

(2Qtr/General) Jul 30

(3rdQ Metrics) Oct 30

(Jul-Sept ’20) Jan. 30

(Oct-Dec ‘20) Apr 30

(Jan-Mar ‘20)

Jul 30 (Apr-Jun ‘20)

Oct 30 (Jul-Sept ‘20)

Jan. 30 (Oct-Dec ‘20)

Apr 30 (Jan-Mar ‘21)

Jul 30 (Apr-Jun ‘21)

8. Receive Quarterly Reports from NNSA Mar 13|1stQ19

(Oct-Dec ‘18)

April 30

(2nd Qtr)

Jul 31

(3rd Qtr)

Oct 30

(4th Qtr)

Jan. 30

(1st Qtr)

April 30

(2nd Qtr)

Jul 31

(3rd Qtr)

Oct 30

(4th Qtr)

Jan. 30

(1st Qtr)

April 30

(2nd Qtr)

Jul 31

(3rd Qtr)

9. Receive evaluation results from consortia Optional Data collected Oct 1 thru Aug 30

(End of Fiscal Year Due Sept 30 | See #19)

Optional Data collected Oct 1 thru Aug 30

(End of Fiscal Year Due Sept 30 | See #19)

Optional Data collected Oct 1 thru Aug 30

(End of Fiscal Year Due October 30 | See #19)

10. Send Quarterly Updates to NNSA Aug 15th Dec 15 Feb 15 May 15 Aug 15 Dec 15 Feb 15 May 15 Aug 15

11. Administer Surveys to Faculty, Students, & POCs

@ MSIPP Institutions and Labs/Plants Summer 2019 (After IRB Approval) End of Summer 2020 End of Summer 2021

12. Focus Group/Interviews with Students Spr/Summer Spr/Summer Spr/Summer

13. Interviews w/ NNSA, POCs @ MSIs, Labs/Plants Spr/Summer Spr/Summer

14. Data Analysis (Metrics Roll-Up and Surveys) Nov 2019 Nov 2020 Nov 2021

15. Data Analysis (Qualitative) Nov 2019 Nov 2020 Nov 2021

Meetings & Training

16. Monthly Consortium Meetings TBD TBD TBD

17. Biweekly/Monthly Evaluation Meetings/NNSA Thursdays @ 11:00 AM Thursdays @ 11:00 AM Thursdays @ 11:00 AM

18. Annual MSIPP Technical Meetings April 16-18, 2019 April 2020 (?) April 2021 (?)

Evaluation Reports & Status Updates

19. Consortium Specific Evaluation Results (Optional) Sept 30 Sept 30 October 30,

20. Formative Evaluation Reports (Metrics & Non-

Metrics to NNSA) (Process Evaluation Efforts)

Dec 15, Dec 15,

21. Summative Evaluation Report for Entire Project Dec 15, 2021

NOTE: Data collected in the 4th quarter of the final year (2021) must be received by QM no later than October 30, 2021 and will be included in the summative evaluation report. Consortia responsibilities highlighted in green.