Requirement_for_a_statistical_approach_to_training_evaluation.pdf

Department of State Foreign Service Institute

Office of the Director

Requirements Document:

A Statistical Approach to Implementing a Core Construct Training Evaluation

Background:

Training evaluation at the Department of State Foreign Service Institute (FSI) is relatively decentralized as the responsibility for evaluation resides with the Schools/Center leadership.

Nevertheless, FSI leadership requires a way to analyze training effectiveness beyond the individual Schools/Center to encompass the entire Institute. To enable this analysis, FSI currently uses FSI-wide core questions that address all four Kirkpatrick evaluation levels. The core question data, along with School/Center-specific data are collected and analyzed by each School/Center. The Schools/Center core question data are rolled up and analyzed at the Institute level to gauge Institute-level performance.

FY16 was FSI’s evaluation pilot year where the FSI-wide core questions were implemented.

Data were collected and analyzed and the first evaluation reports were submitted in December 2016. In 2017, FSI reflected on lessons learned from the evaluation pilot year and made recommendations for moving forward.

One of the lessons learned was that the FSI-wide core questions failed to provide data that enabled evidence-based decision making at both the School/Center and Institute levels.

Therefore, a revised core question strategy is being developed through a working group.

The proposed alternative solution will replace the FSI-wide core questions with FSI-wide core constructs – constructs that are key to making FSI a preeminent diplomatic training academy.

The working group is identifying and clearly defining those constructs. Once the constructs are defined, Schools/Center will have the flexibility to determine how they measure those constructs.

FSI recognizes the importance of robust data collection and analysis processes to ensure 1) the Schools/Center are measuring the intended core constructs and 2) the data, collected by various methods, can be combined in a way to allow FSI to gauge performance on each of those core constructs at an Institute level. FSI requires a statistically robust approach for collecting and analyzing data from FSI’s Schools/Center.

Required Deliverable:

The deliverable is a technical report that details an FSI-specific, statistically robust approach for collecting and analyzing the data from FSI’s Schools/Center to implement the core construct training evaluation. The report must include:

1) A School/Center level specific process that enables them to collect data they need and still allow FSI to analyze data across Schools/Center addressing the core constructs.

The process must address at a minimum:

a. The need for Schools/Center to test whether an item is measuring the core construct it is intended to measure and whether a group of items are measuring the same core construct. And if this is necessary, the steps to conduct the analysis.

b. Data collection requirements and constraints Schools/Center must follow (e.g., minimum number of items per construct, consistency of scales, use of construct anchor questions, minimum sample size) to enable analysis at both the Schools/Center and Institute levels.

c. Item construction requirements that allow the necessary statistical analyses.

d. Any statistical analyses must be executable with the analysis tools available to

FSI schools/center– primarily Excel.

2) An Institute-level specific process to combine and analyze varied data formats from the

Schools/Center to report on Institute performance within each of the core constructs. The process must address at a minimum:

a. Data format requirements for Schools/Center to submit data for Institute-level analysis.

b. Data transfer requirements for the Schools/Center (e.g., spreadsheet format) to enable Institute-level analysis.

c. An analysis process to ensure that when data are combined at the institute level, we can be confident that the Schools/Center are measuring the intended core constructs so we can make a statement about Institute performance on each core construct (for example, the Schools/Center are measuring effectiveness in a way that allows us to combine their items into an Institute-level measure of effectiveness).

d. Any statistical analyses at the Institute level must be executable in one of the analysis tools available in Educational Policy (SPSS and/or Excel).

3) An implementation plan to enable FSI to test and execute the recommended overall approach.

The offeror should propose a process for developing a statistically robust approach for collecting and analyzing data to implement our FSI-wide core construct training evaluation.

Deliverable must be completed by December 15, 2017

NOTE: The offeror must demonstrate experience in a relevant field, such as educational research statistics, or measurement and evaluation, and proficiency in SPSS and Excel for statistical analysis.