PWS for Foreign Service Test Development.pdf

DRAFT

Performance Work Statement Foreign Service Test Development

Background:

The Department of State (DOS), Bureau of Human Resources (HR), Board of Examiners (BEX) requires the development of legally and professionally validated written examinations to be the initial screening in the selection process to identify qualified candidates for Foreign Service employment (Foreign Service Officer and Foreign Service Specialist). Pursuant to the Foreign Service Act of 1980, as amended, the Secretary of State is charged with prescribing written and other examinations for appointment to the Foreign Service of the United States. The DOS requires that the examinations be nondiscriminatory and conform to professional testing guidelines. Test development shall be based on the results of Job Analyses reports. Test development for the Foreign Service Officer examination (FSOT) includes test item development for up to three test administrations per year and compilation of a bank of essay prompts for up to three administrations. For the on-demand tests to include the Consular Fellows Program Test (CFPT) and the Diplomatic Technology Officer Test (DTOT)) and potentially contingent on available funding with no specified timeline—the Office Management Specialist Test (OMST), and the Human Resources Officer Test (HROT) and other selection processes requested by the State Department. The contractor will provide the following work: validation, report generation for fully accessible (i.e., conformant with the requirements of Section 508) and statistical reporting.

Statement of Objectives

BEX requires a Contractor to ensure DOS hiring and promotion systems legally and professionally address its unique context and changing requirements. This should include the following tasks:

1. Create test materials for DOS Foreign Service Generalists and Specialists Tests

2. Provide a timeline plan for the development and production of test materials;

3. Develop Job Knowledge; English Expression; and Logic Reasoning.

4. Job Analyses for all Generalists’ cones

5. Develop validation plan for compliance with test blueprints.

6. Travel as needed (domestic or international)

CLIN 1: Foreign Service Officer Test (FSOT)

Description - Develop a computer-based testing instrument known as the Foreign Service Officer Test (FSOT) for administration as the initial part of the selection process for Foreign Service Officers. This instrument shall be based on the results of the most recent job analysis of Foreign Service Officer generalist positions.

The FSOT is considered a generalist “gateway” test. Candidates applying for any of five career tracks (Consular, Economic, Political, Public Diplomacy, and Management) must pass the “gateway” before taking additional assessments.

FSOT Test specifications are:

1) Job Knowledge (JK): JK forms consist of 50 operational items and 10 pre-test items for a section 40 minutes in length.

2) English Expression (EE): EE forms consist of 45 operational items and 20 pre-test items for a section 50 minutes in length.

3) Logical Reasoning (LR): LR forms consist of 20 operational scenarios and 5 pre-test scenarios for a section 42 minutes in length.

4) Essay Prompts: As a separate testing event, the essay prompts shall be on a substantive topic to allow candidates to draft an essay of sufficient length to evaluate each candidate's ability to analyze a substantive topic, organize their ideas, and express them in correct and readable English prose. However, only writing skill, and not the essay content, will be scored.

Prompts used operationally in one window cannot be used in the next test window. The time allotted candidates to draft the essay will be 45 minutes (total seat time is 60 minutes).

Task 1.1: FSO (Generalist) Job Analysis

1) Update the most recent job analysis to include identifying tasks, knowledge, skills, abilities, and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs from each career track so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the basis for the blueprints used to create test questions and for documenting content validity.

4) Analyze JAS data and write final job analysis report.

Task 1.2: FSOT Item Development

1) Review the 2018 Job Analysis Update and FSOT specifications (blueprints) for each test component, review items in each item bank, and identify deficiencies in each item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint.

5) Conduct test expert review of new FSOT items to confirm content accuracy.

6) Conduct SJT review workshops with DOS experts to ensure that the scenarios response options for the situational judgment tests are plausible.

Task 1.3: FSOT Form Assembly

1) Assemble operational forms for each test that meet statistical and content criteria using blueprints for each test.

2) Facilitate a yearly workshop for BEX to review the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic administration.

5) Conduct pre- and post- publishing quality control of the test forms.

Task 1.4: FSOT Validation and Scoring Plan

1) Facilitate the collection of content validity (CV) data by collecting ratings on new items by Foreign Service Officer subject matter experts (SMEs) identified by DOS (typically five experts per management track). This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, and for development and validation of test items, as needed.

4) Maintain the item bank and import test items.

Task 1.5: FSOT Analytics

1) Analyze psychometric data (e.g., difficulty level, item-total correlations) from each administration/cohort to inform inclusion of items in future operational test forms. For FSOT, provide post-administration reports (e.g., passing and failing candidate score frequency distributions) within approximately two weeks following receipt of the DOS determination on those candidates who are to be invited to the Foreign Service Officer Assessment (FSOA);

standard reports shall include breakdowns by career track.

CLIN 2: Foreign Service Officer Assessment (FSOA)

Description – Develop/maintain computer-based testing instruments that comprise the Foreign Service Officer Assessment (FSOA). The FSOA is taken by those who pass the FSOT. A Case Management Exercise is administered to specialists who pass their respective “gateways.”

Task 2.1: FSOA Development

1) Review the most recent job analysis and discuss with BEX/DOS the relevance/datedness of FSOA testing instruments and make suggested changes requested.

2) Provide updates/refinements that include the following components:

a) Virtual Negotiation Exercise

i. Work with BEX to develop materials for Negotiation exercise

b) Virtual Structured Interview: (60 minutes) Four components:

i. Critical Thinking (CT)

ii. Experience and Motivation (E&M)

iii. Hypotheticals (HYPO)

iv. Past Behavior Questionnaire (PBQ)

3) The Case Management Exercise is administered as part of a separate testing event (for specialists).

CLIN 3: Consular Fellows Program Test (CFPT)

The Consular Fellows Program is a limited non-career appointment of up to five years. Its mission is to provide overseas services that protect U.S. citizens, ensure U.S. national security, facilitate the entry of legitimate travelers, and foster economic growth at home. Consular Fellows adjudicate visa applications of foreign citizens seeking to come to the United States to visit, study, and work, and assist U.S. citizens traveling or living abroad.

CFPT is considered a “gateway” test. Candidates must pass the “gateway” before taking additional assessments.

CFPT test specifications are:

1) Job Knowledge (JK): JK forms consist of 45 operational items and 5 pre-test items for a section 18 minutes in length.

2) English Expression (EE): EE forms consist of 25 operational items and 5 pre-test items for a section 18 minutes in length.

3) Comprehending Regulations (CR): CR forms consist of 15 operational items and 5 pre-test items for a section 27 minutes in length.

4) Situational Judgment Test (SJT): SJT forms consist of 20 operational scenarios and 5 pre-test scenarios for a section 18 minutes in length.

5) Case Management Exercise (CME): single exercise 60 minutes in length delivered as a separate testing event.

Task 3.1: CFPT Item Development

1) Review the most recent job analysis and CFPT specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct test expert review of new CFPT items to confirm content accuracy.

6) Conduct SJT review workshops with DOS experts to ensure that the scenarios response options for the situational judgment tests are plausible.

Task 3.2: CFPT Form Assembly

2) Facilitate reviews for CFP experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic administration.

5) Conduct pre- and post- publishing quality control of the test forms.

Task 3.3: CFPT Test Validation and Scoring Plan

1) Facilitate the collection of content validity (CV) data by collecting ratings on new items by subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, and for development and validation of test items, as needed.

4) Maintain the item bank and import test items.

Task 3.4: CFPT Analytics

1) Analyze psychometric data (e.g., difficulty level, item-total correlations) from each administration/cohort to inform inclusion of items in future operational test forms.

CLIN 4: Office Management Specialist Test (OMST)

Office Management Specialists (OMS) with the U.S. Department of State provide office management and administrative support to U.S. diplomats in embassies and consulates abroad, as well as some domestic locations. They coordinate the efficiency and effectiveness of the organization using organizational, time management, problem solving, and decision-making skills, along with tolerance of intensive world travel, living away from family, and working and living in difficult and/or isolated conditions.

OMST is considered a “gateway” test. Candidates must pass the “gateway” before taking additional assessments.

OMST test specifications are:

1) English Expression (EE): EE forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

2) Situational Judgment Test (SJT): SJT forms consist of 30 operational scenarios and 5 pre-test scenarios for a section 31 minutes in length.

3) Reading Comprehension (RC): RC forms consist of 15 operational scenarios and 5 pre-test scenarios for a section 25 minutes in length.

4) Case Management Exercise (CME): single exercise 60 minutes in length delivered as a separate testing event.

Task 4.1: OMS Job Analysis and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

Task 4.2: OMST Item Development

1) Review the most recent job analysis and OMST specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct academic and test expert review of new OMST items to confirm content accuracy.

6) Conduct SJT review workshops with DOS experts to ensure that the scenarios response

Task 4.3: OMST Form Assembly blueprints for each test.

2) Facilitate reviews for OMS experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic administration.

5) Conduct pre- and post- publishing quality control of the test forms.

Task 4.4: OMST Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, and for development and validation of test items, as needed.

4) Maintain the item bank and import test items.

Task 4.5: OMST Analytics

CLIN 5: Foreign Service Specialist Assessment System (FSSAS) (with Job Knowledge)

The Foreign Service Specialist Assessment System (FSSAS) is used for jobs with relatively low numbers of job incumbents where new positions are advertised through vacancy announcements.

Provide tests for administration as the initial part of the selection process for the following specialist positions:

• Human Resources Officer (HRO): JK, EE, SJT

• Construction Engineer (CE): JK, EE, SJT

• Facilities Manager (FM): JK, EE, SJT

• Financial Management Officer (FMO): JK, EE, SJT

• General Service Officer (GSO): JK, EE, SJT

FSSAS test specifications are:

1) Job Knowledge (JK): JK forms consist of 30 operational items and 5 pre-test items for a section 25 minutes in length.

2) English Expression (EE): EE forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

3) Situational Judgment Test (SJT): SJT forms consist of 30 operational scenarios and 5 pre-test scenarios for a section 31 minutes in length.

Task 5.1: FSSAS (with JK) Job Analysis

1) Update the most recent job analyses to include identifying tasks, knowledge, skills, abilities, and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs from each specialty so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

4) Analyze JAS data and write final job analysis reports.

Task 5.2: FSSAS Item Development

1) Review the most recent job analysis and test specifications (blueprints) for each job and each test component, review items in the current bank, identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct test expert review of new specialist items to confirm content accuracy.

Task 5.3: FSSAS Form Assembly blueprints for each test.

2) Facilitate reviews for relevant specialist experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 5.4: FSSAS Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Facilitate a workshop/survey for specialist experts/BEX to review the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Provide ongoing consultation on the best methods for assessing the required content areas, Task 5.5: FSSAS Test Validation and Scoring Plan

CLIN 6: Foreign Service Specialist Assessment System (FSSAS) (without Job Knowledge)

The Foreign Service Specialist Assessment System (FSSAS) is used for jobs with relatively low numbers of job incumbents where new positions are advertised through vacancy announcements.

Provide tests for administration as the initial part of the selection process for the following specialist positions:

• Medical Provider (MP): EE and SJT only

• Regional Medical Lab Scientist (RMLS): EE and SJT only

• Regional Medical Officer (RMO): EE and SJT only

• Regional Medical Officer-Psychiatrist (RMO_P): EE and SJT only

• Regional English Language Officer (RELO): EE and SJT only

• Regional Public Engagement Specialist (RPES): EE and SJT only

Note that job knowledge tests are not provided for the MED jobs or for the RELO or RPES because they are highly specialized jobs that require advanced training as part of the minimum qualifications. Because these are relatively low-volume positions, tabular reports are not provided for these tests.

FSSAS test specifications are:

1) English Expression (EE): EE forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

2) Situational Judgment Test (SJT): SJT forms consist of 30 operational scenarios and 5 pre-test scenarios for a section 31 minutes in length.

3) Case Management Exercise (CME): single exercise 60 minutes in length delivered as a

Task 6.1: FSSAS (without JK) Job Analysis

1) Update the most recent job analyses to include identifying tasks, knowledge, skills, abilities, and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs from each specialty so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance

4) Analyze JAS data and write final job analysis reports.

Task 6.2: FSSAS Item Development

1) Review the most recent job analysis and test specifications (blueprints) for each job and each test component, review items in the current bank, identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct academic and test expert review of new specialist items to confirm content accuracy.

Task 6.3: FSSAS Form Assembly blueprints for each test.

2) Facilitate reviews for relevant specialist experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 6.4: FSSAS Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Facilitate workshops/surveys for specialist experts/BEX to review the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Provide ongoing consultation on the best methods for assessing the required content areas,

4) Maintain the item bank and import test items.

Task 6.5: FSSAS Test Validation and Scoring Plan

CLIN 7: Diplomatic Technology Officer Test (DTOT)

The Diplomatic Technology (DT) Bureau enables the provision of secure information technology infrastructure and services in a complex international IT environment. DT plays a critical role in enhancing the productivity, efficiency, and global communication capabilities of the Department by offering modern IT tools, approaches, systems, and information services to best support secure, data-informed diplomacy.

DTOT test specifications are:

1) Job Knowledge (JK): JK forms consist of 30 operational items and 5 pre-test items for a section 25 minutes in length.

2) English Expression (EE): EE forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

3) Situational Judgment Test (SJT): SJT forms consist of 30 operational scenarios and 5 pre-test scenarios for a section 31 minutes in length.

Task 7.1: DTOT Job Analysis and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

Task 7.2: DTOT Item Development

1) Review the most recent job analysis and DTOT specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct test expert review of new DTOT items to confirm content accuracy.

Task 7.3: DTOT Form Assembly blueprints for each test.

2) Facilitate reviews for DTO experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 7.4: DTOT Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, Task 7.5: DTOT Analytics

CLIN 8: Diplomatic Security Special Agent Test (DSSAT)

The Bureau of Diplomatic Security (DS) is the law enforcement and security arm of the U.S.

Department of State. Operating from a global platform in more than 30 U.S. cities and more than 170 foreign countries, DS ensures the United States can safely and securely conduct diplomacy.

DS is a world leader in the protection of people, property, and information – international investigations, threat analysis, cyber security, counterterrorism, and security technology. As members of the Foreign Service, DS has more than 2,500 Special Agents (DSSATs), Security Engineering Officers (SEOs), Security Technical Specialists (STSs), and Diplomatic Couriers worldwide. Provides tests for DSSATs, SEOs, and STSs (see CLINS 8, 9, and 10 respectively).

DS Special Agents are sworn federal law enforcement officers and specially trained Foreign Service security professionals. While serving overseas as a Regional Security Officer, a DS Special Agent advises the U.S. Ambassador on all security matters and manages a complex range of security programs designed to protect personnel, facilities, and information. DS Special Agents protect the U.S. Secretary of State and visiting foreign dignitaries, investigate passport and visa fraud, and conduct personnel security investigations.

DSSAT test specifications are:

1) Job Knowledge (JK): JK forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

2) English Expression (EE): EE forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

3) Situational Judgment Test (SJT): SJT forms consist of 30 operational scenarios and 5 pre-test scenarios for a section 31 minutes in length.

Task 8.1: Special Agent Job Analysis and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

Task 8.2: DSSAT Item Development

1) Review the most recent job analysis and DSSAT specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct expert review of new DSSAT items to confirm content accuracy.

Task 8.3: DSSAT Form Assembly blueprints for each test.

2) Facilitate reviews for DSSAT experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 8.4: DSSAT Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, Task 8.5: DSSAT Analytics

CLIN 9: Security Engineering Officers (SEO)

Security Engineering Officers are responsible for the protection of U.S. Department of State personnel, facilities, and sensitive information from acts of crime, terrorism, and technical espionage. They help keep the technical resources that U.S. diplomats use safe and secure, as they work to advance our nation’s interests overseas.

SEO test specifications are:

1) Technical Knowledge (TK): TK forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

2) English Expression (EE): EE forms consist of 25 operational items and 5 pre-test items for a section 25 minutes in length.

3) Situational Judgment Test (SJT): SJT forms consist of 25 operational scenarios and 5 pre-test

4) Essay prompt: single exercise 60 minutes in length delivered as a separate testing event.

Task 9.1: SEO Job Analysis

5) Update the most recent job analysis to include identifying tasks, knowledge, skills, abilities, and other characteristics (KSAOs), and precepts from previous that are still applicable.

6) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

7) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

8) Analyze JAS data and write final job analysis report.

Task 9.2: SEO Item Development

1) Review the most recent job analysis and SEO specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items contain no graphics.

5) Conduct test expert review of new SEO items to confirm content accuracy.

Task 9.3: SEO Form Assembly blueprints for each test.

2) Facilitate reviews for DSSEO experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 9.4: SEO Test Validation and Scoring Plan

1) Facilitate the collection of content validity (CV) data by collecting ratings on new items by subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas,

4) Maintain the item bank and import test items

Task 9.5: SEO Analytics

CLIN 10: Security Technical Specialist (STS)

Security Technical Specialists install, troubleshoot, and maintain electronics and electromechanical systems, and develop both professionally and personally. Security Technical Specialists help keep the technical resources that U.S. diplomats use safe and secure, as they work to advance our nation’s interests overseas.

STS test specifications are:

1) Technical Knowledge (TK): TK forms consist of 35 operational items and 5 pre-test items for a section 25 minutes in length.

2) English Expression (EE): EE forms consist of 25 operational items and 5 pre-test items for a section 25 minutes in length.

3) Situational Judgment Test (SJT): SJT forms consist of 25 operational scenarios and 5 pre-test

4) Essay prompt: single exercise 60 minutes in length delivered as a separate testing event.

Task 10.1: STS Job Analysis

9) Update the most recent job analysis to include identifying tasks, knowledge, skills, abilities, and other characteristics (KSAOs), and precepts from previous that are still applicable.

10) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

11) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance

12) Analyze JAS data and write final job analysis report.

Task 10.2: STS Item Development

1) Review the most recent job analysis and STS specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items may contain graphics.

5) Conduct academic and test expert review of new STS items to confirm content accuracy.

Task 10.3: STS Form Assembly blueprints for each test.

2) Facilitate reviews for DSSTS experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic administration.

5) Conduct pre- and post- publishing quality control of the test forms.

Task 10.4: STS Test Validation and Scoring Plan

1) Facilitate the collection of content validity (CV) data by collecting ratings on new items by subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, Task 10.5: STS Analytics

CLIN 11: DS Courier

Diplomatic Couriers safeguard and escort diplomatic pouches containing classified and sensitive material between U.S. diplomatic missions overseas and the U.S. Department of State.

Courier test specifications are:

1) English Expression (EE): EE forms consist of 25 operational items and 5 pre-test items for a section 25 minutes in length.

2) Situational Judgment Test (SJT): SJT forms consist of 25 operational scenarios and 5 pre-test

3) Essay prompt: single exercise 60 minutes in length delivered as a separate testing event.

Task 11.1: Courier Job Analysis and other characteristics (KSAOs), and precepts from previous that are still applicable.

2) Review resulting task lists and KSAO/precept lists with SMEs so that they can add, delete, and modify tasks and KSAOs and finalize lists.

3) Create a job analysis survey (JAS) to be sent to job incumbents for them to rate Importance and Time Spent on tasks and Importance and Needed at Entry for KSAOs. This will form the

Task 11.2: Courier Item Development

1) Review the most recent job analysis and Courier specifications (blueprints) for each test component, review items in the current bank, and identify deficiencies in the item bank.

2) Develop/review item writer's guides and training materials for item writers.

3) Identify item writers and conduct training.

4) Develop items to address deficiencies based on the test specifications/blueprint. These items may contain graphics.

5) Conduct academic and test expert review of new Courier items to confirm content accuracy.

Task 11.3: Courier Form Assembly

2) Facilitate reviews for DSSTS experts/BEX to examine the forms to be administered for the following year to ensure that items are correct, not dated, and free of cueing.

3) Design configuration of pre-test items to be included as part of operational tests forms.

4) Provide test items in the required format (e.g., QTI files) to enable their electronic

Task 11.4: Courier Test Validation and Scoring Plan subject matter experts (SMEs) identified by DOS. This will occur using workshops or surveys as determined by DOS. Analyze rating data collected to identify items that meet content validity criteria.

2) Provide scoring formulae to enable pass rates according to DOS needs.

3) Provide ongoing consultation on the best methods for assessing the required content areas, Task 11.5: Courier Analytics

CLIN 12: Travel

a) Contractor Personnel.

The Contractor shall be responsible for assuring that contractor personnel who are required to travel are physically able to travel to the specified destination, and to remain there without significant health risk for the required period of the travel event. The Contractor shall be responsible for assuring that such individuals receive the proper immunizations and take proper health measures before, during, and after travel. Physical examinations and immunizations will not be provided by the U.S. Government (USG). Additionally, the Contractor is responsible for medical, disability, life insurance, and other requirements, as may be applicable, in order to comply with host nation government labor laws and the Federal Acquisition Regulation (FAR).

b) Travel Terms & Conditions.

Incidental travel may occur in order to execute related contract tasks. Travel costs are governed by FAR 31.205-46, which addresses Federal Travel Regulations, Joint Travel Regulations, and Standardized Regulations (Government Civilians, Foreign Areas). In cases where travel is required, the contractor shall provide the Contracting Officer Representative and Contracting Officer with estimated travel costs and receive approval prior to travel. The Contractor will be reimbursed for reasonable travel costs provided that the travel has been authorized in advance by the Department of State and reimbursement of travel costs is authorized for the specific travel event. Travel costs subject to reimbursement are limited to travel occurring at the direction of the USG and performed in conjunction with a specific requirement for a trip authorized in this contract. The government WILL NOT incur any costs for travel not approved.

c) Travel Advisories.

All Contractor personnel traveling to foreign destinations shall comply with all U.S. Department of State travel advisories for the specified destinations. The Contractor shall coordinate all travel under this contract with the COR.

Other Direct Costs: Travel may be required for vendor’s staff to assist with training, registration, and other testing administration issues.

Government Oversight/Surveillance Management

Performance Monitoring

• Regular review of contractor deliverables against contract requirements

• Site visits and inspections

• Progress reports and status updates

• Quality assurance reviews

Financial Oversight

• Invoice review and payment verification

• Cost tracking against budget allocations

• Audit of contractor financial records

• Review of indirect cost rates

Compliance Monitoring

• Verification of regulatory compliance (labor laws, safety standards, etc.)

• Review of required certifications and licenses

• Monitoring adherence to Federal Acquisition Regulation (FAR) requirements

• Small business subcontracting plan compliance

Administrative Surveillance

• Contractor performance assessment reports (CPARs)

• Risk assessments

• Contract file documentation reviews

• Tracking of contract modifications and changes

Technical Oversight

• Review of technical approaches and methodologies

• Evaluation of personnel qualifications

• Assessment of equipment and facilities

• Quality control inspections

Location: Place of performance.

Security requirements: The Contractor shall implement comprehensive security measures to protect Foreign Service test materials, systems, and candidate data throughout all phases of test administration. All personnel with access to test materials or candidate information must have at minimum a Public Trust and sign an NDA. The Contractor shall establish documented chain of custody procedures for all test materials from receipt through destruction, store materials in locked, secure facilities with controlled access and intrusion detection, and use tamper-evident containers with trackable shipping for all transportation. The Contractor shall report any actual or suspected security incidents to the COR within 2 hours for critical incidents (test compromise, data breach) and 24 hours for all other incidents, maintain detailed incident documentation, and permit announced and unannounced security audits by Department personnel. Upon completion of contract or termination, all test materials must be returned to BEX or destroyed using approved methods with written certification provided within 30 days. Performance will be measured against zero tolerance for unauthorized disclosures, 100% compliance with background investigations and security training, and timely incident reporting, with security violations potentially resulting in corrective action requirements, financial penalties, or contract termination.

Clearances:

This is a Sensitive but Unclassified (SBU) contract. Contractor personnel specifically designated by the Contracting Officer's Representative (COR) will be required to have a Moderate Risk Public Trust (MRPT) determination conducted by the Bureau of Diplomatic Security.

Performance Requirements Summary (PRS):

Required Services

(Tasks)

Performance

Standards

Acceptable Quality

Levels

Methods of

Surveillance

Monthly reports are due by the 5th of each month. All reports should include tasks completed for each test.

Yearly reports on work conducted throughout the year

100% of reports accurately depict current status

Once a year

99%

100%

File reviews, periodic inspections, and random, observations, customer complaints

Biweekly meetings, emails, reports post testing event

Deliverables:

Line Item

Description Product or Service Code

(PSC)

Quantity Delivery schedule

OR

Period of performance

Destination

OR

Place of performance

PWS

reference*

001 Generalists’ tests transferred to vendor for administration.

Foreign Service Officer Tests

4 with various iterations throughout the year

A month prior to

FSOT (4

times a year)

Vendor for Test Administration

Task 1.3

002 Specialists’ Tests transferred to vendor for administration

Foreign Service Specialists Tests

Varies with iterations of test administrations determined by

BEX

Various times throughout the year.

Vendor for Test Administration

CLINS 3-9

003 Reports on the work conducted in a month

Progress Reports

Monthly and yearly

COR

004 Job Analysis for Foreign Service Generalists positions

Report 5 1 year COR, Program Office

CLIN 1.1