CBP_Training_Development_Standards_Oct_2008_107pgs.pdf

Department of Homeland Security

U.S. Customs and Border Protection

Office of Training and Development

Training Development Standards

October 2008

CBP Training Development Standards October 2008

Training Development Standards Table of Contents i

Table of Contents

1. Overview

Related Documents

2. Planning

Business Case (BC) Statement of Requirements (SOR) Management Plan

3. Analysis Participants Training Needs Assessment Target Audience Analysis Current Training Evaluation Job Analysis Task Analysis Media Analysis End of Analysis

4. Design Design Plan Course Purpose and Goals Target Audience and Prerequisites Delivery Method Course Materials Learning Objectives

Learning Objectives for Courses with Scored Assessments Learning Objectives that Are Not Tested

Course Structure Sequencing

Student Activities and Interactions Question Formats Feedback

Images and Media Assessments

Written Tests Practical Tests

Training Development Standards Table of Contents ii

5. Development

Gathering and Organizing Course Content Writing Using Images

Use of the CBP Logo Audio/Video Media

Using Audio/Video Media in Presentation Slides Knowledge Checks and Activities Assessments

Assessment Validation

6. Evaluation

Level 1 – Reaction Level 2 – Learning Level 3 – Behavior Level 4 – Results Distribution of Reports Validations

Phase I Phase II

Program Review Process

7. Delivery Considerations

Instructor-Led Training (ILT) Products Web-Based Training (WBT) Products Train the Trainer Pilot Course Completion

Abbreviations and Acronyms

Training Development Standards 1. Overview

1. Overview U.S. Customs and Border Protection (CBP), Office of Training and Development (OTD), ensures that training delivered to CBP employees meets established quality standards of instruction and evaluation. Training standards apply to all accredited training programs, including e-learning components, and meet Federal Law Enforcement Training Accreditation (FLETA) Standards. The standards also adhere to Federal training mandates such as Shared Content Object Reference Model (SCORM) requirements and Section 508 of the Rehabilitation Act of 1973 regarding accessibility to electronic media.

OTD standards address specific components required for all training developed for CBP and CBP contract personnel. The standards apply to all CBP national training programs and are to be used in concert with CBP training style guides as complete direction for the development of all CBP training.

The objective of the standards is to ensure that training is developed, conducted, and evaluated using a systematic approach that provides continuous self-evaluation and improvement based on analysis, design, development, implementation, evaluation, and revision processes.

The purpose of this document is to:

• Communicate instructional design standards for the development of CBP training materials.

• Outline OTD’s instructional development requirements and acceptance criteria.

This document is divided into the following topical sections:

• Planning – This section covers all required documents and process steps necessary to initiate a training effort. These documents are the foundation for work statements and task orders that result in a final product that matches training requirements.

• Analysis – This section describes the process of identifying the training needs and the critical tasks to be trained. The results from these analyses are the basis for developing the learning objectives and content for the training.

• Design – This section describes the documents that form the backbone of training content, including course goals, lesson objectives (terminal objectives), and enabling objectives, i.e., specific skills and knowledge the trainee will obtain. During the design phase, the course goal, target audience learning objectives, prerequisites, assessment requirements, course topics, instructional strategies, and course structure are documented.

• Development – This section explains the process of creating and submitting draft and final Web-based and instructor-led materials.

• Evaluation – This section covers the review, evaluation, and revision of draft, pilot, and final stages of materials.

• Delivery Considerations – This section deals with the final acceptance of materials.

Training Development Standards 1. Overview

Related Documents This Training Development Standards document is to be used in conjunction with the following related guidance, some of which are referenced in this document:

• CBP Instructor-Led Training (ILT) Style Guide, September 2007

• CBP Instructor-Led Training Development Process Guide, December 2007

• CBP Web-Based Training (WBT) Standards, October 2008

• CBP Web-Based Training Process Guide, October 2008

• Customs Directive No: 51250-001A, Mandatory Successful Completion of Basic

Training, September 3, 1999

• U.S. Department of Homeland Security Seal and Signature Usage Guidelines

• DHS Training Glossary, version 1.2, December 2007

Training Development Standards 2. Planning

2. Planning Planning is the cornerstone of all successful training projects. It involves information-gathering and documentation of project-related decisions and activities so that any completed training course can be traced back to the requests, reviews, revisions, and approvals that led to its development.

Three basic planning documents are involved in initiating an OTD training course:

1. Business Case

2. Statement of Requirements

3. Management Plan

Each document follows in succession after client acceptance, building upon the details in each preceding item. The three planning documents are detailed below.

Business Case (BC) The BC originates with the sponsoring customer. It is the initial description of the proposed course. It explains the customer’s training need, the course’s intended audience, its mission-related justification, skill and knowledge requirements, project budget, proposed delivery media, desired course length, and tentative delivery date.

The BC reflects signed approval by the customer’s Assistant Commissioner and funding review and authorization by the OTD Training Advisory Board (TAB).

Statement of Requirements (SOR) After funding authorization, an SOR is developed based on project details in the BC. The SOR describes skills and activities required of the development team. It lists deliverables and explains budget parameters and constraints. It is the formal communication to a developer of the government’s requirements for the course.

The SOR is the basis of detailed project costing, instructional design, and course development.

OTD and customer approval of the SOR signals that course development can officially be launched during a “kick-off.”

Management Plan The Management Plan expands on the SOR, specifies development tasks, and establishes the project budget and delivery schedule. The document reiterates the course’s purpose and targeted audience and lists assumptions and constraints – reflecting any changes or clarifications generated during the launch meeting, or kick-off. (Typically the course developer produces a Management Plan after the kick-off.)

The Management Plan describes roles and responsibilities of developer(s), customer(s), and reviewer(s); defines project deliverables, and specifies costs associated with each deliverable.

Training Development Standards 2. Planning

The Management Plan describes the course development activities and the responsible parties for each. It presents the final production schedule, including delivery dates and customer review cycles. It also identifies the development team, including the OTD representative, and includes their resumes and contact information.

Training Development Standards 3. Analysis

3. Analysis Analysis is the initial stage of structured curriculum development. Analysis is the process of identifying the learning audience’s needs and creating a studied basis for the content of the proposed training.

Analysis consists of:

1. Identifying the training need

2. Defining the target audience

3. Evaluating the required training content

4. Job analysis

5. Task analysis

6. Media analysis

Participants The analysis team includes an OTD project manager, subject matter experts (SMEs) and other designees of the course sponsor/customer, and others (e.g., instructional design vendors). The team is engaged in developing a task list that reflects the jobs, duties, and tasks of the occupation, plus any necessary physical skills, special skills, and equipment.

Training Needs Assessment The purpose of this analysis is to identify knowledge/skill gaps and suggest the content for new and/or follow-on training.

A number of analysis techniques are involved in gathering background information about the necessary instructional content. Examples include:

• Interviews with SMEs

• Focus groups (moderated group interviews) involving SMEs and/or representatives of the learning audience

• Observation of and interviews with those performing a particular job or task in the field

• Review of course critiques, test results, performance evaluations, etc.

• Instructional review, course audit, or content review of existing training

• Review of field incident reports, critical factors identified, and lessons learned

Target Audience Analysis This analysis examines the projected learning audience. The analysis team reviews relevant background information such as:

• Policy, laws, and regulations

• Training requirements

• Job descriptions

Training Development Standards 3. Analysis

• Anticipated special skills, physical skills, and necessary equipment required for full performance

• Learners’ previous experience and training

• Recommendations from past performance evaluations, training validations, test analyses, and learner feedback

Current Training Evaluation The analysis team uses its audience examination findings and the resulting knowledge/skill baseline to describe a targeted performance level. That performance level description then leads to the development of a task/knowledge list. The task/knowledge list becomes the basis of a proposed scope and sequence of training, plus assessment instruments (i.e., scored and unscored tests).

Job Analysis A job analysis is conducted to identify the tasks that comprise each job. Job incumbents rank the tasks by difficulty, importance, and frequency. After the data is collected and compiled an SME panel of job incumbents develop the Critical Task List (CTL) that reflects the jobs, duties, and tasks of the occupation. The panel then reviews and confirms the final critical task list.

Task Analysis Once the team constructs the CTL for the occupation, it is thoroughly examined and a scope and sequence for the training is constructed. The scope and sequence consists of a hierarchical flowchart or topic list building from the most basic skills for performing the duties of the occupation to the most complex skills, each categorized according to function performances in the field. Tasks are often broken into subtasks and objectives and enabling objectives created for each task or subtask.

Media Analysis The media selection component involves considering the instructional, student, and cost aspects of each objectives category. Many decisions are made by considering the end users as well as the cost-benefit of the solution.

End of Analysis All analysis materials are reviewed and approved by the OTD project manager and appropriate client representatives prior to subsequent phases of course design and development.

Training Development Standards 4. Design

4. Design Design is the process of decision-making by an instructional systems designer (ISD) – in consultation with the customer and SMEs – to determine a training course’s structure, content flow, tone, instructional strategies, testing strategies, interactivity, and navigation, based on the training needs analysis.

Design Plan The results of the design phase are documented in a design plan, sometimes called a design guide. The design plan establishes the framework for the training course and includes the following information:

• Course purpose and goals

• Target audience and prerequisites

• Delivery method

• Course materials

• Learning objectives

• Course structure

• Student activities and interactions

• Requirements for images and audio/video media

• Design and scoring criteria for Level 2 assessments (tests), if applicable

Course Purpose and Goals The course purpose states why the training is needed. This statement can be as simple as, “The purpose of this course is to teach all armed CBP officers and agents to follow correct procedures when flying on a commercial airline with a firearm,” or it can also include more detailed background information.

Course goals describe the desired result of the training. Goals can be stated as terminal learning objectives, as derived during the analysis phase, or the goal statement can summarize those objectives. For example, “After completing this training, the learner will recognize the laws and regulations governing flying armed on a commercial airline and will identify the correct procedures to follow.”

Target Audience and Prerequisites The target audience for the training is identified during the planning and analysis phase, and is specified in the design plan. Examples of target audiences include:

• All CBP employees

• All CBP Officers and Agents

• CBP Officers – Agriculture Specialists

• All CBP first-line supervisors, GS-12 and above

Prerequisites are anything the target audience needs to know or have done before they take the training. They can include knowledge or skills, educational level, previous training, and/or job experience. For example, if a course assumes that learners have knowledge gained from other courses, those prerequisite courses and TRAEN codes must be listed.

Delivery Method The design plan documents the method of course delivery determined during the analysis phase.

The delivery method – whether it is web-based training (WBT), instructor-led training (ILT), blended (i.e., both web- and classroom-based), standalone video, etc. – affects the way the training is designed and developed.

WBT is delivered via the CBP Virtual Learning Center (VLC) that is accessible online to CBP employees at most work locations, either at individual computers or in training rooms. In some cases, CD-ROMs are also provided for students with no VLC access. While most VLC courses are self-paced and do not require a facilitator or instructor, some VLC courses are facilitated.

ILT courses are generally delivered in a classroom setting and/or specific work environment where equipment is available for hands-on training. Logistics for delivering ILT are more complicated than for WBT and include the following considerations:

• Location of training, e.g., ports of entry, Border Patrol stations, CBP headquarters, etc.

• Type of facility where the training will take place, e.g., classroom, work environment, a combination of both.

• Need for and availability of special equipment.

Course Materials Most WBT courses are self-contained and no other course materials are needed. Courses facilitated by an instructor, however, require an instructor guide. Printed student materials such as study guides, reference manuals, or job aids might also be required.

Course materials for ILT generally consist of the following:

• Presentation slides

• Instructor guide

• Student guide

• Reference manual (as needed)

(Refer to the ILT Style Guide for specific content and format guidelines for those documents.)

Other printed materials might include:

• Assessments and/or personal qualification standard (PQS) checklists

• Job aids

• Materials handed out during the class, such as worksheets and other paper-based exercises

In addition to instructional materials, other items might be needed for exhibits, demonstrations, and hands-on practice. For example, a class about fraudulent documents would include examples of both genuine and fraudulent documents so that students can see and feel the difference.

Learning Objectives Learning objectives describe what the learner should know or be able to do after completing a unit of training. These objectives are based on the results of job, task, and training needs analyses. In the absence of formal analysis, course developers derive learning objectives through interviews with the customer and subject matter experts (SMEs) and by studying existing content about the subject, such as user manuals, web sites, previous courses, etc.

Learning objectives are developed from the point of view of the target audience and take into account entry-level knowledge. The process of developing learning objectives begins with these questions:

• What is the knowledge or performance gap that needs to be filled?

• What must learners be able to do after taking this training?

• What knowledge and skills does the learner require?

• Which knowledge and skills need to be taught in this particular course?

Generally, two levels of learning objectives are defined: terminal learning objectives (TLOs) and enabling learning objectives (ELOs). A TLO is the goal for a discrete unit of instruction, usually a lesson. ELOs are the supporting knowledge or skills that lead to achievement of the TLO.

The foundation of an ELO is that it is observable and measurable, and addresses a single, demonstrable knowledge or skill.

Learning Objectives for Courses with Scored Assessments Scored assessments (tests) measure learners’ mastery of the training content. Test questions are generally constructed at the ELO level. Typically, each ELO is tested to the extent possible;

otherwise questions are constructed at the TLO level.

The assessment may take the form of a scored test or may be a case study that requires the student to apply his or her new knowledge and skills. Depending on customer requirements, course assessments may result in a pass/fail score and have employment ramifications.

Test questions assess the mastery of each learning objective according to the desired behavior stated in the objective. That is why all learning objectives must be observable and/or measurable. For example, in a lesson on U.S. states and their capitals, it is not enough to say, “You will know all the U.S. states and their capitals” because it doesn’t describe how to measure whether the objective was met. A well-written learning objective consists of a statement of the task or action, conditions of performance, and the standard to be met. For example:

• Given a blank map of the U.S. containing outlines of the states, you will be able to place the names of at least 35 states (70%) in the correct outlines.

• Given the name of a state, you will correctly name its capital.

Learning Objectives that Are Not Tested CBP fields a number of courses for which scored testing is neither required nor necessary. Even if a scored test is not required, establishing clear learning objectives is essential. Learning objectives identify what will be taught, reinforced, and practiced during the training, regardless of whether there is a scored test.

Course Structure The design document should describe the course structure and flow. This can be done graphically, as shown in the examples below. Text description should also be included that identifies the lessons and topics, approximate duration, and associated TLOs and ELOs.

Example 1.

Lesson 1 Course Overview

Lesson 2

Lesson 3

Lesson 4

Assessment

Lesson 6 Course Summary

Lesson 5

COURSE

Lesson 1

Course Overview

Lesson 2

Lesson 3

Lesson 4

Assessment

Lesson 6 Course Summary

Lesson 5

COURSE

Example 2.

Module 1

Module 2 2-1 Module Introduction

Lesson 2-2

Lesson 2-3

2-4 Module Summary

3-1 Module Introduction

Lesson 3-2

Lesson 3-3

Lesson 3-4

Module 3

Lesson 3-5

Module 3 Assessment

Course Overview

Module 4 4-1 Module Introduction

Lesson 4-2

Lesson 4-3

4-4 Module 4 Summary

4-4 Module Summary

End-of-Course Assessment

Module 5 Course

Summary 3-5 Module Summary

COURSE

Module 1

Module 2 2-1 Module Introduction

Lesson 2-2

Lesson 2-3

2-4 Module Summary

3-1 Module Introduction

Lesson 3-2

Lesson 3-3

Lesson 3-4

Module 3

Lesson 3-5

Module 3 Assessment

Course Overview

Module 4 4-1 Module Introduction

Lesson 4-2

Lesson 4-3

4-4 Module 4 Summary

4-4 Module Summary

End-of-Course Assessment

Module 5 Course

Summary 3-5 Module Summary

Courses typically are divided into lessons. Each lesson addresses one or more TLOs. Lessons are composed of topics, each of which addresses one or more ELO. The length of a course is determined by the content necessary to support the objectives. A complex course might require a more complex structure, with lessons grouped into larger units of instruction such as modules.

The duration (sometimes called “seat” time) of WBT lessons should typically be no longer than 30-40 minutes. Short lessons help learners focus their thoughts and make the material more understandable and accessible.

With ILT, it is generally accepted that people need to take a break every 50 minutes or so.

Classroom lessons should be limited to an hour, taking into account the time needed for group interactions and discussion. Lessons that require more than an hour should be divided into smaller parts.

Sequencing The first lesson is the course overview, explaining the course purpose and structure. The last lesson, the course summary, reviews the main instructional points and gives learners an opportunity to provide feedback on the course. The course may or may not include an assessment.

Instructional content typically is sequenced chronologically, if it is performance-oriented, or according to the need for the knowledge. That way, learners are taught what they need when they need it.

Student Activities and Interactions A tenet of effective adult training is that people learn and retain more through active participation. All learning objectives should be accompanied by appropriate exercises or other opportunities for the learner to interact with the instructional material and apply the new knowledge or skill.

For WBT, examples of interactivity include:

• Icebreakers to encourage people to participate and get to know each other.

• Questions to assess the audience’s current experience and knowledge.

• Knowledge-check questions to review content and reinforce learning.

• Case studies and scenario-based questions.

• Open-ended questions to elicit discussions and share viewpoints and real-life experiences.

• Hands-on practice.

• Role playing to provide practice in specific types of interactions and dialog, such as interviewing techniques.

Activities can be verbal, written (by hand or on a computer), or practical. A game format may be used where appropriate. In the design of interactions, the following factors should be considered:

• Activities should support the learning objectives.

• Activities should match the target audience.

• Especially for pass/fail courses, frequent knowledge-check questions throughout the course should be asked to reinforce the content well before the learner is scored.

• Consider the learning environment and additional supplies or equipment that may be needed.

For WBT, it is equally important to engage the learner by providing similar opportunities for decision-making and practicing skills. In this case, the interactivity is technology based and computer-supported.

Question Formats For informal classroom situations, questions can reflect a variety of formats, including:

• Multiple choice, single correct answer

• Multiple choice, multiple correct answers

• True/false, yes/no

• Matching

• Sequencing

• Fill-in-the-blank or labeling

• Essay

For WBT courses, see CBP Web-based Training Standards for specific guidance.

Feedback All questions are accompanied by feedback that presents the correct answer(s) and the reasoning behind it. Knowledge checks in the form of simulations and case studies should also provide feedback to let learners know the results of their actions or decisions.

Images and Media Whether ILT or WBT, courses are enhanced with visuals – photos, graphics, and audio or video media, including audio clips, narration, music, movies, Flash animations, etc. Any images or media included should help illustrate, clarify, or otherwise enhance the content and support the learning objectives.

During design, the following is established:

• Types of images and graphics

• Source of the images (i.e., CBP, stock art, or custom created)

• Types of audio or video media

Assessments End-of-course assessments are to:

• Assess students’ mastery of the learning objectives

• Allow students to monitor their learning progress

• Evaluate the overall effectiveness of the training

Written Tests When designing written (paper or computer-based) end-of-course assessments, the following factors should be considered and documented in the design plan:

Pass/fail criteria and consequences

Not all assessments require a pass/fail outcome. Establishing a pass/fail score is appropriate if the main purpose is to assess student’s mastery or if the training falls under Customs Directive No: 51250-001A:

Mandatory Successful Completion of Basic Training.

If a pass/fail assessment is selected, the customer establishes the consequences for not passing and identifies the policy addressing those consequences. If there are no such consequences, the test should not be pass/fail.

The consequences of failing a test range from mild to severe, including:

• Not receiving a certificate of course completion

• Not proceeding to the next course

• Being denied a promotion

• Being demoted or terminated

The pass/fail score is based on the percentage of questions the student answers correctly. The passing score at CBP is typically 70% but there might be exceptions for specific courses. You will need to calculate the number of questions that need to be answered correctly in order to pass the test. If the calculation does not result in a whole number, round DOWN to the next number. For example, if the test consists of 35 questions, the student must answer 24 correctly to pass.

Remediation and re-testing

For a WBT course with a non-pass/fail test, trainees may repeat the course and retake the test as many times as they wish. For ILT, students might not have the opportunity to retake a test administered in the classroom.

For a pass/fail test, the customer establishes the remediation to be provided and how many times the student may attempt the test.

Remediation may include repeating the entire course, retaking selected parts of the course, or undergoing individual coaching.

Remediation and re-testing for Basic Training courses

Customs Directive No: 51250-001A: Mandatory Successful Completion of Basic Training states remediation and retesting procedures for courses that fall under this directive.

For these courses, three scored tests (written exams or practical exercises) are developed. The employee must score at least 70% on the test. Test questions on each of the three versions must be 80% different from one another. If a trainee fails the first version of the test, he or she is retested with a second version. A third version is used in case one of the other two tests is compromised.

Number of test questions

In general, tests should include at least one question for each enabling learning objective. However, course duration and available training time may limit the number of questions. (With ILT, the trainees need time to take the test and the instructor needs time to score answers and provide learner feedback.) If test questions cannot cover every ELO, there needs to be a question for each terminal learning objective.

Duration Straightforward multiple-choice questions require approximately one minute per question. Two minutes or more are allowed if the question is complex, requires a lot of reading, contains multiple parts, or involves looking up reference materials.

Open or closed book

Whether students may refer to course materials and/or references while completing the assessment should be established based on what is expected on the job.

Question formats Most of the question formats applied to student interactions can be used for written tests that are scored by an instructor. Question formats in tests delivered by computer or Scantron may be limited by those technologies.

The use of true/false or yes/no questions is discouraged because students have a 50-50 chance of getting them correct just by guessing.

With multiple-choice questions, answer choices should not include overlapping choices such as “all of the above,” “A and C only,” or “none of the above.”

Free-form formats such as essays, which rely on human judgment to score them, should generally be avoided. However, essay tests with well defined scoring criteria can be better measures of content mastery because they involve higher levels of thinking, such as analysis and synthesis, not just simple recall.

(Refer to the ILT Style Guide and the CBP Web-based Training Standards for specific guidance.)

Practical Tests End-of-course assessments also include practical tests that involve hands-on or other performance competency instead of written questions. These types of tests are called Personal Qualification Standards (PQS).

In a PQS, an instructor provides a checklist of actions or procedures to be performed. The instructor observes and assesses the student’s performance.

For practical (hands-on) tests, the same considerations regarding pass/fail criteria, testing of each ELO, and remediation and re-testing apply as for the written tests.

Training Development Standards 5. Development

5. Development Course materials described in the design plan are created during the Development phase.

Several interim products are also developed during this phase. For WBT, those products include:

• Prototype

• Storyboards

• Assessment/test items

• Alpha (draft) version of the course

• Beta version of the course

For ILT, interim deliverables are the draft versions of the course materials. For both WBT and ILT, if audiovisual media are included, scripts and video “rough cuts” – i.e., initial edited versions – will also be needed.

Web-Based Training Instructor-Led Training Prototype Audio/video script/rough cut* Storyboards Presentation slides Audio/video script/rough cut* Instructor guide Assessment test items* Student guide

Reference manual* Programmed courseware and assessments (Alpha, Beta, and Final) Assessment test items*

* Not applicable for all courses

Each WBT and ILT product has an associated cycle of review by the customer/SME and OTD project manager for accuracy, instructional soundness, and conformance to standards. WBT courseware also undergoes functionality testing for conformance with the requirements of the online Virtual Learning Center. Details about requirements, formats, and review cycles are documented in the Web-Based Training Process Guide, Web-Based Training Standards, Instructor-Led Training Process Guide, and Instructor-Led Training Style Guide.

Gathering and Organizing Course Content Much of the content is gathered during the Analysis and Design phases of the project, but SMEs must remain engaged during the entire development process to provide input as needed and review interim deliverables. The instructional systems designer (ISD) organizes the content, culling redundant or unnecessary information – i.e., content that does not support the learning objectives – and divides the content into a logical structure of lessons and topics.

Content is gathered from a number of reliable sources, including:

• Interviews with SMEs

• Interviews with those who perform the job at various levels of experience

• Existing online courses, training materials, presentations, etc.

• Relevant laws, policies, and directives

Writing Writing is involved in storyboards for WBT screens, ILT presentation slides, instructor and student guides, and scripts for audio/video pieces. Specific presentation requirements and constraints are different for each type of writing, but the following guidance applies to all:

• Address the learner directly. For example, instead of saying, “The student will be able to,” say, “You will be able to.”

• Limit narrative to what is relevant and necessary to support the learning.

• Don’t talk in generalities. Use job-related examples to explain concepts.

• Avoid the “textbook or manual on a screen” approach to WBT.

• Use clear, concise wording and keep sentences short. Use bulleted lists where appropriate.

• Onscreen text for WBT should be limited in its visual density and allow “white space” rather than filling the screen. Long paragraphs should be broken up into shorter ones.

• Onscreen text for presentation slides must be large enough to be readable by everyone in the room.

• With ILT, present the key points on the slides. Provide details in the instructor/student guides content notes areas.

(More specific guidance on writing for ILT is in the Instructor-Led Training Style Guide.)

Using Images Still images include photographs, charts and graphs, and digital artwork. Incorporating images into WBT or ILT requires locating or creating suitable images, editing them as necessary, applying consistent treatments, and placing them on the screen.

Photographs and other images can be obtained from a variety of sources, including:

• CBP public photo gallery (http://www.cbp.gov) and intranet site (http://cbpnet.cbp.dhs.gov )

• Other CBP courses

• CBP shared drives

• Government and other Web sites

• Licensed commercial stock art

When suitable images aren’t readily available, new photographs and digital artwork may need to be created.

The following factors should be considered when selecting or creating images for a course:

Relevance Images support the content and learning objectives by illustrating and clarifying the information being presented. Inserting generic images for the sake of having images only teaches learners to ignore the instructional significance of visuals.

Format, resolution, and size

Most images downloaded from the Web are in JPG format, which is generally suitable for WBT and PowerPoint slides. JPG images usually have been compressed to reduce file size, sometimes resulting in loss of resolution. Sharp, clear images are required. Downloaded images must be large enough so they need not be stretched when placed on the screen.

(Refer to the Web-Based Training Standards for further guidance on file formats for digital artwork.)

Images for printed materials

Images to be used for printed materials, such as for the cover page of the instructor guide, must be high resolution. The CBP photo gallery contains high-resolution images. Commercially purchased stock images are available in various resolutions.

Current standards Images depicting CBP employees must adhere to current standards for uniforms, grooming, and weapons.

Consistency of style A lesson might contain a variety of existing or newly created images from various sources. Regardless of images’ origins, their overall look and feel must be cohesive. There is no haphazard mixture of artistic styles and appearance. Similar treatments, such as frames, are applied to provide continuity.

Placement on screen

Placement of images on screen should also be cohesive. That does not mean that every image has to be the same size and in the same position on every page. However, images of similar size and position should not appear to shift from page to page.

Copyright Copyright restrictions must be followed for images acquired from non- CBP sources. Appropriate permissions must be obtained before using any copyrighted images.

Permissions If WBT or ILT images depict recognizable members of the public, individuals’ written release is required for their use. Written release by CBP employees is not required, but CBP employees do not have to agree to being photographed.

Use of the CBP Logo

The CBP logo, consisting of the DHS seal and U.S. Customs and Border Protection word mark, is used in accordance with DHS guidelines:

“The seal and signature of the U.S. Department of Homeland Security were created to embody our mission and values.

As the public presentation of our department, it is important that it be reproduced with care and precision. Consistent reproduction across all types of publications – print materials, presentations, advertisements, Web sites, signage and stationery – will reinforce our mission and result in a greater awareness of the Department of Homeland Security.”

-- U.S. Department of Homeland Security Seal and Signature Usage Guidelines

Note: The resolution of the CBP logo in printed materials that will be converted to Adobe Acrobat (PDF) format must be at least 200 dots per inch (dpi) to retain image clarity.

Several variations of CBP logos are available in encapsulated postscript (EPS) format. These official logos may be scaled proportionately and converted to other graphic formats, but cannot be altered in any other way. For further details, refer to the U.S. Department of Homeland Security Seal and Signature Usage Guidelines.

Audio/Video Media If audio or video are to be included in a course, a treatment and script are typically the first deliverables. A treatment describes, in storytelling style, the themes, actions, tone, main visuals, and message points the video will convey. Scripts are split-page documents: on the left are descriptions of the visuals, including the types of shots and actions that are to occur onscreen; on the right is the precise narration and/or dialog and other audio (such as music or sound effects) that the video will contain. A video script takes the place of storyboards.

Video can be a slide show that combines still images with audio and animation, or it can be a full scale movie shot in a studio or on location, featuring professional or non-professional onscreen talent and/or voiceover narration.

Using Audio/Video Media in Presentation Slides Various types of audio/video media files can be imbedded directly into a slide. The media file should not be programmed to play automatically upon entering the page. Instead, the media should feature an instructor-activated control -- such as a “Play” button or text that prompts the user to select an image to play the video.

Visual cues and instructions used for playing a media file must be consistent for all lessons throughout the course.

Knowledge Checks and Activities Frequent and varied interaction with the instructional material is essential for engaging participants and reinforcing learning. Activities may be included throughout a lesson or are grouped at the end of a lesson.

In WBT, activities are presented on screen and learners interact with the built-in functionality.

In ILT, each activity is described on a slide so that learning activities are presented in a consistent manner. Instructions for conducting the activity are provided in the instructor guide.

Feedback explaining the correct answer is provided for both WBT and ILT. With ILT, this feedback might be displayed on the next slide or contained in the instructor guide.

Assessments During the Development phase, specific test items are created. (For further details, see the Design chapter.) Question formats may be constrained by test delivery format. If the test is computer-based or uses Scantron forms, not all the assessment formats are available.

In ILT, there are two products: the student test and the instructor’s answer key. For each test item, the answer key shows the question, correct answer, the learning objective being tested, and the location in the course where the content is covered.

A similar answer key should also be developed for WBT for reviewers to determine the validity of questions. The answer key would not be part of the final product.

Assessment Validation Using the answer keys, test questions must be validated by the customer and OTD to ensure the following:

• Questions and answers are accurate and correct.

• Wording for each question, correct answer(s), and distracters is clear and unambiguous.

• All questions are matched to learning objectives.

• Content is covered in the course.

Training Development Standards 6. Evaluation

6. Evaluation Evaluation of training programs assesses the effectiveness of training in preparing new employees to perform specified job duties. Evaluation identifies training requirements that may have been missed by previous training and/or job analyses. Evaluations can also signal the need to change training based on a new or modified mission, duty requirements, lessons learned, or dramatic events.

All Federal training is evaluated in one way or another. For CBP, the Research and Evaluation Branch (REB) of the Office of Training and Development (OTD) establishes sound procedures for conducting evaluations and reporting the results to the Director of Training.

The evaluation process that OTD REB follows is the system developed by educational theorist Donald Kirkpatrick1. The Kirkpatrick approach employs four levels of evaluation:

Level 1 – Reaction Level 2 – Learning Level 3 – Behavior Level 4 – Results

OTD relies on evaluation levels 1-3. Level 4 is more suited to business applications and does not generally apply to CBP training.

Level 1 – Reaction Level 1 is a subjective evaluation by students during and immediately after instruction. Level 1 evaluation gathers “reactive feedback” from the trainees. It asks them to express their satisfaction with the training and to judge the quality of the instructors, the content, training material and modalities, and the learning environment. Results of Level 1 evaluation can be used to enhance or correct training material and improve instructor skills.

A common method for gathering learner feedback is through “bubble” scan-sheets on which students indicate their reactions about the training using a scale of 1-5, in which 1 is Poor and 5 is Excellent, plus anonymous written comments. Standardized interviews with students can also be employed.

Level 1 evaluations are administered, tabulated, and reported through the CBP National Training Evaluation Program (NTEP). OTD and/or the academies determine the frequency of evaluating various instructional programs. During a given training year, all courses are expected to be assessed at least once.

Level 2 – Learning Level 2 evaluation assesses learners’ mastery of the training immediately after completing the course. Evaluations are conducted by administering objective, valid, and reliable written tests that are graded against a minimum passing score. Some of those tests can determine whether a student remains in the program or is terminated from employment.

Level 2 tests objectively measure how well the training program has prepared the employee to perform on the job. Level 1 and 2 results typically are combined and reported to the Director of Training or designated official. The report highlights areas of the training in which learners tested and reported well, in addition to areas of the training that should be reviewed for improvement, according to respondents’ feedback.

Level 3 – Behavior Level 3 evaluations determine whether the training has prepared graduates to perform various tasks and duties while on the job. The process functions as a check on the accuracy of validation studies and permits corrections to shortcomings uncovered through job analyses.

Elements that comprise Level 3 evaluations are:

• Surveys

• Interviews

• Tests

• Incorporation of Level 1 and Level 2 information and data

Objective performance tests are administered to trainees by a neutral group of OTD REB evaluators near the end of Academy training. Those tests measure to what extent and how effectively the students believe they have applied the training to their actual work in the field.

Surveys are sent to graduates between 8 weeks and 6 months after trainees have completed the course and have had an opportunity to apply the skills learned at the Academy. Interviews with graduates and their immediate supervisors also are conducted as part of evaluation.

All the evaluative feedback is analyzed and a concise report is prepared for the course sponsor, combining input from questionnaires, interviews, observations, and Levels 1-3 evaluation results.

A final report is prepared that highlights areas of training that appear to have problems or offer noticeable benefits. In the report, problems and benefits revealed in Level 3 evaluations are supported by evidence from Levels 1 and 2, tests, and interviews.

Level 4 – Results Level 4 evaluations concern results of training that contribute to the overall organization.

Results are normally reported as a return on investment (ROI). According to the Kirkpatrick model, Level 4 attempts to quantify how much training contributes to the profit margin of a business.

Distribution of Reports Upon approval of the Level 3 Report by the Chief of REB, the Evaluation Report is transmitted to the Director of Training Production and Standards Division (TPSD).

For information purposes, an executive summary is forwarded to the operational office that oversees the occupation being trained. Level 3 reports are produced for internal CBP purposes only. (Distribution outside the OTD is prohibited, except by special permission of the Director.)

REB retains copies of all Level 3 reports for a period of seven years. After seven years, all copies are destroyed and one is retained in electronic format.

Validations The standard procedure for validating CBP training programs is a two-phase process. OTD REB provides the technical leadership for both phases.

Phase I This phase commences with an examination of existing job tasks by a task review panel composed of subject matter experts (SMEs) representing the CBP Program Office, a union representative a representative from the academy, and an REB project manager. As needed, the critical task list (CTL), which is the basis of training development, is revised to reflect any changes in mission, law, regulations, policy, and special skills used by the occupation.

Generally, a survey is prepared and distributed to field locations to verify that the CTL addresses the occupational requirements. The compiled survey data serve as an empirical decision-making aid for the Critical Task Selection Board charged with the final selection and refinement of the CTL, specifying where each task will be trained. Like the task review panel, the Critical Task Selection Board is generally composed of SMEs from field locations; an TPSD REB evaluation expert, and representatives of the union, the Academy training unit, and the affected CBP Program Office.

Phase II This phase of the validation process consists of a detailed review of the existing training curriculum, cross-referenced to the selected critical tasks. This process eliminates extraneous content from the curriculum and assures that only the critical tasks will be addressed.

Prior to curriculum revision, the following are constructed:

• Hierarchical scope and sequence of the program to ensure optimal learning progression.

• Task-to-Training Matrix showing the relationship of the CTL, objectives, test items, taxonomy classification, and where trained. (Those components are accreditation documentation standards.)

The first offering of this training, normally referred to as the pilot, will be attend by a representative of TPSD and REB. During the pilot offering, these representatives will be responsible for ensuring the following is performed before validation of a pass/fail course can occur:

• Ensures the instruction is presented in the manner in which it was developed.

• Instructor presents all content tested in the examination.

• Delivered test is fair and equitable.

• Item analysis and impact study is conducted for all tests and validated.

Finally, the new curriculum is implemented. Program validation concludes only after both phases of the validation process are complete and the appropriate verification and approval forms are signed by the Director of Training Production and Standards Division.

The normal cycle for conducting a validation study is once every three to five years, subject to various internal and external factors such as changes in agency mission or operations, or significant impact of legislation and policy.

Program Review Process The Program Review Process determines whether:

• Curriculum content is current

• Curriculum content has face and content validity

• The curriculum has met the established OTD standards of training of development

Therefore, the Program Review Process encompasses a traditional “expert team” type of evaluation review. Inputs, processes, outputs, and products from each phase of course development are examined to ensure that they produced top quality training.

A Program Review Process checklist guides the review team, using inspection standards derived from the Instructional Systems Design model itself. (If there is no documentation of the curriculum content or ISD standards, or if existing documentation does not meet CBP OTD standards, the CBP team must create or modify them.)

A Training Review Panel (TRP), a Critical Task Selection Board (CTSB), and other expert panels then sequentially review the curriculum with the goal of achieving consensus on the critical tasks for training.

The CTSB initially validates the CBP critical task list, the Physical Skills List, the Special Skills List, and the Required Equipment List pertaining to the requirements of the CBP occupation.

The TRP, comprised of about 7 individuals, and the CTSB, comprised of about 12-14 individuals, are composed of subject matter experts (SMEs) from the particular occupation. The SMEs include field officers, training unit staff from the Academy, OTD headquarters, and the OTD Research and Evaluation Branch (REB).

Numerous panels – nicknamed “Murder Boards” – are conducted to review and revise the CBP course test items developed by the Academies, depending on the number and complexity of the test items. FLETC provides the test item analysis on the three required tests in its test item bank.

Training Development Standards 7. Delivery Considerations

7. Delivery Considerations OTD training development projects require the following products for final delivery.

Instructor-Led Training (ILT) Products

1. ILT Pilot Version of Instructor Guide and Student Materials:

A pilot is a “test run” of a complete and approved course in a…