Attachment 8- TSA MD 1900.14 Training Standards Handbook.pdf

MD 1900.14 Training Standards Handbook

Policy Effective: October 23, 2020

Handbook Published: October 23, 2020

APPROVAL

Kim Hutchinson

Assistant Administrator

Training and Development

TSA HANDBOOK 1900.14

TRAINING STANDARDS

This page intentionally left blank.

Page i

TABLE OF CONTENTS

A. Introduction

B. Purpose

C. Definitions

D. Scope

E. Authorities and References

F. Standards for Developing Training

F.1 Analysis

F.2 Design

F.3 Development

F.4 Implementation

F.5 Evaluation

G. Standards for Reviewing Training

G.1 Review of Documents and Evaluation Data

G.2 Curriculum Review Conference Activities

G.3 Curriculum Review Records Management

Attachment A

Course Risk Assessment/Job Hazard Analysis

Page ii

LIST OF FIGURES

Figure F-1: ADDIE Model

LIST OF TABLES

Table F-1: Intake Process Deliverables

Table F-2: Analysis Phase Deliverables

Table F-3: Design Phase Deliverables

Table F-4: Development Phase Deliverables

Table F-5: Implementation Phase Deliverables

Table F-6: Evaluation Phase Deliverables

Table G-1: Curriculum Review Deliverables

A. INTRODUCTION

These training standards are designed to provide the guidelines and tools necessary to ensure that training for our employees and stakeholders meets industry-established benchmarks.

These standards were founded upon a proven instructional systems design method that encompasses a training project’s entire lifecycle through the distinct, successive phases and associated activities of analysis, design, development, implementation, and evaluation

(ADDIE). Moreover, these standards have been designed to foster the principles of team spirit and collaboration through the active involvement of employees, partners, and other stakeholders during training development. Documenting processes and their outcomes promotes consistency within programs, across Training and Development (T&D), and throughout the Transportation Security Administration (TSA).

This handbook is structured using the core elements of ADDIE as a basic context for the chapters. This structure is not meant to restrict or limit creativity, or to prescribe that ADDIE is the only allowable methodology or process. Instead, the handbook provides standards or a framework for curriculum development to allow for the application of various industry proven models based on the specific requirements of the project. By following these standards, TSA will be able to satisfy the requirements for accreditation. The standards also support project succession and overall positive record-keeping principles.

Regards, Kim Hutchinson

Assistant Administrator

Training and Development

B. PURPOSE

This document outlines the agency standards for the analysis, design, development, implementation, evaluation, revision, and documentation of all TSA-developed training products. This handbook provides a comprehensive and flexible framework as opposed to a prescriptive set of procedures. In cases where an established process, procedure, or specific form has been approved already for use to achieve any of the standards described herein, there is no requirement to stop their use.

Adhering to the processes described in this document will provide for a solid foundation of standards and training excellence for the agency.

Standardization of course, instructor, and student materials leads to standardization of instruction delivered to students—no matter which session they attend.

Safety is enhanced, as training is based on actual job tasks with assessments of risk.

Further, safety and environmental hazards are taken into account to ensure the well-being of students, instructors, other TSA employees, and the public.

C. DEFINITIONS

1. Analysis, Design, Development, Implementation, and Evaluation Model: An instructional systems design model comprised of five phases: analysis, design, development, implementation, and evaluation.

2. Assessment Guide: A standardized framework for designing an effective Level II

(Learning) student assessment plan. The resulting plan should effectively measure the degree to which students achieved the objectives and acquired the intended knowledge, skills, abilities, and attitudes based on their participation in the training event.

3. Class: One instance or delivery of a course.

4. Course: A series of lessons that provides the knowledge and skills necessary to perform a job or activity.

5. Course Design Guide (CDG): Document created during the design phase that provides an overview of the training product, illustrating all product elements and appropriate facilitation methods while specifying training materials.

6. Course Sponsor: The management official or requestor who provides the justification for the development of any given course or training program, who gives feedback throughout the effort on behalf of the target audience, and who approves all curriculum deliverables.

Note: The course sponsor should be a branch manager or higher, approved by a division director to carry out the role.

7. Criticality: The importance of a task to job effectiveness.

8. Curriculum: A sequential group of related training courses or materials.

9. Curriculum Development Conference (CDC): A kick-off meeting to establish communication protocols and to determine phase-specific tasks and responsibilities for the development of a course or curriculum.

10. Curriculum Review Conference (CRC): A recurring meeting or series of meetings attended by the designated project lead, as well as sponsors, users, subject matter experts

(SMEs), instructional designers (IDs), and other stakeholders to discuss curriculum-related issues including program and course subject matter content, instructional methodology and technology, sequencing, student needs, and other issues as needed.

11. Curriculum Review Report (CRR): A detailed, documented account of the curriculum review process that contains information about the review, reviewers, findings, and recommendations.

12. Curriculum Risk Assessment: Process of evaluating the curriculum associated with a particular job task or training project for associated safety and environmental risks in order to prescribe the necessary safety equipment and procedures to mitigate risk.

13. Federal Law Enforcement Training Accreditation (FLETA): An independent accreditation process that provides law enforcement agencies with an opportunity to voluntarily demonstrate that they meet an established set of professional standards and receive appropriate recognition.

14. Federal Law Enforcement Training Accreditation Board: A federal organization that sets law enforcement training accreditation standards and establishes the process that accredits federal law enforcement training academies and programs.

15. Formative Evaluation: A process designed to collect data and information for improving a program, product, or instruction; this evaluation is conducted while the program is still being developed or delivered.

16. Instructional Designer: A professional with knowledge of learning theory and the principles, methods, practices, and techniques of one or more specialties of the instructional systems field, who designs and develops training material using the instructional systems design methodology.

Note: For ease of usability and understanding, this document employs the industry standard “instructional designer” to refer to both government Instructional System

Specialists and contractor Instructional Systems Designers employed in this capacity.

17. Instructor: An individual who provides knowledge or information to learners in a systematic manner by presenting information, directing structured learning experiences, and managing group discussions and activities.

18. Instructor-Led Training: Learning in which interaction between trainers and students occurs without a time delay and can be conducted in-person or from a distance/virtually.

19. Intake Consultant: The T&D analyst or other representative who manages the intake process.

20. Intake Meeting: A stakeholder discussion designed to identify program or training needs, address and clarify project risks, and collect additional data to create requirements.

21. Intake Process: A procedure initiated by a training request through the completion of an

Intake Requirements Document (IRD) that analyzes training needs to define project requirements and potentially triggers curriculum development and the identification of project personnel.

22. Intake Requirements Document: A record that captures all the requirements from the intake meeting.

23. Job Analysis Tool (JAT): An official document used to record core duties and responsibilities and competency requirements of a position, as well as title, series, category, and pay band. This document is not designed to capture all duties performed.

24. Job Task Analysis (JTA): A formal process for developing a list of tasks for a specific job or part of a job (function) in which the duties, tasks, knowledge, skills, and abilities are determined for a set of job responsibilities for a particular position of employment. The analysis is usually developed in consultation with incumbent employees, supervisors, and others familiar with the job. All aspects of the job—mental, physical, and attitudinal—are included in the analysis. The JTA provides reasonable assurance that tasks essential to a job are identified for training.

25. Kirkpatrick Evaluation Model: A training evaluation process developed by Donald

Kirkpatrick for determining the effectiveness of training. It consists of four levels: Level I

(Reaction), Level II (Learning), Level III (Behavior), and Level IV (Results).

26. Lesson: The smallest stand-alone segment of a course.

27. Level I Evaluation (Reaction): A measure of the degree to which students react to a learning activity.

28. Level II Evaluation (Learning): A measure of the degree to which students acquire the intended knowledge, skill, abilities, attitudes, confidence, and commitment based on their participation in a learning event or training event.

29. Level III Evaluation (Behavior): A measure of the degree to which students apply what they learned during training when they are back on the job.

30. Level IV Evaluation (Results): A measure of the outcomes that occurred because the students participated in the training program.

31. Online Learning Center (OLC): The official Learning Management System (LMS) for applicable training and related performance or development accomplishments for TSA employees. The OLC serves as the delivery platform for online training and is the official repository for TSA training records via its Learning History tool.

32. Project Lead: The T&D staff member assigned to oversee TSA-developed training projects with primary responsibility for a training curriculum or individual courses within a curriculum. Among other duties, the project lead manages the training development project schedule; coordinates with the Business Management Office (BMO), contract resources, and stakeholders; and ensures all deliverables meet TSA standards and requirements.

33. References: Real-world examples, documentation, and additional reference materials provided to help instructors ensure accurate and meaningful context for instruction and learning activities.

34. Request for Quotation (RFQ): A formal document detailing specifics for contractor support to develop and deploy the training or other intervention. The RFQ details the extent of contractor support and may be used by the contractor to determine expenses associated with the training request.

35. Rough Order of Magnitude (ROM): An estimate of project costs and development time provided in the early stages of the training project as its scope and requirements are being defined. This estimate is developed using industry standard labor rates, estimated timeframes for developing the course, and number of resources.

36. Sharable Content Object Reference Model (SCORM): A set of technical standards for web-based e-learning products which governs how online learning content and LMSs communicate with one another (i.e., registration functions, reporting of scores and course progress, and access to online player and course functionality). Using SCORM ensures interoperability between all e-learning content and LMSs.

37. Stakeholder: A person or organization (e.g., customer, sponsor, or performing organization) actively involved in or affected by a training project.

38. Standardized Job Description (SJD): An official document used to identify and record core duties, responsibilities, and competency requirements of a position. It is also used to document the assignment of title, series, job category, and pay band of positions located at TSA Headquarters, with the exception of L-Band and Transportation Security

Executive Service positions.

39. Subject Matter Expert: An individual with the special skill or knowledge representing mastery of a particular subject.

40. Summative Evaluation: An evaluation designed to obtain data regarding student reaction and student performance, assess the worth of the program or product, and make recommendations for improvements.

41. Technical Reviews: Formal reviews staged throughout a training development project, during which designated personnel review key deliverables to authorize the project to advance.

42. Test Item Matrix: A type of course-tracking tool specifying job tasks and competencies to be trained, objectives associated with job tasks and competencies, and assessment requirements.

43. Training: Instruction designed to teach an employee how to do a task or improve job performance. Training may be delivered in a classroom or laboratory, as a field exercise or simulation, on the job, at professional seminars and conferences, or virtually. Training also includes self-study courses, computer and web-based courses, and seminars.

Note: Shift briefings, information circulars, and general information-sharing events are not considered training for the purposes of this MD.

44. Training Program: The sum of learning activities identified and supported by an organization to achieve mission requirements.

45. Training Request: For purposes of this MD, the online single point of entry for all TSA offices to initiate training development projects using the T&D iShare site.

https://team.ishare.tsa.dhs.gov/sites/otwe4/intake/WebPages/RequestsByStatus.aspx

D. SCOPE

These standards are applicable to all TSA personnel and contractors who are involved in the development of TSA training materials.

E. AUTHORITIES AND REFERENCES

1. The Aviation and Transportation Security Act, Public Law 107-71

2. Government Performance and Results Act of 1993, Public Law 103-62

3. Executive Order 11348, Providing for the Further Training of Government

Employees

4. DHS Management Directive (MD) 3210, Training

F. STANDARDS FOR DEVELOPING TRAINING

Figure F-1 illustrates the five-phase ADDIE model, which consists of analysis, design, development, implementation, and evaluation. Within this model, review and revision are conducted during each phase. Other industry-approved models also may be used for developing training as long as the process is properly documented and archived.

Figure F-1: ADDIE Model

Each phase of ADDIE produces various outcomes and deliverables that feed into the next phase in the process. While each phase builds on the outputs of the preceding phase, the phases are not always sequential. For example, the development of a new training product may require a more linear process, whereas a revision of an existing product may be much less linear. However, a change in any phase will require a review and may drive adjustments to the preceding phases.

Management and evaluation responsibilities are critical during all phases of ADDIE.

Continuous formative evaluation from each phase feeds an iterative review and revision of products, along with approvals, serving to eliminate or reduce duplicative efforts. The following sections identify the standards for creating training products for each phase of

ADDIE.

F.1 ANALYSIS

The analysis phase—the foundation of an effective training program—determines specifically what a student must know and perform on the job through careful exploration of the training needs for the job, target audience, performance requirements, current performance levels, gaps between performance requirements, what is needed to close the gap, and occupational tasks to be trained. The T&D intake process initiates the analysis phase by establishing a baseline of mission needs, assessing existing capabilities, and determining mission risks and vulnerabilities.

F.1.1 Training Request

All training requests for T&D training services (e.g., new training, modifications to existing training, and maintenance of training materials) initiate with a T&D Training

Request, using the online form available through the T&D iShare site. The intake consultant reviews training requests to ensure they contain enough detail to develop training requirements.

F.1.2. IRD

All curriculum development efforts and projects initiate with an IRD. The intake consultant and course sponsor should schedule an intake meeting, generally within five days of receipt or submission of a training request, to produce an IRD. The following actions should be performed as part of the IRD:

Review of program-specific documents such as current agency policies, procedures, manuals, operational reports, and other related materials, as applicable, to help assemble the IRD;

Completion of the IRD, including:

o Intake identification number;

o Stakeholder information;

o Scope of project;

o Target audience analysis;

o Risks and constraints;

o Problem statement (gap in knowledge or performance, baseline metrics, success factors, and existing training);

https://team.ishare.tsa.dhs.gov/sites/otwe4/intake/WebPages/RequestsByStatus.aspx o Timelines and constraints;

o Evaluation strategy; and o ROM.

Circulation of the IRD to the attendees of the intake meeting, as well as the appropriate T&D Division Directors, for validation and approval;

Consultation with senior leadership to determine whether the development effort will be an internal or external effort, when applicable; and

Review and approval of the IRD.

Table F-1 describes the deliverables that result from the intake process.

Table F-1: Intake Process Deliverables

Intake Process Outcome Description

Training Request Types of training requests may include new training, training updates, or revisions among other items. A variety of reasons may prompt a training request, including (but not limited to) the needs listed here:

baseline training for new employees; new or changing technology; new or changed laws, policies, directives, or requirements; safety issues; threats, feedback concerning the content or delivery of particular training programs;

and operational concerns or reports of necessary remediation.

Intake Requirements Document The IRD collects information needed to direct efforts for the training request. This document outlines the stakeholders, purpose of the training, target audience, performance gap or deficiency the training is addressing, scope, training methodology, timelines and constraints, evaluation strategy, resources, and ROM.

The IRD serves as the formal record prepared for the sponsoring or requesting office to communicate: a) the requestor’s requirements; b) T&D’s evaluation of the training request based on a combined consideration of mission criticality, risk, liability, and visibility; and c) the document approvers.

The final output of an IRD is the decision to develop training materials or use another intervention. It also includes the level of support offered.

F.1.3 Project Initiation

Curriculum development activities often begin with a kick-off meeting to establish communication protocols and determine phase-specific tasks and responsibilities. The project lead is responsible for scheduling and facilitating the CDC or kick-off meeting, which could consist of a whole team of SMEs, IDs, graphic designers, or other roles. A kickoff meeting could be as simple as one SME and an ID deciding on roles and responsibilities among other topics. One of the products resulting from the CDC or kick-off meeting is the project plan.

F.1.4 Project Plan

The project plan includes (but is not limited to):

Scope of the effort;

List of internal project team members with roles and responsibilities and an internal team communication plan (e.g., documents emailed or posted on iShare);

Deliverables required and target completion estimates;

Project risks and mitigations;

Baseline metrics (e.g., number of customer complaints); and

Major milestones, which include:

o Specifications on how to work with the BMO to develop an RFQ, if contractor support is required; and o Documented approval.

F.1.5 Additional Analyses

After intake and project planning, it might be determined that additional analyses are needed to clarify further what students must know and perform on the job. Program-specific documents such as current policies, procedures and manuals, operational reports, and program-related materials are used to conduct essential analyses with the results documented. The following describes additional types of analyses.

F.1.5.1 Gap Analysis

If needed, a gap analysis should include (but is not limited to):

Current state of knowledge, skills, or practices;

Desired or ideal state of knowledge, skills, or practices; and

Lists of identified gaps or needs that exist and potential learning objectives to address such needs.

F.1.5.2 Job Task or Competency Analysis

A copy of the current SJD, JAT, or JTA documentation should be included in the project file.

Note: Training should not be developed on anything outside the current scope of the position. Personnel should contact Human Capital to obtain current SJD, JAT, or JTA documentation.

If a job task analysis does not exist or if the existing job task analysis is not current, a listing of duty areas, tasks, procedures and steps, or competencies is compiled. SMEs are used to determine the tasks, as well as applicable levels of competency to be trained, the grouping of job-related tasks, and special conditions under which tasks will be performed (e.g., low light while firing a weapon). They also ensure validation of the final task list. Tasks are defined in sufficient detail to derive training objectives.

F.1.5.3 Metrics

To determine if the training intervention is closing the performance gap, it is helpful to collect or develop metrics to measure the effectiveness of the training program.

For instructor-led training, metrics could include (but are not limited to):

Enhanced written test scores;

Improved outcomes of practical evaluations (PEs);

Increases in student satisfaction ratings;

Graduates’ abilities to perform tasks back on the job as reported by supervisors;

Graduates’ confidence in ability to perform tasks back on the job;

Change in performance ratings over time;

Employee engagement;

Employee turnover rates;

Percentage of promotions; and

Productivity rates over time.

For online learning courses, metrics could include (but are not limited to):

Time spent on each activity;

Number of attempts each learner requires to achieve a passing score;

Enhanced test scores; and

Increase in student satisfaction ratings.

F.1.6 Analysis Documentation

As mentioned earlier, results of the analysis process must be documented. After all analysis phase deliverables are approved, all signed master copies should be archived along with any pertinent correspondence (e.g., emails, meeting requests, rosters of attendance) in T&D’s official curriculum management system. These records will be used as reference material for future training development projects. The physical security, management, retention, release, and destruction of all training-related records must comply with TSA Records Management and Sensitive Security Information (SSI) standards.

Table F-2 describes the deliverables that result from the completion of the analysis phase of the ADDIE model.

Table F-2: Analysis Phase Deliverables

Analysis Outcome Description

Project Plan The document created at the onset of a training project to prepare a strategic blueprint for the effort

Additional Analyses

Gap Analysis Report The documented comparison of actual performance or state against desired performance or state

Listing of Tasks and

Competencies

A list of the roles and tasks, along with required knowledge, skills, abilities, and competencies, for a particular position identified for the training need— usually developed in consultation with employees, supervisors, SMEs, and others familiar with the job

Metrics Measures for how well the training solution closes the performance gap

F.2 DESIGN

After successful completion of the analysis phase, training projects proceed to the design phase. During design, information from analysis is used to map the course structure, identify potential course materials and strategies, determine possible assessment techniques, and organize the sequence of the training product.

F.2.1 Course Design Guide

The CDG is the primary deliverable of the design phase and will be supported by additional deliverables as needed to identify instructional strategies and resources in sufficient detail to estimate program costs. A CDG is written for instructor-led training, online learning, blended (a combination of instructor-led and online) delivery, or any other delivery option. It should include (but is not limited to):

Instructional strategies:

o Description of the:

business need or opportunity the training will address;

course;

target audience for the course;

instructional setting to ensure the appropriate space, equipment, and materials to support the program—for instructor-led training;

o Use of plain language (reading grade level);

o Special considerations for instructor-led training (e.g., whether the training will involve physical activity, associated safety and environmental risks, SSI concerns); and o Risk mitigation strategies for instructor-led training (e.g., prescribing the use of a medical clearance process, safety equipment, and procedures to mitigate risks for inherently dangerous training).

Course or program goal;

Outline of the course, consisting of learning objectives derived from job tasks sequenced to facilitate student progress from one level of skill and knowledge to another;

Descriptions of the course materials and learning strategies in sufficient detail to further estimate costs;

Descriptions of role players, demonstrations, case studies, small- or large-group discussions or activities, or other engagement activities; and

Course development schedule.

In addition to most of the above standards, if the course includes online content (e-learning), the CDG should follow the TSA OLC Content Developer Integration Guide and should include (but is not limited to):

A description of course design elements, as listed below:

o Graphical user interface (GUI);

o Fonts;

o Color scheme;

o Style guides;

o Technical specifications (e.g., LMS, SCORM Standard, Section 508 compliance—as appropriate for the intended audience);

o TSA-approved authoring technology;

o Whether the training may be viewed on a mobile platform;

o List of documents to be linked to, within the course; and o Additional notes or special requests.

Course-mapping elements (includes but are not limited to):

o Outline or visual representations for each screen of the online learning course, so that reviewers of the course can easily grasp the look and feel for each frame;

and o Logic flow (programming of an online course).

A description of engaging elements (e.g., quizzes, scenarios, and other aspects of the online experience to ensure that students are interactive and participatory);

Scripts and Shot Lists (for both instructor-led and online learning)—if video segments are used, a thorough script with an accompanying shot list is important to create during design, be approved, and occur well before production is to occur;

Sequential Learning Objectives—inform what will be taught, when the instruction will occur, and how students will be assessed;

https://office.ishare.tsa.dhs.gov/sites/OTWE3/OLC/Course%20Sponsor%20Information/Forms/AllItems.aspx?RootFolder=%2Fsites%2FOTWE3%2FOLC%2FCourse%20Sponsor%20Information%2FContent%20Developer%20Integration%20Guide&FolderCTID=0x012000EB4C97F52B4A424BAE1DBF12E8BC8430&View=%7B83B2E866%2D264F%2D49D2%2DBEE3%2D817B702F9E58%7D

Student Assessment—describes how students will be assessed. Options for student assessment include: formal examinations (e.g., verbal, written, or proctored tests);

practical evaluations (e.g., checklists specifying student task achievement criteria or rubrics explaining the varying degrees of successful student performance); and informal assessments (e.g., role plays, quizzes, knowledge checks, and projects).

Strategies should comply with the following (but not all-inclusive) standards:

o Students are evaluated on each learning objective—at the terminal level at a minimum.

o All written examination questions and practical evaluation criteria are referenced to one or more learning objectives.

o Multiple test versions are available and evaluate the same objectives when written examinations, such as those required for certification, are used.

o When different scenarios (or versions) are used for practical evaluations, the evaluated objectives and criteria demonstrating achievement of the performance objectives remain the same.

o All written examinations are developed in accordance with TSA’s Organizational

Assessment Service Item Writing Guidelines. These guidelines are located on the

T&D iShare site.

o The preferred method for formal examinations is delivery via the OLC. If using an alternative assessment strategy, the alternative strategy must be approved by the project authorities.

o The pass-fail, or “cut” score, for all formal examinations is set in a consistent manner with the purpose of the credential and standard of competence for the profession, occupation, role, or skill. Specifically, cut scores:

Should be set at the minimal acceptable standard of competence for each objective being tested in a particular exam or exercise.

Should consider both the criticality of the objectives being tested and the impact of poor performance (e.g., administrative tasks versus firearm or medical tasks).

Should be set at the minimum acceptable levels of competence for each function performed within the transportation security community

(international, Federal, State, and local).

All formal examinations should be evaluated in the following ways to validate the cut score:

Draft examinations are evaluated with a formative evaluation to compare test population scores with those estimated by SMEs.

Cut scores are evaluated during the pilot offering of the new course, during the first use of new or revised exams or exercises for existing courses, and as part of the comprehensive curriculum review process.

When designing tests or practical evaluations, test pools should meet the following criteria:

https://office.ishare.tsa.dhs.gov/sites/OTWE3/SIO/Shared%20Documents/DES_QRG_AIG_20161005_D1.pdf

All test questions are developed in accordance with TSA’s Organizational

Assessment Service Item Writing Guidelines, (located on the T&D iShare site).

All final questions and practical evaluations are approved by the close of the development phase.

All questions must be secured and maintained for the program.

Final questions will be approved, administered, secured, and maintained.

All test materials and associated documents containing actual test items and answers must be treated as sensitive testing material and protected according to the rules below, at a minimum:

The distribution of sensitive testing material must be limited to a “need-to-know” basis.

All sensitive testing material must be password-protected (non-SSI password) when shared.

Remediation and retesting strategies should be included as a component of the assessment design.

o In addition to the above standards, consult with Human Capital or Chief Counsel, as necessary, when designing assessments for courses with job retention impacts.

o A course tracking tool, such as a test item matrix, can serve as a helpful means for ensuring every objective in every lesson is tested whether by written examination or PE.

Course Evaluation—the CDG ensures that the course is properly evaluated at Levels

I through III of the Kirkpatrick Evaluation Model, and that the resulting data are incorporated in the comprehensive curriculum review process. Electronic data collection and submission are standard. If the data are not captured in the OLC, users should contact OTD-SI Evaluations@tsa.dhs.gov for data submission options.

o Level I (Reaction)—TSA uses a standard form to capture student feedback at the end of a course. If additional information is needed, it is noted in the plan.

o Level II (Learning)—this type measures achievement of objectives and should be conducted for all training, whether or not the course provides a certification or has associated job risks.

o Level III (Behavior)—this evaluation measures transfer of learning and behavior as a result of the course. TSA uses a set of standard questions across the Level III evaluations to compare courses across the enterprise and then tailors the remaining questions to measure application of learning on the job. This evaluation is performed three to six months after the course is completed.

Implementation—instructions should describe how the course will be piloted (if instructor-led training) or tested (if online learning).

Logistics—if the course is for instructor-led training, resource requirements ensure the needs are communicated early enough to allow sufficient time for the authorized https://office.ishare.tsa.dhs.gov/sites/OTWE3/SIO/Shared%20Documents/DES_QRG_AIG_20161005_D1.pdf https://office.ishare.tsa.dhs.gov/sites/OTWE3/SIO/Shared%20Documents/DES_QRG_AIG_20161005_D1.pdf mailto:OTD-SI%20Evaluations@tsa.dhs.gov

TSA training location to review requests and ensure the appropriate space, equipment, and materials are available to support the program:

o Recommended instructor-to-student ratio;

o Training space;

o Training materials;

o Equipment;

o Printing and shipping requirements;

o Training certificates; and o Special considerations (e.g., scheduling concerns or confirmation of guest speakers).

If the training is an online or blended learning course, there should be a description how to deploy the instruction. Resource requirements that need to be documented are the target audience (to whom the course will be assigned), how it will be assigned

(by whom, and frequency), and if the course is required or optional.

Communication—planning should include how to market the curriculum or class— to prepare a “sales pitch” including (but not limited to):

o Consideration of who should attend, the benefits of attending, any prerequisites, any expectations upon returning to the job, and what support is needed from supervisors;

o Identification of the target audience and key stakeholders with responsibilities pertaining to course implementation; and o Specification of the methods and timelines of communication with all applicable parties—who, what, when, where, why—including (but not limited to):

Target audience;

List of key stakeholders, their training project roles and positions, and contact information; and

Specified communication methods and frequencies.

F.2.2 Design Documentation

There must be documented approval of all design phase deliverables. After all design phase deliverables are approved, signed master copies should be archived along with any pertinent correspondence and records of contribution (e.g., meeting requests, rosters of attendance) in T&D’s official curriculum management system. Design phase records will be used as reference material for future training development projects and may serve as supporting evidence for audits or accreditation. The physical security, management, retention, release, and destruction of all training-related records must comply with TSA

Records Management and SSI standards.

Table F-3 identifies the deliverables that result from the design phase.

Table F-3: Design Phase Deliverables

Design Outcome Description

CDG The primary output of the design phase, providing an overview of the training product, appropriate facilitation or online delivery methods, and all training materials produced during the development phase

Components of the CDG

Instructional Strategies Techniques that will be used to help students learn or gain a better understanding of the course material

Online Design Elements (for online courses)

Visual, navigational, layout, technical, and other components for e-learning modules

Scripts and Shot Lists (for both instructor-led and online learning)

Details for what students or viewers will hear

Sequential Learning Objectives Terminal and enabling performance objectives written in three parts: condition (“givens”), behavior (action verb with an object), and standard (criterion)

Student Assessment Written tests, informal knowledge checks, or PEs

Note: Course tracking tools, such as a test item matrix, are helpful for ensuring every objective in every lesson is based on a real-world task and is assessed by written examination, PE, or some other check.

Course Evaluation Details of how the course will be evaluated for effectiveness using the Kirkpatrick Evaluation Model

(Levels I through III)

Implementation Instructions for how the course will be executed that highlight all major milestones and decisions that need to be made

Logistics Outlines of the specific request for course facilitation or deployment; specifically the logistical requirements including (but not limited to) instructor-to-student ratio, classroom size, equipment, printing needs, image requests, and OLC technical considerations http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel

Design Outcome Description

Communication Identification of who should attend, the benefits of attending, any prerequisites, expectations upon returning to the job, any support needed from supervisors, target audience, and key stakeholders with responsibilities pertaining to course implementation

Methods and timelines of communication with all applicable parties

F.3 DEVELOPMENT

During the development phase, the learning objectives, assessments, and strategies that were planned during the design phase are incorporated into the curriculum. Depending on the requirements, development may include activities such as constructing training materials, producing media, programming online training, and performing internal technical reviews to monitor quality of the resulting training project.

Whether training is to be delivered in the classroom or online, developers should consider following Gagne’s Nine Events of Instruction:

1. Gain attention

2. Inform learner of objectives

3. Stimulate recall of prior relevant knowledge

4. Present information

5. Provide guidance

6. Elicit performance

7. Provide feedback

8. Assess performance

9. Enhance retention and transfer

The goal of the development phase is to produce materials needed to conduct the intervention.

F.3.1 Course Administration Guide

The Course Administration Guide provides instructors with direction on the necessary tasks to successfully administer the course, including (but not limited to):

Course synopsis;

Description of the course in its entirety (day-by-day activities);

Instructional staff requirements (e.g., minimum number of facilitators required to effectively conduct the training);

Multimedia requirements and recommendations:

o All training supplies, learning resource materials, program-related equipment, and space requirements;

o Guidance on how to make students aware of training resources;

o Guidance on how to ensure students are properly trained to use learning resource materials and program-related equipment; and o Pre-course activities.

F.3.2 Instructor Guides or Lesson Plans (for instructor-led training)

Effective instructor guides provide direction on the learning and performance objectives, any corresponding skills and ideas, and the facilitation of student mastery and strategies for accomplishing these goals.

An instructor guide or lesson plan should be completed for each lesson that is part of the course, including a standard introductory lesson or student orientation that incorporates:

o Specific information on training location, including (but not limited to):

Fire emergency and safety rules and regulations;

Code of conduct rules and requirements; and

Disciplinary procedures.

o Course goals and objectives;

o Training schedules;

o Performance expectations; and o Practical evaluations or written examination requirements.

All lesson plans describe learning activities in sufficient detail to ensure consistent delivery of instruction and should include (but are not limited to):

o Syllabus, consisting of the following items:

Curriculum name;

Lesson name;

Lesson number;

Lesson time (hours):

Instructor preparation time

Presentation

Practice/activity

Lab

Assessment

Total hours

Description of course; and

Learning/performance objectives (knowledge/skills).

o Lesson overview:

Methodologies (e.g., lecture, role play, facilitated discussion);

Prerequisites;

Training aids and equipment;

Instructor to student ratio;

Required instructional staff;

Special instructor requirements or considerations (e.g., whether the training requires physical activity and associated safety risks, performance environment);

Risk mitigation strategies for lessons that involve inherently dangerous training;

Attachments (if any); and

References (if any).

o Directions for conducting activities or exercises (if any) o Detailed information to teach:

Bulleted or outlined information for the instructor;

Knowledge checks;

Visual or other aids to reinforce instruction; and

Appropriate level of complexity for the target audience.

o Documentation for instructors or other training personnel concerning role plays, demonstrations, case studies, small or large group discussions or activities, or other engagement activities in a classroom, lab, practical exercise, or other training environment:

Required skills (if needed), scripts, or any other relevant information that is clear and easily followed;

Any requirements for briefing and rehearsing; and

If the training center provides role players or resources during training delivery, a routing and scenario package with the scenario, script, and any other requirements.

o All associated course materials must not contain objectionable material and must comply with applicable licensing and copyright laws.

F.3.4 Student Guides and Other Course Materials

Student guides serve as documents in which students can take notes and use back on the job as a reference. Remember: if course materials contain any SSI content, they cannot be taken out of the classroom. Considerations should be made to limit print materials and to share items electronically whenever practicable.

Student guides and other course materials, designed to enhance the training experience, may include (but are not limited to):

Slides;

Handouts;

Training aids;

Practical exercise plans;

Quick reference guides;

Manuals and reference material;

Interactive multimedia instruction files;

Visual presentations; and

Other audio and visual media.

F.3.3 Authoring Online Courses

Any TSA-approved authoring tool can be used to program online or blended learning courses. Some training centers accept a sample GUI with a content outline as approved products before programming can begin. Others will build the course in PowerPoint and import into rapid design software. In any case, the output is an interactive media file.

F.3.5 Risk Assessment

Risk assessment evaluates whether a particular instructor-led course, lesson, or other training intervention will include inherently dangerous training, so appropriate mitigation strategies are incorporated into the course design and lesson plan.

Risk assessments should be conducted for all training projects.

Note: When conducting a risk assessment on an existing course, available information and resources such as test results or incident reports should be utilized as part of the analysis.

If the risk assessment reveals training to be inherently dangerous, these steps should be accomplished:

o Updating the CDG to include both the level of associated safety and environmental risks identified and any specific risk mitigation strategies (e.g., the use of safety equipment and procedures to mitigate those risks); and o Ensuring the appropriate safety measures are incorporated into applicable instructor guides, lesson plans, and materials during the development phase.

Please see Attachment A for the Course Risk Assessment Template that can be used for conducting risk assessments.

F.3.6 Updated Course Tracking Tools

After development of course materials is complete, course tracking tools, such as a test item matrix, should be reviewed to ensure all objectives are covered in the course content and are tested.

F.3.7 Development Phase Records Management

The Course Administration Guide, all lesson plans, all training guides, and all other associated learning materials should be checked prior to their use to verify approval via signature (hard copy or electronic). Use of a single, consolidating signature page is acceptable. After all development phase deliverables are approved, signed master copies should be archived along with any pertinent correspondence and records of contribution

(e.g., meeting requests, rosters of attendance) in T&D’s official curriculum management system. These records will be used as reference material for future training development projects and may serve as supporting evidence for audits or accreditation. The physical security, management, retention, release, and destruction of all training-related records must comply with TSA Records Management and SSI standards.

Table F-4 describes the deliverables produced during the development phase.

Table F-4: Development Phase Deliverables

Development Outcome Description

Course Administration Guide Specific guidance on the instructor requirements, training supplies and materials, pre-course activities, multimedia requirements, facilitation schedule, and checklists

Lesson Plans (for instructor-led training)

Descriptions of learning activities in sufficient detail to ensure consistent delivery of instruction; typically include instructor guidance and supporting media with specific direction for accomplishing the learning objectives

Student Guides and Other Course

Materials (for instructor-led training, as needed)

Enhancement for learning, presentation of course information, and reference on the job

Interactive Media Files Instructional content, images, video clips, animation, and audio for online or blended learning courses

Risk Assessment A determination of the physical, safety, and environmental risks of a task, course, or lesson that prescribes the necessary safety equipment or procedures to mitigate risk

Updated Course Tracking Tool Assurance of alignment of all objectives with real-world tasks and their assessment, either by written test or PE

F.4 IMPLEMENTATION

After training has been developed, it must be implemented using a controlled and monitored methodology that applies to lessons, blocks of instruction, and complete training programs. This controlled process ensures that the released training meets operational needs revealed through analysis and addressed in the resulting CDG. The implementation phase includes a pilot for instructor-led training or testing for online learning, as well as any resulting modifications. It culminates with the course entering into a consistent deployment or delivery schedule.

The goal of the implementation phase is to deliver the training program or blocks of instruction that teach content compliant with the lesson objectives. The standards addressed in this section apply to a general process for testing new training activities, regardless of their size. The majority of standards described in this section focus on observing lessons and considering their readiness for operational delivery. Lessons may be combined into blocks of instruction, and blocks of instruction may be combined to form training programs (sometimes referenced as a course).

F.4.1 Course Pilots

The course pilot tells the story of whether the training was successful and constitutes the final step in the validation process. The purpose of a pilot is to determine if the instructional system actually works under normal conditions; to obtain feedback from a sample of the target audience regarding the quality of the instructional system prior to implementation; to address any implementation, logistical, or operational problems such as equipment, facilities, and timing of activities; and to ensure that implementation is consistent with approved training materials.

Pilots may vary from a single block or module of instruction to an entire course. The instruction to be validated will depend largely on whether it is a new course or a revision to an existing course. Newly developed programs or blocks of instruction that have undergone major modifications must be piloted before formal implementation. A pilot must be representative of full implementation.

F.4.1.1 Instructor-Led Pilot

Standards to apply to course pilots for instructor-led training include (but are not limited to):

Piloting of training prior to formal implementation of any courses or blocks of instruction;

Completion of an orientation covering the curriculum, learning outcomes, method of delivery, testing procedures, and instructor rehearsals (train-the-trainer);

Observation of the delivery of a training pilot;

Participation in rehearsals;

Assurance that the pilot audience is representative of the target audience;

Capture of observation data, ideas, and impressions;

Completion of a pilot report; and

Updates to all applicable training deliverables and materials after the pilot has been satisfactorily conducted and evaluated.

F.4.1.2 e-Learning Testing

Standards to apply to testing online or blended learning courses include (but are not limited to):

Internal review for correction of errors;

SME review to ensure…