Attachment_27 _Instructional_Systems_Design_Manual.pdf

Training Delivery Services

Attachment 27

Instructional Systems Design Manual

CDP T&E MANUAL 016-4-00

Instructional Systems Design

APPROVAL DATE: 08/20/2014

DEPARTMENT OF HOMELAND SECURITY

FEDERAL EMERGENCY MANAGEMENT AGENCY

CENTER FOR DOMESTIC PREPAREDNESS

TRAINING AND EDUCATION DIRECTORATE

Denis Campeau Director Training and Education Directorate

Date: August 19, 2014

Bernice Zaidel Assistant Director, Curriculum

Development and Evaluation

Date: August 18, 2014

Charles D. Medley Assistant Director, Training Delivery

Date: August 19, 2014

Foreword

This Manual provides policy and guidance for the Center for Domestic Preparedness (CDP) instructional systems design process. The ISD process provides the CDP the necessary steps to develop quality materials that enhances the learning process and also provides robust evaluation data that aids in the improvement of the training materials and delivery methodologies. The ISD process is also flexible to meet the changing needs of the audience, changing priorities of the agency, and adaptable to new technologies and staff capabilities.

Table of Contents

CHAPTER 1: GENERAL INFORMATION

1-1. Purpose

1-2. Applicability and Scope

1-3. Supersession

1-4. Authorities/References

1-5. Policy

1-6. Definitions

1-7. Responsibilities

1-8. Questions

CHAPTER 2: Analysis Phase

2-1. Definition and Purpose

2-2. Conduct Needs Assessment

2-3. Conduct Front-End Analysis

2-4. Course Approval Submission to NTED

CHAPTER 3: Design Phase

3-1. Definition and Purpose

3-2. Design Steps

3-3. Course Catalog Submission to NTED

3-4. Finalize

CHAPTER 4: Development Phase

4-1. Definition and Purpose

4-2. Development Steps

CHAPTER 5: Implementation Phase

5-1. Definition and Purpose

5-2. Implementation Steps

CHAPTER 6: Evaluation Phase

6-1. Definition and Purpose

6-2. Levels of Evaluation

6-3. Information Collection

6-4. Improvement Issues

6-5. Course Review

Appendix A: Course Implementation Checklist

CHAPTER 1: GENERAL INFORMATION

1-1. Purpose

This manual provides policy and guidance for the procedures applied in the CDP instructional systems design process. Conformance with these procedures is essential to ensuring the CDP sustains instructional programs relevant and responsive to the needs of responders and receivers. Also following these procedures will ensure we provide quality courses that meet the IACET standards.

1-2. Applicability and Scope This manual applies to all CDP Training and Education (T&E) Directorate federal employees, and contractor personnel.

1-3. Supersession A. Directive Number: CDP-T&E-016-1, Revision Number: 00, Issued 9/14/2012

Instructional Systems Design Policy and Process B. CDP-T&E-04-008 Courseware Implementation SOP

1-4. Authorities/References A. CDP 540-1-00 Quality Directive B. CDP 540-2-00 Quality Management System C. CDP 540-3-00 Quality Assurance/Quality Control Program D. CDP 542-1-00 Internal Control Program E. CDP T&E Manual 016-1-00 Registrar/Admissions Manual F. CDP T&E Manual 016-2-00 Training Support Manual G. CDP T&E Manual 016-3-00 Training Approval Manual H. CDP T&E Manual 016-5-00 Course Evaluation Manual I. CDP T&E Manual 181-1-00 Training and Education Records Management

Manual J. CDP T&E Manual 530-1-00 Data Flow and CTAS Requirements K. CDP T&E Manual 563-1-00 EAS Records Manual L. Evaluating Training Programs: The Four Levels, by Donald L. Kirkpatrick M. Making Instructional Design Decisions by Barbara Seels and Zita Glasgow, N. ANSI/IACET Standard of Continuing Education & Training 2012 O. The Systematic Design of Instruction by Dick, Carey, and Carey 2005

1-5. Policy A. Overview

The purpose of instructional systems design (ISD) is to determine instructional needs and priorities, to develop effective and efficient solutions for achieving those needs, to implement those solutions in a competent manner, and to assess the degree to which the outcomes of the system meet the specified needs. The goal of instructional design within the CDP is to develop performance-based, criterion-referenced instruction that efficiently promotes transfer of learning from the instructional setting to the job or function.

B. CDP Educational Philosophy The CDP supports a constructivist educational philosophy most commonly considered experiential learning. Experiential learning is based upon providing participants with a meaningful experience followed by critical thought and reflection that when combined with previous learning and experience leads to a significant change in their knowledge and level of skill.

Since the CDP has a limited time to interact with responders and receivers, the goal of this meaningful experience and critical thinking is to rapidly assimilate new knowledge, skills, and behavior/ability to enable participants to undertake action aimed at affecting a positive change within their state, agency, tribe, or jurisdiction. This is most effectively accomplished when instruction is learner-centered and when concrete experiences, as realistic as well as practical, are the foundation for each instructional program. To this end, the primary goal of the CDP ISD process is to provide the tenets of experiential learning in all relevant programs.

C. The CDP ISD model follows the five phases of the ADDIE Model consisting of analysis, design, development, implementation, and evaluation. The critical elements of the deliberate process are outlined in each chapter by phase.

However, this model may be abbreviated, as necessary, to support critical mission requirements and to ensure timely training in support of responders and receivers.

1-6. Definitions A. Analysis – According to Seels and Glasgow, “Analysis is collecting and analyzing data to determine needs, tasks, and content, and instructional requirements.

B. AAR – After Action Review C. CD&E Branch – Curriculum Development and Evaluation Branch D. CEU – Continuing Education Unit E. COR – Contracting Officer’s Representative F. FEA – Front-end analysis is comprised of defining the instruction or training needs, describes the performance requirements to meet the training need and identifies possible alternatives for filling the need or gap.

G. ISD – Instructional Systems Design

H. IT – Information Technology I. Job analysis – Job analysis is the process of gathering, analyzing, and synthesizing descriptions of what people do in their jobs.” (Dick, Carey and Carey 2005)

J. LCMS – Learning Content Management System K. NTED – National Training and Education Division L. POI – Plan of Instruction M. T&E Directorate ─ Training and Education Directorate N. Task analysis – According to Dick and Carey, “Task analysis begins when the tasks chosen for further review are broken down in component elements, the relationships among elements are detailed, the tools and conditions involved in performing each element are described, and standards for successful performance are written.

1-7. Responsibilities A. The Director of Training and Education is responsible for determining policy in regards to CDP curricula.

B. The Assistant Director of the Curriculum Development and Evaluation Branch of the Training and Education Directorate is responsible for implementing the analysis, design, development, and evaluation policies, processes, and procedures identified in this manual.

C. The Assistant Director of the Training Delivery Branch is responsible for implementing the policies, processes, and procedures related to training delivery identified in this manual. Training Delivery is responsible for the implementation phase of the ADDIE Model.

1-8. Questions Questions concerning the policies and guidance detailed should be addressed to the Assistant Director of the Curriculum Development and Evaluation Branch, Training and Education Directorate.

CHAPTER 2: Analysis Phase 2-1. Definition and Purpose

The purpose of the analysis phase is to determine if a need exists for training and to define job or function performance requirements. The following procedures are used in conducting the front-end analysis.

2-2. Conduct Needs Assessment A needs assessment may be conducted to validate the relevance of a new course development to meet national preparedness needs of responders and receivers. A needs assessment will not typically be required prior to updating an existing course provided the evaluation data indicates continued relevance.

Steps for conducting a needs assessment are:

A. Determine Purpose of Assessment. Determine if the purpose for the analysis is a performance problem, new system or capability, or a mandated requirement. A needs assessment is not typically necessary for the development of new system or capability, or mandated training provided the target audience is apparent and job or function performance requirements are clearly defined.

B. Identify Sources of Information. Sources of information may include publications, books, records, subject matter experts, intended target population members, position description, laws and regulations, and standards documents.

C. Select Tools and Conduct Assessment. Based upon the identified sources of information, conduct research, statistical analysis, interviews, performance observations, focus groups, questionnaires, and/ or surveys to determine actual and desired performance requirements.

D. Analyze Findings. Analyze findings and use the results to make a decision regarding needed training or the requirement for further analysis. If a need for training or further analysis is not determined, conclude the analysis process.

E. Normally the needs assessment process includes research and investigations, interviews, questionnaires/surveys, and performance observations. Depending on the depth of the study and analyses, this process can take from five to 60 days. Much of the time will be accessing subject matter experts for interviews and focus/working groups and researching various documents such as reports and standards. On average, the needs assessment process for CDP will take about two to four weeks to include the creation of the needs assessment report.

2-3. Conduct Front-End Analysis A. Conduct job or function task analysis to include articulation of conditions, behavior, standard, performance steps, resources, and references. These tasks will provide the basis for the terminal and enabling learning objectives for instruction.

B. Validate job or function tasks through documentation, observation, or subject matter expert review.

C. Prioritize job or function tasks based upon difficulty, importance to job or function performance, and frequency of performance.

D. Conduct a learning analysis of each task by identifying required knowledge and skills to achieve each performance step. These knowledge and skills will provide the bases for the determination of enabling learning objectives for instruction.

E. The front-end analysis is conducted in some instances in conjunction with the needs assessment. If the training need is well identified and defined the FEA can start immediately with the needs assessment validation process. If a training need is not well defined, it is recommended the FEA is delayed until the normative need is determined. The FEA includes many aspects that can be conducted simultaneously:

1. Task/job/function analysis

2. Subject matter/content analysis

3. Instructional analysis

4. Learner/audience analysis

5. Cost analysis

F. The FEA should take about 30-60 days depending on availability of SMEs, access to resource such as standard documents, and the skill of the ISD professional.

2-4. Course Approval Submission to NTED A. Complete a NTED Needs Assessment Form by providing

1. Course Title

2. Purpose of Training

Describe the training you propose to develop. Identify the training goals that will be addressed. This information should come from your analysis of need.

3. Analysis of Need Describe the techniques used in your analysis to arrive at the need for the proposed training. You should consider using surveys, interviews, and focus groups.

4. Results of Analysis Describe the results of your analysis and explain what gaps were identified and why they point to a need for training in this area.

5. Target Audience Describe who you plan to train, and at what level – awareness, performance, management, etc. Please be specific about what discipline(s) and what level of training the audience requires.

6. Audience Analysis Describe the techniques you used and the results of your data collection and research efforts that point to the target audience described in # 5.

7. Prerequisites Describe the general skills and knowledge this audience needs to bring to this course.

8. Duplication of Effort Describe the type of literature search and course search you did to ensure that this type of training does not exist elsewhere.

9. National Scope Describe how this training has applicability across the nation. What type of capability or capabilities does this training address? How closely does the information you propose to teach coincide with the tasks from the core capability?

10. Additional Comments B. Submit to the appropriate Program Manager at NTED for approval.

CHAPTER 3: Design Phase 3-1. Definition and Purpose

This phase results in the determination of the instructional approach and the development of a POI and an assessment and course evaluation plan.

3-2. Design Steps A. Determine purpose, scope, and desired outcomes for instruction.

B. Develop a course outline or outlines that include modules, learning activities, exercises, administrative requirements, and significant or unique training facility and/or resource requirements.

C. Present recommended course or courses of action to the Assistant Directors of Curriculum Development and Evaluation and Training Delivery for concurrence or decision.

D. Develop learning objectives from the task and learning analysis.

E. Develop student assessment items for each terminal and enabling learning objective, as appropriate.

F. Develop detailed learning activity descriptions.

G. Develop an assessment plan that measures performance standards and defines graduation criteria.

H. Develop a course evaluation plan that supports collection of all data required to sustain the course relevant to the needs of the target audience.

I. Develop a detailed course schedule to ensure that proposed learning activity sequencing and resources provide for the efficient and effective implementation and execution of instruction.

J. The design phase generally can take five to 30 days to create the Plan of Instruction (POI). The industry standard for producing the POI is one hour of training to one hour of developing the POI – a 1:1 ratio. An example: a 40-hour course should take only 40 hours to create the POI.

3-3. Course Catalog Submission to NTED A. Complete a NTED Course Catalog Form by providing

1. Name (submitter)

2. Training Partner (organization)

3. Date

4. Status

5. NTED Course Number (this is assigned by NTED upon receipt of this form)

6. Course Title

7. Course Description

8. Course Objectives

9. Course Level (awareness, performance, or management)

10. Course Delivery Method (resident, non-resident/mobile, or indirect)

11. Course Length (in hours)

12. Course Length (in days)

13. Start and End times of class day

14. Training Certificate

15. CEUs (if applicable)

16. Course Point of Contact (POC)

17. E-mail address for course POC

18. Phone number for course POC

19. Registration Contact

20. Target Audience/Discipline

21. Mission Areas

22. Core Capabilities

23. Prerequisite(s)

24. Course Location

25. Room Requirements

26. Class Size (optimal)

27. Course Assessment

28. Cost

B. Save with the NTED course number as the file name, if known.

C. Submit to NTED Program Manager

3-4. Finalize Finalize the POI and conduct a comprehensive review with the course project team to include:

A. Training Delivery representative(s) B. Training Delivery contractor representatives C. Third-party subject matter experts, as appropriate

CHAPTER 4: Development Phase 4-1. Definition and Purpose

All instructional materials and assessment and evaluation instruments are constructed and initially reviewed and validated during this phase.

4-2. Development Steps A. Based on the POI, develop required instructional materials using the currently approved templates and writing and style guide. Instructional materials include but are not limited to:

1. Instructor guide

2. Participant manual

3. Supporting media

4. Examination(s)

5. Performance assessment(s)

6. Experiential learning guide

7. Job aid, or other performance support tool(s)

8. Training aid(s)

9. Computer or web-based training or blended training (if appropriate)

10. Simulations or scenarios

11. Other necessary or required items

B. Construct formative and summative evaluation instruments.

C. Ensure materials are 508 compliant D. CD&E Branch conducts a detailed review and validation of final draft course with the course project team, incorporates consensus improvement recommendations, and finalizes course for implementation or pilot testing.

The course project team typically includes:

1. Training Delivery representative(s)

2. Training Delivery contractor representatives

3. Third-party subject matter experts, as appropriate

4. CD&E Branch representatives who serve as leads and facilitates the development phase.

E. Development time for the instructional materials will vary greatly depending on the skill of the development project team. The industry standard for creating a new training/instructional materials package (instructor guide, participant guide etc. (see items listed in 4-2.A.) is 1 to 40 hours. For every one hour of instruction, it will take 40 hours of development. For example, a 40-hour course can take 1600 hours more or less depending on the capabilities of the team, availability of SMEs, and other priorities of the organization. The course revision ratio is 1 to 16. For every hour of instruction, it will take 16 hours of revision (major) work. For example, a 24 hour course can take 384 hours more or less depending on the type of revision work, capabilities of the team, SME availability, and need (priority) of the course.

CHAPTER 5: Implementation Phase 5-1. Definition and Purpose

This phase addresses the necessity to implement new or revised curriculum.

Implementation of new or revised curriculum into the training delivery schedule requires a systematic approach that encompasses all stakeholders. During this phase, the staff and faculty collaborate to prepare for and conduct pilot class(es) in order to optimize instruction. The number of pilots required is dependent upon whether the course is new or revised. For a revised course, the extent of the revision will determine the need for and the number of required pilot classes.

5-2. Implementation Steps A. The Chief, Training Management Section schedules a meeting to walk-through the new or revised program and to develop an implementation plan.

1. Factors to consider

a. Training Schedule

b. CTAS time line requirements

c. Hardcopy course material on-hand

d. Resource requirements

e. Facility requirements

f. Instructor skill and preparation requirements

g. Budget requirements

2. Attendees typically include:

a. CD&E Branch representative(s)

b. Training Delivery Branch representative(s)

c. Training Delivery contractor representative(s)

d. Training Support contractor representative(s)

e. Training Support Section representative(s)

f. COR, as appropriate

g. COBRATF representative(s), as appropriate

h. Other stakeholders, as appropriate

3. Implementation plan must be accompanied by the course POI and include a schedule for the following:

a. Course implementation and pilot class dates

b. Date for completion of master template in CTAS

c. Course added to Official Course Information List and posted to Current

Classroom Materials site on SharePoint

d. Non-Resident Training Materials Matrix update, if needed, and posted to Current Classroom Materials site on SharePoint

e. Posting all course materials to SharePoint Current Classroom site in the approved Adobe Reader format. (e.g., instructor guide, participant manual, evaluation products, handouts, etc.)

f. Posting required course materials to CTAS B. CD&E Branch, in accordance with the implementation plan:

1. Uploads validated course to Current Classroom Materials site on SharePoint and to CTAS.

2. Imports the final course materials to the Learning Content Management System (LCMS). Once uploaded, the materials will require validation of 508 compliance; metadata tagged, and ensure the files are imported into the templates correctly. For planning purposes a 40 hour course takes about 12 to 16 hours to import, 508 validation, learning objects metadata tagged, and checking the import into the templates. This is normally done after the materials have been pilot tested and finalized.

3. Updates the Non-Resident Training Materials Matrix to Current Classroom Materials site on SharePoint, if required.

4. Updates the Official Course Information List and posts to Current Classroom Materials site on SharePoint.

5. Notifies stakeholders (as listed in A.2) that all courseware is loaded on SharePoint and CTAS.

6. Provides Training Support contractor with appropriate courseware, if necessary.

7. Provides Training Delivery contractor with appropriate courseware, if necessary.

C. Training Support/Registrar’s Office updates the following areas and information:

1. Notify IT contractor to update website with correct information

2. Updates the Continuing Education Units list

3. Updates the CDP Course List

4. Updates the Course Programs list

5. Updates the Training Calendar

D. Pilots If the conduct of pilot classes is required, the CD&E and Training Delivery Branch and Training Delivery contractor representatives observe program performance during pilot classes and document course performance and associated recommendations. When multiple pilot classes are conducted, an interim, post-pilot AAR will be conducted following each pilot and a final AAR conducted after all pilots. Interim, post-pilot AARs are conducted to determine significant improvements or changes required between pilot classes and, based upon program performance, to affirm continuation of further pilots. The final AAR is used to determine consensus of changes to the course content and methodologies, resourcing needs, and execution improvements to achieve in order to optimize the program.

1. A post-pilot AAR includes an analysis of formative evaluation data and a consensus determination of course improvement recommendations based upon observations and course performance. Post-pilot AAR recommendations are presented to the Assistant Directors for Training Delivery and Curriculum Development and Evaluation for final approval, as appropriate.

2. Post-pilot AAR members typically include:

a. CD&E Branch representative(s)

b. Training Delivery Branch representative(s)

c. Training Delivery contractor representative(s)

d. Training Support contractor representative(s)

e. Training Support Section representative(s)

f. COR, as appropriate

g. COBRATF representative(s), as appropriate

h. Other stakeholders, as appropriate

CHAPTER 6: Evaluation Phase 6-1. Definition and Purpose

This phase consists of the collection and analysis of evaluation data, determination of course improvement issues, and the conduct of a course review board. See CDP T&E Manual 016-5-00 Course Evaluation, which details the evaluation processes.

6-2. Levels of Evaluation A. Process, consolidate, stratify, and analyze end-of-course evaluation data, typically referred to as level 1 evaluation.

B. Process, consolidate, stratify, and analyze learner performance data, typically referred to as level 2 assessment.

C. Administer and analyze post-graduate evaluation data, typically referred to as level 3 evaluation.

D. Administer and analyze post-graduate supervisor evaluation data, if appropriate and/or feasible.

6-3. Information Collection A. Collect anecdotal evidence such as testimonials and incident observations from graduates, civic leaders, the CDP website, and other sources and identify any trends that may support the need for course change.

B. Research existing and emerging performance requirements, drivers, lessons learned, and influences on the job/function that may support the need for course change.

6-4. Improvement Issues Identify valid course improvement issues based upon evaluation data, assessment data, observations, trends, and existing and emerging requirements, drivers, and influences.

6-5. Course Review Conduct a course review board in collaboration with relevant stakeholders to identify course content, course execution, resource, and logistical issues requiring resolution and determine recommended program adjustments and revisions to resolve these issues. Present course review board recommendations to the Assistant Directors for Training Delivery and Curriculum Development and Evaluation for approval, as appropriate. Every course should be recertified every three years; thus, the course review board should review courses every 18 to 24 months depending on when the course requires recertification.

Appendix A: Course Implementation Checklist

√ Information Needed

Catalog Course Number Full Name/Title of the course Acronym Version Number Contact Hours Course Description (for web site and CDP course list) Maximum Fill Optimum Fill Minimum Fill Target Audience Course Level Accreditations POI – Training Modules Instructor Qualification Requirements Classroom/Training Facility Requirements Toxic Agent Training Yes/No Respirator (determines medical requirements) Yes/No PPE Yes/No # of times for vitals Implementation Training Week/Date Printed Material or iPad Test Yes/No Performance Checklist Yes/No INFORMATION Updated

CEU List Course List Brochure Programs List Web Site Training Calendar

Appendix A: Course Implementation Checklist