Attachment 18 - CDP Training Audit and Evaluation Standards Guide.pdf

CDP Course Development & Delivery

Attachment 18

CDP Training, Audit, and Evaluation Standards Guide

CDP Training, Audit and Evaluation Program

12/20/2004

VERSION CESD112904V1.1

CDP WMD Training, Audit and Evaluation Program

CESD112904V1.1

CDP WMD TRAINING, AUDIT AND EVALUATION PROGRAM

POLICIES AND PROCEDURES

TABLE OF CONTENTS

Training Audit and Evaluation Program Frequency of Audits and Evaluations___________________________________________________ 5 CESD Evaluator Qualifications _______________________________________________________ 6

Instructor Evaluations Competencies and Standards for Instructors _____________________________________________ 7 Procedure for Instructor Evaluation ___________________________________________________ 8 Professional Foundations____________________________________________________________ 9 Planning and Preparation ___________________________________________________________ 9 Instructional Methods and Strategies___________________________________________________ 9 Assessment and Evaluation __________________________________________________________ 9 Management ______________________________________________________________________ 9

Instructor Presentation Audit Form Course Objective(s) _______________________________________________________________ 12 Communication Skills______________________________________________________________ 12 Facilitation/Trainer Skills __________________________________________________________ 13 Technical Skills___________________________________________________________________ 14 Safety __________________________________________________________________________ 14 Personal ________________________________________________________________________ 15 Learning Management Skills ________________________________________________________ 15

Instructor Qualifications CDP Certified____________________________________________________________________ 17 Professional Development __________________________________________________________ 17

Course Objectives Audit Procedure for Course Objective Audit _________________________________________________ 24

Training Environment Audit Procedure for Training Environment Audit _____________________________________________ 27

Quarterly Reports Procedure for Completion and Submission of the AAR ____________________________________ 33

Responder Testing Procedures for Responder Pre- and Post-Testing ________________________________________ 35 Procedures for Tests and Reports ____________________________________________________ 35

CDP/EAI End-of-Course Critique Procedure for Responder End-of-Course Critique _______________________________________ 39 Procedures for Reports ____________________________________________________________ 39 Government Deliverables per contract ________________________________________________ 39

OSLGCP Student Course Evaluations Procedure for Student Course Evaluations _____________________________________________ 48

Supervisor/Student Surveys Procedure for Follow-up Surveys ____________________________________________________ 58

Analysis of Information Purpose of Analysis _______________________________________________________________ 66

Evaluation Program Cycle

Evaluator Audit

Instructor Techniques Course Objective Audit Training Environment

Student Audit/Survey Testing OSLGCP Evaluation EAI Evaluation Supervisor/Student Evaluation

Analysis

Report

Analysis

Report

Instructor Professional

Development Mgr.

Course Manager DPM Training

CDPM

Solutions

Training Audit and Evaluation Program

The training audit and evaluation program evaluates the effectiveness of the training and learning activities, measures the ability of EAI Corporation to assist the CDP in meeting the training objectives, fulfills its training commitment, and identifies any areas of training requiring improvement. The program objectively and subjectively evaluates the instructor competencies, course objectives, student learning, student reaction, and training environment. The training audit and evaluation program is to be a positive and constructive experience through meaningful, objective reporting from the evaluator to both the instructor and the training department. The Curriculum, Evaluations and Standards Department (CESD) conduct all audits and evaluations and then provide copies and summaries to the appropriate departments/individuals.

Frequency of Audits and Evaluations Each course receives a full course audit by an evaluator annually. In the case of deficiencies, the CESD audits the course quarterly until deficiencies are eliminated.

To conduct a more thorough audit and evaluation, the following elements are included in this process:

Instructor evaluation -- conducted by members of the CESD and designed to evaluate an instructor’s training presentation and competencies in an objective manner.

Course objectives audit – conducted at the same time as the instructor evaluation to ensure that the course instructors are delivering all course and learning objectives.

Training environment audit -- conducted at the same time as the instructor evaluation to ensure the instructors have updated and correct training materials and the training environment is conducive to learning.

Responder testing -- measures the level of knowledge of the responder prior to the course and the level of knowledge after the course.

End-of-course critiques -- assesses at a more subjective level the students’ knowledge and reaction to the course through attitude surveys prior to leaving the training and before the return to work.

OSLGCP course evaluation -- assesses at a more subjective level the students’ knowledge and reaction to the course through attitude surveys prior to leaving the training and before the return to work.

Survey -- assesses at a more subjective level the students’ and supervisors’ knowledge and reaction to the course through attitude surveys after leaving the training and returning to work.

Document, literature and materials audit -- assesses the effectiveness of the training materials and captures any edits that need to be made to those materials.

Quarterly Reports – consolidates and summarizes evaluation information collected from training activities over the quarter. Following the audits and evaluations, summaries and reports are generated to provide the instructor, the course manager, the training department and CESD the analyzed data.

The following documents are generated through the audits and evaluations:

CESD Evaluator Qualifications The objective of the Training Audit and Evaluation Program is to produce evaluation data, through reports, to identify training program strengths and improve training materials, program delivery and student performance. Evaluators must meet career development education, training and experience requirements outlined below:

Experience: One year of auditing or program evaluation experience Education (Desired): Baccalaureate Degree Training: Organizational training and instructor certification

Typical duties include: Perform audits, gather data, develop tests, conduct analyses, evaluate results, draw conclusions concerning program results, and make recommendations to improve or meet program objectives.

**Evaluators will maintain instructor certification and be evaluated on an annual basis to maintain status as an evaluator.

Instructor Evaluations

On an annual basis, instructor evaluations are conducted to measure the performance of the instructor in a training setting and to evaluate the continued development of the instructor. An evaluator will measure the instructor competencies through observation and then indicates whether certain training standards are met. In addition to the classroom audit, the evaluator with the professional development manager, the human resources manager, and the quality assurance manager, reviews the instructor’s development folder to indicate whether the certifications, qualifications and development standards are met.

Competencies and Standards for Instructors The competencies selected for measurement during the observation portion of the instructor evaluations are the objective measures taken from the CDP’s Instructor Training Course and national standards established by International Board of Standards for Training Performance and Instruction (IBSTPI).

Positive Behavior Modeling Skills – The instructor demonstrates the ability to encourage responders to participate in class discussions and respond to the students and their observations in a non-threatening manner.

Communication Skills – The instructor listens to students, uses clear and methodical speech, is open to questions, accepts the background of others as valid experience, and uses proper speaking voice, body language and eye contact.

Ability to Summarize Information – The instructor responds to issues that come up in class and reflects upon them.

Learning Management Skills – The instructor uses allotted time well, prioritizes material and adapts to changes. He/she appears well prepared and utilizes visual aids correctly. The instructor demonstrates the ability to train under adverse conditions, whether caused by a student or technical difficulties. Instructors should also remain professional, not using offensive jokes, language or mannerisms.

Facilitation/Trainer Skills – The instructor demonstrates good conflict resolution skills, controls discussions and group dynamics, gives positive feedback and suggestions for improvement, and uses local examples applied to the technical subject and encourage students to debate points.

Application of Adult Learning Theory – The instructor understands how adults learn best, recognizes that adult students need to have all objectives defined and met.

Technical Skills – The instructor is technically credible and able to act as a resource, answering questions that may not be answered in course materials and able to demonstrate technology and subject matter expertise.

Because of the potential hazards with CDP WMD training, included in the instructor evaluation is the need to present the safety issues for each course applicable.

The instructor must meet the CDP certification standards that are in line with Office for State and Local Government Coordination and Preparedness (OSLGCP) as listed in the CDP Instructor Certification Policies and Procedures Handbook and in the National Domestic Preparedness Consortium (NDPC) Instructor Certification Program for SLGCP Training Partners. Instructors within the CDP and outside consultants are held responsible for continuing professional education in both the subject matter and in adult education. Ways to pursue professional development are:

Attendance at workshops and seminars in both adult education and the subject matter taught.

Membership in professional organizations.

Advanced degrees/College credits.

Presents at professional meetings.

Procedure for Instructor Evaluation The CESD is responsible for conducting instructor evaluations on an annual basis.

1. An evaluator schedules the various instructors and courses for observation, ensuring that, as many instructors for that course as possible will be evaluated during one scheduled rotation. CESD evaluator will schedule instructor evaluations with the course manager and the scheduling department.

2. The evaluator attends the course, observing the instructor’s techniques, the course content, and that the instructor standards and competencies are met using the Instructor Presentation Audit Form {See Page 10}.

3. Following the course observation, the evaluator with the Professional Development Manager reviews the instructor development files and determines whether the instructor meets the instructor qualifications/development standards.

4. Once the CESD evaluator has completed the instructor evaluation, he/she writes the Instructor Evaluation Report (IER).

The IER indicates what percentage of competencies was fulfilled and indicates if improvement is required and a quarterly evaluation is necessary.

The IER also indicates the level of instructor certification and any professional development activities completed in the past twelve months. The CESD provides this report to the instructor, the course manager, Human Resources, the Chief Deputy Program Manager, and the Deputy Program Manager. The CESD maintains all originals.

5. As needed, the CESD conducts follow-up meetings with the instructor, the course manager, the Chief Deputy Program Manager, and the Deputy Program Manager to provide an opportunity for feedback.

2003 Instructor Competencies From IBSTPI 2003 List of Instructor Competencies, www.ibstpi.org/instructor_competencies.htm

Professional Foundations

1. Communicate effectively.

2. Update and improve one’s professional knowledge and skills.

3. Comply with established ethical and legal standards.

4. Establish and maintain professional credibility.

Planning and Preparation

5. Plan instructional methods and materials

6. Prepare for instruction.

Instructional Methods and Strategies

7. Stimulate and sustain learner motivation and engagement.

8. Demonstrate effective presentation skills.

9. Demonstrate effective facilitation skills.

10. Demonstrate effective questioning skills.

11. Provide clarification and feedback.

12. Promote retention of knowledge and skills.

13. Promote transfer of knowledge and skills.

14. Use media and technology to enhance learning and performance.

Assessment and Evaluation

15. Assess learning and performance.

16. Evaluate instructional effectiveness.

Management

17. Manage an environment that fosters learning and performance.

18. Manage the instructional process through the appropriate use of technology.

From Building Professions Development Partnerships for Adult Educators Project Sponsored by U.S. Department of Education, Division of Adult Education and Literacy

1. Develops and maintains a knowledge base in adult learning and development

2. Develops and maintains an in-depth knowledge base in own content area and in other relevant areas

3. Know how to instruct and/or refer adults who have learning disabilities and other special needs

4. Know and is sensitive to demands and responsibilities of adults as workers, family members, citizens and community members

5. Knows how technological systems work and how to apply that knowledge to instructional and administrative functions

6. Knows about and/or knows how to access information about own organization, community resources and issues, relevant laws and regulations

7. Assesses own need for professional growth and develops and monitors own professional development plan

8. Engages in a variety of self-directed and collegial professional development activities and incorporates new skills and knowledge into learning environment to enhance the quality of instruction

9. Plans instruction that is consistent with the program’s mission and goals

10. Identifies and responds to learners’ individual and group needs, interests and goals when developing instructional plans

11. Creates a physical and interpersonal climate that is conducive to learning by drawing on adult learning theory, and knowledge of learners’ cultures, and interpersonal dynamics

12. Instructor applies knowledge of teacher-directed and learner-centered instruction

13. Models communication, negotiation, decision-making and problem-solving skills for learners

14. Employs individual, group and team learning

15. Sequences and paces lessons appropriately

16. Is sensitive to and accommodates diverse learning styles, abilities, cultures, and experiences, including learners who have disabilities and other special needs

17. Provides frequent and varied opportunities for students to apply their learning

18. Monitors and adjusts teaching strategies based upon student needs and performance

19. Effectively integrates current and appropriate media and technology as a tool for instruction

20. Integrates employment, family and community-related activities into instruction

21. Selects and uses a variety of resources for the learning environment (print, human and technological)

22. Acquires, accesses and uses technology for effective adult learning

23. Works with learners to identify their needs, strengths and goals, and advises or refers them to appropriate programs and levels of instruction

24. Uses results of assessment data (diagnostic and needs) on a regular basis to plan lessons, develop curricula, monitor progress towards objectives and goals and to verify learning

25. Monitors learning beyond simple recall of information using a variety of assessment strategies

26. Structures and facilitates ways for learners and peers to evaluate and give feedback on their learning and performance

27. Guides learners in the development and ongoing review of their educational plans

28. Collects and manages accurate data for program improvement and accountability

29. Suggests and/or collaborates in modifying the program organization and in developing program alternatives

30. Shares information with learners and colleagues about additional learning resources, educational opportunities, and options for accessing support services

31. Makes referrals to appropriate resources when guidance and counseling needs are beyond own expertise

Instructor Presentation Audit Form

Course Objective(s)

1. Objective(s) presented verbally or visually at beginning of course Instructor explained in detail the course objective(s) found in the course materials using verbal or visual methods at the start of the course

2. Objective(s) defined for the audience Instructor discussed meaning of course objectives to students to convey course intent and direct student learning

Examples:

Communicate effectively.

Display course objective(s) on Power Point Slide or Black Board.

Organize content effectively.

Provide handouts to students with course objective(s) for the entire course.

State a rationale for the judgment.

Verbally discuss course objective(s) with students.

Communication Skills

1. Uses clear and understandable speech Instructor used correct grammar, was articulate and was able to relay course materials and information to the student

2. Listens well Instructor understood and listened to students’ questions and concerns with course material and content. Can provide feedback to students’ questions

Communicates effectively by using appropriate verbal and nonverbal language, eye contact and level of voice.

Comply with established ethical and legal standards.

Determine when and how to respond.

Establish and maintain professional credibility.

Judge the adequacy of feedback and responses.

Use anecdotes, stories, analogies and humor effectively.

Facilitation/Trainer Skills

1. Demonstrated good conflict resolution skills Demonstrated ability to address student needs. Addressed their interests so that they are satisfied with the outcome and mediated discussion

2. Able to control discussions and group dynamics Instructor included students in discussion and controlled group discussion and debate

3. Gave positive feedback and suggestions for improvement Instructor and support staff provide students with feedback from questions about course content

4. Used examples and applied them to the technical subject Instructor used examples, demonstrations, and case studies to aid in student learning and retention of course material

5. Encouraged involvement in class participation Instructor encouraged classroom participation by asking questions and using students in demonstrations

6. Encouraged responders to debate points Instructor encouraged students' viewpoints on course material

Demonstrate acceptable social practices.

Determine if learners understand messages.

Determine when and how to respond.

Direct questions appropriately.

Identify learners with clarification and feedback needs.

Involve learners in establishing an appropriate level of learner comfort.

Involve students in class demonstrations.

Judge the effectiveness of the communication.

Manage group interactions and participation.

Plan and deliberately use feedback and reinforcement during instruction.

Provide opportunities for learner success.

Provide opportunity and adequate time for students to state questions, comments and concerns and respond to questions.

Provide prompt, timely and specific feedback.

Resolve learner behavior problems.

Use appropriate question types and levels.

Use frames of reference familiar to the learners.

Technical Skills

1. Technically credible Instructor has skills from education, training and certification programs that demonstrate technical competence in a subject matter and delivery

2. Able to answer questions not included in course material Instructor can answer questions directly related to the course material

3. Explains difficult concepts, ideas, or theories Instructor explains concepts and theories in a format, which aids students in learning and retention of course material

Demonstrate content expertise.

Promote transfer of knowledge and skills.

Recommend revisions and changes to existing materials and provide suggestions for new programs and activities.

Stimulate and sustain learner motivation and engagement.

Update and improve one’s professional knowledge and skills.

Anecdotes, stories, analogies and humor are free of bias (sexual, racial, religious, cultural, age, etc.).

Use gestures, silence, movement, posture, space and props effectively.

Safety

1. Training hazards identified and presented to responders An instructor or safety personnel informs students of the hazards associated with the course

2. Safety briefing presented and reinforced Instructor or safety personnel gives safety briefing and enforces the safety requirements

Explain the safety requirements for the course.

Inform students of possible hazards.

Insure that all safety requirements are being adhered too.

Manage an environment that fosters learning and performance.

Personal

1. Professional Appearance Instructor dresses to EAI dress code standards

2. Professional Demeanor Instructor conducts him/her self in a professional manner. Instructor is polite, helpful, informative and technically credible in front of students.

Wear EAI shirt and casual dress pants.

Prepare for instruction.

Establish and maintain professional credibility.

Learning Management Skills

1. Managed time well Maintains a consistent instructional flow

2. Prioritized material and adapted to changes Instructor prioritizes pertinent information in the material and when necessary adapts to student's learning needs

3. Appeared well-prepared Instructor is alert and has all course material and visual aids available when necessary

4. Trained during and under adverse conditions Instructor is flexible when training under conditions that are not ideal for training.

Example: Hallway is noisy, instructor closes classroom door

5. Utilized visual aids Instructor uses handouts, PowerPoint, or other visual aids to emphasize course objectives

Assure proper disposition of equipment, materials and furniture.

Confirm the physical arrangement of the instructional site, materials, equipment and furniture.

Demonstrate effective questioning, facilitation, and presentation skills.

Demonstrate flexibility in response to learner needs and interests.

Employ instructional techniques appropriate to methods and instructional situations prescribed.

Judge the effectiveness of the use of media.

Make minor adjustments to learning materials to promote retention of knowledge and skills.

Match learning outcomes to learner and organizational needs and goals.

Monitor student progress during instruction.

Organize content effectively.

Select initial presentation strategies.

Substitute for, add to, switch or create media as required.

Use media and technology to enhance learning and performance.

Instructor Qualifications

CDP Certified Documentation of education/degree and training required.

1. Education/Degree & School in comment column All faculty members must possess teaching certification or professional license from a qualified institution, i.e. college or university, United States military service school faculty member, a national training institution (FBI, ATF, DEA, NFPA, DOD, etc.), state or municipal training academy, or other institution recognized for its faculty member certification program.

2. Prior experience in ODP recognized fields/No. years & where

All faculty members shall have at least ten (10) years of experience in their specific professional field and background/working knowledge of weapons of mass destruction. Working knowledge includes related experience with a WMD incident and/or with an incident classified as a terrorist act.

Faculty should be presently employed or retired from related professions.

Prior occupations may include firefighters, WMD specialists, emergency medical professionals, law enforcement, hazardous material technicians, and emergency management personnel.

3. ITC course completed/When & where

Teaching certification, post-secondary teaching experience, or teaching degree required

4. Prior training experience/Where & number of years

5. Prior training and educational certifications/Where & when

Professional Development Documentation required.

1. Academic or training courses attended/completed past 12 months

2. Seminars attended past 12 months

3. Active membership in professional organizations

Instructor Evaluation Summary Report Instructor Name

Course Module(s)

Dates of Observation Evaluator

I. Executive Summary

The Instructor Evaluation measures the competencies met by the instructor(s) in each course. The evaluator follows an entire course, measuring each instructor in each module, checking yes or no for each competency listed. This type of evaluation is objective, with no ranking for the instructor. This evaluation is designed for use in any course to indicate areas in which an instructor may excel or may need additional training. Please describe the results of the Instructor Evaluation and any conclusions or recommendations to be made to the course manager, the instructor and the Director of Training.

II. Process/Procedures

The Instructor Evaluation was developed by the Curriculum, Evaluations and Standards Department to measure different competencies listed in the Instructor Training Course and in the IBITSPI standard. The competencies are those that can be measured as met or not met and are not subjectively assigned by the evaluator. The Instructor Evaluation is completed on each instructor annually and may incorporate two separate course sessions to properly evaluate the instructor.

The Instructor Evaluation incorporates the Course Objective Audit and the Training Environment Audit, which provides a more rounded evaluation of the training provided.

III. Data and Data Analysis

A. Course Objectives Comment on the results of this section – did the instructor meet the objective(s) of the module(s) taught; were the objectives defined; did the presentation address the objectives. Please comment on any exemplary or unsatisfactory observations.

B. Facilitation/Trainer Skills

Comment on the results of this section regarding the facilitation skills of the instructor. If the instructor had an opportunity to resolve conflict, point out how and whether he/she was successful. Give examples of how instructor gives feedback and encourages participation. Please comment on any exemplary or unsatisfactory observations.

C. Communication Skills

Comment on the results of this section regarding the speech patterns and listening skills, i.e., did the instructor pay attention to the question and prepare answers for that question. Give specific observations, i.e. speech impediments, fast or slow speech, misunderstanding questions. Please comment on any exemplary or unsatisfactory observations.

D. Technical Skills

Comment on the results of this section regarding the credibility of the instructor, including ability to answer questions not specific to text and ability to explain difficult concepts or ideas. Give specific observations. Please comment on any exemplary or unsatisfactory observations.

E. Learning Management Skills

Comment on the results of this section regarding the ability of the instructor to manage the classroom and the course material. Give specific observations, especially regarding time management, adaptation to changes, and appearance of being well prepared. Please comment on any exemplary or unsatisfactory observations.

F. Safety

Comment on the results of this section regarding the safety issues of CDP training. Please comment on any exemplary or unsatisfactory observations.

G. Personal

Comment on the results of this section regarding professional appearance and demeanor. Please comment on any exemplary or unsatisfactory observations.

H. Additional Observations

Comment on any additional observations, positive feedback and constructive criticism. All observations are important.

I. Instructor Qualifications/Certifications – (please include information under the following and indicate if there is a deficiency in any category)

a. Post-Secondary Education – Include Degree and School

b. Prior Experience in OSLGCP recognized field – Include No. of years and where

c. ITC course completed – Include when and where

d. Prior training experience – Include where and number of years

e. Prior training and educational certifications – Include where and when

J. Professional Development – (please include information under the following and indicate if there is a deficiency in any category)

a. Additional college courses attended/completed in past 12 months

b. Seminars attended in past 12 months

c. Active membership in professional organizations

IV. Recommendations Please list the itemized recommendations for instructor improvements.

In this section, the evaluator needs to itemize the improvement recommendations for instructors based on course objectives, facilitation/trainer skills, communication skills, technical skills, learning management skills, safety, and personal. These recommendations are forwarded to the Chief Deputy Program Manager for review and implementation.

E. Learning Management Skills

G. Personal

Instructor Evaluation Summary Report Instructor: Joe Smith

Technical Emergency Response Training Terrorist Threat Update July 19 and 26, 2004 Evaluator: Jane Doe

Overall, Mr. Smith provided the students with a well-delivered training experience. He met almost all of the competencies listed, with just a few that can be developed. Mr.

Smith’s primary weakness is controlling the class discussion. He is however, knowledgeable in the subject, having obviously done additional research and including outside material and examples in his presentation. In addition to previous experience and education, Mr. Smith has continued his professional development by completing a college course and attending a seminar, as well as maintaining membership in the Fraternal Order of Police.

The instructor evaluation for Mr. Joe Smith was conducted during the July 19 and 26, 2004, TERT course as part of an annual audit of the instructors and course objectives for that course.

Mr. Smith presented all of the objectives for this module on both observations at the beginning of the module. The objectives were presented on a slide and then were mentioned verbally. Mr. Smith did not address completely learning objectives 3 and 5.

There were no obvious reasons for not doing so, and time was not a factor. The objectives were not restated at the conclusion of the module.

Mr. Smith seemed capable of controlling the discussion most of the time. He did allow the class to take over the discussion once on the first observation, during which time they strayed from the subject matter somewhat. Mr. Smith did use examples from current events in discussing the different terrorist threats to the United States and to other countries. Mr. Smith was successful in getting the class to participate in the discussion by asking questions and engaging different students in slight debate. Mr.

Smith’s only weakness is maintaining total control of the class, as shown during the class discussion on protective equipment, and keeping the students on track.

While Mr. Smith spoke at an acceptable tone and speed, he did use some word choices that did not fit the topics causing some confusion. He was adept at listening to the students and responding correctly, giving the indication that he was paying attention to the students’ needs and responses.

D. Technical Skills Mr. Smith appeared to have knowledge of this module, whether through experience or research. He was able to respond to several questions that were outside of course materials, including some regarding more difficult concepts, such as….

Mr. Smith did manage his time well with the exception of leaving out information within the two learning objectives. He definitely appeared well-prepared and utilized the visual aids well. Mr. Smith needs to work on adapting to changes, as indicated by the class veering off discussion.

During this module, there were no safety issues to be addressed.

G. Personal

Mr. Smith maintained a professional demeanor at all times; however, his appearance was less professional, wearing his company shirt with jeans and sneakers.

H. Additional Observations Mr. Smith did at times appear monotone, losing some of the interest of the class. He may consider bringing some lightness or humor (clean) into what can be a somewhat less interesting and dark subject.

I. Instructor Qualifications/Certifications – (please include information under the following and indicate if there is a deficiency in any category)

a. Post-Secondary Education – Include Degree and School

-- B.S. in Criminal Justice, Jacksonville State University (1984)

b. Prior Experience in ODP recognized field – Include No. of years and where

-- 15 years with Alabama State Police

c. ITC course completed – Include when and where

-- Northeast Alabama Police Academy (1990)

d. Prior training experience – Include where and number of years

-- Northeast Alabama Police Academy (9 years)

e. Prior training and educational certifications – Include where and when

-- Alabama POST (1990-2000) -- Jacksonville State University (2001) -- Northeast Alabama Police Academy (1990-2000)

J. Professional Development – (please include information under the following and indicate if there is a deficiency in any category)

a. Additional college courses attended/completed in past 12 months -- JSU Adult Education, ED 503, 3 hours

b. Seminars attended in past 12 months -- Terrorism at Home, FBI, Montgomery, AL, 2 day seminar, Jan 2004

c. Active membership in professional organizations -- Fraternal Order of Police

IV. Recommendations

A. Course Objectives Mr. Smith will receive an instructor evaluation and course objective audit within two weeks to insure that all learning objectives are meet.

Assist Mr. Smith’s with his only weakness of maintaining total control of the class and keeping the students on track.

Assist Mr. Smith with the choice of words used that caused confusion.

Mr. Smith appeared to have knowledge of this module, whether through experience or research and requires no additional improvements.

Mr. Smith needs to work on adapting to changes, as indicated by the class veering off discussion.

During this module, there were no safety issues to be addressed.

G. Personal

Mr. Smith needs to focus on his appearance, by wearing business casual attire during training sessions.

Course Objectives Audit

The Course Objectives Audit (COA) is conducted annually and/or 90-days after the completion of the course’s annual curriculum review by the Curriculum, Standards and Evaluation Department. The purpose of this audit is to evaluate whether the course manager and course instructors present the learning objectives listed in the Program of instruction (POI) for the course.

Procedure for Course Objective Audit The COA uses the POI for each course, listing all modules, terminal learning objectives, and enabling learning objectives. The audit is conducted using the Course Objective Audit Form for the given course, which is simultaneously completed along with the Instructor Evaluation and Training Environment Audits.

The COA is a simple yes/no checklist form {See Page 18}. Upon completing the audit, the evaluator will itemize any objectives not meet by module and discuss his/her findings with the course manager and instructor. If there are mitigating circumstances for not covering the objective(s) they will be noted on the audit form.

Upon completion of the audit, the CESD evaluator reports his/her findings using the Course Objective Audit Report format.

Course Objective Audit Report is a detailed summary of the findings of the audit, which is an attachment to the report. The report details the degree of presentation of the courses terminal and enabling learning objectives, issues concerning the learning objectives from the instructional staff, and recommendations concerning measure to be taken to overcome the shortfalls. A copy of this audit report is provided quarterly to the instructor(s), course manager, Chief Deputy Program Manager, Deputy Program Manager, and Assistant Manager for Curriculum. CESD maintains all originals.

Training Objectives Audit Link

Course Objective Audit Summary Report Course

Course Manager Dates of Observation

I. Executive Summary:

The Course Objective Audit Report (COAR) measures whether the course manager and the course instructors cover the course learning objectives listed in the Program of Instruction (POI). If certain objectives are not included during the instruction period, the COAR indicates the reason why the instructor omitted the objective and the material supporting it. The COAR is performed at the same time as the Instructor Evaluation and the Training Environment Audit. It is a simple yes/no checklist of all modules, terminal learning objectives and learning objectives for the course. Please describe the results of the COAR and any conclusions or recommendations to be made to the course manager, the instructor and curriculum.

The COAR was developed using the POI for each course, listing all modules, terminal learning objectives and enabling learning objectives. An evaluator from the Curriculum, Evaluations and Standards Department completes it during an annual evaluation or following any course revision.

Please list the learning objectives not met and reasons why.

In this section, the evaluator needs to itemize any objectives not met, identifying by module, terminal learning objectives and enabling learning objective. The evaluator should also identify the reasons given by the instructor for not reaching that objective.

IV. Recommendations

Please list the recommendations for improvement in course objective deficiencies.

In this section, the evaluator needs to itemize the improvement recommendations for course terminal learning objectives and enabling learning objectives. These recommendations are forwarded to the Chief Deputy Program Manager for review and implementation.

Course Objective Audit Summary Report Technical Emergency Response Training

Course Manager: John Jones

During the Technical Emergency Response Training course offered July 19 through 26, 2004, most of the learning objectives were met. However, two were omitted – one for time constraints and one because of material included in the course. Whenever objectives are not met due to time constraints, the course manager needs to work with the instructor to make better use of the time. For objectives not being met because of a disagreement on content, the instructor, course manager and writer need to work together with a subject matter expert. In this case, Mr. Doe needs to bring his research to Ms. Davis, the writer. The meeting is set for August 16 at 9:00 a.m.

The COAR was developed using the POI for the Technical Emergency Response Training course, listing all modules, terminal learning objectives and enabling learning objectives. It was completed by Ms. Doe, an evaluator from the Curriculum, Evaluations and Standards Department, during an annual evaluation on July 19 through 26, 2004.

Please list the learning objectives not met and reasons why.

A. Module: Terrorist Threat Update, Learning Objective 5 – Identify and discuss potential terrorist and tactics – this learning objective was left out according to Joe Smith, the instructor for the module, because of time constraints.

B. Module: Radiological Materials, Learning Objective 3 – Identify and recognize various delivery methods of “weaponized,” radiological/nuclear material – this learning objective was left out according to Bob Doe, instructor for this module, because he disagreed with the content of the section in the text.

Please list the recommendations for improvement in course objective deficiencies.

A. Module: Terrorist Threat Update, Learning Objective 5 – Identify and discuss potential terrorist and tactics – this learning objective must be implemented in future classes.

B. Module: Radiological Materials, Learning Objective 3 – Identify and recognize various delivery methods of “weaponized,” radiological/nuclear material – this learning objective should be discussed with Bob Doe, instructor for this module, to modify or eliminate the contents in this section.

Training Environment Audit Training Environment Audits are conducted at the same time as the annual Instructor Evaluations and should be conducted any time a course revision occurs. The audit ensures that all visual aids are correct, clear and standardized, that the equipment is working properly, that course materials, handouts and training aids are available and in sufficient quantity, and that training support personnel are available and utilized properly.

Procedure for Training Environment Audit The CESD is responsible for conducting training environment audits on an annual basis.

1. An evaluator schedules, through course manager and scheduling department, the various training environment audit at the same time as the instructor evaluations for a course.

2. The evaluator attends the course, observing the training environment, in particular the equipment used for training, the course support materials and the training support personnel.

3. Following the course observation, the evaluator writes a Training Environment Audit Report.

a. Training Environment Audit Report indicates what percentage of competencies was fulfilled and indicates if improvements are required and a quarterly evaluation is necessary. The evaluator provides this report quarterly to the course manager, the Chief Deputy Program Manager, the Deputy Program Manager, and Production. The CESD maintains the original in its evaluation files.

4. As needed, the CESD conducts follow-up meetings with the course manager to provide an opportunity for feedback.

Training Environment Audit Form

Training Environment Summary Report Course

Course Manager Date(s) of Observation

The Training Environment Audit is completed at the same time as the Instructor Evaluation. The audit measures whether the training support in the classroom is available, current and correct. The audit is designed to indicate if the instructors have available to them the necessary tools to increase the students’ ability to learn.

Please describe the results of the Training Environment Audit and any conclusions or recommendations to be made to the course manager, the instructor, production and Curriculum.

The Training Environment Audit was developed by the Curriculum, Evaluations and Standards Department to measure the availability, currency and accuracy of the support materials available to the instructors in the classroom environment. The Training Environment Audit is completed for each course annually and is completed at the same time as the Instructor Evaluation and the Course Objective Audit.

A. Audio Visuals Comment on the results of this section – are the graphics clear and text correctly spelled; are graphics standardized; is sound clear and with appropriate volume;

are all pieces of equipment operable. If there are problems with graphics or equipment, be specific.

B. Course Materials

Comment on the results of this section regarding the availability and quantity of course materials (texts), handouts and other training aids. Be specific about overages or shortages.

C. Training Support Personnel

Comment on the results of this section regarding the availability and use of training personnel in the training environment. Give specific observations.

D. Additional Comments

Comment on any additional observations, positive feedback and constructive criticism. All observations are important.

Please list the recommendations for improvement of training environment deficiencies.

In this section, the evaluator needs to itemize the improvement recommendations for the above stated training environment deficiencies. These recommendations are forwarded to the Chief Deputy Program Manager for review and implementation.

A. Audio Visuals

Training Environment Summary Report Technical Emergency Response Training

John Jones

The two biggest observations in this particular course were the need for the correct number of course materials provided and the appropriate slides used. The recommendation for this course is for production, the course manager and student services to work more closely together to ensure the correct number of students and materials needed. The course manager, instructors and curriculum must work together to create the correct slides to be used in instruction.

The Training Environment Audit is completed for each course annually; this audit was completed at the same time as the Instructor Evaluation and the Course Objective Audit on July 19 – 24, 2004.

A. Audio Visuals The slide graphics are standardized throughout the presentation with one exception of one that may have been added by the course manager or instructor, titled, “Waste Materials Hazards.” The word “terrorist” is misspelled on the slide titled, “Future Terrorist Threats.” All equipment was operable. The audio volume tended to be low. The problem seemed to be on the videotape.

Instructors had to request additional texts and handouts from production due to a shortage of approximately ten.

Two additional support personnel were available during lectures, but were not used often. A minimum of three instructors were present who were not being utilized at the time.

D. Additional Comments

Curriculum needs to correct the one slide, and the course manager needs to work with curriculum on the added slide. The additional staff in the classroom was somewhat distracting. If there is not a suitable use for the personnel and instructors, then they should not be in the class.

IV. Recommendations Please list the recommendations for improvement of training environment deficiencies.

A. Audio Visuals

The slide titled, “Waste Materials Hazards”, needs to be standardized with the presentation. The spelling of the word “terrorist” should be corrected on the slide titled, “Future Terrorist Threats.” The audio volume should be evaluated on the videotape and possibly re-recorded to increase the low audio.

Instructors should verify the correct amount of texts and handouts are available before class begins.

Additional support personnel available during lectures must be utilized or removed from the classroom during lectures.

Quarterly Reports

The quarterly reports will summarize data and provide recommendations for action steps collected from the following evaluations, surveys and tests:

• Pre- and Post- Responder Test information,

• CDP/EAI End-of-Course Critique

• OSLGCP Student Evaluations

• Follow-up Surveys

• After-Action Report

Recommendations made in the reports that require action steps will be presented to the CDPM and PM for further action. Sample reports from testing, evaluations and surveys are contained in Responder Testing, CDP/EAI End-of-Course Critique, OSLGCP Evaluations, and Follow-Up Survey sections of this manual. Quarterly reports are submitted to the CDPM and PM and the end of each quarter.

The After-Action Report (AAR) is the principal post-exercise document. The course manager with input from the instructors completes AARs after each course taught. It provides a historical record of findings and forms the basis for refinements to policies, procedures, training, equipment, and overall performance of EAI Corp. within the CDP.

The AAR provides the CESD, the Program Manager and the CDP Director with information about the course that may or may not affect the training environment.

Following the course, the course manager discusses with the instructors any course material and training aid issues and lessons learned during the course.

Contractual Statement (3.8.1) - Individual Course After Action Reports (AAR) – After action reports shall be prepared for each course presented. The AAR shall include recommendations for instructional material revisions, change in class schedules, modification of training aids and responder materials, lessons learned, safety and medical issues or incidents, responder comments, responder critique sheets and analysis of these sheets. Course AARs shall be submitted to the CDP on the second working day following course completion.

In the case of CDP courses, lessons learned provide course managers, instructors, the CESD and others the information to make the training more beneficial to the students who attend the courses.

Procedure for Completion and Submission of the AAR

1. The course manager meets with the instructors upon completion of each course to make recommendations for delivery and instructional material revisions and to develop the lessons learned during the course.

2. The course manager completes the individual course AAR.

3. Upon completion of the individual course AAR, the course manager provides the original data to the CESD.

4. On a quarterly basis, the CESD will develop the quarterly report with data from the Responder End-of-Course Summary Report, the Pre- and Post-Test Summary Report and individual course AARs.

5. The CESD provides the report, on a quarterly basis, to the Deputy Program Manager and Chief Deputy Program Manager for submission to the CDP

Director, maintaining a copy within the department.

Center for Domestic Preparedness After Action Report

Course Name/Number…