Attachment_35 _Third_Party_Review_Procedure.pdf

Training Delivery Services

Attachment 35

CDP NTED 3rd Party Review Procedure

Federal Emergency Management Agency

Standard Operating Procedure (SOP)

Date Organization 11/12/2014 Center for Domestic Preparedness (CDP)

Training and Education Directorate NTED Third Party Review

I. Purpose This document outlines the Standard Operating Procedures (SOP) that facilitates the content and structure reviews of National Training and Education Division (NTED) courses which are submitted by grantees from State, Local, Tribal, and other federal agencies and organizations. This SOP is to guide reviewers in the execution of the NTED Third Party Review program within the Center for Domestic Preparedness (CDP) in Anniston, Alabama. Specific structure and content review standards are provided to assist reviewers in lessening the ambiguity and inconsistency of the review process. A simplified, fundamental guide designed to be user-friendly and task-specific, the ultimate goal for the SOP is to implement and supplement the NTED Course Review and Approval Process guidelines (December 2008, ed.) as a “best practices” document.

II. Authority A. CDP 540-1-00 Quality Directive

B. CDP 540-2-00 Quality Management System

C. CDP 540-3-00 Quality Assurance/Quality Control Program

D. CDP 542-1-00 Internal Control Program

III. References NTED Course Review and Approval Process (December 2008, ed.)

IV. Definitions A. A/V ─ Audio/Visual

B. CDD ─ Course Design Document

C. CD&E Branch ─ Curriculum Development and Evaluation Branch

D. CDP ─ Center for Domestic Preparedness

E. COR ─ Contracting Officer’s Representative F. ELO ─ Enabling Learning Objectives

G. HQ ─ Headquarters

H. MSEL ─ Master Scenario Events List

I. NTED ─ National Training and Education Division

J. POC ─ Point of Contact

CDP-T&E-04-005v1 Page 1 of 21

CDP-T&E-04-005v1 11/12/2014

K. POI ─ Plan of Instruction L. SME ─ Subject Matter Expert

M. TLO ─ Terminal Learning Objectives

N. TPR ─ Third Party Review

O. TSP ─ Training Support Package

V. Responsible Office Training and Education Directorate, Curriculum Development and Evaluation Section

VI. Standard Operating Procedures A. Process Overview

1. Course TSP mail package(s) are received via mail or FEDEX at 40 Twill Lane, Anniston, AL 36205 and are taken by a clerk/carrier to the CD&E Branch Administrative Assistant. The Administrative Assistant, the Evaluation Lead Instructional Systems Specialist, or the CDP NTED POC (the Senior Instructional Systems Specialist) opens the packages and reviews the contents to ensure the complete TSP is included. A complete TSP consists of:

a. Instructor guide/instructor outline/instructor lesson plans

b. Participant/Student manual, guide, or workbook

c. Audio/visual support materials (e.g., PowerPoint presentation, DVDs, CD- ROMs, videos)

d. Evaluation materials such as written tests and answer keys or a performance checklist (if applicable), and

e. Additional support materials (e.g. descriptions of practical and hands-on exercises)

NOTE: There should be two electronic copies (CD ROM, DVD, flash drive, etc.) of the complete TSP in MS Word or Adobe PDF format with each review request submitted.

NOTE: Folders and spreadsheets for documenting data pertaining to Third Party Review requests are found in SharePoint, under the Third Party Review Tab, Shared Documents section.

2. Incomplete TSP submissions

a. If the TSP is incomplete, the CDP NTED POC will notify the HQ NTED POC (Business Operations Analyst, National Preparedness Directorate, National Training Education Division) within one working day.

b. Incomplete TSPs are kept in a secure location in the CD&E Branch in the Structure Reviewer’s office. From within CDP SharePoint, under the Third Party Review Tab, Shared Documents section the Excel spreadsheet titled “Incomplete NTED Courses,” is opened and populated with the following information:

1) Course Title

2) NTED Tracking Number CDP-T&E-04-005v1 Page 2 of 21

3) Submitting Agency or State

4) Date Received

5) Issues (i.e., a full description of why the TSP is incomplete)

NOTE: If the course does not include an NTED tracking number, the course is incomplete until one is received. The HQ NTED POC is contacted via email, to inform of missing material(s) so the submitter can be asked to forward the missing data electronically.

3. Complete TSP submissions

a. If the TSP is complete, the Submitting Agency Compilation spreadsheet in SharePoint under the Third Party Review Tab, is opened and populated under two separate tabs:

1) Under the separate tab associated with the submitting entity/State, populate each column with the data indicated (Course Number, Course Title and Agency).

B. Labels are created and placed on each individual component of the TSP. The labels will include Course Title, NTED tracking number, and submitting agency or state.

C. Each NTED TSP review is allocated 30 working days from the receipt of the materials. The Structure Reviewer or the CDP NTED POC will count the 30 working day review period on the calendar (not including Federal holidays or weekends) and deduce the government due date. From there, the Structure Reviewer calculates the content reviewer due date and the structure reviewer due date. Traditionally, course content reviewers have been assigned the same number of review hours as the length of the course.

D. A labeled folder with all course materials and documents is kept in the office of the structure reviewer until all reviews are finished, the course report is submitted to the CDP NTED POC, and the course is ready for the archive room (within the Building 61 Archive Vault). Course TSPs returned following review completion are kept in Curriculum NTED Archives, located within the Building 61 Archive Vault.

E. Forms and Templates Folder

The Forms and Templates folder, located on SharePoint (Third Party Review Tab) contains all of the forms needed for processing a review request. Files from within this folder will be used to populate the NTED course folder, which are created in the applicable Fiscal year (FY) and month folder. The following forms are in the Forms and Templates folder:

1. Labels – A template used to label all Training Support Package (TSP) materials.

2. CDP-TE-0041 NTED Third Party SME Review

3. CDP-TE-0072 CD&E Branch Contract Task Order Request

4. CDP-TE-0073 NTED Third Party Structure Review

5. CDP Third Party Review Decision Matrix (See Appendix A)

6. Sample e-mail

CDP-T&E-04-005v1 Page 3 of 21

F. Structure Review Purpose:

1. Clarifies structure review areas (e.g., objectives, scope statements, resource lists) within the NTED Course Review and Approval Process (December 2008, ed.)

guidelines

2. Serves as an informational guide for all stakeholders and other interested parties as to the criteria used to perform third-party structure reviews on NTED courses

3. Structure Review Sections

a. NTED Guidelines

Includes the definitions regarding the 12 structure review areas that were taken directly from the guidelines.

b. CDP (structure reviewer) Standards

This section includes an expansion of each of the 12 structure review areas and provides guidance and clarification to the NTED guidelines.

4. Training Support Package (TSP)

a. NTED Guidelines

The NTED Guidelines define the TSP as “all of the materials associated with the delivery of the training course.” The TSP must include the following items:

1) Instructor Guide/Instructor Outline/Instructor Lesson Plans

2) Assessment Instruments and Answer Keys

3) Participant/Student Manual/Guide/Workbook

4) Audio/Visual Support Materials

5) Special Support Materials such as Exercise Handouts

b. CDP Standards

1) TSPs are accepted in softcopy (e.g., CD ROM, flash drive) only.

2) Common file formats, such as .doc or .docx (Microsoft Word), .pdf (Adobe Acrobat), .xls (Microsoft Excel), .ppt (Microsoft PowerPoint), .wmv (Windows Media Video), and .wav (Windows Audio), are accepted.

If the majority of these files cannot be opened and are pertinent to the TSP, the TSP is considered to be an incomplete submission.

3) Incomplete TSPs are stored in the secure location in the Curriculum (Evaluations) Branch in the Structure Reviewer’s office until the remaining elements of the TSP arrive. TSPs that are received without an NTED tracking number (“course number”) are also considered incomplete submissions.

5. Instructor Guide/Instructor Outline/Instructor Lesson Plans

a. NTED Guidelines

The NTED Guidelines define Instructor Guide/Instructor Outline/Instructor Lesson Plans as “the published instructor material that contains course text

CDP-T&E-04-005v1 Page 4 of 21 and special instructor notes that provides the information to deliver the material.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the ranking follows:

1) Acceptable: Contains all the information needed to conduct the course, exercises and evaluations.

2) Marginal: Contains the majority of the information needed to conduct the course, exercises and evaluations.

3) Unacceptable: The information is insufficient to conduct the course.

b. CDP Standards

1) Instructor Guide/Instructor Outline/Instructor Lesson Plans must arrive as a softcopy (e.g., CD ROM, flash drive).

2) Ideally, the structure reviewer receives the Instructor Guide/Instructor Outline/Instructor Lesson Plans in an organized fashion.

1) Instructor files should be separated numerically by module/unit or by folder. The contents of the electronic file must be clearly organized to avoid confusion on behalf of the structure reviewer. Deviation from this standard results in a lower ranking for this structural area.

3) The structure reviewer performs a review of the Instructor Guide/ Instructor Outline/Instructor Lesson Plans for completeness (e.g., all materials are present for instructor facilitation and utilization). If the instructor materials are complete, the structure reviewer begins a thorough structure review of the TSP.

1) If any structure area receives an unacceptable ranking, a completed NTED Independent Third-Party Course Structure Review form is completed and sent to the HQ NTED POC attached to a disapproval recommendation.

2) If the structure review passes (e.g., every area gets an acceptable or marginal ranking), a content review is requested.

4) Modules/units/sessions/lessons, etc., must be in the order of classroom delivery. Each must include a page at the beginning of each module/unit/session/lesson with all of the required NTED components, which are (at a minimum):

a) Scope Statement

b) TLO

c) ELO

d) Resource List

e) Instructor-to-Participant/Student Ratio

f) References List

g) Practical Exercise Statement

h) Evaluation Strategy CDP-T&E-04-005v1 Page 5 of 21

6. Participant/Student Manual/Guide/Workbook

a. NTED Guidelines

The NTED Guidelines define the Participant/Student Manual/Guide/Workbook as “the published student material that contains the supporting information in booklet, electronic, or handout form that the participant/student has available for reference.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: The information contained is described in detail, appropriate, and well-described.

2) Marginal: Some of the supporting information is incomplete or needs additional clarification.

3) Unacceptable: The majority of the supporting information is out dated or irrelevant.

b. CDP Standards

Participant/Student Manual/Guide/Workbook must arrive as a softcopy (e.g., CD ROM, flash drive). Ideally, this document is structured in an organized fashion parallel with the IG or Instructor Materials. An explanation follows:

1) Student files should be separated numerically by module/unit or by folder.

The contents of the electronic files must be clearly organized to avoid confusion on behalf of the structure reviewer. Deviation from this standard results in a lower ranking for this structural area.

2) Modules/units/sessions/lessons must be aligned with the instructional materials. A page is included at the beginning of each module/unit/session/lesson with all the required NTED components.

a) Scope Statement

b) TLO

c) ELO

d) References List

e) Evaluation Strategy

7. Audio/Visual Support Materials

a. NTED Guidelines

The NTED Guidelines define A/V Support Materials as “any audio/visual components that are part of any learning module, session, lesson or that supports the overall training being delivered.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

CDP-T&E-04-005v1 Page 6 of 21

1) Acceptable: A/V materials are of good quality, comprehensively described, and consistent with learning objectives.

2) Marginal: A/V materials are of good quality, consistent with learning objectives but incomplete.

3) Unacceptable: A/V materials are of poor quality, non-existent or are not consistent with learning objectives.

b. CDP Standards

1) A/V materials must arrive as a softcopy (e.g., CD ROM, flash drive).

Ideally, the A/V material is structured in an organized fashion and parallel with the instructional material. An explanation follows:

a) A/V files should be separated numerically by module/unit or by folder.

The contents of the electronic files must be clearly organized to avoid confusion on behalf of the structure reviewer. Deviation from this standard results in a lower ranking for this structural area.

b) At a minimum, videos and pictures must be up-to-date, relative to all target audiences, and of good quality and consistent with learning objectives. Federally-sponsored or developed courses must also be 508 compliant (either with closed-captioning or a transcript included), and with copyright permission (if needed). A failure in these specific areas will result in a lower ranking.

8. Special Support Materials

a. NTED Guidelines

The NTED Guidelines define Special Support Materials as “any descriptions of practical exercises, table-top exercises, hands-on exercises or other material that supports the learning objectives.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the ranking follows:

1) Acceptable: Exercise materials are professionally done, comprehensive, and focused on achievement of the learning objectives.

2) Marginal: Exercise materials are incomplete but address the learning objectives.

3) Unacceptable: Exercise materials do not support the achievement of the learning objectives.

b. CDP Standards

1) Special support materials must arrive as a softcopy (e.g., CD ROM, flash drive). Ideally, the special support material is structured in an organized fashion. An explanation follows:

a) Special support materials should be separated numerically by module/unit or by folder. The contents of the electronic files must be clearly organized to avoid confusion on behalf of the Structure Reviewer. Deviation from this standard results in a lower ranking for this structural area.

CDP-T&E-04-005v1 Page 7 of 21

b) Materials such as a MSEL (if applicable), exercise scenario, required resources, and instructor notes must be included. Failure to provide all information necessary to execute exercises will result in a lower ranking.

9. Module/Unit/Session/Lesson Content

a. NTED Guidelines

According to the NTED Guidelines, Module/Unit/Session/Lesson Content in training courses “should be designed based on a building block approach with each sub-component in the course titled as a module, unit, session, or lesson.”

Additionally, each module/unit/session/lesson must have an administrative page or front page that outlines the following:

1) Scope Statement

2) TLO

3) ELO

4) Resource List

5) Instructor to Student Ratio

6) Reference List

7) Practical Exercise Statement (if applicable)

b. CDP Standards

1) Module/Unit/Session/Lesson content must arrive as a softcopy (e.g., CD ROM, flash drive).

2) Ideally, the structure reviewer receives the Module/Unit/Session/Lesson Content in a logical fashion.

i. These files should be separated numerically by module/unit or by folder. The contents of the electronic files must be clearly organized to avoid confusion on behalf of the structure reviewer. Deviation from this standard results in a lower ranking for this structural area.

3) Module/Units/Session/Lessons must be formatted the same throughout the TSP for uniformity and organization. The structure reviewer will note when format is not uniform.

10. Scope Statement

a. NTED Guidelines

The NTED Guidelines define a Scope Statement as “a brief description of the content of the module/unit/session/lesson.” According to the guidelines, this statement will earn a score of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the scores follows:

1) Acceptable: The Scope Statement is a clear description of the module/unit content or topics, and directly supports the course goal. It is referenced and complies or supports applicable standards.

CDP-T&E-04-005v1 Page 8 of 21

2) Marginal: The Scope Statement is clear, but does not align with the content or topics and appears to not support the stated course goal.

3) Unacceptable: The Scope Statement is not provided or is present and does not support and/or comply with the course goal.

b. CDP Standards

The scope statement should be a concise statement or bulleted points outlining the module/unit/session/lesson’s content or topics that support the course goal.

11. Terminal Learning Objective (TLO)

a. NTED Guidelines

The NTED Guidelines defines a TLO as “an action statement that describes what the student is expected to learn or be capable of performing at the conclusion of the course, module/unit. There should be only one TLO per module/unit.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: TLO is clear, uses action verbs, and directly supports the course goal. It is referenced and complies with or supports applicable standards.

2) Marginal: TLO is clear as to what the Participant/Student is expected to learn but does not support the stated course goal, and is not measurable-or-statements are included that could qualify as a TLO but are not identified as such.

3) Unacceptable: TLO or similar statements are not provided or are not present nor do they support and/or comply with the guidelines.

b. CDP Standards

1) TLO may be measurable according to the Knowledge, Skill, or Ability (KSA) associated with the TLO. The following is how the structure reviewers will evaluate each TLO according to KSAs.

2) The knowledge (K) or cognitive aspect of a TLO may involve the lecture instructor-led elements of content that are offered either in an online training environment or in a classroom setting. The TLO supplements the information required for the conduct of instructor-facilitated training (e.g., hands-on exercise, lanes training). Typically, TLOs use action verbs such as “identify,” “define,” “differentiate,” “analyze,” “compare,” “contrast,” and “recognize.” The knowledge component is constructed of background, historical, and factual information or other data where no interpretation is required. This is generally lecture-specific course material.

3) The skills (S) aspect of a TLO involves performance elements, which must accompany classroom delivery in support of a TLO that calls for the demonstration of a skill or task. Typically, these TLOs use action

CDP-T&E-04-005v1 Page 9 of 21 verbs such as “demonstrate” or “perform.” This is comparable to lanes training or demonstrations of skill sets.

4) The abilities (A) aspect of a TLO involve those portions of course material, which are exclusive to a classroom and/or experiential-based training environment in that instructor facilitation is necessary for student learning. Typically, these TLOs use action verbs such as “discuss, describe, or explain,” “perform,” and “demonstrate.” This is generally in the form of small group work such as case study discussions or individual work where the individual performs a specific task or function or a team performs specific tasks associated with a simulation or scenario identified in the TLO.

5) If the action verb does not match the TLO’s outcome, it will reflect a lower ranking: action verbs must be measured in some form or fashion within the module/unit/ session/or lesson. For example, if the action verb “describe” is utilized, the developer must provide some way in which a description of the subject matter is evaluated (e.g., oral or written test using vignettes or case studies) consistently and accurately.

Conversely, if the action verb “demonstrate” is utilized, a measurable demonstration (e.g., properly don PPE Level C according to the XXX standard) must be performed by the student, whereas if the action verbs “identify” or “recognize” are used, a multiple-choice or true/false test is sufficient.

12. Enabling Learning Objective (ELO) (Review the level 2 assessment instruments using these criteria.)

a. NTED Guidelines

The NTED Guidelines define ELOs as “enabling learning objectives that support the TLO. Each ELO must be a measurable performance statement that enables the student to achieve the standard required to meet proficiency of the task(s) and TLO.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: The ELO is clear, measurable, supports the TLO, and supports or complies with the guidelines.

2) Marginal: The ELO supports the TLO but verbs are not measurable or ELOs are present but no TLO is listed to tie the ELOs together or statements are included that could qualify as ELOs but are not identified as such.

3) Unacceptable: The ELO or similar statement is not provided or ELOs are present but do not support a listed TLO or do not comply with the guidelines.

b. CDP Standards

ELO action verbs must be measurable according to the KSA associated with the ELO. The following is how the structure reviewers will evaluate each ELO according to KSAs and TLO(s).

CDP-T&E-04-005v1 Page 10 of 21

1) The knowledge or cognitive aspect of an ELO involves lecture/instructor-led elements of subject matter that are either offered in an online training environment or in a classroom setting. The TLO supplements the information required for the conduct of instructor-facilitated training (e.g., table top exercise, lanes training). Typically, the ELOs use action verbs such as “identify,” “define,” “differentiate,” “analyze,” “compare,” “contrast,” and “recognize.” The knowledge component is constructed of background, historical, and factual information or other data where no instructor-based interpretation is required. This is typically lecture-specific course material.

2) The skills aspect of an ELO involves instructor-facilitated elements, which must accompany classroom delivery in support of an ELO that calls for the demonstration of a skill. Typically, these ELOs use action verbs such as “categorize,” “demonstrate” and “perform.” This is comparable to hands-on training or demonstrations of skill sets through a practical exercise portion of a course, wherein subject matter (Knowledge and/or Skills) learned in another portion of the class is completed in an instructor-facilitated environment.

3) The ability aspect of an ELO involves those portions of course material, which are exclusive to a classroom or experiential based training environment in that instructor facilitation is necessary for student learning.

Typically, these ELOs use action verbs such as “describe,” “demonstrate,” and “perform.” This is usually done through small group work using a case study, individual or team work where learner(s) demonstrate a specific task to a particular standard with proficiency.

4) If the action verb does not match the ELO’s outcome, it will reflect a lower ranking: action verbs must be measured in some form or fashion within the module, session, or lesson. For example, if the action verb “describe” is utilized, the developer must provide some way in which a description of the subject matter is evaluated (e.g., oral or written test using a vignette or case study) consistently and accurately. Conversely, if the action verb “demonstrate” is utilized, a measurable demonstration (e.g., properly don PPE Level C according to XXX standard) must be performed by the student, whereas if the action verbs “identify” or “recognize” are used, a multiple-choice or true/false test is sufficient.

13. Resource List

a. NTED Guidelines

The NTED Guidelines define a Resource List as “a listing of the resources needed to successfully accomplish the module/unit/session/lesson.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: Resource List(s) are comprehensive and consistent with materials needed to accomplish all objectives.

2) Marginal: Resource List(s) is complete enough to accomplish most objectives.

CDP-T&E-04-005v1 Page 11 of 21

3) Unacceptable: Resource List(s) are either not included or lack the understanding of the resources needed to accomplish the objectives.

b. CDP Standards

A complete resource list must be included in the POI/CDD and within each module. Failure to have all of the materials needed to conduct instruction and exercises listed will result in a lower ranking.

14. Instructor to Participant/Student Ratio

a. NTED Guidelines

The NTED Guidelines define an Instructor to Student Ratio as “the instructor to student requirement for successful presentation of the material (e.g., 1:25).”

According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: The ratio of instructors to Participant/Students is appropriate for this level of training and objectives.

2) Marginal: The ratio of instructors to Participant/Students indicates that the instructors will be challenged in meeting the training requirements and objectives.

3) Unacceptable: There is no instructor to Participant/Student ratio listed or the ratio is inappropriate for the level of training and objectives.

b. CDP Standards

Structure reviewers assess the instructor to student ratio based on the TLO and ELOs. If the learning objectives are knowledge-based then an example of an appropriate ratio would be 1:30; however, if a(n) TLO/ELO is skill-or ability-based (e.g., practical exercise) an example of an appropriate ratio might be 1:6. Failure to provide appropriate instructor to student ratios will result in a lower ranking.

15. Reference List

a. NTED Guidelines

The NTED Guidelines define a Reference List as “a listing of all reference materials used to develop the module, unit, session, or lesson. This information may also be included as a bibliography.” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: Reference List(s) are comprehensive, current, and consistent with the materials needed to accomplish all objectives.

2) Marginal: Reference List(s) are incomplete with gaps in resources needed to provide current and relevant information in order to accomplish some objectives.

3) Unacceptable: Reference List(s) are not present or suggest a lack of resources needed to support and accomplish the objectives.

CDP-T&E-04-005v1 Page 12 of 21

b. CDP Standards Structure reviewers evaluate the use of in-text citations and graphics.

Submitters must provide references for all graphics and for any rules, regulations, laws, and/or acts cited (e.g., National Fire Protection Association [NFPA] 1600) within the text. Structure Reviewers will be checking for the following information:

1) Title

2) Author (if applicable)

3) Date

4) Location retrieved (e.g., valid, current web address, if applicable)

16. Practical Exercise Statement

a. NTED Guidelines

The NTED Guidelines define a Practical Exercise Statement as “[a description of] any practical exercises and exercise objectives associated with the module, unit, session, or lesson.” The statement is on the administration page and is a statement or paragraph. According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: Practical exercise descriptions are appropriate, complete, sufficient, and professionally developed.

2) Marginal: Practical exercise descriptions are incomplete but address most objectives.

3) Unacceptable: Practical exercise descriptions are missing major elements and features, and lack assessments needed to evaluate the exercise activity.

b. CDP Standards

1) In order for an exercise or activity to be delivered, it must contain the following information (not just a statement on the Administrative Page) in the module/unit/learning activity in order to receive full ranking:

a) Scenario description

b) Master Scenario Events List (MSEL)—if applicable

c) Required resources—e.g., computers with internet connection, telephone, facsimile, easel pads, equipment, props

2) The structure reviewer examines the practical exercise statement. There is usually not enough information in this statement that an experienced instructor would be able to execute the exercise. The execution details are embedded in the appropriate modules/units/learning activities of the instructor guide.

CDP-T&E-04-005v1 Page 13 of 21

17. Evaluation Strategy (Review the level 2 assessment instruments with this criteria.)

a. NTED Guidelines

The NTED Guidelines define an Evaluation Strategy as “[a description of] the strategy used to evaluate the participant/student’s level of success upon completion of the module, unit, session, or lesson (e.g. written and/or performance tests or assessments).” According to the guidelines, this structural area will earn a ranking of unacceptable, marginal, or acceptable after a thorough structure review. A breakdown of the rankings follows:

1) Acceptable: Test(s) of performance or competence are appropriate, complete, sufficient, and professionally developed.

2) Marginal: Test(s) of performance or competence are incomplete or inconsistent with some objectives.

3) Unacceptable: Test(s) descriptions are missing major elements and features or lack assessments needed to evaluate the KSAs.

b. CDP Standards

1) Each TSP must have an assessment instrument. The following are examples of acceptable assessment instrument.

a) 15-25 question post-test with a 70% passing rate

b) Performance assessment checklists with a set passing standard

2) All of the evaluation strategies must include some type of remediation strategy.

G. Structure and Content Review Standards

1. Structure Reviewers may refer to CDP-TE-0073 NTED Third Party Structure Review, when conducting the structure review. The questions on this form reduce the ambiguity of the Structure Review standards in the NTED guidelines and provide guidelines that are utilized by the structure reviewers.

2. The questions on form CDP-TE-0041 NTED Third Party SME Review form reduce the ambiguity of the content review standards in the NTED guidelines and provide information that are utilized by the content reviewers. Content reviewers answer these questions while reviewing the NTED course TSP.

3. NTED Standards

“Independent Third Party Subject Matter Experts (SMEs) will be assigned to provide detailed course content and materials review, the SMEs will determine the accuracy and appropriateness of the course content and materials.” Form CDP-TE-0041 is utilized to record their findings and recommendations.

4. CDP Standards

Structure reviewer(s) utilize the CDP-TE-0073 NTED Third Party Structure Review located in the G common drive under Training and Education Current Documents/CDP-TE-Forms.

CDP-T&E-04-005v1 Page 14 of 21

VI. Forms Prescribed A. CDP-TE-0041 NTED Third Party SME Review

B. CDP-TE-0072 CD&E Branch Contract Task Order Request

C. CDP-TE-0073 NTED Third Party Structure Review

VII. Attachments Appendix A, CDP Third Party Review Decision Matrix

APPROVED BY:

November 10, 2014 Bernice Zaidel Date Assistant Director of Curriculum Development and Evaluation Center for Domestic Preparedness

CDP-T&E-04-005v1 Page 15 of 21

CDP-T&E-04-005v1 Appendix A 11/12/2014

Appendix A: Decision Matrix for CDP Recommendations of Approval or Disapproval for Structure an Content Review

Structure Review Element Question Examples of issue/concern Possible Decision Training Support Package (TSP) Assessment

Does it contain all the components? IG, SG, Visuals, Assessment Instruments, etc.

Missing assessment instruments and visuals

If no, then contact the HQ NTED POC for submitter to send missing components.

Item 1 – TSP Objectives Does the course material, as presented in the provided TSP, support the objectives for each module/session/lesson?

The objectives were stated with no measureable criteria or standards and were not supported in the content adequately. The content failed to discuss both academically and pragmatically salient concepts surrounding the objectives and topics.

If NO - the comments identifying objectives that are not measurable support recommending disapproval of a course. Course material that is incorrect, outdated, or lacking sufficient content data to fully support objectives are valid reasons to recommend disapproval as well.

TSP Objectives continued Visuals supporting the objectives?

The content of most slides was too simplistic for participants.

If NO - the course materials do not support the learning objectives, and then disapproval should be recommended with possible resubmission after materials are aligned with objectives.

Item 2 – Course Delivery Generally not an issue for structure and content review.

Item 3 – Missing Elements Are there missing topics, content, or elements from the

TSP?

Time limits for lecture and exercises are missing or the Planning Objective is not met in the Planning Module;

however, through the

If YES - however, these comments could support the approval recommendation of a course – these are minor comments or issues.

CDP-T&E-04-005v1 Page A-1 of 21 completion of the course all objectives are eventually met.

Missing elements continued Missing topics, content, or elements from the TSP?

Missing content regarding nuclear terrorism. Need more technical information and data on RED, RDD, and IND especially the differences and how these could pose a threat to a community. Reinforcement of the concepts of each threat is missing or is so limited that it can be mis-construed or misunderstood and lead to poor decision making and response actions.

If NO - Comments such as this would support a disapproval recommendation. The lack of information is critical to the comprehension of the information and how it can be transferred to the organization and community. Do they have enough information to make decisions about Threats, Hazards, and Risks? The content is inadequate for decision makers and responders/the audience.

Item 4 – Audio/Visual Support Are the AV materials of good quality, comprehensively described, and consistent with learning objectives?

Generally quality issues are included in the comments regarding size and clarity of the video or audio.

If YES - generally recommend for approval because the submitter can easily improve the quality and provide better products.

AV Support continued See above AV source is not included that supposedly supports a learning objective

If NO - would recommend disapproval especially if the AV source is critical to the application and achievement of the learning objective.

Item 5 – Incorrect or Misleading Content

Are there areas in the content that are not correct or

IG Module 1 and 2 are not consistent with the actual

If NO - The sheer volume of missing, misleading, CDP-T&E-04-005v1 Page A-2 of 21 misleading such as terminology, reference information, laws, procedures, or practices/protocols?

course presentation material. The Instructional materials are not aligned to the course objectives, agenda, or content outline.

Missing details on specific ELOs, titles within the IG and Participant Manual/Guide are inconsistent.

inconsistent, and outdated content to the learning objectives should support a disapproval recommendation. For example 26 specific items were incorrect, misleading, and outdated. Also look at the number of grammatical errors that could also contribute to disapproval when aligned with above examples.

Item 6 – Instructor Requirements Does an instructor who will teach this course need special qualifications?

Sometimes strong suggestions for pre-requisite training are included.

Generally does not impact the structure review elements or criteria.

Item 7 – Safety Issues Is there any point in the execution of the course that the instructor or course lead would need to be concerned or on the lookout for safety issues?

The exercises are not defined or documented. PPE activities are conducted in a mock response environment that also includes “a cleaning and disinfection procedure” that is not described. An interactive mapping exercise is proposed, but not described.

Instructions are limited and not tied back to the specific objectives nor includes info about risks or hazards that could be associated with the activities.

If YES - This type of information and comments would lead to a disapproval recommendation. Safety details of exercise and activities are critical to the safety of the students but to all involved in the activities

– instructors, evaluators, role players, etc. This is critical information that must be included and documented.

CDP-T&E-04-005v1 Page A-3 of 21

Safety Issues continued ….. lookout for safety concerns?

All activities require a student to be in the water and/or around boats involve some type of risk. These risks can be managed with a strong safety plan. A practical Daily Briefing Safety Sheet is required to be completed for each practical session, which will help in the mitigation of a lot of risks in open water operations.

If NO - This example provides the details that support the approval recommendation. Details that are reminders, job aids, and other actions are examples of what needs to be documented and included in exercise and activity instructions.

Item 8- Benefit to Others Will participating in this course benefit the target audience?

The course is highly beneficial for first response personnel.

This generally does not affect the overall decision process regarding an approval or disapproval recommendation.

Item 9 – Offensive Content Are there any areas in the content that can be perceived as offensive?

The term “X” is inaccurate, unprofessional, and offensive. The use of such a term may give insight into the course author’s stance/bias concerning social media and other electronic technologies.

Other credible sources/SMEs in this content area stated that the use of the term “X” is offensive and inappropriate.

If YES - Statement and phrases such as the example are good indicators that a disapproval recommendation would be warranted. Not all words, terms, or phrases are necessarily offensive; be wary of being so “politically correct” that you lose sight of the factual data or educational information that supports the objectives.

CDP-T&E-04-005v1 Page A-4 of 21

Item 10 - Documents, Standards Were documents or standards

(references) used to review the course?

NFPA 1670, 1472, 1600,

etc.

NIMS, PPD#8, NPD Goals, etc.

These may be cited, but generally does not impact a structure review decision.

Item 11- Lecture/Valuable Training

Do the lecture portions of the course provide valuable training?

The lectures gives the students a basic level of understanding of the forces they will experience when confronted with moving water rescue, etc.

Not generally an issue affecting the decision regarding approval or disapproval with the structure or content review.

Item 12 – Lecture/Training Agency

Could any training agency teach the lecture portion?

Generally does not impact the decision regarding approval or disapproval.

Item 13 – Hands-on Exercise Value

Does the practical exercise portion of the course provide valuable training?

The practical exercise allows the student to take the information learned in the classroom setting and apply it in a real word setting while in a safe environment.

Generally does not affect the decision process.

Item 14 – Hands-on Exercise/Training Exercise

Could any training agency teach the practical exercise portion?

The practical exercise must be completed in a water environment, including both pool and moving water locations.

Generally does not affect the decision process;

although SMEs will provide comments to the yes and no responses.

Item 15 – Evaluation/Overall Strategy in TSP

Does the course contain a complete evaluation strategy to include learner assessment and an end-of-course feedback? Do the assessment questions align with the objectives in the course?

A posttest is provided, but no pretest was found within the documents. The posttest does not have a place for the participant’s name.

Questions were not included in the TSP; therefore the

If NO - Need the assessment and evaluation documents with answer keys to determine and justify the decision regarding approval or disapproval recommendation. If the

CDP-T&E-04-005v1 Page A-5 of 21

SME is unable to determine whether they measure the learning.

tests do not measure the learning against the objectives, this would support disapproval.

Item 16 – Evaluation/Hands-on Exercise

Does the practical exercise include any type of performance checklist or observation checklist?

If NO - If the exercise does not measure the performance objectives, then a checklist is a good reminder of what should be accomplished. Therefore, the lack of a checklist could support disapproval unless otherwise indicated in the exercise instructions or the overall evaluation strategy.

Item 17 – Evaluation/Course Content Appropriate at training level

Is the course content appropriate at the level of training.

If the course is put together well, it follows a logical progression of lecture and skills-based training and it would be appropriate to the audience.

Generally does not affect the decision process.

CDP-T&E-04-005v1 Page A-6 of 21