Attachment_25 _Course_Evaluation_Manual.pdf

Training Delivery Services

Attachment 25

Course Evaluation Manual

CDP T&E MANUAL 016-5-00

Course Evaluation Manual

APPROVAL DATE: 06/17/2015

DEPARTMENT OF HOMELAND SECURITY

FEDERAL EMERGENCY MANAGEMENT AGENCY

CENTER FOR DOMESTIC PREPAREDNESS

TRAINING AND EDUCATION DIRECTORATE

Bernice Zaidel Assistant Director, Curriculum

Development and Evaluation Training and Education Directorate

Date: June 17, 2015

Charles D. Medley Assistant Director, Training Delivery Training and Education Directorate

Date: June 17, 2015

Foreword

This manual defines the levels and processes of evaluation within the Curriculum Development and Evaluation (CD&E) Branch. It documents the appropriate procedures for administering tests/evaluations for Resident, Non-Resident, and Indirect classes for officially scoring, recording, and archiving all tests/evaluations and for reporting student performance to headquarters. Tests and evaluations are a critical component used to assess student learning and the effectiveness of Center for Domestic Preparedness (CDP) instruction and curriculum products.

Table of Contents

CHAPTER 1: GENERAL INFORMATION

1-1. Purpose

1-2. Applicability and Scope

1-3. Supersession

1-4. Authorities/References

1-5. Definitions/Acronyms

1-6. Responsibilities

1-7. Forms Prescribed

1-8. Questions

CHAPTER 2: Level 1 Evaluation Procedures

2-1. Evaluation Instruments

2-2. Automated Evaluation Instruments

2-3. Printed Evaluation Instruments

2-4. Analysis of Data

2-5. Consolidated Reports

2-6. Corrective Action

CHAPTER 3: Level 2 Assessment Procedures

3-1. Test and Performance Assessment Checklist Development, Criteria, and Reviews

3-2. Test Administration

3-3. Automated Test Instruments

3-4. Printed Test Instruments

3-5. Processing of Level 2s

3-6. Official Scoring

3-7. Test Item Analysis

CHAPTER 4: Level 3 and 4 Post Graduate Evaluation Procedures

4-1. Student Information

4-2. Evaluation Requests

4-3. Analysis of Data

CHAPTER 1: GENERAL INFORMATION

1-1. Purpose The purpose of this manual is to define the evaluation/assessment instruments and processes used for the development and administration of CDP Resident, Non-Resident, and Indirect training evaluations/assessments.

1-2. Applicability and Scope This manual applies to all CDP Training and Education (T&E) Directorate federal employees and contractor personnel.

1-3. Supersession A. CDP-T&E-04-003 Course Evaluation Processes and Procedures B. CDP-T&E-04-001 Written Test Procedures

1-4. Authorities/References A. CDP 540-1-00 Quality Directive B. CDP 540-2-00 Quality Management System C. CDP 540-3-00 Quality Assurance/Quality Control Program D. CDP 542-1-00 Internal Control Program E. CDP T&E Manual 016-2-00 Training Support Manual F. CDP T&E Manual 016-4-00 Instructional Systems Design G. CDP T&E Manual 142-1-00 Acronyms and Definitions Manual H. CDP T&E Manual 181-1-00 Training and Education Records Management

Manual I. CDP T&E Manual 530-1-00 Data Flow and CTAS Requirements J. CDP T&E Manual 563-1-00 EAS Records Manual K. CDP-T&E-Pub-004 EAS Master Document Matrix L. Evaluating Training Programs: The Four Levels, by Donald L. Kirkpatrick

1-5. Definitions/Acronyms A. CD&E Branch – Curriculum Development and Evaluation Branch B. CTAS – CDP Training Administration System C. EAS – Electronic Archive System D. FEMA SID – FEMA Student Identification Number E. IT – Information Technology F. Kirkpatrick's Levels of Evaluation Defined:

1. Level 1 is described as the learner’s reaction to the training provided. This is best captured through the administration of an evaluation instrument such as a Level 1 end-of-course survey.

2. Level 2 is the learning assessment. CDP assesses learning by written assessments such as pre- and post-tests and performance is generally assessed using a checklist against specific performance standards.

3. Level 3 is the evaluation instrument that allows the learner and the supervisor to provide feedback on how the learning/training transferred to the job and how the organization’s and/or jurisdiction’s overall preparedness improved because of the training. This evaluation takes place after a period, usually 4-6 months, following the training event.

4. Level 4 evaluations seek to determine tangible results of the learning process in terms of reduced cost, improved job performance, increased production quality, efficiency, and return on expectations.

G. Kurtosis – A measure of the concentration of a distribution around its mean, esp. the statistic B 2 = m 4/m 2 ² where m 2 and m 4 are respectively the second and fourth moment of the distribution around the mean. In a normal distribution B 2 = 3 or a quantity indicative of the general form of a statistical frequency curve near the mean of the distribution.

H. NTED – National Training and Education Division I. pdf – portable document format

1-6. Responsibilities A. Director of Training and Education is responsible for determining policy in regards to CDP curricula.

B. Assistant Director, Curriculum Development and Evaluation is responsible for technical oversight of the CDP total curriculum development life cycle and maintains continuing overview of curriculum and evaluation programs.

1-7. Forms Prescribed A. FEMA Form 092-0-2, CDP Level 1 End-of-Course Evaluation Form B. FEMA Form 092-0-2A, CDP Level 3 Post Graduate Questionnaire Form for

Students C. FEMA Form 092-0-2B, CDP Level 3 Post Graduate Questionnaire Form for

Supervisors D. CDP-TE-0010 Test Wrap E. CDP-TE-0023 Indirect Remediation Letter F. CDP-TE-0071 Test Answer Form

1-8. Questions Questions concerning the policies and processes detailed should be addressed to the Assistant Director of Curriculum Development and Evaluation Branch, Training and Education Directorate.

CHAPTER 2: Level 1 Evaluation Procedures 2-1. Evaluation Instruments

A. The CD&E Branch will customize evaluation instruments for each CDP training course and delivery type. Evaluation instruments include statements and questions rated on Likert-type scales as well as questions seeking narrative responses utilizing FEMA Form 092-0-02, Level 1 End-of-Course Evaluation Form (template).

B. The CD&E Branch will provide automated or hard copy Level 1 evaluation instruments for all direct deliveries except Academies. Indirect Training deliveries do not receive Level 1 evaluations.

2-2. Automated Evaluation Instruments The CD&E Branch will enter evaluation data on a web-based automated evaluation system (e.g., Questionmark) and schedule a time period based on the approved class schedule (Level 1 assessments will be available for the duration of the class). Students will access the automated evaluation instruments during the time period as specified on the instruction sheet provided to each student (see Appendix A).

2-3. Printed Evaluation Instruments A. Resident Training

1. The Training Delivery contract Instructor will distribute evaluation forms to students at the beginning of each course.

2. At the end of each module, the instructors will encourage students to complete the corresponding section on the evaluation form.

3. At the end of each course, a student elected by other students or pre-designated by an instructor, will collect the completed evaluation forms and seal them in an envelope provided by the Training Support contractor.

4. The Training Support contractor personnel will receive the sealed envelope from the instructor or the designated student. Upon receipt, the Training Support contractor will scan the forms into the EAS, email a pdf copy of the individual student evaluations to the training distribution group and deliver the envelope to the CD&E Branch. Envelopes received by the Training Support contractor after regular business hours are archived and delivered to the CD&E Branch no later than 9:00 a.m. on the next business day.

B. Non-Resident Training

1. Training Delivery contract Instructors distribute evaluation forms to students as required.

2. Level 1s are not used for Mentored or Academy deliveries except as noted below.

3. Level 1s are used for Phase 1 of the Mentoring process, the Chief, Training Management, determines at what point an agency or academy transitions to full Mentored status.

2-4. Analysis of Data A. The CD&E Branch scans evaluation instruments using the designated software program to produce a quantitative and qualitative analysis report.

B. The CD&E Branch uses the analysis results to prepare Power Point slides for inclusion in a presentation for the weekly Enhanced Training meeting. The visual contains data about each class, with summary scores listed under specific categories such as instruction modules, facilitation, overall course impression, food service, housing, and transportation.

C. The CD&E Branch provides a copy of the analysis report for each class to the Training Support and Training Delivery contractors. The Training Support contractor archives this report with the class documents per the CDP-T&E- Pub-004 EAS Master Document Matrix.

2-5. Consolidated Reports The CD&E Branch creates Level 1 weekly and quarterly evaluation reports from the analyses of the data that includes demographic information, a categorized average score chart, and any recommendations for corrective action and the resolution to the issue. The CD&E branch submits the reports to the Assistant Director of CD&E Branch for review and distribution to the other T&E staff.

2-6. Corrective Action There are two elements to corrective action: items identified through the weekly level 1 reports to include trends identified in quarterly reports and items identified and discussed through the Enhanced Training Meetings.

A. Review process – The Chief, Training Management will address, by class, any corrective action requirements during the Enhanced Training Meeting and will assign any needed follow-up actions addressing trends and anomalies in class evaluation data. There are also periodic reviews of evaluation data, weekly collaborative meetings, and review of curriculum materials, to identify trends that would affect the overall learning and training process.

B. The trend analysis is captured in the weekly and quarterly level 1 reports and provides the documentation needed for future improvements to the CDP curriculum, training delivery, student support services, and other elements within the CDP structure.

C. Follow up/close out – If assigned a corrective action, the designee will provide a follow-up action report at the weekly Enhanced Training meeting or through the COR (if the function is managed by a contractor).

CHAPTER 3: Level 2 Assessment Procedures 3-1. Test and Performance Assessment Checklist Development, Criteria, and

Reviews A. CDP T&E 016-4-00 Instructional Systems Design Manual requires course reviews and required revisions every 18-24 months by CD&E Branch. This includes written tests and performance assessment checklists.

B. Level 2 assessment instruments are customized for each course based on the learning objectives and expected outcomes.

C. Tests and performance assessment checklists are validated during the first three iterations of a course. Test items are analyzed for validity and reliability at the conclusion of the pilot classes. Test items deemed invalid or unreliable are deleted, modified, or replaced, as appropriate.

3-2. Test Administration A. The course manager or instructor presents or explains each of the following to the class prior to administering an test:

1. Passing grade for the test.

2. Resources, tools, and references authorized for use in completing the test.

3. Provide students with instructions on what to do upon completion of the test.

4. Start and stop time for the test.

B. Pre-test

1. Conducting a test item review of the test is optional. The instructor or course manager may conduct a review of the pre-test items if time allows.

2. The instructional staff utilizes the class pre-test results to determine class subject matter knowledge and uses this information to reinforce relevant instruction. The instructor can run the pre-test through the Remark software station or access the coaching report from the Questionmark system to see the results.

C. Post-test

1. The instructional staff conducts a review of the test. At a minimum, review each test item answered incorrectly by 25% or more students and reinforce the correct answer for the item.

2. The instructional staff utilizes the class post-test results to support preparation for future classes. The instructional staff processes the posttest through the Remark Station as an unofficial score. The official results are run by the CD&E Branch and posted to CTAS. The instructional staff may share results with each individual on a requested basis. These results can be obtained from Remark or the Questionmark as a Coaching Report.

3. Remediation

a. The instructional staff will provide any student that scores below the minimum passing score an opportunity to remediate the failed learning objectives and retest.

b. Test Remediation Procedure: The minimum passing score on written tests is 70%. A student attaining less than the minimum passing score on the course written test must receive instructor-guided remediation prior to attempting a test retake. Students may review the proctor guide and answer key (if available) or coaching report available from Questionmark, which contains rationales for answer choice options for each question. Instructors shall allow the student to review their test results, clarifying any question(s) the student may have regarding the correct answer and the rationale for the correct answer to a test question.

c. Consequences for cheating: Notify any student suspected of cheating of the allegations and provide them with a reasonable opportunity to respond to those allegations. After consideration of the student’s response, any student determined to be cheating will receive a 0% for that test. Separate the student from his/her classmates and provide them an opportunity to retake the test as an individual effort. If the student passes this test, his/her official score will be recorded as minimum passing for the course.

d. Retake Procedure: following remediation, failing students may then retake the entire test or performance checklist one time to attempt attainment of a minimum passing score. If a student does not receive a passing score, the lead instructor informs the student, the Registrar’s Office, and the Training Support contractor.

D. Contingencies for pre- and post-test administration

1. Instructors will administer tests via Questionmark for those predetermined classes.

2. If there is a Wi-Fi outage or issue, the entire class must take the test via printed copy, even if they have already completed the test in Questionmark. All students will use the same method of testing.

3. If the pre-test is given via printed copy, the post-test should be given in the same manner. If the pre-test is given electronically via Questionmark, every effort should be made to give the post-test in the same manner.

However, printed copy tests are acceptable if an unavoidable issue arises.

3-3. Automated Test Instruments The following are the procedures for administering electronic tests and retrieving results.

A. Student directions from Instructors

1. Open iPad when instructed.

2. Find the Test & Survey icon and touch it.

3. Enter your name and password as directed by the instructor.

4. After entering your name and password, touch “Enter.”

5. Touch “Tests” on the left side of the screen.

6. Touch the appropriate test for the class you are attending.

7. Touch the button with the arrow to begin taking the test.

8. Proceed to answer all items.

9. Upon completion of the test, touch “Submit” at the bottom right of the screen.

10. Touch “Home” to return to the home page and log out.

B. Instructors will go to a local designated computer and log into Questionmark software to obtain test results and print Coaching Reports to be used with remediation, if necessary.

3-4. Printed Test Instruments The following are the procedures for administering printed tests and for preparing printed test packages to submit for official grading:

A. Resident Training

1. The course manager or designee retrieves the appropriate course printed pre-test or post-test and ensures it is the most current version based upon the Official Course Information Listing.

2. The course manager or designee present instructions for correctly completing the test form as listed below:

a. Inform students to only use a Number 2 pencil to complete the test form, to darken in all required bubbles fully, and to thoroughly erase any changes or mistakes.

b. Inform students to enter their first and last names and darken the corresponding bubbles. Darken the corresponding bubbles at the very top of any unused column signifying a null value. If either portion of their name is too long to fit in the spaces provided, they are to complete as many characters as allowed.

c. Inform students to write in the class number and date on the test form and to circle Pre-Test, Post-Test, or Retest, as appropriate.

d. On the back of the answer form, in the space provided, they are to print their FEMA SID and darken the corresponding bubbles.

e. When answering the test questions inform students to darken the letter of their answer completely and to choose only one answer per test question. For a 15-item test, fill in only the first 15 rows and leave the other rows empty.

f. Do not make copies of the blank forms. Always print the forms from the original file provided.

3. The instructional staff reviews all test forms for completeness, places the test forms in alphabetical order by student last name, and compares the forms to the class roster to ensure that each student has completed the test. Test form reviews will include:

a. Name, FEMA SID, class number, and test date are complete. The

Bubbles are fully darkened and that they contain the correct information.

b. Student responses to each test item are fully darkened, obvious erasures are removed appropriately, and that answers were completed in pencil only.

4. Instructional staff or instructional support staff will:

a. Complete all elements of a CDP Test Wrap Form, for the test.

b. Place the test forms into an envelope with the CDP Test Wrap Form attached to the exterior.

c. Provide the test package(s) to the CD&E Branch for grading by 10:00 a.m. the following business day.

B. Non-Resident Testing

Procedures are the same as Resident except for the following:

1. Instructional staff uses an overlay answer key to unofficially grade the tests in the classroom.

2. The instructional staff and/or Training Support will perform steps 3-4 A 3 and 3-4 A 4 after receipt of the testing package.

C. Indirect Testing

Procedures are the same as for Resident except for the following:

1. Upon receipt of an indirect course written test package from a Trainer or external agency, Training Support will log in the test to include class number, date package received, and date submitted for official grading.

This log tracks receipt of class testing packages and supports responses to outside agency inquiries.

2. When the official scoring is finished, the Training Support contractor completes CDP-TE-0023 Indirect Remediation Letter for any student(s) who failed the course and provides this to the class trainer no later than 30 days after the receipt of the test package by CDP from the agency.

3. The trainer or external agency will coordinate remediation of indirect course test failures. Retests conducted by Trainers or external agencies will be handled using the same procedures as in Resident Training.

3-5. Processing of Level 2s A. Prepare completed Test Answer Forms for scanning.

1. Place forms in alphabetical order by last name.

2. Turn all forms in the same direction with the name blocks face up.

3. Place test key on top of stack.

B. Scan answer forms using a multifunction device or scanner set for two-sided scanning.

C. Open Remark software, select Templates Tab, and click on “Open from template.”

D. Select appropriate template from list, if appropriate template is listed skip step G.

E. If appropriate template is not listed select “Browse.”

1. In window that appears select “Computer.”

2. In window that appears select “Local Disk (C :).”

3. In window that appears select “Program Files (x68).”

4. In window that appears select “Gravic” folder.

5. In window that appears select “Remark Office OMR 8” folder.

6. In window that appears select “Templates” folder.

F. In window that appears highlight “Level 2 15-item answer form” for Pre-Test” or “Level 2 30-item answer form” for Post-Test and click “Open.”

G. In window that appears “Remark Office OMR Data Center” select “Read Wizard.”

H. In the “Read Method” window that appears ensure “Read from image files” is selected and click “Next” button.

I. In “Image Selection” window that appears go to “Select the Images to Read” and choose “Desktop.”

1. Select the file you saved earlier by class name.

2. Click on the green arrow pointing to the column “Images In Read Order.”

3. Click on the “Read” button.

J. Test results will populate in the “Remark Office OMR Data Center” window.

K. Click on the “Easy Grade” button above the test results.

L. In “Remark Quick Stats – Easy Grade” window that appears, you must assign the appropriate grading percentage per the Official Course Information List.

1. Click on the “Tools” menu at the top of the window.

2. Click on “Preferences” in the dropdown that appears.

3. The “Preferences” window will appear, click on “Overall grade scale” down the left hand column. This information will appear to the right.

a. Ensure the “Default scale” dropdown is set to “Pass/Fail.”

b. Refer to the “Course Grading Criteria” located on SharePoint under the tests tab sheet for course pass/fail percentage and enter into the “Percent (Min.)” column on the Pass line and click the “OK” button, system will return you to the “Remark Quick Stats – Easy Grade” window.

M. In “Remark Quick Stats – Easy Grade” window, that appears select “Display Reports” in the “Report Options” column.

1. In the “Reports” column that appears select the “107 – Demographic

Grade Report.”

2. In the “Group Selection” window that appears, select/highlight the “Last

Name”, “First Name”, and “SID” fields by holding the “Ctrl” key down and clicking each one.

3. Click on the " > " button to populate the “Selected groups” column on the right.

4. Click the “OK” button at the bottom of the window.

N. The system will display the “Demographic Grade Report” on the “Remark

Quick Stats – Easy Grade” screen. From this screen, the report can be printed or saved by clicking on the “File” menu at the top of the window.

3-6. Official Scoring A. The CD&E Branch conducts the official test grading for the CDP. The score derived from this grading procedure will serve as the official source for all students’ test scores and for pre and post-test aggregate class scores reported to NTED at FEMA Headquarters through the CTAS system.

B. Written Tests

1. The CD&E Branch will inventory each written test packet to ensure test forms are complete, in alphabetical order, and free of stray or extra bubble marks. Contact the Registrar if any forms are missing or deemed unacceptable.

2. If the package is acceptable, sign for each written test package from Instructional Support or Training Support to ensure a record of receipt and log into the test spreadsheet.

3. Process the test forms for grading.

C. Electronic Tests are automatically submitted for processing within the Questionmark software.

D. Scoring

1. All remediation and retesting will occur during the conduct of the course. If a failure is identified, the CD&E Branch will immediately contact Training Support and the Registrar.

2. The CD&E Branch will produce a Test Report for the class. The report header will include the following information:

a. Course Number, e.g. YY-####LEPMA001, YY-####SAT8VA, YY-

####IL24FFO, YY-####TERT

b. Course Short Title e.g. HT, ICR, ARIO, etc.

c. Test Date

d. Test type – Pre-Test or Post-Test

e. Delivery Type – Resident, Non-Resident, Non-Resident Academy, Indirect, or Indirect Academy

3. The CD&E Branch will ensure the body of the Test Report includes the following:

a. Class “Mean Score” or “Overall Percentage”

b. Student FEMA SID

c. Percentage Grade

d. Student status – Pass or Fail

4. The CD&E Branch will save a copy of the Test Report by class number as a pdf in the proper folder on SharePoint, Training Support contractor archives these reports in the EAS.

5. In instances where there are class retests, the CD&E Branch will recalculate the mean score and generate a corrected report. Add the corrected report by class number as a pdf in the proper folder on SharePoint. For classes with failures, there should be a post-test report “with failures” and a post-test report “with retests.”

6. In instances where there are classes conducted on-line, the CD&E Branch will arrange for the Instructor/Trainer to receive an e-mail with the web address and instructions for Questionmark. The CD&E Branch will retrieve the report and add the report by class number as a pdf in the proper folder on SharePoint.

E. Discrepancies between Unofficial and Official Class Scoring In the event of a discrepancy between unofficial and official class grading, once validated, the CD&E Branch will report the issue immediately to the CDP Registrar for resolution. Unofficial grading is conducted by an instructor or trainer using a paper-scoring template (for paper-based tests) for nonresident and indirect training and use Remark for resident courses.

F. Priorities and suspense timelines for official test scoring The CD&E Branch will complete test scoring, report generation, and CTAS posting of scores within the following timelines:

1. Priority 1 – Resident Classes - First working day of the week following class completion

2. Priority 2 – Indirect Academy Classes – Five working days after CD&E

Branch receipt

3. Priority 3 – Non-Resident Classes – Five working days after CD&E Branch receipt

4. Priority 4 – Indirect Non-Academy Classes – no later than 10 working days after CD&E Branch receipt

5. The CD&E Branch will report extenuating circumstances that prevent the official grading of a class within the above suspense to the Registrar.

G. Aggregate Class Score Entry into CTAS

1. The CD&E Branch records aggregate “Mean Scores” for pre-test and post-test percentages by class for relevant Resident, Non-Resident, and Indirect Classes in CTAS. The CD&E Branch will finalize this posting by checking the Test Data Recorded box in CTAS.

2. The CD&E Branch will check the “Retest Pending” box in CTAS for Indirect Training classes with failures.

3. The CD&E Branch will finalize all Indirect Class retests no later than 31 December for the previous fiscal year by unchecking the “Retest Pending” box in CTAS. This is necessary to close out all classes for the fiscal year prior to the end of the calendar year and to report course attendance data to higher headquarters.

H. Written Test Archiving The CD&E Branch will maintain written test forms, the Roster Report and Master Report, referred to as an test package, for a period of 2 years. The CD&E Branch will maintain the test packages in a standard document storage box stowed in the CD&E Branch office space temporarily or the CDP Vault.

3-7. Test Item Analysis Process test items for all courses in pilot stage and new test revisions through these steps.

A. The test item analysis begins with the pilot course offerings. To validate the test items, the tests will be administered in the pilot course offerings.

B. CD&E Branch will process data from the pilot classes and create reports for analysis and trending.

C. The CD&E Branch will use the results of the analysis report to resolve any invalid or unreliable test items by using the item analysis methods.

D. As necessary, the CD&E Branch will create new test items and track the revised test items during non-pilot classes to validate and resolve any issues.

E. When revision of a test item is complete, the CD&E Branch will assign a new version number and assign an implementation date. The CD&E Branch will post the new test on SharePoint. The CD&E Branch will conduct random spot checks during the 3-year period to assure continuous validity.

F. If it is determined that a test revision is warranted before the recertification date, the CD&E Branch will perform analysis on specific test items or the entire test(s). If revised or recertified, the test analysis process will begin again.

G. During revisions or recertification, the CD&E Branch will conduct test item analysis. The CD&E Branch can provide the following data as justification for modification to test items. This information is provided to the curriculum staff, Instructional Systems Specialist as needed or requested.

1. Median/Mode/Mean

2. Degree of Kurtosis (Questionmark can produce a visual (chart) to demonstrate Kurtosis)

3. Distractor frequency Analysis

CHAPTER 4: Level 3 and 4 Post Graduate Evaluation Procedures 4-1. Student Information

At the end of each course, the CD&E Branch or the Training Delivery contractor’s Course Manager or lead instructor informs the class of the CDP’s interest in how each student improved job performance and skills resulting from training received, and alerts students they and their supervisors will be contacted later to request on-the-job evaluations.

4-2. Evaluation Requests A. Three to six months following training

1. The CD&E Branch will send each student via e-mail, a CDP Level 3 Post Graduation Evaluation Form (FEMA Form 092-0-02A) requesting that they complete the form and return it to the CDP via the designated software directions.

2. The CD&E Branch will send the respective supervisors, via email, a CDP Level 3 and 4 Post Graduate Evaluation Form for Supervisors (FEMA Form 092-0-02B) requesting that they complete the form and return it to the CDP via the designated software directions.

B. Evaluation forms will include statements and questions rated on Likert-type scales and narrative responses.

4-3. Analysis of Data The CD&E Branch will analyze student and supervisor Level 3 and 4 evaluation forms and provide a detailed quantitative and qualitative report quarterly.

4.4 Reports

The Assistant Director of CD&E Branch will submit the reports to stakeholders for review and required action.