Attachment_7b_Systems_Approach_to_Trainiing_NAVMC_1553.1_2.pdf

Student Outline

NAVMC 1553.1

27 Oct 2010

The student outline·provides the student with a general structure to follow during the class and a conceptual framework that highlights the main ideas of the class. It contains the learning objectives, lesson outline, and any references used to build the lesson. It also includes any additional notes or Information, such as graphics (charts, graphs, dip art, photos, diagrams) deemed necessary.

When developing the student outline, it does not have to be in Naval Format or In outline form. It should be developed in a way that the student is able to easily follow and use. Regardless of the format, all pertinent information from the lesson plan should be induded, as described above. Appendix B provides examples of some different formats for student outlines and MeO 1553.2_ Appendix 0-25 provides a checklist to follow when creating a student outline.

Student outlines can be written using one or a combination of the following styles:

~ Completed. This style provides students with a "cut-and-paste" of the body from the Instructor's lesson plan that excludes the administrative information, introduction, any cues, Instructor notes, and/or the summary. This style is desirable when students are expected to read the entire outline during class, are unable to take notes or follow along during class, or when instruction takes place outdoors. It is very useful as a study guide or a job aid.

~ Fill in the blank. This style uses an abridged form of the completed style with key terms or phrases omitted and replaced with blank lines.

It is developed as a skeleton outline of the lesson plan. It is the student's responsibility to follow the lecture and fill in the missing information. When students complete the missing key terms or phrases, they are more likely to remember the material as they heard, saw, and wrote It. Presentation of the lesson must be structured to allow students time to fill In the blanks. This style of outline is not recommended for subjects of a technical nature.

~ Jb!l.I.m. This style incorporates short, informational statements presented in the same sequence as in the lesson plan. The student must take detailed notes to complete the information. Curriculum developers must take this into consideration and leave sufficient "empty space" for student's notes In the outline. The bulleted style is not recommended for those students with little or no knowledge of the subject.

Supplemental Student Materials

Supplemental student materials include handouts, other than the student outline, given to the class in support of the instruction. Supplemental student materials may include advance handouts to prepare the student for class.

Additionally, supplemental student materials may Include answer keys to quizzes, addltlonal articles for reading, and reference materials such as technical manuals, graphs, charts, formulas, figures, and maps. The use and number of supplemental student materials is optional and they can be presented In any format. The distinction needs to be made between supplemental student materials and dassroom instructional aids. The distinctlon is made based on ownership. Supplemental student materials wiff be Items that students are able

4-22

27 oct 2010 to take with them fof/owing the class. Instructlorial aids belong to the Instructor for use in the classroom. Although the students use them during the class, they do not retain them at the end of the lesson. All supplemental student materials

. should support the learning objectives being taught.

4-23

INSTRUCTOR PREPARATION GUIDE

The Instructor Preparation Guide Is a required element of the Master Lesson File (MLF). This guide is created to provide the instructor with Information that Is critical to the preparation for implementing the lesson. Detailed information is given so that the Instructor understands what resources are necessary for the lesson. Much of the information provided under administrative information is copied from the concept card. Though this guide Is a MLF Item, instructors can make a copy so that they can check off items when preparing for the lesson. An example of the Instructor Preparation Guide can be found in MCO 1553.2_ Appendix 0-26,27. ,The minimum components for the Instructor Preparation Guide are listed below. This checklist can be added to as heeded.

Lesson Title and Lesson Designator. The lesson title and lesson deSignator are provided to identify the lesson. Both can be found on the concept card.

Tota' Lesson Time. Refer to the concept card for the total lesson time. This provides the instructor with the amount of time that he/she has to teach the lesson.

References. List all of the references from the concept card.

Locatlpn of Test. The location of the test is provided so that the Instructor will know where to go to gather the test materials.

Personnel Required. list all personnel that will be required to implement the lesson (e.g., instructors, support personnel, Corpsman). Check the student to instructor ratio and notes on the concept card for this Information.

Fadlities. The facilities required for the lesson need to be listed (e.g., classrooms, labs, ranges, etc.). Some faCilities may require prior coordination to ensure availability.

The above components are listed as follows:

LESSON TiTLE: '

LESSON DESIGNATOR:

TOTAL LESSON TIME:

REFERENCES:

LOCATION OF TEST:

PERSONNEL REQUIRED:

FACILITIES:

" 'ASSemblea M~stef LessorlFlle>c, '!.

CD0209,

30 Minutes

MCO 1553.2, SAT Manual, MCTlMS User's Manual

See Course Chief

1 Instructor

30 seat classroom

4-24

NAVMC 1553.1

27 Oct 2010

An example of the Instructor Preparation Guide can be found in MCO 1553.2_ Appendix 0-26,27.

Top portion of the Instructor Preparation Guide.

27 Oct 2010

Review Course Materials. This checkbox Is necessary so that the instructor will review the course materials to identify any potential problems prior to Instruction.

Rm-E\VCOURSE'MATERIAL5:

o Review the course/training schedule, administrative requirements, student background information, lesson pians, student materials, media, and evaluations (tests).

Personalization. This checkbox is necessary so that the instructor adds personalization to the lesson plan.

PERSONALIZATION:

o Pel'SQnalize the leSscnplan by adding subject matter detail, relating personal experiences, providing examples, and/or interactive techniques.

MaterialsiEgulpment. All materials and equipment needed to conduct the lesson are listed here with check boxes so that the instructor can gather materials well in advance of the lesson. Materials may include models, mockups, audiovisual eqUipment, handouts, etc. •

MATERIALS/EQUIPMENT:

o Video dip for Gain Attention DComputer'andPrOjector .

o 30 Brown Binders for Master Lesson Files o 30 Master Lesson File Checklists

Exercise Setup and planning. Each exercise (e.g., demonstration, practical application) is listed here. Underneath each, the setup and planning is described In sequence with check boxes to the side.

EXIERClSE sETup AND pLANNING: i ..

Demonstration· o An MLF binder Is ready to hand out to each student.

o The MLF checklists are ready to hand out to each student.

o Ensure that classroom Is setup so tl'l_a~d~monstratloncan be seen by all.

~. The ORA worksheet is a required element of the MLF and must be reviewed by the Instructor. This checklist also requires that the Instructor reassess the environment for changes (e.g., weather or worn eqUipment) and report findings on the AIR.

SAFETY: .

CrRev.iewO~:work$heE:tlfnMC!$ter,L~ssPtt.File., . ..... . ... ..~. •.

o Reassess the~nvi"onmentfQrcl$hge~1I1ataffectttieoriginaroRA. Document any additional conslderations/conttols.i)rdheAfterJr'lstruqion Report (AIR) for .. :futvre reference.· .."

Other Possible Items. Additional Items can be added to the checklist if deemed necessary by the FLe.

Approylng Signature and Date. A space is provided for the designated approving authority's signature and date. The Formal School's SOP will dictate who approves the Instructor Preparation GUide.

4-25

4005. CONSTRUCT TESTS

Tests are designed to evaluate If the learner has the knowledge and skills required to master the objective or task. Back in the Design Phase, test items were developed for each learning objective. Now, based upon the course structure and when specific learning objectives are to be tested, the test is constructed. Before going Into the steps for constructing a test, there must be an understanding of the methods of tests, purposes of tests, and the types of tests.

METHODS OF TESTS

Knowledge-Based Test

As was discussed in Chapter 2, Section 2207 of the Design Phase, knowledge based testing can be done through oral or written tests. This method of testing does not evaluate the student's ability to perform the required job skills;

however, it does determine if the student knows how to perform the required job skills. Two advantages of this method are its high degree of objectivity in scoring (the capability of measuring a large numbers of facts, Ideas, or prIncipleS in a relatively short time) and the convenience In the development of statistical analysis. There are a number of factors that force schools to administer knowledge tests: time, cost, safety, and resource constraints that do not always permit performance-based testing.

performance-Based Test

This evaluation deals with the assessment of technical skills, usually physical/motor skills. It usually deals with physical performance that follows a procedure or sequence of steps, which Is called a process, or the end result, which is called a~. A test item that requires the student to replicate a task that is performed on the job is conSidered performance-based. A performance-based test will usually have a checklist that clearly defines the steps or procedures that must be completed to master the objective. In some circumstances, a written test can be considered a performance-based test If the student actually performs that Item on the job. For example, filling out a DD Form 214 Is a valid performance test for a student who actually fills those out on the job. A performance test duplicates the actual behavior by using the same eqUipment, resources, setting, or circumstances that the student will encounter on the job.

MCTIMS Student Evaluation Module (SEY). The SEV module is part of McnMS and is a web-based application used to construct tests, record test data, track student scores/GPA's and generate reports. MCO 1553.2_ Ch. 5 par. 2a(S) directs the SEV will be used to the fullest extent possible.

4-26

NAVMC 1553.1

27 Oct 2010

SECTION:·······

CATEGORIES OF TESTS

There are different purposes for giving tests. Below are some categories of testing along with their purpose. Since criterion-referenced testing is the preferred method of evaluation for the Marine Corps, more focus has been given

1. Criterion-Referenced Test. These tests are used to evaluate the student's accomplishment of the criterion objective arid to determine the effectiveness of the Instructional system. Criterion-referenced tests are composed of items based on specific learning objectives. Each indiVidual's ability to demonstrate mastery of the leaming objectives is measured. The leamer's achievement is measured against the predetermined criterion established in the leamlng objectives.

2. Diagnostic Test. The purpose of diagnostic testing Is to measure the achievement of the supporting skills and knowledge that contri~ute to the ability to perform the criterion objective.

3. Survey Jest. These tests are designed to determine what prospective students already know and can do before receiving instrUction. The test Is administered while the instructional system is being developed and provides important design data.

TESTING INTERVALS

A stUdent's knowledge and skill level can be tested at different intervals before, during, and after Instruction. A pretest, progress test, and a posttest ~ccomplish this.

1. Pretest. A pretest Is designed to identify how much the student knows or is able to do w:!Qr to starting the course. This kind of testing is diagnostic In nature. It provides what level the student is at prior to the course.

2. progress Test. A progress test IS administered throughout a course to evaluate student progress and to determine the degree to which students are accomplishing the learning objectives.

3. Posttest. The purpose of posttests is to identify/evaluate the effectiveness of Instruction/and how well the student leamed. It is also a certification process. The student's ability to graduate from the course is generally based on posttest results. Therefore, certification that the student is able to go out in the real world and perform the job 15 provided through graduation.

4-27

NAVMC 1553.1

27 Oct 2010

Different test intervals.

Before During After

STEPS FOR CONSTRUCTING TESTS

The test items have already been written, Now the challenge Is to properly assign and arrange test items, determine the grading criteria, develop scoring method, and develop the testing instructions, Determining Mastery

Mastery Learning. Criterion-referenced testing Is the preferred method of testing for learning objectives taught In the FLC. The criteria for test mastery are established by the learning objectives. The student, when completing a test, receives either a master (pass) or non-master (fail) for each learning objective. The student may be aSSigned an overall score, but it does not remove the responsibility of mastering each learning objective. Students that non-master a learning objective may receive remedial Instruction and retesting until they reach the standard for mastery. The FLC will establish the remediation policy based on school resources (i.e., time, equipment utilization, availability of instructors). Students who do not master the learning objective during the established number of retests could be recycled through the program or dropped from the course. See MCO 1553.2_ Ch. 1, Par. 3, (5) for guidance on remediation policy.

Determination of COurse Mastery. The term "mastery" can be misleading

- mastery does not mean or require that students pass with 100%. Students graduating from a course must, however, master 100% of the learning objectives.

Determination of Learning Objective Mastery. A determination is made by the FLC as to what is the acceptable level of performance for mastery of each learning objective. It may easily be that, for some objectives, a score of 60% is indicative of mastery, while for others a score of 90% or higher would be required. The determination is based upon the criticality of the objective.

Mastery of all ELOs does not necessarily result in the mastery of the TLO, just as receiving a minimum score on each individual event of the PFT does not necessarily mean that you receive an overall passing score on the PFT.

4-28

Assigning Written Test Items

When determining what test Items to use, the idea is to measure all learning objectives. Formal evaluation of learning objectives is accomplished by testing each learning objective. '

Informal evaluation of learning objectives is accomplished through class work, homework, quizzes, and practical application. This does not meet the requirement to test learning objectives in the FLC.

There is no established formula for determining the most appropriate number of test Items required for testing any given learning objective. However, the guidelines listed below are factors to consider:

Criticality of skill. This refers to how Important the skill is in relation to its application to actual job performance. . .

1; High:.S.killls used during job performance.

2. ModeI'aM: ,SkllfinfiU:eneesjob sierforrnal'ice; .'

3. Low: Sklll has little Influence on job performance.

Other Criticality Factors. Refers to a learning objective's/outcome's Importance as related to the performance of a job task.

1. .' $afetytoi",""ne1/equipiftent: . Crltlcaltas'- .... ~~ Ylb,~ are considered high risk or'clal1gerous; .. . '.~',; ':". .

2. . Frequency of perform'ince: The more often a task ispeHonn'ed, • .

ttuJimbfe:CdtltlUt becomes. ' . .. .' '. '.

3. Lean1ln9'bJJJ~i"e$J~pprt:ancetoon~~e~jobperff;tmt~,~~.· ..

4;' .' J.~~nningobJed:iveSimpoifanCeto the overall courSe:mlsSl6n. .

Criticality of the objective. When both most critical and least aitical objectives are measured on the same test, the critical objective should have more items to ensure that the test reflects the critical aspects of the course.

Instructional time allotted to present the material. For example, if the majority of the material covers one objective, then the majority of the test items should cover that objective. This ensures the emphasis In the classroom.

Complexity of the material. The more complex the material is, then the more test items are required to ensure understanding.

4-29

NAYMe 1553.1

Arranging Test Items

When making decisions on how to arrange test items, consider the following:

Test item placement. Test Items should be placed on the page so each item stands out clearly from the others. For example, a true or false item that Is two lines long would have single spacing with doublE!"'spaclng between items. A space should separate the stem of multiple-choice items and the list of answers. The answers should be in a single column beneath the stem and should be indented beyond the paragraph margin.

Example of Multiple Choice Item:

1. What are the three Domains of Learning?

a. Auditory, Visual, Kinesthetic

b. Intellect, Value, Tactile

c. Knowledge, Skill, Attitude

d. Cognitive, Affective, Psychomotor

Arrangement of test items. Items of the same type (e.g., multiple chOice, short answer, essay) are grouped together in a test. Individual test items should also be arranged in approximate order of difficulty, which allows the students to progress as far as they can without spending excessive time on difficuit items at the first part of the test.

Design. A test is designed so that the majority of the students can complete

it. When many students cannot complete a test, efficiency is lost and student morale suffers.

Layout/Format. Below are some guidelines to consider when formatting the test:

1. Space items for easy reading and responding.

2. Provide generous borders.

3. List alternative responses vertically beneath the stem (multiple choice).

4. Do.!lQt split an item onto two separate pages.

5. If an answer sheet is nQt being provided, allow space for student answers.

6. Number items consecutively throughout the text.

7. If separate answers are used, number them so a check can be made for complete sets of materials before and after test administration.

8. Select an arrangement of items that serve the purposes of the test.

4-30

NAVMC 1553 1

.. '~:. ~iL:',.:;X;:<:c , .. , > '''-':/':',' . .• "~~",U~TX.N.TItE'iSSt'"M'ifLEi:),"fES1""~"'::c'",,,~"~, •... • <"~' ". ,.:~·.i :-',,"~ -'- - -.

1. Relevance Do the test Items present relevant tasks?

2. Conciseness Are the test tasks stated in simple, dear language?

3. Soundness Are the items of proper difficulty, free of defects, and do they have answers that are defensible?

4. Independence Are the items free from overlapping, so that one item does not aid in answering .

another?

5. Arrangement • Are Items measuring the same objective grouped together?

• Are Items of the same type grouped together?

• Are Items in order of increasing difflcultv?

6. Numbering Are the items numbered In order throughout the test?

7. Directions • Are there directions for the whole test and each part?

• Are the directions concise and at the proper reading level?

• Do the directions Indude time limits and how to record answers?

8. Spacing Does the spaclnCl on the page contribute to ease of reading and resQQ.ndlng?

9. Typing Is the final co~y-'ree ottypographical errors?

Assessment of Student Achjevement. By Norman E. Gronlund. p. 122.

Developing Grading Criteria

Grading criteria describe the standards by which the student will be measured and factors that will be considered In determining the student's grade on an individual performance or knowledge test/test Items.

Uses of criteria. Grading criteria enable the instructor to determine whether or not the student/group has met the objective. Additionally, it provides an unbiased and non-subjective evaluation of the student's ability with respect to a particular area of performance or knowledge. The primary concern of grading criteria should be that it describes what the student is expected to do and what happens if the requirements are not met. .

Grading Criteria for performance Evaluations. The creation of grading criteria may be the most critical step in performance evaluation test development because It ensures ,standardized grading. The scoring guide contains a description of each step or group of steps to be graded. A pass/fail checklist describes in detail what constitutes satisfactory and unsatisfactory performance. Grading criteria for the course [s a factor if the course is graded Mastery or Non-mastery; a checklist may be the most appropriate to use. If the course Is graded with a numerical grade, a rating scale may be the most appropriate to use. When defining the checklist steps and rating scale decisions, all behaviors have to be written in suffident detail so that all tasks are as precise as possible. The more completely the behaviors are described, the more effective the Job Sheet Checklist/Rating Scale will be. This helps remove instructor subjectivity from the grading process. Performance and knowledge-based testing should not be combined. Multi-part tests can be constructed In MCTIMS to support situations where both forms of testing are needed.

4-31

Other important grading criteria factors should include:

1. Compliance with required safety precautions.

2. Correct operation of equipment after completed assembly.

3. Physical testing if the job is finished.

4. lime required completing the job.

5. Skill in using tools.

6. care and use of the equipment.

Develop a Scoring Method

Manually graded. A key or template needs to be developed to eliminate any subjectivity in the scoring process. Ensure this item is safeguarded against compromise. The essay test requires different scoring criteria. A model answer is required that lists all essential data a knowledgeable student can be expected to provide. This model Is used as the standard answer by which all other answers are scored and the worth of each item or part of an Item is set.

Automated grading system. Some schools utilize bubble sheet scanning devices to do automated grading. Future upgrades to the Student Evaluation Module of MCTIMS will provide an interface between systems ,so grades can automatically be fed to Student Evaluation.

,.'itULES FOR SCORING 'eSSAY' ANSWERS

1. Evaluate answers to essay questions In terms of the learning objectives being measured;

2. Score restricted-response answers by the point method, using a model answer as a guide.

3. Grade extended-response answers by the rating method, using defined criteria as a guide (Rubric).

4. Evaluate all the students' answers to one question before proceeding to the next question.

5. Evaluate answers to essay questions without knowing the identity of the writer.

6. Whenever possible have two or more persons grade each answer.

4-32

Test Instructions for the Student

Once the desired test items are prepared, focus on the required information Identifying the test. A complete set of Instructions, either written, oral and/or by visual aid, must be given to the student. For written tests, a sample test item is given so that students understand how they should answer the question (i.e., circle, write out, "X''). The student instructions should specify the following:

1. References and materials are to be utilized during the test (If any).

2. Any rules for the test (e.g., "No talking.'')

3. Time allowed for each section or for the whole test.

4. How to proceed with the test (i.e., individually, from part to part, from page to page or whether to wait for a signal.)

5. Procedures to follow after completing the test.

6. School's policy on cheating.

Student evaluation instructions are covered In Chapter 4, Section 4400.

Test Instructions for the Test Administrator/Proctor

Specific Instructions need to be written out to the test administrator/proctor so that there is uniformity in how the test is to be administered. The instructions should tell what is required for preparation in giving the test, how the test is to be administered, and how remediation is handled.

InstructionS for PreDaring to Give'Jest Should specify:

1. What the testing materials are.

2. Where to gather the testing materials.

3. How many can be tested at a time if there's a limit.

4. Required testing environment (e.g. computer dassroom, motor pool).

S. Seating arrangements (if applicable).

6. Prepare a "Testing" placard to be displayed outside the testing environment. '

Instructions for Administering the Test should spedfy:

1. Whether the students can use references or other materials during the test.

2. Inform students of the cheating policy for the school.

3. Amount of time the students are given to complete the test.

4. Whether the test administrator/proctor is to answer questions during the test.

Remediation Instructions should specify:

1. Type of remediation that will be conducted.

2. Where the retest will be located.

3. Procedures for giving retest.

By preparing detailed instructions, the administration of the test Is more likely to be standardized. The overall effect of the standardization is more reliable test results on student progress and level of mastery.

4-33

4006. CONDUCT VALIDATION

Validation Is a process of trying out instructional materials and course materials prior to implementation to ensure that mastery of the learning objectives Is possible and reasonable. Validation involves examining theeffectlveness of instructional materials by identifying strengths and weaknesses. The instructional material should be presented to members of the target population to determine Its effectiveness. If the instruction does not enable students to reasonably achieve mastery, It Is revised until it does ..

METHODS OF VALIDATION

There are a variety of methods for validating instruction. Validation of instructional materials should Involve as many methods as possible. If aU methods are to be used, they should be conducted in the order in which they are presented below. Personnel other than the curriculum developer(s) should conduct the validation to enhance objectlvity. The personnel conducting the validation are referred to as curriculum validators.

4-34

NAVMC 1553.1

27 Oct 2010

SECTION

Subject Matter Expert (SHE) Technical Data Review

SME technical data review involves reviewing course materials to ensure the technical accuraCy of Instructional material content. Although the Instructional materials are not in final form at this stage, the content should still support the information provided In technical manuals and orders, job guides, and checklists.

SME participation will help identify specific problem areas and provide additional technical data.

Curriculum Validation Teams (00)

The cvr is a method of validation in which a team comprised of an experienced jobholder, a novice; a supervisor, an Instructor, and a curriculum developer meet to review the instructional materials. The curriculum va/idator will coordinate the meeting as a facilitator only. As with the SME technical data review, the instructional materials are not in final form yet. Each of the partiCipants of the cvr will examine the material from their different perspectives ensuring that materials are technically accurate, instructionally sound, and the learning level Is appropriate to the target audience. For instance, a novice can point out gaps in the conte'nt that may be unnoticeable to SMEs, or vice Versa. If there are disagreements among participants, a technical data review concerning all partidpants may be assembled to resolve the Issue.

Pilot Course

In this validation method, instructional in final form are presented to a target population group. This validation method Is Important because it.takes into account indiVidual student learning differences. Student samples should represent the entire range of the skill and knowledge level of the target population. Instructional materials should be presented under normal environmental conditions. For example, if the materials are intended for classroom use or field use that is the environment in which the trials should be materials conducted. The dedsion to use a pilot course as a validation method Is based on the availability of the necessary members of the target population and time. A pilot course is used at the discretion of the school to validate a developing POI prior to submission to TECOM (within 120 day window of Proof of Concept COD Signature or Direction from TECOM to create a new course).

This pilot course must be run with pre-existing school resources, a representative TPD, and not at the detriment of approved courses. Pilot courses are not meant to be run in multiple iterations but simply as a test bed on the validity of a POI prior to submission for approval.

o ImallGroUp; In a small group validation, the curriculum validator ..

presents'lnstructional materials to a small group (2-4 individuals) of the target population to determine if mastery can be attained.

o Large GrOuP. During large group validation, the lesson plan is presenteo'tOJr group of 5 or more people for validation. Presenting the lesson to cflarge group allows many people from different learning perspectly~$,~~O receive the instruction. If tUne Is a constraint, large group validation can be conducted concurrently with Implementation.

4-35

Validation at First Implementation

27 Oct 2010

This type of validation involves presenting instructional materials, in their final form, to members of the target population at first implementation. In this case, validation and implementation are conducted concurrently for one presentation of a scheduled This is .t:AQI the preferred class. This is MIT the preferred method of validation, and is done only when there method of validation is not enough time to conduct validation of materials prior to Implementation.

Validation at first implementation should only be done as a last resort.

TYPES OF DATA

The following are types of data gathered during validations.· Depending upon the type of validation, data may vary in quantity.

Data Collected from Students. Student data are collected to determine the attitude of students when they are presented with instruction, particularly anything that kept them from attaining mastery of the learning objectives. Additional student background information Including age, time in service, past experience, past academic experience, current job assignment, etc., should also be collected. In the collection of data from students, students should provide their comments on the fotlowlng:

Instructional Material Data. Information on the effectiveness of the Instructional material should be gathered from instructors, SMEs, students, and curriculum developers. These data can include effectiveness of:

r-·------.-.-0:-L~···Pian:---·-·~-·--'--··:·-~'::7,-~·~.,.-;'~"-" ~, r

0" Student outline. . , ..•.• ,... _, I--~;;;~ r . 0 Practical applications., '

4-36

STEP 1

STEP 2

NAVMC 1553.1

27 Oct 2010

Instructional Procedures Pata. Data on the effectiveness of the delivery system (instructional methods and media) should be gathered from Instructors, SMEs, students, and curriculum developers. These data may Include effectiveness of:

r-----·~-0 -MethOd ofl~nstruction~~-'---

I'~-----[tr~=ederol()instruction (tral ... lng---' . ··lMdl U e ..

r---- .@ ;i~~~ttor presentation.

1-·--""0~ :N~inber of instructors.

r-~ 0· i~itivctional Setting-. --

Test Item Pata. During validation, test Items should be analyzed to determine If they measure the knowledge or, skills required of the leaming objectives. Test Items should also be analyzed for reliability to determine If they produce consistent results.

This is done through a process called test item analysis. Test item analysiS is a set of procedures for evaluating the effectiveness of test Items. Item analysis will identify-which test items need to be revised or rejected. IUs critical to conduct Item analysis during validation prior to course implementation to ensure that th.e test items are valid. Chapter 5, Section 5300 presents detailed procedures for conducting test item analysis.

STEPS FOR VALIDATING INSTRUCTION

Review Formal Learning Center (SOP) Standing Operating PrOcedures· .

The information needed to plan validation may be contained in the school validation plan located in the school's Academic SOP. This document may provide additional guidance on types of validation trials, data collection methOds, and appropriate ...

authority for approval.

Plan and Schedule Validation

Plan and schedule validation to allow enough time to incorporate any improvements into the lessons prior to the start of the course. This is a critical step that must be well thought out. Validation IsplannecJ sO that all trials can be conducted, data analyzed, and revisions made prior to implementation of the course. During this step, the type of data to be gathered (see Section 3602) and the type of validation methods (see SectJon 3601) are determined.

4-37

Determine Data Collection Procedures

Once the validation method Is selected, determine the system for collecting data.

These data may be collected using ~urveys, questionnaires, interviews, group discussions, observations or other methods (see Chapter 5, Section 5603).

CUrriculum validators should ask open-ended questions so that participants can genuinely express their feelings, opinions, and perceptions of the effectiveness of the instruction. Curriculum validators must keep in mind that the purpose of validation is to obtain Information that will Improve Instruction.

Implement Validation Plan

USing the validation methods planned in Step 2 and the data collection procedures Identified in Step 3, conduct the validation.

a. SHE Technical Pata Review. Provide SMEs with Instructional materials or Instructional material content. Directions should be provided as well as the objectives of the validation.

b. Q£[. The curriculum validator gathers members for the CVT and serves as the facilitator of the meeting. The curriculum valldator should ensure the following:

o All participants contribute to the meeting.

o Recommendations for revisions are understood by'~npaiticipants and are recorded. .

0' Any other feedback concerning the effectiveness of instruction Is collected and recorded.

c. Pilot Course Trial. A pilot ti'lalis the most comprehensive and time consuming validation to conduct. It involves conducting an actual class with a group of students within or similar to the target population group.

To conduct a pilot trial, the curriculum validator will;

o Gather students from the targel population to receive the instruction.

III Arrange the Instructional setting as it will be arranged for the actual implementation of the class.

o Identify and brief Instructors who will participate in the field trial.

III Develop questionnaires to collect data from students and Instructors concerning their attitudes toward the effectiveness of instruction.

III Ensure the Instruction is conducted as it will be implemented.

4-38

NAVMC 1553.1

27 Oct 2010

STEP]

STEP 4

STEPS

STEP 6

NAVMC 1553.1

27 Oct 2010

Interpret and Record Validation Results

Interpret and record data from the validation. Since there Is no specific format for doing this, curriculum valldators should record the results in a manner that meets their validation objectives. For example, data can be summarized in a brief paragraph annotating how many comments were made and the trends found detailing instructional strengths and deficiencies. If the data were collected using a scaled rating system, the answers should be averaged and presented as an average response for each question. This summation should also include recommendations for solutions to correct for instructional deficiencies. See Chapter 5, Section 5300 for detailed procedures concerning the analysis and Interpretation of data.

Report Validation Results

Once validation data is collected and the results are summarized, make recommendations for correcting problems. The summarized results will indicate what materials, methods, or media need revision report the validation results to the validation authority for approval.

VALIDATION AUTHORITY

The responsibility for validation of instruction ultimately rests with the FlC commander. The method of validation is based on resources available. The commander provides guidance on conducting validations through a validation plan, usually found in the Standing Operation Procedures (SOP). The plan will Identify who has validation authority. Decisions about how to validate are based on resources, as outlined In the table below.

For example, the following decisions concerning validation must be made by the FlC: .

0+ What personnehtre'avallabletO'conduct the validation eSMEs, ·lhStrUctors; cu.J1icl:lluiride"elope~; .etc)?;

. + How many methods of vali~ation (s~ti~cJort;#~. . .. ~ ·~~$edil~ .

validating course material? What spedflcrevisfb1ij~"'o~"::.>. onill materials can ~'ui1dertaken:andstillmeet the planitedcourse

. schedule? . .. .. ··.~,,;'''s~.

~ How do we obtain members of the target .,opulatJOiIfot'Va1idation? If actual members of the target population a,.eliot BVBUable,then the school directorsho"illd select.:indlvlduals with backgrounds assimil.r as possible to those of the desired target population. .

~ How much tlmeJsa'r~irabr~? If your time to de~.',n~~ri~~.Y~IOP ~, course islimltedj Yo" iWIU have:tochoo5e a validation rliethodthat fits within the tllrie'cori$tf.in~.< .. . . . . . _.

4-39

4007. DEVELOP A PROGRAM OF INSTRUCTION

(POI)

Per MCO 1553.2-1 every Marine Corps FLC must have an approved Program of Instruction (POI). This approved POI is the schools authorization to conduct a course. A POI documents a formal school's plan for instructing Individual Training Events (ITEs). Specifically, a POI describes a course in terms of structure, delivery methods and media, length, intended learning objectives, and evaluation procedures. It also serves as a histOrical record that reflects the continual evolution of the course. An Important element of the POI is the Course Descriptive Data (COD) which is section I of the POI. The COO provides a summary of the course

, Including Instructional resources, class length, and curriculum breakdown.

COURSE DESCRIPTIVE DATA (CDD)

DesqiptiOO. An approved POI authorizes the FLC to develop a new course or it authorizes a change to an existing course. The COD is an important component of the POI and does the following:

.:. Indicates the school's concept of how the course will meet the training needs as established in the T&R Manual .

• :. Identifies resource requirements needed to conduct the course from which decisions can be made.

4-40

NAVMC 1553.1

27 Oct 2010

TECOM- rralhfng& Education Command

NAVMC 1553.1

27 Oct 2010

Ii:! New Course of Instruction. A FlC will submit a Proof of Concept COD and a cover letter requesting approval to add a new course. The.

cover letter should address why the course Is required, what deficiendes it will correct, and why it should be conducted in a Formal School setting. MCO 1553.2_ describes the requirements for Proof of Concept COOs for new courses.

o TECOM (FSTD) records the Information contained In the COD along with data collected from the Occupational Field (OccFld) sponsor at Manpower and Reserve Affairs into the Training Resource Requirement Management System (TRRMS) database. TRRMS processes this information and·produces a Training Input Plan mp) reflecting the annual instructional requirements and a four-year instructional plan for each Formal School.

iii The COD gives the details for assignment of students to fortnal courses of Instruction. For each course listed on the TIP, TRRMS generates a Training Quota Memorandum (TQM), which is loaded to the MCTlMS Student Registrar module, an automated information system that enables order-writing commands to assign specific Marines to available course seats. These memoranda translate the annual TIP requirement into actual class seat quotas and form the basis for order ~ritlng.

3. Elements of a CDP (see MCa 1553.2_ Appendix A-1 for a sample COD).

Each element of a COD, in order of presentation (appearing as items #1 through #24 in a COD), is addressed below:

Course Title .. The course title must appear as it is listed in MCO P1080.20_ (JUMPS/MMS Codes Manual) unless a change is required or the POI is for a new course.

LOcation. Record the complete address for each location the course is taught.

Marine Corps Service School Code (Ssel. The SSC must correspond to the SSC listed in MCO P1080.20_ PUMPS/MMS Codes Manual). If the course is new, record "'To be determined."

Other Service Course Number. Use other pertinent service course numbers as provided by other branches of the military. If other service course numbers are not applicable, record UNA."

Militarv Assistance Program Articles and Service List Number. The military assistance program articles and service list number Is a seven digit alphanumeric code used to identify a course intended for foreign military Instruction. If this type of instruction is not applicable, record "NA."

Purpose. Include a concise statement about the goals of the instructional program.

. ~. Provide a list of the main'subjects covered in the course. The list should be comprehensive to Include all topic areas. A list of the T&R events covered by the POI must also be included with the COD as the task list.

4-41

Length (peacetime). Record the total number of Instructional days required for the course. Per MCO 1553.2_ Ch. 1 Par. 6 the peacetime .Instructional week includes an average of 40 hours (8-hour day x 5 work days). Do not Include holidays or weekends where instruction does not occur. TECOM FSTD will reconcile any exceptions, such as holidays, by comparing the number of instructIonal days to the TIP. .

CUrriculum Breakdown (Peacetime). Provide a breakdown of the curriculum in academIc and administrative hours (see Section 3101). The Peacetime instructional week Indudes an average of 40 hours (8-hour day x 5 work days), 35 of which will be academic time (Administrative time exceeding five hours per week must be conducted after hours or justified in a waiver request), For detailed organizational and tracking purposes of Instructional hours, academic hours should be further broken down into methods (e.g., practical application, lecture, demonstration, performance evaluation, written evaluation). Administrative hours should also be broken down into appropriate methods. See MCO 1553.2 ... Ch. 1 pa.3f

Length (Mobilization). Record the total number of instructional days required for the course during wartime mobilization. During mobilization, the instructional week averages 60 hours (10-hour day x 6 days). For courses on three shifts with equipment or facility constraints, the mobilization instructional week averages 48 hours (a-hour day x 6 days). This time Indudes both academic and administrative hours. If the course will discontinue upon mobilization, enter "NA." If the course length Is the same during mobilization as in peacetime, click "Same as peacetime."

Curriculum Breakdown (Mobilization). Provide a breakdown of the curriculum in academic and administrative hours for mobilization. During mobilization, It is likely that academic hours will increase and administrative hours wlll decrease. If the course will discontinue upon mobilization, enter "NA." If the curriculum breakdown is the same during mobilization as in peacetime, click "Same as peacetime."

Maximum Class CapaOtv. Record the maximum number of students who can receive instruction using available resources. Resources include classrooms, meSSing, billeting, equipment, budget, and personnel available.

Optimum Class"Capacity. Record the number of students per class that can take maximum advantage of all the resources (e.g., facilities, equipment, instructional capabilities) available to the school.

Minimum Class capacity. Record the minimum number of students per class that will make the course cost effective.

Class Frequency. Record the number of classes required to support the TIP for the current year.

Student Prerequisites. List the prerequisites that personnel must meet to attend the course. This Information can be found In the Target ,Population Description (TPD) developed In the Analysis Phase and filed at the school.

MOS Received. Record the Military Occupational SpeCIalty (MOS) assigned to the student upon successful completion Of the course. If the course does not result in an MOS assignment, record "None."

4-42

Quota Control. Record the name of the agency managing course quotas. The OccFld sponsor can provide this Information if they are not the controlling agency.

Funding. Record the name of the agency that funds temporary additional duty inCidental for students attending the course. In those instances where the using agency must also bear cost, an explanatory statement must be contained in this section. Courses are funded from a variety of sources, depending upon a number of factors such as student type, length of course, and career track. Basic guidelines for schools to determine the funding source are:

a. Courses over 139 days or 20 weeks at one location are PCS and funded by

MMOA/MMEA.

b. Courses less than' 139 days or 20 weeks may be unit-funded or TECOM-funded. '

c. Entry-level pipeline students - normally funded by MMOA or MMEA.

d.Lateral Move students - may be unit-funded or TECOM-funded.

e. Reserve students - normally funded by MARFORRES.

Reporting Instructions. DeSignate to whom the student will report when arriving for a course of Instruction, to indude information on transportation and directions (both during and after working hours). Contact phone numbers, fax numbers, organizational e-mail, and website addresses are elements that are to be induded.

Also Indude a statement indicating the availability of govemment billeting and messing. Provide telephone number and office contact information to obtain billeting reservations or confirm that govemment quarters are not available. If there is more than one school location, Include a separate set of Instructions for each location.

Instructor Staffing Requirements. Instructor staffing reqUirements are based on the academic course hours and computed in accordance with mo agreements, and are automatically computed by MCTlMS in the Instructor Computation Worksheet of the POI. Although instructor-staffing Increases may be validated based on an approved POI, the POI Itself will not generate a table of organization (T/O) change. A Table of Organization Olange Request (TOCR) must be submitted with the POI to CG, TECOM G-l requesting a T/O change.

1'tiis section of: the COD listS the school's T/O number and Its date/arid the Instructor and IristructOr' supervisor billets by line number"grade,billefname, MOS reqUirements, and number, indicating those line numbers not'CUn'eiitly'~"ed; The.

Instructor Computation Worksheet (ICW) used to compute requlfement5should be ..

induded as an a ppendil< cto the COD with the POI. Additional comments as to ...

whether the billetls't1"jCf.'~I1~~lare required. .. .-.

: ··_w ::~:.:)~i~ ;":. .. .~ ::~:::':i:.

For FLCs 1.:a~6ther,setvice's location, refer to MCO;lS ;: ·:.~hd .•• compute Instructor- )i~tlirements using the Inter-servlcetr~lhr:": ,,'-~' Organlzatlor'l (ITRO)'er:computation fOrmula. The ITRO M~'ri~W~r Analysis Subcommittee Proo· __ ,__ a~~afdetaUing this formula maybe OBtalheciby, . . .... , contacting TECOMG~3:Y:;;~:~>': ' , .,. '._

4-43

PROGRAM OF INSTRUCTION (POI)

The POI serves as a FLC plan for implementing and evaluating a formal course of instruction. A POI is the management tool for conducting a course. At a minimum, a FLC must have a locally approved (by signature of FLC commander) POI for every course of instruction it delivers. For each school, the POI is used as an important element in the documentation and historical record that reflects the evolution of a course. Accordingly, a copy of the POI Is maintained at the school to document this evolution

POI Development Process.t Using the information from the approved COD, the FLC will develop the POI. MCO 1553.2-/ details management of Marine Corps FLCs, contains POI submission and approval reqUirements and procedures. The curriculum module of the Marine Corps Training information Management System (MCTIMS) is used to develop the POI.

POI Content Requirements. Development of the POI primarily Involves the consolidation of materials produced during the AnalySiS and Design Phases. MCO 1553.2_ mandates minimum POI content requirements. Any additional items to the POI must be cleared for inc/usion by TECOM (GTD/ATD) prior to submitting the POI.

POI requirements listed in order:

(1) JiUe Page. The title page provides information necessary to identify the document. This includes the course title, SSIC, school name/address, and effective date. The effective date is left blank until the POI is approved, then the date approved is recorded. Each time a revised POI is approved, the new approval date is recorded.

(2) Certification Page. The signed certification page Signifies that the CG, TECOM has reviewed and approved the POI. The approved. POI directs the school commander/director to Implement the course of instruction. Local approval of POls Is not authorized.

(3) Record of Changes Page. The record of changes page is a chronological log of all changes made to a POI. Each entry must indicate the change number, date of change; date received, date entered, and the signature of the individual entering the change.

(4) Table of Contents. This table details the content of the POI and is arranged by section number and section title. The table of contents should include by section the·followlng: COD, Summary of Hours, Scope of Annexes, Concept Cards, Student Performance Data, and Distribution list.

4-44

NAVMC 1553.1·

See…