319 CTS CDG 24 Feb 21_Redacted.pdf

UNCLASSIFIED

319TH COMBAT TRAINING SQUADRON

Curriculum Development Guide

Current as of: 24 Feb 21

Approval Signature

319TH COMBAT TRAINING SQUADRON

Peterson AFB, Colorado

TABLE OF CONTENTS

1 INTRODUCTION

2 INSTRUCTIONAL SYSTEM DEVELOPMENT (ISD)

2.1 Cognitive Learning Domain

2.2 Level of Learning Objectives

3 319 CTS CURRICULUM DEVELOPMENT PROCESS

3.1 319 CTS Course Material Modification Process

4 319 CTS COURSE MAINTENANCE PROCESS

4.1 Course Alignment

5 319 CTS COURSE CONTROL DOCUMENTS (CCD)

5.1 Course Requirements Document (CRD)

5.2 Course Resource Estimate (CRE)

5.3 Course Development and Maintenance Plan (CDMP)

5.4 Course Syllabus

5.5 Master List of Objectives (MLO)

5.5.1 Bloom’s Taxonomy Verbs

5.5.2 Course Objectives (COs)

5.5.3 Block Objectives (BOs)

5.5.4 Lesson Objectives (LOs)

5.5.5 Samples of Behavior (SOBs)

5.5.6 Criterion Referenced Objectives (CROs)

6 COURSE MATERIALS

6.1 Lesson Plan Master

6.2 Activity Plans

6.3 Instructional Aids

6.4 Optional Course Material

6.4.1 Student Readers ........................................... Error! Bookmark not defined.

6.4.2 Supplemental Reading

7 MEASUREMENTS

7.1.1 What to Measure

7.1.2 Test Length

7.1.3 Selection and Arrangement of Test Items

7.2 Measurement Types

7.2.1 Objectives for Written Tests

7.2.2 Objectives for Performance Tests

7.3 Measurement Plan

7.4 Matching Test Item Types to Levels of Learning

7.4.1 Knowledge

7.4.2 Comprehension

7.4.3 Apply

7.4.4 Analyze

7.4.5 Synthesis

7.4.6 Evaluation

7.5 Written Tests

7.5.1 Assembling Written Tests

7.5.2 Written Test Instructions

7.5.3 Grading Rubric

7.5.4 Quizzes

7.6 Performance Tests

7.6.1 Knowledge/Comprehension

7.6.2 Apply/Analyze

7.6.3 Grading Rubric

7.7 Test Review/Feedback

8 DEBRIEF PROCESS

9 CDG TEMPLATES

10 CONCLUSION

11 GLOSSARY OF ACRONYMS

12 REFERENCES

1 INTRODUCTION

In accordance with AFI 36-2201, Air Force Training Program, Instructional System Development (ISD) will be incorporated in all Air Force training. Therefore, ISD is used to develop the 319th Combat Training Squadron (319 CTS) Course Development Guide (CDG) and course material. The purpose of the CDG is to provide information, tools, and templates for the development and maintenance of courses taught at the 319 CTS.

Accordingly, this guide provides:

• ISD Quick-Reference material

• Guidance for designing, modifying, and maintaining 319 CTS course materials

• Structure for developing new and revising existing instructional materials

• Definitions, examples, and templates for support materials

The CDG is the standard for developing and revising course material and will be used by the 319 CTS Office of the Dean of Academics (DEO) to evaluate products and processes implemented at the school. CDG Guidance and templates are approved by the 319 CTS Dean (319 CTS/DE) and will be used to develop all curriculum at 319 CTS. Any deviation from the CDG guidance and templates must be approved in writing by the 319 CTS/DE.

This CDG applies to all personnel who plan, analyze, design, develop, implement, and evaluate curricula for 319 CTS courseware. This CDG is to be used in conjunction with the 319 CTS OI 36-2203.

2 INSTRUCTIONAL SYSTEM DEVELOPMENT (ISD)

“Instructional System Development (ISD) is the official Air Force process for developing education and training for Air Force personnel.” (AFMAN 36-2234, Instructional System Development, 1 Nov 93).

ISD provides a systematic process for the development of curriculum within the Air Force. The process is a quality improvement process consisting four chronological phases (Analysis, Design, Development, and Implementation). The phases are embedded within the system functions with evaluation being the central feedback network for the total system. In addition to the four phases, there is a fifth phase (Evaluation) which takes place in every phase and throughout the life cycle of the training system to ensure an effective and cost-efficient system is developed in order to produce graduates with the necessary skills, knowledge, and attitudes to successfully perform their jobs. For a breakdown on the activities conducted in each function and phase, reference AFMAN 36-2234 and AFH 36-2235 v10.

Use of the ISD process in the Air Force is directed in Air Force Policy Directive (AFPD) 36-26, Total Force Development and Management and Air Force Instruction 36-2201, Air Force Training Program. In addition, AFMAN 36-2234, Instructional System Development, provides information on the principles and processes for developing Air Force education and training programs using the ISD process, and Air Force Handbook (AFH) 36-2235 Volumes 1 – 13, Information for Designers of Instructional Systems, spells out the application of the ISD process to different types of education and training programs within the Air Force.

The systematic approach ties instruction to what the students will be able to do after instruction that they could not do before. Objectives in educational curricula deal largely with cognitive learning (knowledge) and affective learning (attitudes) rather than with psychomotor skills. 319 CTS measures student success when student performance demonstrates that learning of the subject matter and attitudes took place.

2.1 Cognitive Learning Domain

The “cognitive domain” refers to Bloom’s Taxonomy of Educational Objectives. This domain focuses on content knowledge and the development of intellectual abilities and skills. 319 CTS courses utilize Bloom’s Taxonomy for developing objectives and measurements, planning instruction, and identifying required levels of learning (levels of expertise) within each course. In this domain, learning focuses on the cognitive (thinking) behaviors of learners. The majority of objectives used in 319 CTS courses are cognitive objectives to identify the cognitive (thinking) behaviors required of the students. Cognitive learning behaviors, or learning outcomes, are classified into six levels of learning:

knowledge, comprehension, application, analysis, synthesis, and evaluation. The different levels depict a cumulative hierarchy of cognitive processes that range from the simplest thinking skills (knowledge), also known as lower order thinking skills, to the most complex thinking skills (evaluation), also known as higher order thinking skills. It is a general rule of thumb with Bloom’s Taxonomy that in order to reach a higher level of learning, the level(s) below it must be achieved first.

Air Force Manual (AFMAN) 36-2236, Guidebook for Air Force Instructors, 12 Nov 2003, define each of the levels of learning as follows:

• Knowledge: Recall previously learned material (facts or theories) in essentially the same form taught.

• Comprehension: Identify relationships, concepts, and abstractions beyond the simple remembering of the material. Typically involves translating, interpreting, and estimating future trends.

• Application: Use learned material in new and concrete situations including the application of rules, methods, concepts, principals, laws, and theories.

• Analysis: Break down material into its component parts so the organizational structure may be understood, including identification of the parts, analysis of the relationships between the parts, and recognition of the organizational principals involved.

• Synthesis: Put parts together to form new patterns or structures, such as a unique communication, a plan of operation, or a set of abstract relations.

• Evaluation: Judge the value of material for a given purpose. Learning in this area is the highest in the cognitive hierarchy because it involves elements of all the other categories, plus conscious value judgments based on clearly defined criteria.

2.2 Level of Learning Objectives

319 CTS courses use Course Objectives (CO), Block Objectives (BO), Lesson Objectives (LO), and Criterion Referenced Objectives (CRO). Most 319 CTS objectives are written as cognitive levels of learning objectives because these types of objectives are the most applicable for academic classroom based education and training as defined by Air Force Handbook (AFH) 36-2235, Volume 10 and AFMAN 36-2236. Level of learning objectives contain a specific level of learning and a clearly stated subject expressed in terms of what students are expected to achieve. The level of learning objective has two parts: the objective and the sample of behavior (SOB).

The first part of a level of learning objective is the objective. The objective is written as a statement using a non-observable descriptor, which identifies the level of learning that the student should attain (refer to the six levels of Bloom’s Taxonomy, paragraph 2.1).

The first word of the lesson objective designates the desired level of learning to be attained and is not an observable and measurable behavioral descriptor. However, the statement clearly identifies what the student should learn.

For example: Lesson Objective 1 - Know the six classical orbit elements.

The second part of a level of learning objective is the SOB. SOBs define what the student must do to prove they have met the objective. SOBs will use the verbs listed in Attachment 14 – Bloom’s Taxonomy Verbs and Level of Learning. The verbs used to describe the sample of behavior will inform the measurement. For example: Lesson Objective 1 – Know the six classical orbital elements. SOB 1 – List the six classical orbital elements.

The second part isn’t applicable to CROs.

All objectives for a 319 CTS course will be listed in the course Master List of Objectives (MLO). Further explanation on the three types of objectives can be found in the MLO section of this guide. Reference AFMAN 36-2236 and AFH 36-2235 v10 for further guidance on developing level of learning objectives. Reference Attachment 5 – Master List of Objectives Template (MLO) for more information on the MLO.

3 319 CTS CURRICULUM DEVELOPMENT PROCESS

As previously mentioned, the ISD process has four phases: Analysis, Design, Development, and Implementation.

319 CTS curriculum development process starts in the Analysis Phase by drafting a Course Requirements Document (CRD). The Analysis Phase is focused on defining the training audience and the COs for meeting the training audience’s needs. During the Analysis Phase, task analysis, learning analysis, resource analysis, and audience analysis are conducted to gather information concerning course needs and requirements.

This information is documented in the CRD. The 319 CTS/DE with support from the 319 CTS Flights, will develop the CRD or recommend significant changes to an existing CRD with inputs from stakeholders. A stakeholder is an individual or group with an interest in the success of a course in fulfilling its mission. Generally, stakeholders are external to the schoolhouse but have a strong influence in course outcome. This influence will help determine course requirements and objectives during the Analysis and Design phase.

During the Design Phase, course materials are planned out. To start, LOs are derived from data received during the Analysis Phase (tasks, learning goals). As such, all subordinate LOs created must support attainment of the COs. BOs are required if the course is broken down into blocks. LOs will also support attainment of the BOs. Test items/measurements are also created in the Design Phase with or immediately after the LOs to ensure the test item correctly measures the LO. See AFMAN 36-2234 for further guidance.

The CRD created in the Analysis Phase is used with information obtained in the Design Phase to develop the MLO, the Course Resource Estimate (CRE), and the Course Development and Maintenance Plan (CDMP). Once the MLO, CRE, and CDMP are developed, curriculum developers move to the Development Phase and create the rest of the course materials.

In the Development Phase, course materials including lesson plan (LP) masters, the course syllabus, training aids, and additional course materials are created. This phase culminates with the beginning of validation of the curriculum.

During the Implementation Phase, new lessons are initially taught or additional course materials are used. If problems are found, they are noted for correction using feedback instruments before the next course offering occurs and are tracked through the Lessons Learned Database.

3.1 319 CTS Course Material Modification Process

Changes to LPs and supporting instructional materials are implemented through the course material modification process. 319 CTS Flights must ensure course materials remain current and relevant. Evaluation of course materials will be obtained through peer reviews, student critiques (lesson/end of course), Hotwashes, stakeholder coordination, comprehensive reviews, and periodic reviews. Any issues identified will be analyzed and may drive changes to course material. Changes are broken into three categories;

administrative, minor, and major changes. Descriptions of lesson change categories and reviews can be found in 319 CTS OI 36-2203.

4 319 CTS COURSE MAINTENANCE PROCESS

All 319 CTS Flights are required to maintain their respective courses. This is done through comprehensive reviews of all course materials. This process is driven by the ISD process which requires continuous evaluation of all course materials and is documented in the CDMP. The process starts with a revalidation of Course Control Documents (CCD) starting with the CRD and MLO. Each course will re-analyze these two documents to verify the requirements and objectives are still valid and current. For additional information on the CDMP, reference the 319 CTS OI 36-2203.

CDMP reviews will not exceed 18 months and will review all current course material. In addition, comprehensive reviews will be conducted on a course, its CCDs, and any related course material as determined by the 319 CTS/DO or the 319 CTS/DE.

The level of course modification (Major, Minor, or Administrative) will be determined by the complexity of revisions or rewrites required to update the courses. For additional information on course maintenance reference the 319 CTS OI 36-2203, para 6.7.

4.1 Course Alignment

Alignment of the elements of a course are critical for an effective instructional system and successful student learning. The curriculum (objectives, LPs, teaching aids/materials) and assessments must be in constant alignment throughout every 319 CTS course. If an objective is changed, reviewing the associated assessment item, LP, and teaching materials is necessary to ensure they align in terms of content and level of learning with the revised objective. If an assessment item is changed, the associated objective and LP must be reviewed to ensure proper alignment. Instruction may also have to change depending on the changes made to the curriculum (e.g. may have to change from informal lecture to cooperative learning). Always be thinking about alignment when developing or revising any aspect of a course.

5 319 CTS COURSE CONTROL DOCUMENTS (CCD)

5.1 Course Requirements Document (CRD)

The CRD is the CCD used to initiate course analysis after an instructional need has been identified by an authorizing agency or within 319 CTS. The CRD is also used to direct change to an existing course (shortening the training days, prioritizing the student, number of offerings, etc.). The CRD will include information regarding the latest course review (who attended stakeholders meeting and a summary of changes from the previous version), purpose of the course, course stakeholders, course objectives, target audience, projected annual course offerings, projected annual student throughput, and security classification. The CRD will be coordinated with stakeholders in order to gain feedback and validate the overall course objective. See Attachment 1 for the CRD template.

5.2 Course Resource Estimate (CRE)

The CRE is the CCD that puts a cost on the development, execution, and maintenance of a course. This document helps 319 CTS leaders and higher headquarters determine if the value added by the course justifies the outlay of funding to start and maintain the course. The document outlines costs and resources associated with manpower, facilities, equipment, supplies, etc. The projected throughput from the CRD will be a driver in the CRE. The CRE is used to develop a consolidated Unit Resource Estimate to secure necessary funding for all 319 CTS courses, or to outline shortfalls in funding. See Attachment 2 for the CRE template.

5.3 Course Development and Maintenance Plan (CDMP)

The CDMP is the CCD used to develop new courses and maintain existing courses. It identifies key milestones and decision points, describes course interdependencies, establishes OPRs for key tasks and sets/tracks deadlines for completion. For existing courses, the CDMP establishes a program for a comprehensive review of CCDs and all associated course material. See Attachment 3 for the CDMP template.

5.4 Course Syllabus

The Course Syllabus provides students with an overview of the course, explains 319 CTS academic policies and procedures, and sets unique course standards, policies and procedures. At a minimum, the course syllabus contains the following (see Attachment 4 for the Course Syllabus template):

• Course Title

• Course Overview

• COs

• Attendance Policy

• Student Assignments

• Homework Requirements

• Grading Policy

• Minimum passing score

• 319 CTS academic freedom and non-attribution policy

• Graduation requirements.

• Course Schedule

5.5 Master List of Objectives (MLO)

The MLO is a single source for all objectives associated with a course being taught at the 319 CTS. It includes COs, BOs (if used), LOs with associated SOBs, and CROs. Most objectives written for 319 CTS courses will be level of learning cognitive objectives. SOBs associated with LOs will be written using action verbs as the descriptors of what the students are expected to be able to do in order to show they have met the LOs. LOs state what the students should learn and at what learning level. SOBs provide observable, measurable behaviors that the student should be able to accomplish after learning the material. Objectives are written in a hierarchical process starting with the overall COs, and then BOs, followed by LOs which are further refined using SOBs. If written correctly, the SOBs will ensure achievement of the LOs, the LOs will ensure achievement of the BOs, and the BOs will ensure achievement of the COs which ensure achievement of the course goals. The MLO will be coordinated with stakeholders in order to gain feedback and validate the overall course content. See Attachment 5 for the MLO template.

5.5.3 Bloom’s Taxonomy Verbs

LOs begin with descriptors from Bloom’s Taxonomy which describe levels of learning:

Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation. The SOBs supporting the LOs start with action verbs to create a measurable, observable task/behavior. Attachment 14 provides a brief statement describing each of Bloom’s Taxonomy levels of learning, followed by common corresponding behavioral verbs for stating specific learning outcomes within that level of learning.

5.5.3 Course Objectives (COs)

COs are created first and developed based on the course goals. These are very broad, overarching objectives that identify the learning outcomes expected of students upon completion of the course. A course may have anywhere from 2-6 overarching objectives.

The number of COs depends on the amount of material being taught, the diversity of the material, and what students are expected to learn throughout the course. COs are written using one of the six verbs from Bloom’s Taxonomy cognitive domain (Know, Comprehend, Apply, Analyze, Synthesize, Evaluate) to clearly identify the level of learning required. COs are also listed in the CRD.

5.5.3 Block Objectives (BOs)

To provide a structure to the learning environment and enable proper sequencing, COs may also be broken down into larger, logical instructional units called blocks. Blocks facilitate proper sequencing of multiple lessons as well as provide opportunities for formative evaluations and/or progress checks. Lessons may be clustered into blocks based on COs that form prerequisites to future instruction, related in the same way, require a common skill/knowledge, or other construct that facilitates an effective learning environment. Blocks/BOs are not required but they help break the course down into logical, manageable instructional units for effective learning. BOs are written using one of the six verbs from Bloom’s Taxonomy cognitive domain (Know, Comprehend, Apply, Analyze, Synthesize, Evaluate) to clearly identify the level of learning required.

5.5.3 Lesson Objectives (LOs)

Within blocks of instruction, the topics are broken down into manageable pieces called lessons. In most cases, lessons can be thought of as a single topic to be covered in the classroom by a single instructor in a reasonable amount of time. The behaviors students are required to meet for each lesson are written as LOs. LOs help determine the content for a lesson. The LOs are detailed down to a level where we can break it into pieces small enough to evaluate. There should be a clear link between the levels of learning directed by the COs/BOs and those specified in the subordinate LOs. This is not to say that if a BO calls for analysis level all subordinate lessons must be at the analysis level, but the body of instruction should lead students to reach the desired learning levels. LOs are written using one of the six verbs from Bloom’s Taxonomy cognitive domain (Know, Comprehend, Apply, Analyze, Synthesize, Evaluate) to clearly identify the level of learning required.

5.5.3 Samples of Behavior (SOBs)

LOs are not inherently measurable and observable so they require SOBs to measure them. SOBs translate the directed level of learning into statements of specific, measurable, observable, behaviors. SOBs can be thought of as the behaviors the student should be able to execute (display) if they have successfully met the LO at the appropriate level of learning (desired student outcomes of learning). This is accomplished by appraising the student based on SOBs associated with the LOs. Similar to subordinate LOs, SOBs do not ALL have to match the level of learning identified in the LO, but taken as a whole must provide ample evidence the student has reached the desired level of learning and learning outcome identified in the LO. Since measurement tools are derived from SOBs, lesson developers must ensure the tools assess students at the correct learning level. SOBs are written using action verbs associated with the six levels of Bloom’s Taxonomy cognitive domain to clearly identify the action and level of learning required. 319 CTS approved action verbs are listed in Attachment 14. Note: SOBs are only used with LOs.

5.5.3 Criterion Referenced Objectives (CROs)

CROs are statements that specify precisely what student behavior is expected. Complete criterion objectives are made up of three parts:

• Behavior - What the students will be able to do (for example, characterize the signal parameters).

• Condition - Environment in which they will do it (for example, given a Keysight N9030B PXA Signal Analyzer).

• Standard - How well they will do it in order to demonstrate that they have learned (for example, "...within ten minutes")

CROs are typically used in technical instruction situations (e.g. operating equipment, performance in a simulator, or on the ops floor). This format of objective usually focuses on physical skills and abilities instead of cognitive processes. CROs do not require SOBs to measure them as they are inherently measurable and observable. In 319 CTS courses, they will be used as applicable to identify performance behaviors such as operating equipment or performing tasks in a process. Reference AFMAN 36-2236 and AFH 36- 2235 v10 for further guidance on developing CROs.

6 COURSE MATERIALS

6.1 Lesson Plan Master

A LP Master is a guide for the instructor to use when preparing for and while teaching in the classroom. It contains lesson background information, Lesson Objectives and SOBs, the main points of the lesson, strategy statement, attention steps, student motivation, and other material and/or information that may be needed by the instructor to execute the lesson. The LP Master must contain enough information to adequately present all LOs with sufficient facts, supporting information, examples, and questions attention steps, and student motivation over the specified period of time. The primary purpose of the LP Master is to provide the instructor presenting the material a roadmap of the lesson. In addition, a well-developed LP Master will allow an instructor to properly prepare for a lesson.

PowerPoint is the predominant instructional aide used in 319 CTS courses and all LP Masters are built using it. See Attachment 6 of the CDG folder on the sharedrive for the LP template.

In accordance with the 319 CTS Operating Instruction, Flt/CCs approve lessons in their courses. A Lesson Change Log must be maintained in each lesson that lists lesson changes and Flt/CC approvals. 319 CTS Flights will maintain the LP Master and ensure the file is protected from inadvertent/unauthorized editing. Approved LP Masters will reside solely in the MCF folder. 319 CTS Flights will store LP Masters in the MCF consistent with the 319 CTS/DEO provided structure documented in the 319 CTS OI.

Major or minor changes require two people from the flight for the change process – one to make the changes and one to review.

Instructors may create a personalized LP for use during classroom presentation.

Instructors must ensure personalized LPs are in scope for the course and include all mandatory pieces of an LP. Personalized lessons must not change the LOs or SOBs, measurement methods, template format, or overall length of the course. Personalization is only allowed in the following notes sections of the LP: Attention, Motivation, Summary and Re-motivation. Instructors may add relevant examples, questions, notes, etc. based on their personal experiences or research. Regardless of the amount of personalization, all LOs/SOBs must be taught to ensure all students are taught the same information regardless of who teaches the lesson. Information added to a personalized LP may be considered for incorporation into the LP Master through the formal change process if it is beneficial to meeting the objectives and student learning. The Flt/CC and the lesson SME must approve personalized lessons to ensure lessons meet the intent of the course and scope of the course.

Guest instructors from outside the 319 CTS are not required to prepare formal LPs.

However, 319 CTS Flights will have a LP Master in the MCF covering the LOs and SOBs being presented by the guest instructor. A certified instructor for that lesson will ensure all LOs and SOBs are covered.

6.2 Activity Plans

Activity Plans are used to document activities such as tours or other extracurricular activities that are applicable to course tasks and reinforce previously taught Lesson Objectives and Samples of behavior. Activity Plans are required for all scheduled events/locations that do not have prescribed Lesson Objectives or Samples of Behavior.

Activity plans can be used to strengthen concept understanding or where content is changing so rapidly that formalized objectives are inappropriate. All activities must have an Activity Plan that describes the activity, the justification for the activity and the goal of the activity. While activities can be very beneficial to a course, they can only occupy a very small percentage of the total course.

The activity plans will have a lesson linkages section that identifies with LOs/SOBs the Activity Plan supports. Lessons that are supported by an Activity Plan will state which Activity Plan reinforces the LOs/SOBs in the Lesson Plan. See Attachment 7 for the Activity Plan template.

6.3 Instructional Aids

Instructional aids include anything used in the classroom to assist students in their learning. All instructional aids must be professional in appearance and must not distract from the material being presented. They must also be marked with classification markings as appropriate. Instructional aids can be, but are not limited to the following:

• Whiteboard

• Bulletin Board

• Pictures

• Charts, Diagrams, and Graphics

• Models, Mockups, and Cutaways

• Simulations

• Videos & Audio Recordings

• PowerPoint Presentations

6.4 Optional Course Material

Student Note Takers Guides, Study Guides, and Supplemental Readings are optional material for 319 CTS courses. The decision to have these types of course material should be made by the 319 CTS Flights in coordination with the instructors. Information for using these items in a lesson should be documented in the Lesson Plan.

6.4.1 Student Note Takers: Student Note Takers, if used, are designed to follow the instructor presentations with the goal of assisting the students in retaining knowledge.

They follow the sequence of the material presented in class and are detailed enough to provide students a way to have stand-alone review of the course material. The headings in a Study Guide should be the same as those used in the main topics/subject areas as presented in the classroom. When preparing a Study Guide, start with the LP Master as an outline to place the information in correct order. Study Guides will:

• Always include LOs and SOBs as they appear in the MLO

• Follow the format and sequence of instruction

• Use illustrations and tables as needed to enhance learning

• Contain the required classification markings

• Include enough content to support the lesson and provide the student a method for review

6.4.2 Study Guides

Study Guides are different than Student Note Takers in that they are not designed for the student to follow the material in the same order and depth as the presentation in class.

They are designed to assist the student in studying the course material for knowledge retention, homework assignments and preparation for quizzes and tests. The Study Guide supplements the material presented in class. It provides additional readings and material related to the lesson topics. The material may closely reflect items covered in class but may also include material not directly covered in class. If an instructor assigns material in a Study Guide as homework, it must be related to a Lesson Objective in the course and the Lesson Plan should explain how the reading reinforces concepts in the lesson and provide information to the instructor on how to incorporate the reading.

6.4.3 Supplemental Reading

From time to time, an instructor may find a pertinent article or information that fits well with the lesson they are presenting after the Study Guide and/or Student Reader are printed. The instructor should feel free to give that material or links to access the material to the students in the classroom when the lesson is presented by treating it as Supplemental Reading. They are also not necessarily designed to assist the student in studying the material in preparation of a test.

7 MEASUREMENTS

7.1 Purpose of Testing

The primary purpose of testing is to assess the student's attainment of the behavior specified in the objective. Tests also serve several secondary purposes such as to:

• Identify problems or weaknesses in the instruction

• Indicate whether a class is performing up to standards on specific objectives

• Indicate instructor proficiency

To ensure that tests adequately measure the objectives they support, the performance required in the test must match the performance required in the objective.

7.1.1 What to Measure

Determine what to measure from the objective to be covered by the test. Tests should measure application of principles, knowledge of factual information, ability to perform tasks, and transfer of knowledge and skills to solve similar problems. The performance expected for higher levels of learning is more demanding, complex, and abstract. In addition, supply type test items may be designed so one item measures several Samples of Behavior simultaneously.

7.1.2 Test Length

Adequate coverage of the Course Objective and time available for testing are the major factors in determining test length. If time available for testing does not allow for administering a test large enough to measure all Lesson Objectives, consider creating several smaller tests that could be administered sequentially throughout the block. Longer tests are considered more reliable than shorter tests because they cover more material, more completely. However, the overall content validity of a series of short, related tests can duplicate the validity of a larger test.

7.1.3 Selection and Arrangement of Test Items

Select test items that cover the most important points of the material. Test items should be clear, concise and well written to minimize misunderstandings. Tests can be arranged by topic or by level of difficulty. See para 7.5.1 for additional guidance.

7.2 Measurement Types

Instructors need a means to measure the effectiveness of instruction to identify whether students have attained the LOs and CROs specified in the MLO. There are a variety of ways to measure achievement of both, including formal tests (multiple choice, fill in the blank, short answer, essay, combinations, etc.), capstone projects, written papers, classroom presentations, exercises, etc... However, the majority of 319 CTS course measurements utilize tests. Tests can be classified into two main groups: written tests and performance tests.

7.2.1 Objectives for Written Tests

LOs are generally measured with written tests. However, exercises, capstones and case studies can also be used to measure these objectives depending on the activity required by the student to demonstrate learning. Written tests are typically used to assess the expected behavior in the cognitive and affective domain.

There are two types of written tests:

• Selection Tests: Requires the student to choose the correct answer from several provided. Examples are:

o Multiple choice o True-false o Matching

• Supply Tests: Requires that the student supply the answer from recall. Examples are:

o Essay o Completion (short answer) o Labeling (identification)

7.2.2 Objectives for Performance Tests

CROs are generally measured with performance tests that are inherently observable and measurable by a predefined set of standard, behavior, and conditions. Performance tests are typically used to assess the expected behavior in the psychomotor domain.

There are two types of performance tests:

• Product Tests: Requires the student to produce a product that is tangible, permanent, and observable. The product itself displays the student’s success or failure in mastering the performance. The product can be measured directly and at any time by the test administrator. Examples include assembled a piece of equipment, a filled-in form, data analysis report and a hole in a target.

• Process Tests: Requires the student to use verbal skills or procedures are observable but temporary. The performance must be observed and rated at the time it occurs. Examples include physical actions such as operating equipment or teaching a lesson.

The LOs, SOBs, and CROs dictate test questions and other measurements. Test items are used to measure the attainment of the LOs. All LOs and CROs will be measured in 319 CTS courses. Curriculum developers select test items that cover the most important points of the material. Test items should be clear, concise, and well written to minimize misunderstandings.

7.3 Measurement Plan

A Measurement Plan indicates how the objectives in a course will be measured. In some Air Force schools, the Measurement Plan is a separate document in the form of a chart matching every LO/SOB to specific test questions or other measurement tools. At the 319 CTS, this information is annotated in the LP in the Lesson Overview section. It is extremely important in the ISD process that measurements cover the LOs/SOBs correctly to verify learning occurs at the designated level.

7.4 Matching Test Item Types to Levels of Learning

Test items are used to measure the attainment of LOs by having students demonstrate the associated SOBs. The most fundamental and important part of writing test items is ensuring the test item measures the LOs/SOBs at the correct level of learning.

7.4.1 Knowledge

Bloom’s definition of knowledge in the cognitive domain is, “Recall previously learned material (facts or theories) in essentially the same form taught.” This means the student is required to write, state or perform in exact terms or memorize exactly the content of the instruction. A test item may require the student to look at or read alternatives and recognize the correct answer. The correct answer has been presented during instruction either in the visual aid (Lesson PowerPoint presentation), provided by the instructor verbally or provided in other course materials or activities.

Sample question stems for knowledge measurements:

• How many...?

• Can you name the...?

• Recall what happened when...?

• Define the meaning of...?

• What is...?

• Which is true or false...?

Examples:

• List the GEO satellite constellations.

• Label the satellite subsystems.

• Match the capability with the satellite payload.

• Define electromagnetic interference.

7.4.2 Comprehension

Bloom’s definition of comprehension in the cognitive domain is, “Identify relationships and concepts beyond the simple remembering of the material. Typically involves translating, interpreting, and estimating.”

A test item may require the student to paraphrase or approximate what has been taught during instruction. A test item could present the student with a novel practical example or situation. It asks the student to discuss or produce the correct answer or procedure.

Sample question stems for comprehension measurements:

• Can you write in your own words...?

• Describe how … works.

• Interpret the information found in…

• What do you think could happen next...?

• Can you distinguish between...?

• What differences exist between...?

Examples:

• Translate joint doctrine into operational requirements.

• Summarize the differences between WGS and Milstar.

• Describe the JSIR process.

• Discuss joint space strategy roles and responsibilities.

7.4.3 Apply

Bloom’s definition of application in the cognitive domain is, “Use learned material in new and concrete situations including the application of rules, methods, concepts, principals, laws, and theories.” The student will need to perceive job performance condition cues and generate judgments as to whether what the cue means. Is it an indicator of an abnormal or emergency condition and what action is required?

Sample question stems for application measurements:

• Do you know another instance where...?

• Could this have happened in...?

• Can you organize by characteristics such as...?

• What factors would you change if...?

• Can you apply the method used to another situation...?

• What questions would you ask of...?

• From the information given, can you develop a set of instructions about...?

• Would this information be useful if you had a ...?

Examples:

• Use Newton's Laws of Motion to explain how a model rocket works.

• Using the information you have learned about aerodynamics, construct a paper airplane that minimizes drag.

• Apply the rules of thumb to perform dB math.

• Using the information you have learned about mixed numbers, solve the following questions.

• Demonstrate how changing the location of the fulcrum affects a tabletop lever.

7.4.4 Analyze

Bloom’s definition of analysis in the cognitive domain is, “Break down material into its component parts to understand the structure, including parts identification, analysis of the relationships between the parts, and recognition of the principals.”

Sample question stems for analyze measurements:

• Which events could have happened...?

• If ... happened, what might the ending have been?

• How was this similar to...?

• What was the underlying theme of...?

• What do you see as other possible outcomes?

• Why did ... changes occur?

• Can you explain what must have happened when...?

• How is ... similar to...?

• What are some of the problems of...?

• What were some of the motives behind...?

• What was the turning point in the game?

• What was the problem with...?

Examples:

• Analyze General Hyten’s motives for creating the Space Mission Force.

• Examine the results of the START treaty.

• Dissect the reasons the satellite did not reach its orbit.

7.4.5 Synthesis

Bloom’s definition of synthesis in the cognitive domain is, “Put parts together to form new patterns or structures, such as a plan of operation or a set of abstract relations.”

Sample question stems for synthesis measurements:

• Can you design a ... to ...?

• Can you see a possible solution to...?

• If you had access to all resources how would you deal with...?

• Why don't you devise your own way to deal with...?

• What would happen if...?

• How many ways can you...?

• Can you create new and unusual uses for...?

Examples:

• Revise the plan to colonize Mars.

• Design a new satellite.

• Formulate a process for EMI mitigation.

• Plan a wargame.

7.4.6 Evaluation

Bloom’s definition of evaluation in the cognitive domain is, “Judge the value of material for a given purpose. This involves elements of all the other categories, plus conscious value judgments based on clearly defined criteria.”

Sample question stems for evaluation measurements:

• Is there a better solution to...?

• Judge the value of...

• Can you defend your position about...?

• Do you think ... is a good or a bad thing?

• How would you have handled...?

• What changes to ... would you recommend?

• How effective are...?

Examples:

• Evaluate the accuracy of the movie Gravity.

• Recommend a course of action in response to the North Korean threat.

• Appraise the joint space strategy for a contested, degraded operationally limited environment.

• Assess the likelihood of success.

7.5 Written Tests

When developing written tests, be clear and brief, avoid wordiness and be grammatically correct. Also ensure the distractors are parallel in content. Construct incorrect choices that are plausible to students having varying degrees of information or misinformation.

Incorrect choices can be derived from actual wrong answers given by students during instruction, from knowledge of common misconceptions, or from judgment of misconceptions students are likely to hold based on the instructional material. Keep all alternatives relatively equal in length. List choices containing numbers in descending or ascending order.

Remember, there are two types of written tests, selection and supply. Selection test items require students to choose the correct answer from several provided. Common selection type test items include multiple choice, true-false, and matching questions. Supply items require students to supply the answer from recall or formulation of thoughts and ideas.

Supply type test items include labeling, completion/fill in the blank, short answer, and essay questions. See AFMAN 36-2236, AFMAN 36-2234 and AFH 36-2235 Vol 12 for more information on creating tests and measurements.

Not all written test formats are ideal for every level of learning. The following table is a Knowledge-level/Type of Test Item Grid for Bloom’s Taxonomic levels of learning. Within each cell of the grid, a (++) indicates the particular test item is ideal for testing the particular level of learning. A (+) indicates the test item is satisfactory for testing the level of learning. A (-) cell indicates the test item is not acceptable for testing that particular level of learning (AFH 36-2235 v 12, pg. 205).

Types of Written Test Items for Bloom’s Levels of Learning

True or False

Short Answer/ Essay Matching Multiple

Choice Essay

Knowledge ++ ++ ++ ++ +

Comprehension + ++ ++ ++ +

Application - + ++ ++ ++

Analysis - + + ++ ++

Synthesis - - + ++ ++

Evaluation - - - + ++

When drafting a written test, each separate written test item will include the following information:

• Classification (if part of a classified test)

• Short Title of Lesson

• Lesson Number

• SOB/Lesson Objective the Item is Measuring

• Test Question

• Correct Answer and Distracters o For short answer and essay questions, the anticipated response should be provided with key points in bold or underlined.

7.5.1 Assembling Written Tests

There are different ways to arrange written tests. Two common arrangements include:

Logical Order and Simple to Complex. Logical Order involves arranging the test questions according to the sequence in which the objectives are taught. Simple to Complex involves placing the easy questions first in the test and then progressing to the more difficult ones without regard to the order in which the material was taught. This latter arrangement builds confidence in the test taker, since they start out being surer of the answers to the early questions on the test.

7.5.2 Written Test Instructions

Standardized instructions will be given to the students before the test is administered.

These instructions will be printed on the cover page of the test. In some cases, these instructions may need to be given verbally by the test administrator or may be delivered electronically for software based testing. The instructions should include information on the amount of time allotted, how to get help if needed, what to do when finished, where to record answers, the total point value of the test, how much each question is worth, whether or not resources such as a calculator or book can be used, etc.. See Attachment 8 for the Test Template.

7.5.3 Grading Rubric

Some supply tests require a grading rubric. Grading rubrics should be used for any test type that does not have a specific right or wrong answer (e.g. subjective tests). The grading rubric must identify the number of points available, grading criteria, and detailed criteria for credit. The grading rubric allows for answers to be graded subjectively and reduce variations in grading. Include all important details such as key words, phrases, and response time metrics for each test item. See Attachment 10 for the Grading Rubric Template.

7.5.4 Quizzes

In an educational context, a quiz is usually a form of a student assessment. It often has fewer questions of lesser difficulty and requires less time for completion than a test.

Quizzes can help both the instructor and the student assess understanding as they transition through a block of instruction. There are many different types of quizzes that can be used in the classroom. Use the common sense approach when developing or administering quizzes and use the test section of this CDG as a guide to ensure they meet ISD requirements. See Attachment 9 for the Quiz Template.

7.6 Performance Tests

When the Lesson Objective requires the student to actually demonstrate a skill instead of just theorizing, a performance measurement should be used. Performance test items rate the student on the actual performance output defined in a specific performance objective.

A student’s ability to perform a task is evaluated by observing student behavior and https://en.wikipedia.org/wiki/Test_(student_assessment) judging it against a pre-defined standard. If the specific output is a motor skill (or process), the student is required to display the performance. If the specific output is a product, the student is required to produce the product. Performance tests directly measure training requirements and usually require judgment by a test administrator.

Each student should be tested under conditions that provide the best chance to display the skill or product. The test conditions should not change from one student to another.

The test administrator must observe student performance and rate the performance according to a fixed standard.

Each performance test will have an activity plan that lists:

• Activity Title

• Activity Number

• Anticipated Length of the Activity

• Activity Overview

• Strategy/Instructor Guidance

• Lesson Linkages

• Grading rubric

7.6.1 Knowledge/Comprehension

When measuring knowledge or comprehension levels of learning in a performance test, the student is required to accomplish a job-like task under controlled conditions.

Emphasize non-verbal aspects. May require individuals to look up, read, and use certain job aids. Test items are the psychomotor skills the individual must perform and the judgments and decisions that must be made to perform the procedure under normal and abnormal job performance conditions. Test items are dependent upon the sequence in which they are presented. Errors made early in the sequence may affect the final outcome of the task.

7.6.2 Apply/Analyze

When measuring application or analysis levels of learning in a performance test, the student is required to demonstrate intellectual skills by responding to various types of written or oral questions and emphasize verbal or symbolic aspects. May require individuals to look up, read, and use certain reference materials. Test items measure job related intellectual skills that the individual must know to perform a job or to make judgments or decisions during job performance. Test items can be independent questions that are not dependent upon sequence or they can be dependent, and errors made early in the sequence can affect the final score on the test.

7.6.3 Grading Rubric

A performance grading rubric must identify the number of points available, grading criteria, and detailed criteria for credit. Include all important details such as key words, phrases, and response time metrics for each test item. See Attachment 10 for the Grading Rubric Template.

7.7 Test Review/Feedback

Following the administration of a test, the 319 CTS flights will analyze the results for any high miss or low scoring problem areas. If there are high miss questions, or low scoring questions, then the 319 CTS flights will need to investigate and determine the cause. It could be the result of a poorly written question, information that was not covered (or inadequately covered) in class, or information in the Lesson Plan is incorrect or outdated.

There can be other reasons as well.