Attachment_11_TSAs_Organizational_Assessment_Services_Item_Writing_Guidelines.pdf

TSA’s Organizational Assessment Services Item Writing Guidelines

General Procedural Concerns

1. Best Answer or Correct Answer Format

- Best Answer: This format has more than one option that may be correct.

However, only one alternative is the best correct answer in terms of degrees of correctness.

- Correct Answer: With this format the correct answer is the one and only correct answer to the question.

2. Writing Distractors: Developing distractors may well be the most difficult aspect of item writing. Distractors must be incorrect, yet must also be plausible to the test taker who does not have complete mastery over the content area the item is tapping.

Distractors should resemble the keyed response in grammatical style, content and length.

3. Trick Questions: Items that are designed to deliberately confuse or mislead the test taker are to be avoided. The following is a list of types of questions generally perceived as being trick questions:

- items representing trivial content

- items where test takers are required to make overly fine discriminations between ideas, areas, or concepts

- items containing window dressing not required to answer the question

- items with multiple correct answers in a “correct answer” format

- items tapping principles in ways that are not required by the job

- highly ambiguous items.

4. Overly specific items: Knowledge which can be tapped in a test can range from abstract to concrete. While the item developer should avoid overly simplistic items, items which are highly abstract should also be avoided.

5. Trivial Material: Avoid constructing items which require the recall of insignificant content. The content of the test items must be relevant to the job.

6. Window dressing: It is important that the stem contain all of the information necessary to make the correct judgments and conclusions to answer the item correctly.

However, many item writers include information which has little to nothing to do with the question at hand. Be careful to realize the difference between necessary information and filler when writing test items.

7. Item mapping: Make sure that all test items can be mapped back to specific content that is relevant to the job or learning objectives from the course material. It is important to determine that all items are relevant to the content and should be tested.

HSTS01-15-R-HRM015 - Attachment 11 - TSA’s Organizational Assessment Services Item Writing Guidelines Page 1 of 7 Pages

Specific Item Writing Guidelines

The following list is a guideline for writing items which conform to good professional practice.

1. All items should be checked for spelling and typographical errors. The entire test should be spell checked.

2. Grammar:

a. Contractions are not acceptable, a formal writing style should be utilized (e.g. won’t should be will not)

b. Possessives should be checked to ensure they are properly used. Correctly punctuate singular and plural possessives.

c. Commas should be examined to ensure they are placed properly.

1) Are there too many or too few?

2) Check that commas are placed correctly and consistently in a series (e.g., red, white, and blue).

d. Hyphens should be checked to ensure they are actually required.

e. Parallelism – check that alternatives are of the same tense (following from the stem referent).

f. Subject-verb agreement – Intervening words do not affect the number of the verb. The number of the subject determines the number of the verb. Similarly, the verb tense in all distractors must correspond to the verb tense in the stem so that the examinee cannot rule out an alternative based on poor grammar.

3. Sentence Structure:

a. Complex, awkward, and weak sentence structure should be checked and revised. Break up complex sentences into several shorter, simpler sentences.

b. Ambiguous, vague, and slang terms should be avoided.

c. Center embedding makes items more difficult than is necessary. Keep subject and verb close together in order to avoid ambiguous relationships.

d. “Window dressing” or irrelevant material should not be included. Be concise and clear.

e. Combined words should be checked to ensure they should not be separated.

Also check that two words should not be combined into one.

f. Active voice should be used rather than the passive voice.

4. Item Stem:

a. A central problem or idea should be stated in each item. Items should contain only ONE concept or problem.

b. “A/an” should be used in the stem if alternatives begin with both vowels and consonants.

c. Blanks should be placed as close to the end of a sentence as possible. Blanks in items are to be ten (10) spaces long, not including the spaces at either end.

d. Colons should be used instead of blanks whenever possible (e.g. at the end of the stem).

Item Writing Guidelines Page 2 of 7 Pages

e. Periods should be placed after all alternatives for an item stem ending in a colon.

f. Periods should not be placed after alternatives for which there is a blank.

g. Periods should not be placed after alternatives for which the item stem ends as a question unless the alternatives are complete sentences.

h. Periods should be placed at the end of answer options when they are complete sentences:

o It must include at least one main clause. A main clause contains an independent subject and verb and expresses a complete thought. Put another way, a complete sentence must contain a subject, predicate and a complete thought so that it can stand on its own.

o Keep in mind, it can be very short including only 2-3 words (i.e.

David stood.)

o The subject must be present in the sentence (i.e. The Officer reviews the documents.) in order to add a period. If the subject is implied (i.e.

Review the documents) no period is added.

i. Stem should contain as much of the item as possible. If each alternative begins with the same word, include the word in the stem.

j. The question should be understood without having to reread it. If the question must be reread, then it should be rewritten – simplified for clarity.

k. Double worded alternatives should be avoided (e.g. Two cities in Virginia are ______ and _____ ).

l. All qualifying information should be included in the stem – not in the alternatives.

m. If a word in the item stem negates the meaning, capitalize the word (i.e., NOT, NONE, NO, EXCEPT). Negative prefixes (i.e. un, non) should not be capitalized. Cannot should be written as “can NOT.”

n. If an absolute term is used in the item stem (i.e. ALWAYS, ALL, ONLY, EVERY, NEVER, MUST) it should be capitalized.

o. Words in the answer options should not be capitalized.

5. Direct and Indirect Clue Errors:

a. Same words should NOT be included in the item stem and in the correct alternative.

b. Distractors should not differ systematically from the correct answer.

c. “Clanging associations”, words that sound alike, should not be in the stem and the correct answer.

d. Language duplication, same words or synonyms, should not be in the stem and alternative.

e. Overlapping alternatives should be eliminated.

f. Distractors should be equally plausible and not easily eliminated. Construct distractors of equal difficulty and attractiveness. Distractors that are opposites of the correct answer should be avoided. If a distractor is incorrect only because one word has changed, it should be eliminated or altered.

g. Alternatives should be as concise and clear as possible.

Item Writing Guidelines Page 3 of 7 Pages

h. Alternatives should not include both categories and subcategories. Examinees who merely knew the headings could eliminate alternatives on this basis alone.

i. Make sure all alternatives are on the same page as the question. The alternatives or part of the alternatives should not be split from the question itself.

j. Item stem and alternatives should not give away the answer to another item.

Do not use the identical set of alternatives for more than one item. This may allow the examinee to immediately rule out one alternative (correctly or incorrectly) based on their response to a previous question. If this cannot be avoided, alternatives should be placed in the same order for both questions.

6. Numbers:

a. Inclusive numerical alternatives and logically impossible number series should be avoided.

b. Ascending order should be used for alternatives containing numbers (e.g. 1, 3, 5)

c. Numbers 1 to 10 should be written out. Use numerics for numbers over 10 within the stem. If the alternatives have a mix of numbers above and below 10, use numerics.

d. Units should be used consistently and not be mixed (e.g. months and days).

7. Acronyms:

a. Acronyms should be spelled out every time they are used in item stems and answer alternatives (e.g., TSO (Transportation Security Officer)).

b. For PASS assessments ONLY, acronyms that are very common and well known for a particular test (e.g., BDO in a test for BDOs) do not need to be spelled out. Please note that TSA MUST approve the acronym list for a given test prior to use.

c. If the acronym is spelled out in the item stem, it does not need to be spelled out in the answer alternatives.

d. If an article (e.g., a, an, the) is used in front of the acronym, it should agree with the acronym. For instance, “An STSO” rather than “A STSO.”

e. If a term with an acronym is plural, the acronym should be singular when it is first listed with the term, and plural thereafter.

8. Miscellaneous:

o Absolute terms should rarely be used (e.g. never, always).

o Sexist language should be avoided unless specified by procedure. Use him/her or his/her whenever possible.

o “All of the above/none of the above” should be avoided as an alternative when possible.

o “All of the below . . . except” should not be used in the item stem.

o Abbreviations should be avoided when possible. Use abbreviations only when they are commonly used by the examinees.

o Words or abbreviations should be used consistently.

Item Writing Guidelines Page 4 of 7 Pages o Avoid using names in tests when possible. If necessary, refer to passangers generically (i.e. Passanger A, Passanger B, Passanger C).

o True/False questions should account for NO more than 20% of the items in a test.

o Items should be randomized and NOT grouped by item type (e.g. multiple choice or true/false) or content.

Item Writing Guidelines Page 5 of 7 Pages

Item Format Examples

1. If the alternative completes the stem, use the following format:

The first president of the United States was:

A. Washington.

B. Lincoln.

C. Adams.

D. Jefferson.

NOTE:

- Place a period at the end of each alternative.

- Use a colon rather than the blank at the end of the stem.

- The decision to capitalize or not capitalize the first letter of each alternative is strictly grammatical.

2. If the stem is a question, use the following format:

What is the capital of the United States?

A. Washington, D.C.

B. Philadelphia C. Los Angeles D. New York City

NOTE:

- Place a period at the end of each alternative only if it is a complete sentence.

- Capitalize the first letter of each alternative.

3. If the alternative is imbedded in the stem, use the following format:

The city of __________ is the capital of Ohio.

A. Philadelphia

B. Pittsburgh C. Columbus D. Richmond

NOTE:

- NEVER have a period (or other punctuation) at the end of alternatives. If punctuation is required, place it in the stem.

- The decision to capitalize or not to capitalize the first letter of each alternative is strictly grammatical.

- Blanks should appear as close to the end of the stem as possible; avoid having items where the blank is located at the beginning of the item.

Item Writing Guidelines Page 6 of 7 Pages

4. Answer options entered into the online testing system should be scrambled UNLESS:

− The question is True/False − All of the Above or None of the Above is listed as an answer option (these should be listed as the last choice) − The answer options are in numerical or alphabetical order − Never or Always is listed as an answer option (these should always be listed as the last choice)

Item Writing Guidelines Page 7 of 7 Pages

Specific Item Writing Guidelines