ED_508_FINAL_CDSE_ResearchMethods_DDD.pdf

Final Detailed Design Document Research Methods and Statistics to Support DoD Security

Programs

Developed for

The Department of Defense (DoD) Defense Security Service (DSS)

Center for Development of Security Excellence (CDSE) Education Division

Presented by

Carney, Inc.

Alexandria, VA

Under BPA HS0021-11-A-0091 Order No. HS0021-11-A-0091-0003

July 9, 2012

DSS CDSE Education Division Research Methods and Statistics

Carney, Inc. Page 2

Contents

1 Introduction

1.1 The Requirement

1.2 Purpose of the Detailed Design Document

1.3 Organization of This Document

2 High-Level Course Design

2.1 Course Description/Overview

2.2 Credits Conferred

2.3 Prerequisites

2.4 Student Outcomes/Objectives

2.5 Delivery Method/Course Requirements

2.6 General Course Requirements

2.7 Grading

2.7.1 Class Participation (20%):

2.7.2 Weekly Assignments (40%):

2.7.3 Mock RFP (15%):

2.7.4 Final Research Paper (25%):

2.8 Incorporation of Feedback

2.9 Course Textbooks

2.10 Course Outline

3 Content Outline

3.1 Part 1: Introduction and Basic Concepts

3.1.1 Week 1: Course Overview and Research Methods Basics

3.1.2 Week 2: Research Design Introduction

3.2 Part 2: Analytical Statistics

3.2.1 Week 3: Descriptive Statistics

3.2.2 Week 4: Comparative Statistics

3.2.3 Week 5: Statistical Modeling

3.2.4 Week 6: Time Series Analysis

3.3 Part 3: Data Collection Methods

3.3.1 Week 7: Quantitative and Qualitative Research Concepts

3.3.2 Week 8: Qualitative Research Methods

Carney, Inc. Page 3 July 9, 2012

3.3.3 Week 9: Sampling Concepts

3.3.4 Week 10: Quantitative Research Methods

3.3.5 Week 11: Data Mining

3.4 Part 4: Reporting Statistics

3.4.1 Week 12: Graphing Statistics

3.4.2 Week 13: Writing about Statistical Findings – Comparative Statistics

3.4.3 Week 14: Writing about Statistical Findings – Statistical Models & Time Series . 44

3.5 Part 5: Applying Research

3.5.1 Week 15: Critiquing Statistical Reports

3.5.2 Week 16: Final Projects and Conclusion

Carney, Inc. Page 4 July 9, 2012

1 Introduction

1.1 The Requirement

The Defense Security Service (DSS) Center for Development of Security Excellence (CDSE) has undertaken, through its Education Division, the responsibility to provide graduate-level education to security professionals designed to develop the future security leaders for the Department of Defense (DoD).

The purpose of this project is to design and develop an advanced, semester-long course that will provide an educational opportunity for mid-career security specialists to gain in-depth knowledge and understanding of research methods, statistics, and reporting to support DoD security programs. The course will be conducted via the Sakai Continuous Learning Environment (CLE) and will consist of readings, prerecorded lectures and presentations, online synchronous sessions, a discussion forum, written assignments, and student oral presentations.

1.2 Purpose of the Detailed Design Document

The purpose of this Detailed Design Document is to convey the learning objectives, topic outlines, and planned assignments for each week of the course. It also proposes the grading schema the instructor will use. This document proposes the organization of content to be delivered throughout the 16-week semester and details the proposed instructional methods to be used for each week.

This detailed design will guide the development of the course materials.

1.3 Organization of This Document

The remainder of this Detailed Design Document is organized into several sections:

Components of the High-Level Design Document:

o An overview of the course o A statement of the anticipated credits to be conferred o A list of prerequisite courses, if applicable o The terminal learning objectives of the course o The delivery method(s) to be used o General course requirements o Grading scheme for the course o A course outline showing how the course will unfold over the semester

A content outline detailing the topics, instructional methods, and assignments for each week

Carney, Inc. Page 5 July 9, 2012

2 High-Level Course Design

2.1 Course Description/Overview

The main purpose of the Research Methods, Statistics and Reporting to Support DoD Security Programs course is to introduce students to the fundamentals of statistics, research methods, and presentation of statistical information in the context of the Department of Defense. The course will develop each student’s ability to use this knowledge to become more effective as security leaders in the Department of Defense (DoD). These tasks include:

Collecting data that accurately represents the concepts they are supposed to measure Measuring the effectiveness of a program Using data to make decisions Providing technical guidance to contractors for inclusion in contract documents Evaluating feasibility of research proposals Presenting data to support programs to higher-level officials

The course will provide an overview of the important concepts of research design, data collection, statistical and interpretative analysis and final report presentation. The focus of this course is not on mastery of statistics but on the ability to use research in the DoD Security environment.

Each week, students will work through lessons that present the research and/or statistics-related weekly concepts, as well as security-specific readings that bring to life examples of how the weekly concept applies in the security realm. This will allow students to clearly understand how the course material relates to their jobs as security professionals.

2.2 Credits Conferred

This course will be designed to equate to three credit hours at the graduate level.

2.3 Prerequisites

This course has no prerequisites.

2.4 Student Outcomes/Objectives

This course will be designed to enable students to meet the following final terminal learning objectives (TLOs):

1. Act as an educated consumer of data

2. Prepare a preliminary research design for projects in their subject matter areas

3. Promote clear procedures for collecting, documenting and reporting data

4. Present complex data or situations clearly

5. Critically review and analyze research findings that affect their agency

Carney, Inc. Page 6 July 9, 2012

2.5 Delivery Method/Course Requirements

This is a graduate-level distance-learning course in research methods and statistics for security professionals. The course will consist of readings, prerecorded lectures and presentations, participation in the discussion forum, exercises, and written assignments.

The assigned course readings draw from a variety of resources, such as DoD and GAO reports, articles and essays on research methods, and examples of effective and ineffective presentation of statistical information. Students are expected to familiarize themselves with the assigned topic and readings each week and should be prepared to participate in the online discussion forum to discuss the readings critically.

2.6 General Course Requirements

Class participation is both important and required. If, due to an emergency, students are not able to respond to a discussion prompt in the week it is assigned, they must contact the instructor by e-mail and will be expected to post their response in the following week.

It is expected that assignments will be submitted on time (by midnight the day they are due).

However, it is recognized that students occasionally have serious problems that prevent work completion. If such a dilemma arises, students should contact the instructor in a timely fashion.

The completion of all readings assigned for the course is assumed. Since the class will be structured around discussion and includes multiple small-group activities, completion of readings and all class assignments is crucial.

2.7 Grading

The following provides an approximate breakdown of how each assignment contributes to the overall performance in the class.

Class Participation 20% Weekly Assignments (ten) 40% Mock RFP 15% Final Research Project 25%

Carney, Inc. Page 7 July 9, 2012

Pass/fail grading criteria for each graded assignment are listed below.

PASS FAIL

Exceed Expectations

Meets Expectations

Somewhat Meets Expectations

Below Expectations

Class Participation

Engages other students;

Encourages more participation

Provides meaningful insights;

responds to other students

Only asks questions in the discussion forum; Only engages with the instructor

Does not participate in discussion forum

Initiates discussions to link lessons to the DoD and the workplace

Keeps discussions active

Tries to dominate discussion forum

Is argumentative and off-topic

Weekly Assignment

Demonstrates a mastery of the concepts

Demonstrates an understanding of the concepts

Submits a completed assignment with conceptual errors

Does not understand the concepts presented.

Essay responses are well thought-out, and presented in a compelling fashion

Essay responses include all elements in the questions.

Essay responses include the main point but do not include all elements

Essay responses provided are simplistic

MOCK RFP

PASS FAIL

Exceed

Expectations Meets

Expectations Somewhat Meets

Expectations Below

Expectations Research Design

Research purpose/program need is defined and clearly described.

Research purpose/program need is defined, and described at a high level.

Research purpose/program need is not well defined, and is not well described.

No research purpose/program need is included.

Overall research design is described accurately and completely.

Overall research design is described accurately, but lacks some detailed description.

Overall research design is correctly identified, but narrative includes errors in the description, or description is not included.

Overall research design is neither described accurately nor completely.

Selected research design is justified, and is appropriate to meet the program need.

Selected research design is appropriate to meet the program need, the choice is not justified.

Selected research design only partially meets the program need.

Selected research design is not appropriate to meet the program need.

Data Collection Method

Data collection method is defined and described accurately.

Data collection method is defined and described somewhat accurately.

Data collection method is defined, but not described accurately.

Data collection method is not defined or described accurately.

Carney, Inc. Page 8 July 9, 2012

PASS FAIL

Exceed

Expectations Meets

Expectations Somewhat Meets

Expectations Below

Expectations Data collection method is appropriate for the research design, and is justified.

Data collection method is appropriate for the research design, and choice of approach is partially justified.

Data collection method is not appropriate for the research design, and choice of approach is partially justified.

Data collection method is not appropriate for the research design, and choice of approach is not justified.

Description of the data collection method includes accurate representation of the sample population and coverage issue from the target population.

Description of data collection method includes accurate representation of the sample population.

Description of data collection method includes representation of the sample population with some inaccuracies.

Description of data collection method does not include representation of the sample population.

Description of Data

A complete description of the desired output, including rationale, is included.

A complete description of the desired output is included.

Data description is not detailed or some data is not included.

A complete description of the desired output is not included.

Statistical Analysis

Desired statistical analysis is accurately defined and described.

Desired statistical analysis is described accurately, but lacks some detailed description.

Desired statistical analysis is correctly identified, but narrative includes errors in the description, or description is not included.

Desired statistical analysis is neither described accurately nor completely.

Statistical analysis method aligns with the type of data collected.

Statistical analysis method does not align with the type of data collected.

Statistical analysis method aligns with the research design.

Statistical analysis method does not align with the research design.

Overall No significant writing errors

Minor errors cause few disruptions in meaning

Errors do not cause the writing to be unclear, but weaken the effectiveness of the communication

Severe and/or frequent errors cause writing to be unclear and difficult to read

All supporting research is properly cited

Research is properly cited most of the time

Research is inconsistently cited

Research is not cited at all

Carney, Inc. Page 9 July 9, 2012

FINAL RESEARCH PAPER

PASS FAIL

Exceed Expectations

Meets Expectations

Somewhat Meets Expectations

Below Expectations

Statistical Analysis

The statistical methods are defined and clearly described.

The statistical methods are defined, and described at a high level.

The statistical methods are not well defined, and are not well described.

The statistical methods are not defined.

Overall research design is described accurately and completely.

Overall research design is described accurately, but lacks some detailed description.

Overall research design is correctly identified, but narrative includes errors in the description, or description is not included.

Overall research design is neither described accurately nor completely.

Statistical analysis presented includes multivariate analysis.

Statistical analysis includes bivariate analysis

Statistical analysis only includes univariate analysis

Statistical analysis is misapplied.

Data Collection Method

Data collection method is defined and described accurately.

Data collection method is defined and described somewhat accurately.

Data collection method is defined, but not described accurately.

Data collection method is not defined or described accurately.

Description of the data collection method includes accurate representation of the sample population and coverage issue from the target population.

Description of data collection method includes accurate representation of the sample population.

Description of data collection method includes representation of the sample population with some inaccuracies.

Description of data collection method does not include representation of the sample population.

Presentation The paper is addressed to High Level Officials

The paper is presented as a student paper

Hypothesis Testing

A well-stated hypothesis is the core of the presentation.

A research hypothesis is clearly presented

Research questions are presented instead of a hypothesis; or hypothesis is not clearly stated

Neither hypothesis nor research questions are presented.

All statistical analysis is derived from the research hypothesis

Statistical analysis supports the testing of the hypothesis

Some statistical analysis tests the hypothesis; some addresses additional issues

Statistical analysis is not related to hypothesis

Topic The topic addresses a current issues of great concern to the DoD

The topic addresses a current concern of the DoD.

The topic is related to DoD security

The topic has already been resolved

Carney, Inc. Page 10 July 9, 2012

PASS FAIL

Exceed

Expectations Meets

Expectations Somewhat Meets

Expectations Below

Expectations The paper presents a new solution that is supported by data

The paper provides statistical to support one alternative solution over another

The paper uses data to confirm previous decision

The paper presents no solutions.

Overall No significant writing errors

Minor errors cause few disruptions in meaning

Errors do not cause the writing to be unclear, but weaken the effectiveness of the communication

Severe and/or frequent errors cause writing to be unclear and difficult to read

All supporting research is properly cited

Research is properly cited most of the time

Research is inconsistently cited

Research is not cited at all

Carney, Inc. Page 11 July 9, 2012

2.7.1 Class Participation (20%):

Participation includes participating in the class discussion forum, conducting peer reviews, and participating in small group work. To achieve full credit for participation, students must respond thoughtfully to all weekly discussion prompts, post a response to at least two other students’ discussion posts each week, and provide constructive criticism when conducting peer reviews of other students’ work.

2.7.2 Weekly Assignments (40%):

Most weeks, the instructor will assign small-scale exercises aimed at helping students to apply the lessons of week. The time burden for each writing assignment is not expected to exceed four hours per week.

2.7.3 Mock RFP (15%):

The first major assignment is to prepare the technical section of a request for proposal for a research or statistical data collection. The students should address procedures for collecting, documenting and reporting the data. The RFP should include a sample design, data collection method, data description and required statistical analysis. The Mock RFP is due at the conclusion of Week 11.

2.7.4 Final Research Paper (25%):

The final research paper requires students to write a report for “High Level Officials” that uses statistical data. Topics should be on a security-related issue for which the students have access to statistical data. Potential topics include: trends in violations, serious security incidents, financial reports, and proposals for new programs or policies. Prior approval of the topic for the final research paper is required. Students should submit a one-paragraph written description of their proposed topic for approval no later than Week 4.

Students will research the topic thoroughly in order to fully explore and analyze the varying perspectives regarding the selected issue. They must then formulate their own recommendations for resolution of the issue, including justifications and specific strategies for implementation of the recommendations. Students will properly cite all research referenced in the report, using the format laid out in the Chicago Manual of Style.

Students must have a draft of the report at least 75% complete and ready for peer review by another student in Week 13 of the course. During Week 14, students will review each other’s reports and provide constructive criticism. Students will have the remainder of the semester to complete the report. The Final Report is due at the end of Week 16.

2.8 Incorporation of Feedback

The course instructor will provide multiple opportunities for students to provide constructive feedback on course delivery and content over the period of the course. These may be in the form of group sessions or one-on-one sessions with the instructor. Students will be afforded the

Carney, Inc. Page 12 July 9, 2012 opportunity to provide written feedback following each assignment, to include general feedback on the course or specific feedback on an individual assignment. Additionally, students will provide feedback on one another’s work through posts in the discussion forum and formal peer reviews of identified writing assignments. Finally, the instructor will provide written feedback to students on all assignments.

2.9 Course Textbooks

There will be three textbooks required for this course. Additional readings will include academic papers and DoD reports. The texts are:

Statistics for People Who (Think They) Hate Statistics by Neil J. Salkind, published by Sage Publishing, $65 in paperback.

How to Measure Anything: Finding the Value of Intangibles in Business by Douglas Hubbard, published by Wiley, Hardback – $49.95, Kindle - $27.47

Research Methods: The Basics by Nicholas Walliman, published by Routledge;

Paperback $19.95 -- Kindle $9.99

Carney, Inc. Page 13 July 9, 2012

2.10 Course Outline

The course is divided into five topic areas:

Part 1: Introduction and Basic Concepts (2 weeks) Part 2: Analytical Statistics (4 weeks) Part 3: Data Collection Methods (5 weeks) Part 4: Reporting Statistics (3 weeks) Part 5: Applying Research (2 weeks)

The following table outlines the 16-week course agenda. Graded assignments are in bold. Items in italics are ungraded but are required for a later, graded assignment.

Lesson/ Week

Topics Student Assignments Due

1 Course Overview Basic statistical concepts:

probability, central limit theorem, statistical power

Discussion Forum: Introduction Discussion topic: Review

“Targeting U.S. Technologies” report, statistics used and presented.

2 Research Design Introduction Hypotheses creation – research question vs. straw man Experimental, quasi-experimental, cross-sectional, time-series

Discussion Forum Exercise 1a: Identifying research designs Exercise 1b: Creating hypotheses

3 Descriptive Statistics means & standard deviations percentages and ratios histograms

Discussion Forum Exercise 2: Descriptive statistics

4 Comparative Statistics t-test analysis of variance correlations chi-square

Discussion Forum Exercise 3: Comparative statistics Selection of topic for final research paper

5 Statistical Modeling Regression and its off-shoots

Discussion Forum Exercise 4: Identifying components and describing results of statistical models

6 Time series analysis Discussion Forum Exercise 5: Describing results of time series analysis

Carney, Inc. Page 14 July 9, 2012

Lesson/ Week

Topics Student Assignments Due

7 Quantitative and Qualitative research concepts

Quantifying the issue Discovering the “why” and “what ifs” Grounded Theory vs. Scientific method

Discussion Forum

8 Qualitative Research Methods Ethnographic research In-depth interviewing Group interviewing Interpretation and limitations

Discussion Forum Exercise 6: Describe qualitative research possibilities in your area

9 Sampling Concepts Defining the target population, coverage issue with the sample population

Representative sample Potential consequences of unrepresentative sampling (gaming the system)

Over representative subgroups / weighting

Design effect Sampling methods (cluster, stratified, simple random)

Discussion Forum Exercise 7: Sampling questions

10 Quantitative research methods Measurements Surveys Operations research Administrative records

Discussion Forum Draft Mock RFP for peer review

11 Data mining – finding the patterns in the world of data

Final Mock RFP

12 Graphing statistics Discussion Forum Exercise 8: Graph preparation

13 Writing about statistical findings – comparative statistics

Discussion Forum Draft Final Paper for peer review Exercise 9: Write-up results from data provided

Carney, Inc. Page 15 July 9, 2012

Lesson/ Week

Topics Student Assignments Due

14 Writing about statistical findings – statistical models & Time Series

Review two other students’ papers

15 Critiquing statistical reports

Exercise 10: Critique results from assigned paper

16 Applying Research in the Security Environment

Final paper

Carney, Inc. Page 16 July 9, 2012

3 Content Outline

3.1 Part 1: Introduction and Basic Concepts

3.1.1 Week 1: Course Overview and Research Methods Basics

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Identify the expectations for successful course completion Identify the requirements of the final project Identify the needs for research skills for security professionals (TLO 1) Identify and describe the use of the building blocks of statistics: probability, central limit theorem, and statistical power (TLO 1, 2)

2. Rationale:

The first week provides an overview of the course structure and requirements and introduces students to the fundamental concepts of research and statistics, within the context of the DoD security field.

3. Topic Outline:

Research in DoD Security environment o Data reports o Trends o Vulnerabilities

Statistical Concepts o Representation o Randomization o Replication

Probability o Nothing is certain o Coin flips and dice o Conditional probability

Central Limit Theorem o Bell-shaped curve o Graphic introduction to standard deviation

Statistical Power o Confidence interval (plus/minus %) o The n’s justify the means o Can there be too much?

Carney, Inc. Page 17 July 9, 2012

4. Discussion Questions:

To be refined during content development. Possible topics include:

Introduction to the class, including name, position, and organization. May ask students to share the security issues of most interest to them, and what they hope to get out of this class.

The importance of data to the success of the DoD security program Specific factors about data most common to DoD security professionals Discussion of the statistics in the 2011 Targeting U.S. Technologies: A Trend Analysis of

Reporting from Defense Industry

5. Class Agenda:

Activity Duration Review the syllabus and read the instructor welcome to the course

20 minutes

Review the basic statistical concepts 60 minutes Respond to Discussion Forum Question 1:

Introduction

15 minutes

Read the article and Discussion Forum Question 2: Trend Analysis

30 minutes

Total 125 minutes

6. Reading Assignments:

Assigned readings will cover topics similar to those in the sources listed below. Specific reading assignments will be identified during content development.

2011 Targeting U.S. Technologies: A Trend Analysis of Reporting from Defense Industry http://www.dss.mil/counterintel/DSS_UNCLASS_2011/index.html

Statistics for People Who (Think They) Hate Statistics by Neil J. Salkind; Chapter 1

How to Measure Anything: Finding the Value of Intangibles in Business, Hubbard, Chapters 1 and 3

Report of the Defense Science Board Task Force on Basic Research (January 2012) Office of the Under Secretary of Defense for Acquisition, Technology and Logistics http://www.acq.osd.mil/dsb/reports/BasicResearch.pdf

7. Deliverables for this Week:

To be finalized during content development

The required deliverables for this week:

Discussion Forum: Introduction. Students must post a response to the discussion question and respond to at least two other students’ postings.

Discussion Forum: Article Review – Student must post a comment on the statistics in the

2011 Targeting U.S. Technologies: A Trend Analysis of Reporting from Defense Industry

Carney, Inc. Page 18 July 9, 2012

In addition, students may post to the following Discussion Forum threads:

Questions on Mock RFP (optional) Questions on Final research paper (optional)

8. Assignment for Next Time:

None.

Carney, Inc. Page 19 July 9, 2012

3.1.2 Week 2: Research Design Introduction

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Articulate the concepts of research hypotheses and research questions (TLO 1, 2) Create their own research hypotheses (TLO 2) Distinguish between the different types of research design including experimental design, quasi-experimental design, cross-sectional design, and time-series (TLO 1, 2)

2. Rationale:

In the second week, students will be introduced to the basic starting points of research including developing testable hypotheses. The students will learn the types of research designs that we will encounter and some of the pros and cons of using each design.

3. Topic Outline:

Hypotheses basics

Hypotheses creation o What is the research question?

o A priori knowledge

What do we already know?

What assumptions are we making?

o A Hypothesis is a statement not a question Making the hypothesis testable o What is the alternative hypothesis?

o What measures can support the hypothesis from the alternative?

Analysis of data without a hypothesis o Random noise o Confirmatory analysis

Research Designs

Experimental Design o Random assignment o Treatment vs. non-treatment o Statistical test

Quasi-experimental Design o Non-random assignment o Treatment vs. non-treatment o Correlating factors

Identifying Controlling

Cross-sectional o Single point in time o Statistical issues

Carney, Inc. Page 20 July 9, 2012

Correlating factors Models

Time Series o Repeated measures o Correlation from time 1 to time 2

4. Discussion Questions:

To be refined during content development. Possible topics include:

In DoD research reports, what are the stated a priori hypotheses? Are they presented?

What assumptions are presented?

Which research designs are commonly employed in DoD analysis

5. Class Agenda:

To be refined during content development.

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Assignment 1a: Identifying Research Design exercise

20 minutes

Assignment 1b: Creating Hypothesis exercise 20 minutes Total 120 minutes

6. Reading Assignments:

Assigned readings will cover topics similar to those in the sources listed below. Specific reading assignments will be identified during content development.

Research Methods: The Basics by Nicholas Walliman, Chapters 1 & 2

Measuring Personnel Security Investigation Quality: A Review of Two Approaches; Eric L. Lang, Leissa C. Nelson; Defense Personnel Security Research Center, Management Report 10-02 http://www.dhra.mil/perserec/reports/mr10-02.pdf

There is one required deliverable for this week:

Discussion Forum: Research Design and Hypotheses

In addition, students may post to the following Discussion Forum threads:

Questions on Exercise 1a or 1b (optional)

Carney, Inc. Page 21 July 9, 2012

8. Assignment for Next Time:

To be refined during content development.

Exercise 1a: Research Design Exercise This exercise will consist of a series of scenarios in which the student must identify the type of research design that should be used.

Exercise 1b: Creating Hypotheses Exercise This exercise requires the student to create hypotheses from a list of research questions.

Carney, Inc. Page 22 July 9, 2012

3.2 Part 2: Analytical Statistics

3.2.1 Week 3: Descriptive Statistics

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Distinguish between the three types of data: continuous, discrete, and ordinal (TLO 2, 3) Calculate means and standard deviations (TLO 3, 4) Calculate percentages and ratios (TLO 3, 4) Create histograms (TLO 3, 4)

2. Rationale:

The third week will begin the 4-week section on analytical statistics. In this week, students will learn the statistics used when examining a single variable.

3. Topic Outline:

Types of Data

Continuous

Discrete

Ordinal

Descriptive Statistics Means o Difference from median o Calculation

Standard deviation o Central limit theorem and variance o Standard error o Effect of sample size

Discrete data o Percentages o Confidence interval around percentage

Ordinal data o The mean trap o More than discrete o Histograms

Other important descriptive statistics o Range o Quartiles o Skewness

Carney, Inc. Page 23 July 9, 2012

4. Discussion Questions:

To be refined during content development. Possible topics include:

Descriptive statistics in DoD Security Reports - is context provided?

5. Class Agenda:

To be refined during content development.

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Assignment 2: Descriptive Statistics 40 minutes assignments will be identified during content development.

Statistics for People Who (Think They) Hate Statistics by Neil J. Salkind; Chapters 2 & 3

Demographics 2010: Profile of the Military; Department of Defense;

http://www.militaryhomefront.dod.mil/12038/Project%20Documents/MilitaryHOME FRONT/Reports/2010_Demographics_Report.pdf

Changes in Espionage by Americans: 1947-2007; Katherine L. Herbig; Defense Personnel Security Research Center, Technical Report 08-05, March 2008 http://www.dhra.mil/perserec/reports/tr08-05.pdf

There are three required deliverables for this week:

Discussion Forum: Research and Analysis Exercise 1a: Research Design due to instructor Exercise 1b: Hypotheses due to instructor

In addition, students may post to the following Discussion Forum threads:

Questions on Exercise 2: Descriptive Statistics (optional)

To be refined during content development.

Exercise 2: Descriptive Statistics Students using Excel or Google Spreadsheet will provide descriptive statistics on a series of variables. The assignment will give a verbal description of the data and students will have to determine if the variable is continuous, ordinal or discrete and provide the appropriate statistics.

Carney, Inc. Page 24 July 9, 2012

3.2.2 Week 4: Comparative Statistics

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Identify the correct method to compare two variables depending on the variable type

(TLO 2, 3)

Test the key variable in an experimental or quasi-experimental design (TLO 3, 5)

During Week 4, students will learn the basic statistics used in simple statistical analysis. The students will learn the pre-post t-test, analysis of variance correlations, and chi-square tests.

Experimental Tests

Repeated measures o Variance assumptions o T-test

Comparing Subgroups

T-test for 2 groups o Continuous variable of interest o Ordinal variable of interest

ANOVA for 3 or more groups

Relationship Between Variables

Continuous/Continuous o Correlation

Positive association Negative association

Continuous/Discrete o T-test or ANOVA

Discrete/Discrete o Table

Expected values o Chi-square statistics

4. Discussion Questions:

To be finalized during content development. Possible topics include:

Statistical differences between items may not be meaningful. Discuss issues of correlation and causation.

Carney, Inc. Page 25

5. Class Agenda:

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Exercise 3: Comparative Statistics 40 minutes assignments will be identified during content development.

Statistics for People Who (Think They) Hate Statistics by Neil J. Salkind; Chapters 8, 9, Utilizing the Elements of National Power to Predict Ungoverned Space, MAJ John E.

Elrich, Advanced Military Studies Program http://www.dtic.mil/dtic/tr/fulltext/u2/a470656.pdf

There are two required deliverables for this week:

Discussion Forum:

Exercise 2: Descriptive Statistics

In addition, students may post to the following Discussion Forum thread:

Exercise 3: Comparative Statistics Using the same variables from the previous week, identify the appropriate statistical test for a combination of variables. Using Excel or Google Spreadsheet, 1) perform t-tests on the pre-post variables; 2) produce a table from 2 discrete variables showing the predicted values and the actual values.

Carney, Inc. Page 26 July 9, 2012

3.2.3 Week 5: Statistical Modeling

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Describe the purpose and use of statistical modeling (TLO 2, 3) Interpret the results of a statistical model (TLO 3, 4, 5) Develop a simple recommendation to a defined problem (TLO 3, 4, 5)

In Week 5, students will learn the basics of statistical modeling. Modeling is used to predict future outcomes from past performance and is therefore one of the key concepts of using research.

Statistical Modeling

Slope/Intercept: y = a + bx o If x=0, then “a” is the intercept o Slope is the change in y for a single unit increase in x o Predicted values

Amount of variation accounted for – the r-square

Control variables o Add to the model items that might affect the relationship

• Y=a + bx +cz o Control variables can change the slope and/or the intercept o Discrete variables as controls

• Dummy variables

• Comparison group o R-square and adjusted r-square

Discrete/Ordinal Variable of Interest o Logit

• Predicting propensity

• Odds ratios

4. Discussion Questions:

To be finalized during content development. Possible topics include:

Discuss of the role of making predictions in your environment. What factors do you think can aid in prediction?

Carney, Inc. Page 27 July 9, 2012

5. Class Agenda:

To be refined during content development.

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Exercise 4: Statistical Models 40 minutes Total 180 minutes assignments will be identified during content development.

Statistics for People Who (Think They) Hate Statistics by Neil J. Salkind; Chapters 14

Indicators of NGO Security in Afghanistan, United States Military Academy, The Combating Terrorism Center, http://www.dtic.mil/dtic/tr/fulltext/u2/a467616.pdf

Structural equation modeling of associations among combat exposure, PTSD symptom factors, and Global Assessment of Functioning; Mark W. Miller, Erika J. Wolf, Elaine Martin, Danny G. Kaloupek, Terence M. Keane;

http://www.rehab.research.va.gov/jour/08/45/3/pdf/Miller.pdf

Predicting Industry Personnel Security Investigation Requirements; Eric L. Lang, Leissa C. Nelson; Defense Personnel Security Research Center, Technical Report 07-01 http://www.dhra.mil/perserec/reports/tr07-01.pdf

There are two required deliverables for this week:

Discussion Forum: Recommendations and Presentation of Ideas of Final Project Exercise 3: Comparative Statistics

Exercise 4: Statistical models Students will identify components of statistical models and describe the results of statistical models, including control and dummy variables. Produce a simple slope/intercept equation for data using Excel.

Carney, Inc. Page 28 July 9, 2012

3.2.4 Week 6: Time Series Analysis

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Identify the appropriate use of time series analysis (TLO 1, 2) Describe the results of time series analysis (TLO 3, 4) Explain the effect of auto-correlation on time series analysis (TLO 1, 3, 4, 5)

In Week 6, the students will be introduced to the analysis of time series and the problem of autocorrelation on analysis.

3. Topic Outline:

To be finalized during content development

Time series o Trend Analysis o Significant Event Interruption o Paradigm Shifts

Cycles o Regular cycles

Seasonal employment Military recruitment by months o Irregular Cycles Business cycles

Autocorrelation o This month’s numbers relationship to last month’s o This month’s numbers relationship to this month last year

Statistical analysis of time series o Changes over time o Correlates o Comparing subgroups

4. Discussion Questions:

Discuss recent trends in security. What are the seasonal relationships? What significant events have occurred to change trends – is it an interruption or a paradigm shift?

Carney, Inc. Page 29 July 9, 2012

5. Class Agenda:

To be refined during content development. Note: These activities will be divided between two weeks.

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Exercise 5: Time Series 40 minutes assignments will be identified during content development.

Dynamic Modeling for Persistent Event-Count Time Series; Patrick T. Brandt, John T.

Williams, Benjamin O. Fordham and Brain Pollins; American Journal of Political Science , Vol. 44, No. 4 (Oct., 2000), pp. 823-843

Democracy and Diversionary Military Intervention: Reassessing Regime Type and the Diversionary Hypothesis; Jeffrey Pickering and Emizet F. Kisangani; International Studies Quarterly; Vol. 49, No. 1 (Mar., 2005) (pp. 23-43)

After 9/11: Is it All Different Now? Walter Enders and Todd Sandler; The Journal of Conflict Resolution , Vol. 49, No. 2, The Political Economy of Transnational Terrorism (Apr., 2005), pp. 259-277

There are two required deliverables for Week 6 Discussion Forum: Legal Case Analysis Exercise 4: Statistical models

Questions on Exercise 5 (optional)

Assignment 5: Time Series Students will be given a table and/or graphic with data collected monthly or a period of years across several domains. Students will need to address the changes in the data through time and identify any seasonal or domain effect.

Carney, Inc. Page 30 July 9, 2012

3.3 Part 3: Data Collection Methods

3.3.1 Week 7: Quantitative and Qualitative Research Concepts

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Articulate the advantages of using each type of research design (TLO 1, 2, 5)

2. Rationale:

During Week 7, we begin a 5-week section on data collection methods. The first week is an introduction of the concepts of qualitative research and quantitative research.

Quantifying the Issue o How many o How often o What is the effect o Context?

Discovering the “why” and “what ifs” o Qualitative research o Provides context o Provides insight to the numbers

Grounded Theory vs. Scientific method o Hypothesis based on theory

How is theory developed?

o Grounded Theory

Research to develop theory No preconceived notions Qualitative method, not quantitative (statistical noise)

4. Discussion Questions:

To be finalized during content development

Discuss the areas in which security personnel would need qualitative research as opposed to quantitative research.

Are there research areas in which it is not possible to quantify?

Carney, Inc. Page 31 July 9, 2012

5. Class Agenda:

Activity Duration Review the report format presentation materials

80 minutes

Respond to Discussion Forum question 40 minutes assignments will be identified during content development.

Research Methods: The Basics by Nicholas Walliman, chapter 6

How to Measure Anything: Finding the Value of Intangibles in Business, Hubbard, Chapter 8

Identifying Personality Disorders that are Security Risks: Field Test Results; Olga G.

Shechter, Eric L. Lang; Defense Personnel Security Research Center http://www.dhra.mil/perserec/reports/tr11-05.pdf

There are two required deliverables for Week 7:

Assignment 5: Time Series

None

Carney, Inc. Page 32 July 9, 2012

3.3.2 Week 8: Qualitative Research Methods

1. Lesson Goals/Objectives:

At the end of this week, students will be able to—

Identify the main qualitative research methods and when each is appropriate for use

(TLO 1, 2, 3, 5)

Week 8 offers students a deeper look at the types of qualitative research methods. Many of the methods are used to understand organizations and may be especially useful to DoD security personnel to better understand the environment of all DoD personnel.

3. Topic Outline:

Ethnographic research o Embedded researcher o Shadowing o Relationship documentation

In-depth interviewing o Probing questions o Language-used

Group interviewing o Shared experiences o Prompts from peers o Focus groups

Interpretation and limitations

4. Discussion Questions:

Describe qualitative research possibilities in your area

5. Class Agenda:

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Exercise 6: Qualitative Research 40 minutes assignments will be identified during content development.

Research Methods: The Basics by Nicholas Walliman, chapter 10

Cyberculture and Personnel Security: Report II — Ethnographic Analysis of Second Life;

Defense Personnel Research Center; Technical Report 11-03; July 2011

Carney, Inc. Page 33 July 9, 2012

7. Deliverables for this Week:

To be finalized during content development

There is one required deliverable for Week 8:

Exercise 6: Qualitative Research Write a two-page ethnographic report for a unit in which you worked (be vague on the specifics). Describe the relationships within the unit and the groups in which the unit interacts. Does the unit have to rely on other units for it to fulfill its mission? What are the strengths and weaknesses of the relationships?

Carney, Inc. Page 34 July 9, 2012

3.3.3 Week 9: Sampling Concepts

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Articulate the basic concepts of representative samples (TLO 2) Analyze the soundness of a proposed sampling methodology (TLO 1, 2, 3, 4, 5)

In Week 9, the focus switches to quantitative research and the concept of sampling from a population.

Defining the target population, coverage issue with the sample population

Representative sample

Potential consequences of unrepresentative sampling (gaming the system)

Over representative subgroups o weighting

Sampling methods o Simple Random sample o Cluster sample o Stratified sample

Design effect o Simple random sample o Effect of deviation from simple random sample

Uneven probabilities of selection Gains from stratified sample

4. Discussion Questions:

To be refined during content development. Possible topics include:

Census versus sample. Discuss the advantages and disadvantages either asking every unit to report incidents or selecting a percentage.

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Exercise 7: Sampling 40 minutes

Carney, Inc. Page 35 July 9, 2012

6. Reading Assignments:

Assigned readings will cover topics similar to those in the sources listed below. Specific reading assignments will be identified during content development.

How to Measure Anything: Finding the Value of Intangibles in Business, Hubbard, Chapters 9

The Central Limit Theorem Under Simple Random Sampling; D. R. Bellhouse; The American Statistician , Vol. 55, No. 4 (Nov., 2001), pp. 352-357

DoD Personnel Security Program Performance Measures; Leissa C. Nelson, Kent S.

Crawford, David A. Richmond, Eric L. Lang, John E. Leather; Defense Personnel Security Research Center http://www.dhra.mil/perserec/reports/mr09-01.pdf

7. Deliverables for Week 9:

There are two required deliverables for Week 9:

Discussion board Exercise 6: Qualitative Research

Questions on Exercise 7: Sampling (optional)

Exercise 7: Sampling Two part assignment. In part 1, the student selects a stratified sample form a list of units and characteristics on an Excel spreadsheet. In part 2, write a sampling plan for selecting units to be part of a trial for a new incident reporting mechanism. Explain what factors you used to determine the most effective and representative sample.

Carney, Inc. Page 36 July 9, 2012

3.3.4 Week 10: Quantitative Research Methods

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Identify the various types of methods used to collect quantitative data (TLO 2, 3, 5)

In Week 10, the students learn various methods to collect quantitative data.

Measurements o Calibration o Compliance

Surveys o Cooperation / Nonresponse o Sources of error

Operations research o Maintenance of records o Comparability across sites o Self-preservation error

Administrative records o Applicability o Reliability o Accessibility

4. Discussion Questions:

To be refined during content development. Possible topics include:

Discuss the data sources used in your area. What are the sources of error that need to be acknowledged?

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 40 minutes Total 100 minutes

Carney, Inc. Page 37 July 9, 2012

6. Reading Assignments:

Assigned readings will cover topics similar to those in the sources listed below. Specific reading assignments will be identified during content development.

Research Methods: The Basics by Nicholas Walliman, chapter 9

How to Measure Anything: Finding the Value of Intangibles in Business, Hubbard, Chapters 11-12

Preferences and Priorities for Professional Development in the Security Workforce: A Report of the Professional Development Survey; Alissa J. Kramen, Lynn F. Fischer;

Defense Personnel Security Research Center

Public Opinion of Selected National Security Issues: 1994-2000; Suzanne Wood;

Defense Personnel Security Research Center, Management Report 01-4, October 2001

7. Deliverables for Week 10:

There are two required deliverables for Week 10:

Discussion Form Exercise 7: Sampling

Questions on Assignment 5: Written Statement (optional)

Mock RFP for peer review

Students prepare the technical section of a request for proposal for a research or statistical data collection. The students should address procedures for collecting, documenting and reporting the data. The RFP should include a sample design, data collection method, data description and required statistical analysis. A draft of this proposal will be peer reviewed by two students.

Carney, Inc. Page 38 July 9, 2012

3.3.5 Week 11: Data Mining

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Identify the processes used in data mining (TLO 2) Articulate the benefits of data mining in the DoD security environment (TLO 1, 5) Discuss some of the considerations of the government use of data mining (TLO 1, 4, 5)

In Week 11, we address the 21st century research method of data mining. Data mining consists of creating computer algorithms to sift through large amounts of data to detect patterns.

Data Mining Concepts

Data Mining Tools

Data Mining Restrictions

4. Discussion Questions:

To be finalized during content development

What types of data bases are assessable for data mining in the DoD security realm What restrictions exist for DoD data mining operations

Activity Duration Review presentations 60 minutes Respond to Discussion Forum question 20 minutes Provide feedback to others Mock RFP 40 minutes assignments will be identified during content development.

How to Measure Anything: Finding the Value of Intangibles in Business, Hubbard, Chapter 13

Data Mining for Fun and Profit; David J. Hand, Gordon Blunt, Mark G. Kelly and Niall M. Adams; Statistical Science , Vol. 15, No. 2 (May, 2000), pp. 111-126

Privacy and Confidentiality in an e-Commerce World: Data Mining, Data Warehousing, Matching and Disclosure Limitation; Stephen E. Fienberg; Statistical Science , Vol.

21, No. 2, A Special Issue on Statistical Challenges and Opportunities in Electronic Commerce Research (May, 2006), pp. 143-154

Carney, Inc. Page 39 July 9, 2012

Safeguarding Privacy in the Fight Against Terrorism. The Report of the Technology and Privacy Advisory Committee. (March 2004) http://epic.org/privacy/profiling/tia/tapac_report.pdf

There are three required deliverables for Week 11:

Discussion Forum:

Draft Mock RFP for peer review Review two other students Mock RFP

Mock RFP due to instructor

Carney, Inc. Page 40 July 9, 2012

3.4 Part 4: Reporting Statistics

3.4.1 Week 12: Graphing Statistics

1. Lesson Goals/Objectives:

At the end of this topic, students will be able to—

Determine the most effective way to illustrate statistical data to convey information (TLO 4)

Analyze how well or how poorly graphics reflect statistical findings (TLO 4)

2. Rationale:

In Week 12 begins the series on how to report statistical findings. Security personnel need to be able to present statistical information to their superiors in way that is both accurate and understandable. Done correctly, graphs can display complex data in a way to increase comprehension.

Types of Graphs o Bar charts o Point graphs o Line graphs o Pie charts

Multiple Variables Graphs o Relationships o Complexity / readability

Distorted Graphs o Base o 3-D effects

Graphing statistical models o Regression line in scatter plot o Predicted values versus actual values

4. Discussion Questions:

Share…