Att_4_-_ED509_-_Assessment_and_Evaluation_of_a_Department_of_Defense_(DoD)_Security_Program__DDD.pdf

Attachment 4

Final Detailed Design Document

Assessment and Evaluation of a Department of Defense

(DoD) Security Program (ED 509)

Developed for

The Department of Defense (DoD)

Defense Security Service (DSS)

Center for Development of Security Excellence (CDSE)

Education Division

Presented by

Carney, Inc.

Alexandria, VA

Under BPA HS0021-11-A-0091

Order No. HS0021-12-Q-0034

June 24, 2013

DSS CDSE Education Division Assessment and Evaluation of a DoD Security Program

Detailed Design Document

Carney, Inc. Page 2

Document Version

Author Version Date

Carney Inc. 1.0 8/15/2012

Carney Inc. 2.0 6/24/2013

DSS CDSE Education Division Assessment and Evaluation of a DoD Security Program

Carney, Inc. Page 3

Contents

1. Introduction

1.1. The Requirement

1.2. Purpose of the Detailed Design Document

1.3. Organization of This Document

2. High-Level Course Design

2.1. Course Description/Overview

2.2. Credits Conferred

2.3. Target Audience/Prerequisites

2.4. Student Outcomes/Objectives

2.5. Delivery Method

2.6. General Course Requirements

2.7. Grading

2.8. Course Textbooks and Other Readings

2.9. Course Outline

3. Content Outline

3.1. Week 1: Course Overview and the Role of Evaluations in Organizational

Strategic Planning and Management

3.2. Week 2: Evolution of Assessments and Evaluations as a Leadership and

Resource Tool

3.3. Week 3: Examining a Representative DoD Program and Evaluation

3.4. Week 4: Evaluating A Second Defense Program: Older But Still Instructive . 24

3.5. Week 5: Setting Standards and Goals

3.6. Week 6: The Complexity of Devising Standards

3.7. Week 7: Cost Benefit Analysis I

3.8. Week 8: Cost Benefit Analysis II

3.9. Week 9: Designing An Evaluation

3.10. Week 10: Case Study and Exercise on Program Implementation, Improvement and Assessment

1. Lesson Goals/Objectives:

3.11. Week 11: Conducting Evaluations and Determining Measurements

3.12. Week 12: Evaluation Data Analysis

3.13. Week 13: Leadership and Managerial Issues in Conducting and Using

Program Assessments

3.14. Week 14: What is Special About Evaluating Security Programs: Focus on

Assessing Communications Security, Information Security, Personnel, and Physical

Security Programs

3.16. Week 16: Inspectors General, Course Wrap-Up, and Quiz 3

DSS CDSE Education Division Assessment & Evaluation of a DoD Security Program

Carney, Inc. Page 4

June 24, 2013

1. Introduction

1.1. The Requirement

The Defense Security Service (DSS) Center for Development of Security Excellence

(CDSE) has undertaken, through its Education Division, the responsibility to provide graduate-level education to security professionals designed to develop the future security leaders for the Department of Defense (DoD).

The purpose of this project is to design and develop an advanced, semester-long course that will provide an educational opportunity for mid-career security specialists to gain in-depth knowledge and understanding of how to assess and evaluate DoD security programs. The course will develop each student’s ability to use this knowledge to become more effective as security leaders in the DoD. The course will be conducted via the Sakai

Continuous Learning Environment (CLE) and will consist of readings, prerecorded lectures and presentations, asynchronous sessions, a discussion forum, written assignments, and midterm and final exams.

1.2. Purpose of the Detailed Design Document

The purpose of this Detailed Design Document is to convey the learning objectives, topic outlines, and planned assignments for each week of the course. It also proposes the grading schema the instructor will use. This document proposes the organization of content to be delivered throughout the 16-week semester and details the proposed instructional methods to be used for each week.

This detailed design will guide the development of the course materials.

1.3.Organization of This Document

The remainder of this Detailed Design Document is organized into several sections:

Components of the High-Level Design Document:

o An overview of the course o A statement of the anticipated credits to be conferred o A list of prerequisite courses, if applicable o The terminal learning objectives of the course o The delivery method(s) to be used o General course requirements o Grading scheme for the course o A course outline showing how the course will unfold over the semester

A content outline detailing the topics, instructional methods, and assignments for each week

Carney, Inc. Page 5

June 24, 2013

2. High-Level Course Design

2.1. Course Description/Overview

Security professionals occupy a unique position within the DoD. They are highly valued for their contributions to national security; yet because of their perceived narrow focus, they are often times relegated to the periphery of organizations and key decisions. How or where security lands on this scale often depends on the senior civilian or military leadership responsible for advocating and protecting security equities. As with so many things in life, credibility is the key. The higher the degree of professional expertise and credibility a security professional holds, the stronger his or her influence is within an organization.

Accordingly, CDSE has established as a primary goal the development of future security leaders who are adept generalists across the wide range of DoD security responsibilities rather than being focused specialists. To accomplish this goal, CDSE is creating a graduate program in Defense Security Studies, with the Assessment and Evaluation of a

Department of Defense (DoD) Security Program course as an important optional course in the overall program.

The Assessment and Evaluation of a DoD Security Program course contributes to the need for security managers to assess and improve programs under their supervision. It also helps managers understand how and why their programs would be evaluated for effectiveness by outside organizations. The course will:

1. Examine how the evolving field of program evaluation provides an important tool for assessing and evaluating security programs and help decision-makers identify short- and long-term objectives and methodologies to bring about or adopt change.

2. Examine ways in which a senior security manager can collect and analyze data, employ measurements (metrics), and establish a business case to optimally convey performance of security programs, improve performance in effective programs, and determine whether to fix, reduce, or eliminate ineffective programs.

3. Address approaches in which a senior security manager can effectively demonstrate accountability to commanding officers, agency heads, and policy leadership in a credible, objective, and persuasive manner.

4. Explore whether security programs require unique approaches to assessment and evaluation.

This is not a “skills” training course designed to produce expert inspectors. The course will begin with an appreciation of program evaluation basics and the continued evolution of this field. Evaluations, with a goal of improving performance, are not just a managerial nicety. In the American defense establishment, they have a long history rooted in wartime success, expediency, and desire to achieve objectives at the lowest cost. The course will examine part of this historical background. It then moves to a thorough understanding of the processes for establishing standards and criteria. Without

Carney, Inc. Page 6

June 24, 2013 this context, it is impossible to understand what a quality assessment program can and should do. Assessment concepts and processes will be the next focus area as it is a field that has evolved greatly and rapidly in the last two decades. The course will also examine closely the concept and challenges of “cost-benefit analysis.” Finally, the students will apply this background knowledge to explore the unique aspects of program evaluation in the defense arena. The students will be exposed to the broader aspects of assessing and evaluating programs necessitated by expanded security manager responsibilities for a wider range of security programs.

Consistent themes and questions that will be addressed throughout the course include:

1. Student understanding of the evolution and growing role of program evaluation and assessment to assist decision-makers in the best allocation of resources among competing programs

2. Use of metrics and data that demonstrate change and effectiveness

3. Establishment of a business case for building, improving, modifying, or eliminating a program

4. Data collection and analysis to assess success or shortcomings of a program

5. Demonstration of accountability to commanding officers, agency heads and policy leadership

6. Reporting evaluation data in a credible, objective, and persuasive manner

7. Program evaluation challenges in determining successful program performance

8. Exploration of unique methods and challenges to the assessment of the effectiveness of security programs

Because this class is designed for security professionals with varying levels of expertise in differing security disciplines, it takes the combined efforts of all class participants to stimulate discussion and exchange ideas in the learning environment. Adequate class preparation will be required to successfully complete this course.

2.2. Credits Conferred

This course will be designed to equate to three credit hours at the graduate level.

2.3. Target Audience/Prerequisites

This course is intended for DoD civilian and military personnel who perform security leadership and management duties. It is assumed that all students will be prepared to take on graduate-level work in the security field.

2.4. Student Outcomes/Objectives

This course will enable students to:

Examine representative methods used in assessing and evaluating security programs in DoD

Describe how effective assessments and evaluations have become key parts of security programs for mission assurance

Carney, Inc. Page 7

June 24, 2013

Assess the impacts of policies and plans on assessments and evaluations on security programs

Examine the impact of assessments and evaluations from higher echelon and installation level perspectives

Analyze and validate collected data and metrics from assessments and evaluations to effectively justify, modify, or reduce (when appropriate) expenditures for security program requirements

Articulate how cost-benefit analysis, as challenging as it can be, can support establishing and presenting an effective business case for resource allocation

Examine the importance of solid data collection and analysis prior to presenting recommendations to decision-makers

Assess the effectiveness of new and existing security policies and procedures relative to other options and opportunities

2.5. Delivery Method

This is a graduate-level distance-learning course in assessment and evaluation of a DoD security program. The course will consist of readings, prerecorded lectures and presentations, asynchronous sessions, participation in the discussion forum, three graded take home assignments, two short research papers and three quizzes.

Because this is a 3 credit hour equivalent course, the contact time over the 16 weeks should be approximately 30 hours. A typical week will include a prerecorded lecture; it will be followed by a quiz (about one hour duration to complete), a “take home” assignment, or an on-line discussion forum. Generally a discussion will be based on instructor-provided discussion question(s) with each student providing a response and then commenting on other student inputs. Students should be prepared to critically discuss and debate the readings as well as analyze them for biases and multiple perspectives. Students should also be examining how other disciplines relate to the readings and be prepared to discuss this aspect.

The assigned course readings will draw from a variety of resources, such as authoritative readings (legislation, executive orders, policies, plans and strategies, and journals), implementation readings (government products that are responsive to or attempt to fulfill the requirements of authoritative documents), and external reviews (from the U.S.

Government Accountability Office, Congressional Research Service, or other agency or office). Students will be provided with a large number of open access and password protected sites yielding a tremendous number of research assets.

Students will be expected to do research at the graduate level in this course. To provide a substantial research capability to all students in the program, a number of internet-accessible research sites will be sent to each student prior to the first lesson. The primary research site will be the CiteULike virtual library. Students will also receive information for signing on to approximately a dozen other research sites or databases relevant to security and defense studies; one example would be opening an account with the Defense

Technical Information Center (DTIC). This will ensure that every student has more than enough resources to do the research expected in this course. The instructor may provide

Carney, Inc. Page 8

June 24, 2013 additional research sources or sites. Students are also encouraged to make use of library and research sources available to them in their own geographical area or through their own professional or academic networks (such as the Defense Acquisition University and

National Defense University libraries).

2.6. General Course Requirements

Class participation is important and required. If, due to an emergency, students are not able to respond to a discussion in the week it is assigned, they must contact the instructor by e-mail and will be expected to post their response in the following week.

Weekly assignments must be posted in the Sakai CLE by 2359 on the day they are due. It is expected that assignments will be submitted on time; however, students occasionally have serious problems that prevent work completion. If such a dilemma arises, students should contact the instructor in a timely fashion.

2.7. Grading

The following provides an approximate breakdown of how each assignment contributes to the overall performance in the class.

Class discussion (for eight lessons) 15%

Quizzes (lessons 5, 9, 16) 31%

Take home assignments (lessons 4, 6, 10) 14%

Research paper 1 20%

Research paper 2 20%

A letter grade will be assigned to each graded assignment, following the grading scale below:

A = 90% – 100%

B = 80% – 89%

C = 70% – 79%

D = 60% – 69%

F = 59% and below

Individual graded assignments with a score lower than 80% are acceptable; however, a student’s final grade at the end of the semester must be 80% or higher to pass the course.

Evaluation criteria for discussion question responses are listed below.

ASSIGNMENT EVALUATION CRITERIA

Uses complete sentences

Uses proper grammar structure

Responses reflect depth of thought and critical thinking skills

Carney, Inc. Page 9

June 24, 2013

ASSIGNMENT EVALUATION CRITERIA

Integrates material from class/readings into responses

Provides coherent and reasoned responses to all questions

Integrates real world examples into responses

Meets submission timeline

Draft Detailed Design Document

Carney, Inc. Page 10

Evaluation criteria for research papers (grading matrix adapted from design by Professor Jay Aronson, Carnegie Mellon University)

Excellent Good Needs Improvement Poor F

Counter-

Evidence

The author considers the evidence, or alternate interpretations of evidence, that could be used to refute or weaken his/her argument, and thoughtfully responds to it.

Author acknowledges that counter-evidence or alternative interpretations exists, and lists them fully, but does not effectively explain to reader why his/her argument still stands.

Author acknowledges some of the most obvious counter-evidence and alternative explanations, but is not comprehensive in this task. There is little or no attempt made to respond to them.

No acknowledgement of counter-evidence or alternative interpretations.

P

L

A

G

I

A

R

I

S

M

Sources

Note: You should always consult the assignment description to find out what kinds of sources are required.

Evidence is used from a wide range of sources, including lectures and course readings (as appropriate to the specific paper topic). When required, author also consults scholarly books, websites, journal articles, etc. not explicitly discussed in class.

Evidence is used from many sources, but author relies heavily on a more limited set of sources.

Some effort is made to go beyond material presented in class when required, but not much. If outside sources are used, they are primarily non-scholarly

(i.e., intended for a general audience) and/or web-based.

Uses only a few of the sources provided in class, or does not go beyond what has been provided by professor when required to do additional research.

Does not use sources, only minimally uses sources provided by instructor, or relies exclusively on non-scholarly outside sources.

Citations All evidence is properly cited in footnotes or endnotes.

All evidence is cited in footnotes or endnotes, but there are some minor problems with completeness or format of

Some pieces are unreferenced or inaccurately referenced, and there are problems with completeness and

No attempt is made to cite evidence.

Carney, Inc. Page 11

June 24, 2013 some citations. format of citations.

Excellent Good Needs Improvement Poor F

Organization Paper contains an intro, main body, and conclusion.

Introduction lays out main argument and gives an outline of what the reader can expect in the paper. The conclusion brings everything together, acknowledges potential shortcomings of the paper, and gives the reader a sense of what further work might be done to advance the subject matter described in the paper.

Paper contains an intro, main body, and conclusion. The introduction lays out the main argument but gives the reader little idea of what to expect in the paper. The conclusion nicely summarizes the main argument and evidence, but does not move beyond what has already been presented in the paper.

Paper contains an intro, main body, and conclusion. The introduction gives the reader an idea of what to expect in the paper, but does not effectively lay out the main argument.

It may begin with a set of rhetorical questions, or an anecdote that is never fully explained.

The conclusion does little more than restate the problematic introduction. Intro and/or conclusion may be too wordy or short.

Paper has no clear organizational pattern.

Clarity and

Style

All sentences are grammatically correct and clearly written. No words are misused or unnecessarily fancy. Technical terms, words from other languages, and words from other historical periods are always explained. All information is accurate and up-to-date.

Paper has been spell-

All sentences are grammatically correct and clearly written. An occasional word is misused or unnecessarily fancy.

Technical terms, words from other languages, and words from other historical periods are usually, but not always, explained. All information is accurate and

A few sentences are grammatically incorrect or not clearly written.

Several words are misused. Technical terms, words from other languages, and words from other historical periods are rarely explained. Not all information is accurate

Paper is full of grammatical errors and bad writing. Several words are misused.

Technical terms, words from other languages, and words from other historical periods are rarely explained. Not all information is accurate and up-to-date. Paper has

Carney, Inc. Page 12

June 24, 2013 checked AND proofread

(ideally by you and somebody else), and contains no errors.

up-to-date. Paper has been spell-checked AND proofread, and contains no more than a few minor errors, which do not adversely affect the reader’s ability to understand the paper.

and up-to-date. Paper has been spell-checked

AND proofread, but still contains several errors.

Reader’s ability to understand paper may be compromised by these errors.

not been spell-checked or proofread, and contains numerous errors. Reader has a difficult time understanding paper because of errors.

Carney, Inc. Page 13

June 24, 2013

Class Participation (15%):

To meet the requirement for sufficient contact time each week, there will be a combination of recorded lectures by the instructor along with online discussions by and among the students. This approach will be true for eight of the lessons. In a typical weekly lesson, the recorded presentation will be up to 60 minutes long (the student can listen to the presentation in smaller periods if desired). The students will then be presented discussion questions for response to the instructor and then comment on the inputs from other students. The time to complete this online response/comment is generally one hour; exceptions will be noted in individual lessons.

Quizzes (31%):

Three quizzes will take place during the course at select intervals. Each quiz will be the equivalent of one hour of contact time. The first two are worth 100 points and the last is worth 110 points.

Take Home Assignments (14%):

The course will include three take home assignments that focus more deeply on the evolution of and performance of standard setting and program evaluation. Each is distinctly different and is an individual student effort. There will be sufficient readings provided to do each assignment though the student is free to include other materials that he or she finds. There will be additional instructions for each of these assignments.

Research Papers (40%):

Two research papers (approximately 10 pages each) will allow the students to delve more deeply into the challenges of establishing, maintaining, and improving meaningful standards and assessment programs. While each of the papers will be important to the student’s future management and leadership responsibilities in the defense security field, the possible topics for the papers can come from a number of historical or future-oriented perspectives and from the experience or challenges in a wide array of enterprises. The papers will be focused on strategic level work and not tactical (e.g. building an inspection checklist). Each paper will be worth 20% of the total course grade and will be written/submitted in stages. Outside research will be required and the Chicago style of writing used to ensure the instructor’s ability to check sources.

2.8. Course Textbooks and Other Readings

The bulk of the readings for this course will draw from a variety of resources, such as authoritative readings (legislation, executive orders, policies, plans and strategies, and journals), implementation readings (government products that are responsive to or attempt to fulfill the requirements of authoritative documents), and external reviews

(from the U.S. Government Accountability Office, Congressional Research Service, or other agency or office).

Unless otherwise noted, the readings that are in addition to the three textbooks will be in the appropriate readings folders on Sakai for the applicable week.

Required textbooks:

Carney, Inc. Page 14

Boulmetis, John & Dutwin, Phyllis. (2011). The ABCs of Evaluation: Timeless

Techniques for Program and Project Managers, 3rd Edition. San Francisco, CA:

Jossey-Bass

Lockhart, Paul. (2008). The Drillmaster of Valley Forge. New York, NY: Harper

Collins

Wholey, Joseph S., et al. (2010). Handbook of Practical Program Evaluation

(Essential Texts for Nonprofit and Public Leadership and Management), 3rd

Edition. San Francisco, CA: Jossey-Bass

Additional primary sources:

GAO Guide: Designing Evaluations, 2012 Revision (GAO-12-208G, January 2012)

Additional But Not Required Texts:

The Chicago Manual of Style, 16 th

Edition, Chicago: University of Chicago Press, 2010.

Carney, Inc. Page 15

June 24, 2013

2.9. Course Outline

The following table outlines the 16-week course agenda. Graded assignments are in bold.

Week Topics Instructional Method Student Assignments

Due

Course overview

Introductions: Instructor and students

Overview of strategic planning and program objectives

Reading

Asynchronous presentation

Discussion and sharing of research resources

Discussion Forum 1

Turn in the following on first day of class:

1. Academic Integrity statement

2. Student Introduction

3. Student reports on access to research databases and sites

Evolution of assessments and program evaluations

Introduction to assessing and evaluating programs

Introduction to cost benefit analysis

Reading

Asynchronous presentation

Discussion

Discussion Forum 2

Thesis 1 due

In-depth examination of a representative DoD program assessment

Reading

Asynchronous presentation

Discussion

Discussion Forum 3

Bibliography 1

Evolution of assessments in the defense establishment

Defining the evaluation’s scope presentation

Take Home

Assignment #1

History/evolution of standards setting processes and assessment processes presentation

Paper 1 Outline

Quiz 1

The business of standards, criteria, measures, and metrics

Reading

Asynchronous presentation

Take Home

Assignment #2

Carney, Inc. Page 16

June 24, 2013

Week Topics Instructional Method Student Assignments

Cost benefit analysis I Reading presentation

Discussion

Discussion Forum 4

Bibliography 2

Cost benefit analysis II Reading presentation

Discussion

Discussion Forum 5

Paper 1: Rough

Draft of Paper 1

Designs for assessing programs

Reading

Asynchronous instructions and guidance

Thesis 2

Quiz 2

DCIP Exercise on

Challenges in Assessing

Programs

Reading

Asynchronous presentation

Take Home

Assignment#3

Conducting evaluations and assessments

Measurements

Reading

Asynchronous presentation

Final Draft of

Paper 1

Assessment analysis Reading presentation

Discussion

Annotated

Bibliography 3

Discussion Forum 6

Business case establishment

Use of assessment recommendations

Managerial and leadership perspectives

Reading

Asynchronous presentation

Discussion

Annotated

Bibliography

4/Paper 2 Outline

Discussion Forum 7

Focus on assessment challenges in communications security, information security, personnel security and physical security programs presentation

Discussion

Paper 2: Rough

Draft

Discussion Forum 8

Carney, Inc. Page 17

June 24, 2013

Week Topics Instructional Method Student Assignments

Focus on assessment challenges in R&D, operations security, industrial security, counterintelligence, and special access programs

Reading

Asynchronous presentation

Research Paper 2

Final Draft

Inspectors General, ombuds-men, and whistleblowers

What’s ahead in program evaluation

Course Wrap-Up and

Critique

Reading

Asynchronous presentation

Quiz 3

Carney, Inc. Page 18

June 24, 2013

3. Content Outline

3.1. Week 1: Course Overview and the Role of Evaluations in Organizational

Strategic Planning and Management

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Describe the scope of the course, administrative requirements, instructional methodology, evaluation criteria, and feedback processes

• Describe the role of assessments and evaluations in government, non-profit organizations, and the private sector

• Describe the part that assessments and evaluations play in larger management, resource allocation, and performance improvement arena

• Recognize the course contributions potentially available from classmate’s professional backgrounds and current assignments

2. Rationale:

This class introduces basic concepts of evaluations and assessments as part of the larger context of management and performance improvement. While the course does not focus on strategic planning and management per se, evaluations are an important aspect of leadership, change, and efficient allocation of scarce resources.

3. Discussion Questions:

• Why assessments and evaluations?

• What is driving the tremendous push on them in recent decades?

• What can and should be done with the results of assessments and evaluations?

• How new are the concepts of assessment and evaluation?

• How do we distinguish between evaluating a program and evaluating an evaluation system or process?

4. Class Agenda:

• Course introduction: 30 minutes (recorded instructor presentation)

• Review the syllabus and course requirements: 15 minutes (recorded instructor presentation)

• Review of class demographics and professional background/experience: 15 minutes (recorded instructor presentation)

5. Class Activity:

For lesson 1, there is a short Discussion Forum to give everyone practice in using Sakai.

This discussion forum is worth 10 points (which is half what all the others in the course will be worth later).

Carney, Inc. Page 19

June 24, 2013

6. Required Reading:

• The President’s 2014 Budget, pages 47-57, (http://www.whitehouse.gov/sites/default/files/omb/budget/fy2014/assets/budget.

pdf )

• Boulmetis et al, pages vii-ix, 1-15

• Wholey et al, pages 1-8

7. Additional Reading: Scan Lockhart book, Chapters 1 and 2 (about 20-25 minutes worth)

8. Deliverables for this Week: Discussion Forum #1 consists of four items. They are:

1) the biographical sketch per registration/welcoming letter instructions, 2) return of the academic integrity letter to CDSE staff, 3) a report from each student where he or she stands on getting access to the various research sites and databases that will be provided prior to the start of the course. And, 4) there is one discussion question for each student to answer individually plus post a thoughtful reply once others have made inputs to the forum.

9. Assignment for Next Time:

• Complete the lesson 1 and lesson 2 readings; the first week is challenging because you have to make sure you have all the week 1 requirements done plus the readings for lesson 2.

• Thesis 1 for Research Paper 1

Carney, Inc. Page 20

June 24, 2013

3.2. Week 2: Evolution of Assessments and Evaluations as a Leadership and

Resource Tool

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Understand the history of evaluations and assessments as management tools in the government, private, and non-profit sectors

• Describe basic elements of the 1993 Government Performance and Results Act

(GPRA)

• Describe a basic example of cost-benefit analysis as a prelude to writing the first research paper.

2. Rationale:

The concepts of assessment and evaluation and their basic elements are not complicated.

Many of these elements have been around for centuries and are already familiar to students whether they know it or not. But, the impetus has grown very strong in the last several decades to more accurately assess the effectiveness and outcomes of programs.

Practitioners need to know how and why this impetus has grown institutionally.

Entwined with program assessment is the technique of cost-benefit analysis; this too is a relatively simple concept but can be very challenging to execute. It is introduced in this lesson but covered more thoroughly in later lessons.

3. Discussion Questions:

• What are the major factors that accelerated the growth of program assessments and evaluations in the 20 th century?

• What are the basic elements of the assessment and performance improvement process envisioned and outlined by GPRA?

• How does the GAO guide advise executives to approach GPRA?

• What does the GAO guide emphasize as the ultimate purpose of conducting assessments?

• Using the 1984 GAO report on EPA’s environmental regulations as a guide, what are the basic elements of cost-benefit analysis?

4. Class Agenda:

• Recorded instructor presentation: 45 minutes

5. Class Activity:

Discussion forum #2

6. Required Reading:

• Excerpt from Rossi textbook, pages 2-15 (pdf); access via Sakai

Carney, Inc. Page 21

June 24, 2013

• Government Accountability Office, Performance Measurement and Evaluation:

Definitions and Relationships, GAO-11-646SP. (Tri-fold brochure, 2011)

• Government Accountability Office, Executive Guide: Effectively Implementing the Government Performance and Results Act, GAO/GGD-96-118. (June 1996), pages 1-11

• Government Accountability Office, Cost-Benefit Analysis Can Be Useful In

Assessing Environmental Regulations, Despite Limitations, GAO/RCED-84-62.

(April 1984), pages 1-14, 27-33

7. Additional Recommended Reading:

• Scan Lockhart book, Chapter 3 (about 20 minutes; slow down and read more carefully pages 55-59)

8. Deliverables for this Week:

Discussion forum #2

Thesis 1 for Research Paper 1

9. Assignment for Next Time:

See required readings for Week 3. The primary reading is quite long (about 72 pages).

Allow enough time for it.

Annotated Bibliography #1 (for Research Paper 1)

Carney, Inc. Page 22

June 24, 2013

3.3. Week 3: Examining a Representative DoD Program and Evaluation

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Describe, based on an actual recent review of a DoD program, how a notional assessment would look at a program to determine the key factors in its effectiveness.

2. Rationale:

At the outset of the course, even when only operating from a basic understanding of assessments, it is possible to explore a notional (but real) program and how it could be evaluated. A program need not be large or complex to still yield a host of vexing issues and controversial conclusions in the evaluation process.

3. Discussion Questions:

• What are the criteria that an evaluator might use to conduct an assessment or evaluation?

• What are some key issues that quickly emerge between the evaluators and the program owners during and after an assessment?

• At what points can the outside observer begin to see where the evaluation leads to program improvement?

• How do program owners react to evaluation results?

• Are there any obvious lessons learned from how the DoD IG designed the evaluation of the Voting Assistance Program?

4. Class Agenda:

• Recorded instructor presentation: 45 minutes

5. Class Activity:

Discussion forum #3. Expect this to be a more extensive discussion forum as there is potentially a lot to discuss on this lesson (if you have read the assignment).

6. Required Reading:

• DODIG-2013-074 , Assessment of Voting Assistance Programs for Calendar

Year 2012, April 29, 2013

• GAO Guide: Designing Evaluations, Chapter 1

7. Additional Reading:

None

Carney, Inc. Page 23

June 24, 2013

8. Deliverables for this Week:

Discussion Forum #3

Annotated Bibliography #1 (for Research Paper 1)

9. Assignment for Next Time:

There is a large amount of reading for the next lesson but it is easy and will be quick.

Take Home Assignment #1

Carney, Inc. Page 24

June 24, 2013

3.4. Week 4: Evaluating A Second Defense Program: Older But Still

Instructive

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Describe the evolution of assessments within the Department of Defense

• Review an early use of evaluation and performance improvement for conceptual constants in the assessment world

• Demonstrate an understanding of the Government Performance and Results Act

(GPRA)

• Use a tool designed as part of the GPRA process to evaluate a defense program

• Describe the challenges in defining an evaluation’s scope (and the evaluator’s scope of responsibilities)

2. Rationale:

Though the course will expand on the whole field over the subsequent lessons, an early student opportunity to correlate core principles with a well-known American military figure helps build a context for later lessons. During this lesson, students will appreciate why there are different types of evaluations that can look at a low organizational level and/or focus on compliance with task type standards or, conversely, look at much higher levels of an organization and focus on broader, more difficult to examine goals/outcomes.

3. Discussion Questions:

• The discussion questions for this lesson are encompassed in the Take Home

Assignment

4. Class Agenda:

• Recorded instructor presentation: 30 minutes

5. Class Activity:

Take Home Assignment #1 (Instructions will be posted in Sakai)

6. Required Reading:

• Lockhart book, pages 69-168

• Government Accountability Office, Executive Guide: Effectively Implementing the Government Performance and Results Act, GAO/GGD-96-118. (June 1996), pages 12-46

7. Additional Reading:

None.

Carney, Inc. Page 25

June 24, 2013

8. Deliverables for Today:

Produce a 600 min-700 max word (about 2 double-spaced typewritten pages) analysis of the applicability of current GPRA to General Von Steuben’s role in the American

Revolution. Detailed instructions will be posted in Sakai and reviewed during the Lesson

5 class presentation. This will be an individual student effort.

9. Assignment for Next Time:

See required readings for Week 5

Paper 1 Outline

Quiz 1

Carney, Inc. Page 26

June 24, 2013

3.5. Week 5: Setting Standards and Goals

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to describe:

• Why a program evaluator needs to understand how goals are set

• Why a program evaluator needs to understand the process of standards setting

• Why some evaluations will examine compliance with standards

• Why some evaluations will examine the appropriateness of standards

• Why some evaluations will focus on goal achievement

• Why some evaluations will focus on goal appropriateness for certain outcomes

2. Rationale:

In many cases, particularly in the security world, the first place that specialists encounter

“evaluators” are during inspections focused upon compliance with externally drafted and mandated standards of performance or achievement. While this is an important aspect of evaluations and assessments, it leads to the question of “where do the standards come from and why?” Moreover, as one assumes higher levels of responsibility, the standards-based evaluations are replaced (or joined) by evaluations focused more on the achievement of organizational goals and overall performance in the most effective manner.

3. Discussion Questions:

• What does it mean to “set a standard?” Who can do so?

• What would be the effect on processes if standards were not set?

• What would be the effect on assessments that assess processes if standards were not set?

• Most of the literature and guidance on government evaluations and performance improvement discusses measurements against goals or missions rather than standards – is this an important distinction? Why or why not?

4. Class Agenda:

• Recorded instructor presentation: 60 minutes

5. Class Activity:

Quiz #1

6. Required Reading:

• LeMay, Curtis with MacKinlay Kantor, Mission with LeMay: My Story, 1965, Doubleday and Company, Garden City, NY (pages 429-447 excerpted as pdf file and posted with lesson 5 readings)

Carney, Inc. Page 27

June 24, 2013

• “Federal Engagement in Standards Activities to Address National Priorities:

Background and Proposed Policy Recommendations,” Report of the National

Science and Technology Council, October 2011

• Office of the Inspector General, Department of Defense, FY 2012-FY 2017

Strategic Plan (no date)

7. Additional Recommended Reading:

The following will be used as part of the quiz:

• Lt Gen Jim Kowalski, Commander of Global Strike Command, speech, 9 May

2012 (venue unknown) (posted in Sakai)

8. Deliverables for Today:

Outline for Research Paper 1

Quiz #1

9. Assignment for Next Time:

See Week 6 readings

Take Home Assignment #2

Carney, Inc. Page 28

June 24, 2013

3.6. Week 6: The Complexity of Devising Standards

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Understand how standards are devised

• Understand the relation between a standard and a desired or anticipated outcome

• Understand the parameters and uncertainty inherent in standards setting

• Demonstrate in an exercise (Take Home Assignment) the relation between a standard and its anticipated future evaluation

• Understand why evaluators are simultaneously (usually) evaluating both compliance with a standard and the appropriateness of the standard itself

2. Rationale:

While to some, a standard is an absolute and, perhaps, is thought to be the result of a highly rigorous process that set it, that is not always the case. Standard setting is at least as much an art as it is a science. It is ultimately subjective. There are numerous solid organizational and management reasons for setting, promulgating, training, enforcing, and evaluating standards. Yet there has to be a process that is also understands where standards came from and their intended purposes. Other processes may be required that evaluate standards themselves to determine which to keep, which to delete, and which to modify.

3. Discussion Questions:

• The discussion questions for this lesson are encompassed in the Take Home

Assignment

4. Class Agenda:

• Recorded instructor presentation: 60 minutes

5. Class Activity:

Take Home Assignment #2

6. Required Reading:

• U.S. Department of Transportation, National ITS Standards Program, “Setting

Standards for the Future.”

• NHTSA, Updated Review of Potential Test Procedures for FMVSS No. 208, October 1999

• NHTSA, Quick Reference Guide (2010) Version to Federal Motor Vehicle Safety

Standards and Regulations, February 2011.

• NHTSA, NHTSA Vehicle Safety and Fuel Economy Rulemaking and Research

Priority Plan 2011-2013, March 2011.

Carney, Inc. Page 29

June 24, 2013

7. Additional Recommended Reading:

None

8. Deliverables for Today:

Take Home Assignment #2

9. Assignment for Next Time:

See Week 7 readings

Annotated bibliography #2 (for Research Paper 1)

Carney, Inc. Page 30

June 24, 2013

3.7. Week 7: Cost Benefit Analysis I

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Discuss the background and evolution of cost benefit analysis

• Discuss the role that cost benefit analysis plays in program assessments

• Demonstrate familiarity with a number of challenges in measuring costs of a program

• Discuss the critiques of cost benefit analysis

2. Rationale:

A number of courses in the program and much of program assessment are based on the view that informed decision making and wise resource allocation should be informed by

“cost benefit analysis.” The term is used so frequently that it implies that it a well understood and well-practiced technique. This is a highly debatable assumption. It turns out that cost benefit analysis is not only quite challenging to do but it is highly controversial.

3. Discussion Questions:

• Where did the concept of cost benefit analysis as a fundamental part of program assessment and decision making come from?

• Why is it often difficult to determine the costs of a program? How many different factors need to be included in such a determination?

• Why do many critics believe that cost benefit analysis is not only fundamentally flawed as a process but actually a practice that should be reduced or banned?

4. Class Agenda:

• Recorded instructor presentation: 60 minutes

5. Class Activity:

Discussion forum #4

6. Required Reading:

• Government Accountability Office, Cost-Benefit Analysis Can Be Useful In

Assessing Environmental Regulations, Despite Limitations, GAO-05-252.

(February 2005)

• Gerald Shively reading, “An Overview of Benefit-Cost Analysis,” (posted in

Sakai)

• David Schmidtz reading, “A Place for Cost-Benefit Analysis,” (posted in Sakai)

• Steven Landau et al reading, “Applying Benefit-Cost Analysis for Airport

Improvements: Challenges in a Multi-Modal World” (posted in Sakai)

Carney, Inc. Page 31

June 24, 2013

7. Additional Readings:

None.

The following will be used as basis for discussion questions:

• Environmental Protection Agency, Lean Government Metrics Guide, (July 2009)

8. Deliverables for Today:

Discussion forum #4

Annotated bibliography #2 (for Research Paper 1)

9. Assignment for Next Time:

See Week 8 readings

Rough draft for Research Paper 1

Carney, Inc. Page 32

June 24, 2013

3.8. Week 8: Cost Benefit Analysis II

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Discuss the challenges in determining benefits and how to measure or “monetize” them.

• Determine methods for determining the scope of benefits for inclusion in the analysis of a program.

• Explain the responses of cost benefit advocates to common criticisms of cost benefit analysis.

• Discuss how cost benefit analysis should fit into an assessment.

2. Rationale:

Since the goal of many assessments is to determine whether intended program objectives are being achieved at what level and at what cost, nearly all assessments include some version of a cost benefit analysis. Such analysis supports both the determination of a program’s effectiveness and efficiency. As difficult as it can be to determine all the associated costs of a program, it can be even more vexing to determine (and quantify) the actual or intended benefits. Moreover, there are a number of critics who question the rationale, processes, and validity of cost benefit analysis as a tool for decision-making and resource allocation.

3. Discussion Questions:

• Why is it so difficult to measure “benefits” when conducting cost benefit analysis?

• How wide a scope should an analysis pursue in including benefits in the equation?

• What is meant by the “ethical” cost benefit analysis?

4. Class Agenda:

• Recorded instructor presentation: 60 minutes

5. Class Activity:

Discussion forum #5

6. Required Reading:

• Hristos Doucouligiagos et al reading, “Are Estimates of the Value of a Statistical

Life Exaggerated?” (posted in Sakai)

• OECD paper, “Valuing Mortality Risk Reductions in Regulatory Analysis of

Environmental, Health, and Transport Policies: Policy Implications.” (posted in

Sakai)

Carney, Inc. Page 33

June 24, 2013

• Transcribed interview with Frank Ackerman and Lisa Heinzerling, “Pricing the

Priceless.” (posted in Sakai)

• Denise Caruso, “The Myth of Cost-Benefit Analysis.” (posted in Sakai)

• Richard Zerbe, “Ethical Benefit Cost Analysis as Art and Science: Ten Rules for

Benefit-Cost Analysis.” (posted in Sakai)

7. Additional Reading:

None.

8. Deliverables for Today:

Discussion forum #5

Rough draft for Research Paper 1

9. Assignment for Next Time:

See required readings for lesson 9

Quiz #2

Thesis for Research Paper 2

Carney, Inc. Page 34

June 24, 2013

3.9. Week 9: Designing An Evaluation

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to articulate:

• The importance of evaluation design

• Defining the evaluation’s scope

• Clarifying a program’s goals and strategy

• How to think about relevant evaluation questions

• How to select an evaluation design

• Key components of an evaluation design

• Why design is an iterative process

• Criteria for a good evaluation design

2. Rationale:

In this lesson, we pull together a number of themes that have emerged in the first part of the course (and were briefly addressed in lesson 3) on evaluation design. When millions

(or billions) of dollars are at stake or, for example, a program’s readiness to support the warfighter in question, then any assessment of the program requires a carefully thought out evaluation plan from any outside assessors. Those on the evaluation team have a number of choices to make on what type of evaluation to construct and execute.

Questions of breadth and depth are crucial. Assessments are not free; someone has to pay for them. While assessments are generally looking at the effectiveness and quality of a program, the assessment itself can be of higher or lower effectiveness and quality depending upon how well it is designed and resourced.

3. Discussion Questions:

• What are some typical designs for implementation evaluations?

• What are some typical designs for outcome evaluations?

• Which designs help draw causal inferences about program impacts?

4. Class Agenda:

• Recorded instructor presentation: 45 minutes

5. Class Activity:

Quiz #2

6. Required Reading:

Students will read the following items:

• GAO Guide: Designing Evaluations, Chapters 2 and 3

• Wholey et al, rest of Chapter 1(pages 9-29)

• Boulmetis et al, rest of Chapter 1, Chapters 4 and 5

Carney, Inc. Page 35

June 24, 2013

• American Evaluation Association, Guiding Principles for Evaluators (July 2004)

(posted in Sakai)

7. Additional Recommended Reading:

None

8. Deliverables for Today:

Quiz #2

Thesis for Research Paper 2

9. Assignment for Next Time:

See Week 10 readings; the lesson has a large amount of reading and the student is free to do more of it in advance if he or she chooses.

Take Home Assignment #3

Carney, Inc. Page 36

June 24, 2013

3.10. Week 10: Case Study and Exercise on Program Implementation, Improvement and Assessment

1. Lesson Goals/Objectives:

At the end of this lesson, learners will be able to:

• Describe many of the issues involved in trying to build a successful program, improve it, and submit it to the scrutiny of an assessment

• Discuss how the Department of Defense’s Defense Critical Infrastructure

Program is representative of the many challenges embedded in strategic planning for vast and complex programs across many partners

2. Rationale:

A recent in-depth assessment by the Department of Defense Inspector General found fault with the implementation of a major portion of the Defense Critical Infrastructure

Program. The assessment found many issues at work including missing or unclear guidance and partners in the process with different interpretations or incentives to take part in the program. As such, this is a useful case study to show how a program is challenged in its design from the beginning, how the design can impact successful implementation (or the ability to accurately see the implementation), and how a program assessment attempts to clarify what actually has or has not occurred. Moreover, the case study demonstrates that the end of an assessment only means that there are further decisions for leaders to make.

3. Discussion Questions:

• What are the choices available to the evaluator for how to gather data and what are the respective benefits?

• Why is it often difficult to determine the “what should be measured” and “how”, and to ensure consistency in the process?

• Where does the evaluator go for data, and what is it worth?

4. Class Agenda:

• Recorded background and guidance from instructor: 15 minutes

• Exercise: 1 hour and 45 minutes

5. Class Activity:

The instructor will post a background presentation to start the exercise. The primary purpose of the presentation is to review the case study materials (which will be posted on

Sakai) and to emphasize that the case study is to examine both the problems with DCIP and the assessment process that searched out the problems and made corrective recommendations.

Carney, Inc. Page 37

Students will then have a structured process to follow to do their own study of this case and, though they may work in groups, each student will submit his/ her assignment for grading.

6.…