PWS 2 1 - NGC Assessment Conops_home.pdf

01 Feb 10/mod15

National Geospatial-Intelligence College (NGC) Assessment CONOPS

1. References:

a. ODNI DRAFT ICS 2008 600-3, IC Standard for Training and Evaluation, 2008

b. CJCSI 3500.01B, Joint Training for the Armed Forces of the US, 31 Dec 99

c. Council on Occupational Education (COE) Handbook of Accreditation, Mar 09

d. DoD MIL-HDBK 29612-2, ISD/SAT Approach to Education & Training, 30 Jul 99

e. DoDD 1322.18, Military Training, 1 Mar 05

f. DoDD 3305.2, General Intelligence Training, Draft Apr 05

g. DoDI 1430.17, DoD Civilian Education and Professional Development, Mar 06

h. Executive Order (EO) 11348, 20 Apr 67 & EO 13355, 27 Aug 04

i. Kaplan, R. and Norton, D. The Balanced Scorecard. Harvard Press. 1996

j. Kirkpatrick, D., Evaluating Training Programs, Berrett-Koehler Publishers, San

Francisco: CA, 1994

k. Mager, R., Measuring Instructional Results, Lake Publishing, Belmont: CA, 1984

l. Management Accountability and Control, OMB Circular A-123, Aug 05

m. National Geospatial-Intelligence College (NGC), Draft Strategic Plan,, Feb 09

n. National Geospatial-Intelligence NGC CONOPS 2, Curriculum Management CONOPS, Jan 09

o. National Intelligence University, Draft Guidance on Evaluation, Jan 07

p. Scriven, M. Methodology of Evaluation in Gagne’s Perspectives on Eval. 1967

q. Title 5 Code of Federal Regulations, Part 410, paragraph 601. Evaluating Training

r. USAF Handbook 362235-Volume 12; November 2002, Information for Designers of

Instructions Systems, Test and Measurement Handbook

2. Purpose: To establish an approach for implementing assessment policies and procedures for the National Geospatial-Intelligence College (NGC).

3. Guidance: To promote learning and enhance the value of the academic experience, NGC will systematically develop and deploy an institution-wide program for assessment.

4. Applicability/Scope. This guidance applies to all Schools, all training/education programs, and all faculty and staff funded by the NGC.

5. Definitions: Terms are generally defined as they are used throughout the CONOPS. For simplicity, the term evaluation is considered synonymous with assessment. Other frequently used terms are defined below:

a. Accreditation: The status granted an educational institution that has been found to meet or exceed stated criteria of educational quality. The Council on Occupational Education (COE) accredits the NGC and has done so since 2001.

b. Assessment: Assessment is a means for focusing our collective attention, examining our assumptions, and creating a shared culture dedicated to continuously improving the quality of teaching and learning. Assessment requires making expectations and standards for quality explicit and public; systematically gathering evidence on how well performance matches those expectations and standards; analyzing and interpreting the evidence; and using the resulting information to document, explain, and improve performance." (Angelo, AAHE Bulletin, April 1995, p. 11).

c. Instruction System Development (ISD): A deliberate, yet flexible approach for analyzing, designing, developing, and implementing high quality student centered instructional programs.

Using a systems approach, ISD also considers functions within the College such as management, support, administration, delivery, and evaluation. Evaluation is continuous and inherent throughout our approach to ISD (as noted in the graphic). The COE accrediting standards closely align with ISD to help ensure the college operates as a system while delivering the highest possible level of academic quality to our students, customers, and stakeholders. Additional information about ISD can be found within the NGC Curriculum Management CONOPS.

d. Level-One Evaluation: Standardized survey instruments provided to students at the end of a course or block to solicit their reaction to instruction.

e. Level-Two Evaluation: Course specific instruments (tests, exercises, simulations, case studies, self assessments, and/or observations, etc.) that are employed by an instructor to assess how well students have advanced in terms of gaining new knowledge, skills, or attitudes associated with the satisfactory achievement of the course objectives.

f. Level-Three Evaluation: Course specific instruments (post-graduate survey, interviews, etc.) that are employed to assess the degree by which knowledge, skills, or attitudes are transferred and/or applied into the every day environment of the learner 3-9 months after graduation. Such instruments may be given to the graduate and/or the supervisor.

g. Schools: Schools are considered to include: the School of Leadership and Professional

Development (SLPD) and the School of Geospatial Intelligence (TSG).

h. Programs: Programs represent a collection of courses that are hierarchically arranged to achieve specific programmatic outcomes. At the time of this writing NGC had three formal programs to include: Geospatial-Intelligence; Leadership and Professional Development, and Acquisitions.

i. Provost: The Office of the Provost includes the following functional areas: Curriculum Management, Assessments, Faculty Development, Registrar, E-Learning, and Web Services.

j. Enabling Offices: Two internal offices enable the College’s academic programs:

Business Executive (BX) and the Office of Community Engagement (OCE).

k. Steering Committees: Three steering committees help shape the College’s policies and actions going forward. The Executive Steering Committee (ESC) is composed of seven senior College members: the D, the DD, SLPD Dean; TSG Commandant; the Provost; the BX; and the Chief of Community Engagement. The Accreditation Steering Committee is composed of 15-20 members of the College who maintain responsibility and oversight for specific accrediting standards. The Faculty Development Steering Committee (FDSC) has a similar role, but focused on faculty-related issues. All three committees and their subordinate working groups strive to assure quality and continuous improvement.

6. Responsibilities:

a. The Office of the Provost will: (focal point within the Provost Office)

(1). Serve as a focal point for Institutional Accreditation. (Assessments)

(2). Help the schools define the standards and processes used to review/assess instructors, students, and curricula. (Assessments, Faculty Development, and Curriculum Management)

(3). Design, develop, and approve the end-of-course level-one survey instruments and associated collection methods used by the Schools. (Assessments)

(4). Process all level-one end-of-course surveys and provide summary reports back to the Schools and the Curriculum Management Branch. (Assessments)

(5). Develop assessment strategies and, as required, validation and testing instruments for courses under development. (Curriculum Management)

(6). Develop a NGC-wide approach for the use of post-graduate surveys and closely coordinate that approach with the Schools, Stakeholders, and the Curriculum Management Branch. (Assessments)

(7). Maintain an evaluation database and publish quarterly reports based on collected data back to the Schools and Curriculum Management Branch.

(Assessments)

(8). Publish comprehensive trend data for each course and distribute to the School focal points upon the conclusion of each session. (Assessments)

(9). In coordination with the Faculty Development Branch and the Schools establish standards for instructor performance. (Assessments)

b. Schools: The Schools are responsible for designing and implementing a responsive and meaningful assessment program that aligns with the guidance established by this policy. Each School will:

(1). Ensure their educational programs align with accrediting standards.

(2). Appoint a POC to interface with the Provost Offices

(3). Provide students an overview of the NGC Evaluation Program as part of their initial course orientation (see ANNEX 1).

(4). Ensure all courses administer a level-one (L-1) survey approved by the Assessment Branch.

(5). Ensure all courses implement a valid and reliable assessment strategy to ensure the satisfactory attainment of course objectives. Coordinate any changes to this strategy through the Curriculum Management Branch.

(6). Conduct and administer student evaluations (L-2) to measure learning and assure student proficiency consistent with the objectives. Such assessment may be formal through testing and practical exercise or informal through observation, questioning or peer critiques. If formal in nature, the Schools should strive to implement the controls and techniques addressed in Annex 11 at the earliest opportunity.

(7). Assure all courses have a learning transfer strategy. Coordinate with the Assessment Branch prior to the deployment of any non-standard level- three (L-3) surveys.

(8). Take action, where appropriate, to improve instructional programs based on L-1, L-2, and/or L-3 feedback; coordinate such actions with the Curriculum Management Branch if it pertains to or changes the curricula.

(9). Participate in data calls consistent with accreditation standards.

(10). Participate in existing internal control efforts (Program Reviews, Advisory Boards, Course Manager Checklists, Program Supplement Updates, Catalog Updates, Self Assessments, Data Calls etc), while developing and/or refining additional controls (Monitoring and reporting student proficiency data; and reporting instructor assessment status to date) which may be unique to the school.

(11). Conduct internal academic program reviews at least annually.

(12). Comply with all standards and policies governing NGC operations, while striving to enhance congruency, currency, consistency of instruction, and documentation of competencies and associated course documentation.

7. Accreditation and Evaluation: Simply stated, accreditation frames the NGC approach to institutional assessment.

Consistent with our status as an accredited institution, the NGC adheres to certain community standards. Alignment with these standards ensures the continued quality of our educational programs and assists in the continuous improvement of the College. Every year the NGC conducts a comprehensive Self-Assessment based on the ten COE standards (see themes below) and related education criteria found in ANNEX 2.

1. Mission 6. Physical Resources

2. Educational Programs 7. Financial Resources

3. Institutional/Program Outcomes 8. Human Resources

4. Strategic Planning 9. Organizational Structure

5. Learning Resources 10. Student Services

11. Distance Learning

The Self Assessment may take many shapes depending upon the issues raised and gaps identified from January to August. Regardless of the method used to conduct the assessment, the results are then documented in an annual report that is reviewed by senior leadership and forwarded to

COE.

Although the formal Self Assessment process occurs in October of each year, there are many events that occur throughout the year leading up to the assessment and, in turn, following the assessment to include periodic reviews sponsored by the Schools, the Provost, and the BX.

Further, every 6-7 years, COE Examiners conduct on-site visits to the College to verify adherence to those published standards. To assist in these visits, the College publishes a comprehensive Self Study Report to document its alignment to the criteria and orient the visiting examiners. Additionally, the College uses that report as a living document for professional development and as a baseline for the conduct of recurring annual self assessments.

To ensure alignment to the standards, at least quarterly, the Provost convenes a meeting of the Accreditation Steering Committee (ASC). The ASC is chaired by the Provost. The forum is composed of senior members from within the NGC (TSG, SLPD, Matrix, Provost, OCE, and BX). The ASC meetings serve as a means to:

Encourage familiarity with the COE conditions and standards.

Promote faculty and staff involvement in the accreditation process.

Reaffirm the NGC’s alignment with its accrediting criteria by providing strengths and areas for improvement relevant to the standards.

Shape the College and its programs consistent with the standards Solicit information needed to write the Annual Report.

Provide guidance relevant to addressing deficiencies.

Appoint teams, as required, to lead the Self Study/Self Assessment efforts.

During its quarterly meetings, the ASC will review a variety of issues to include the standards associated with each program/service area. The ASC will seek evidence of alignment with all relevant accrediting standards and work to reinforce that alignment wherever required. Sample themes covered during the ASC meetings include:

Educational program objectives and course offerings are reviewed annually.

All course offerings, regardless of location, time, and/or mode of delivery are comparable with those offered at the main campus.

Programs and courses are adequately reflected in the course catalog.

Programs provide instruction in the competencies essential to success within an occupation or leadership area, including job knowledge, skills, work habits, and attitudes. Non-productive courses are terminated or improved.

A systematic and objective means to evaluate student achievement is in place based on required competencies and objectives.

Academic and student services are responsive to the needs of the faculty/students.

Faculty actively participate in a systematic program of customer engagement and professional development, which helps assure congruency and currency in their technical areas.

8. The Role of the Accreditation Steering Committee in Substantive Changes: The ASC also plays a key role in assuring that change within the College is orderly and made consistent with standard. To do so, the ASC and the Executive Steering Committee partner to review and ultimately approve all potential substantive changes before they are enacted. A substantive change is defined by COE as “any change which significantly alters an institution’s scope, programs, locations, and/or standing within its community.” A list of common substantive changes reported to COE is found at ANNEX 3. The review process for these changes is as follows:

ASC is advised or becomes aware of a potential substantive change

If the change is indeed substantive, then the ASC Standard Leads review the proposed action to assure it is taken consistent with standard. For example, the creation of a new program or branch campus would first be reviewed by the ASC membership (using the standards) to ensure that the proposed change aligns to the mission and organizational structure; is consistent with our academic programs; is linked to assessment and strategic planning; and is adequately resourced relative to equipment, money, manpower, facilities and student services.

The ASC would then make one of three recommendations to the ESC:

a. Recommend ESC approval and COE notification

b. Recommend ESC hold any action in abeyance until issues are resolved

c. Recommend ESC disapprove

9. Strategic Planning, Metrics, and Management Controls: Strategic Planning, Metrics, and Internal Controls represent key themes embedded within our accrediting standards that further help support and frame our overall approach to evaluation.

a. Planning: The NGC strategic planning efforts consist of vision, mission, and goal statements that represents the direction for the future. Such a direction, is, in turn, documented and aligned within the recently published NGC Strategic Plan (shown to the right).

Going forward, what does the future of the NGC look like? At its core, NGC is and always will be a learning institution -dedicated to meeting the learning needs of GEOINT professionals and continuing NGC’s transition to a premier learning organization.

That transition begins with a well aligned vision and mission statement:

Building upon the framework above, NGC has defined four strategic goals leading to 2015:

o Goal 1: Align learning opportunities to the Agency’s mission, strategy, and goals o Goal 2: Effectively and efficiently manage NGC to meet the GEOINT mission training requirements o Goal 3: Enable learning opportunities that are innovative, flexible and tailored to the needs of GEOINT professionals o Goal 4: Leverage resources, capabilities and technology to serve learners in accomplishing their mission worldwide

Going forward, NGC will continue to refine objectives, measures, and milestones to track initiatives associated with these strategic goals.

b. Metrics: Within this system, metrics are used to determine the health of our major processes. The Office of the Provost and the Schools work to establishes cascading measure of student and course performance which collectively provide insight into key academic metrics to include:

o L-1 Indicators of Student Reaction (Provost):

Satisfaction Relevance Instructor Quality Course Quality Academic Support Pace, Length and NPS values o L-1 Indicators of Learning (Provost):

Understanding Confidence Improvement Extent of improvement o L-1 Indicators of Transference (Provost):

Intent to use % intend to use Perceived impact o L-2 Indicators of Student Learning (School):

Results from course-specific assessments to include, but not limited to::

o Pre and post tests o Post tests o Practical exercises o Games/simulations o Questions o Observations o L-3 Indicators of Learning Transfer (School and Provost):

Post-graduate survey of students:

o Learning o Confidence o Performance Improvement o Time to use

Post graduate survey of supervisors o Performance improvement o Supervisory recommendation o Throughput Indicators: Fill rates and no-show rates (Registrar) o Quality Progress Indicators: The Annual Self Assessment process provides opportunities for the faculty and staff to reflect upon the status of existing processes and explore opportunities for improvement. NGC uses four measure to assess each of its over 120 accrediting criteria:

Best Practice Meets Standards Needs Improvement Non-compliant

Additionally, during the annual review initiatives are identified, which strengthen the College’s overall approach to quality and continuous improvement.

c. Internal Control Program (ICP): Internal controls represent the integration of policies and procedures that provide reasonable assurance that our operations, programs, and activities are effectively and efficiently managed in conformity with laws and regulations, and that these programs achieve their intended outcomes.

Every member of the NGC cadre has a role within the college’s ICP; some establish NGC policy, some execute procedures, and some provide feedback or oversight. Within the NGC, our ICP is centered around the accreditation standards. The criteria embedded in the COE standards serve as informal control objectives that focuses everything the NGC does around the eleven central themes and associated criteria previously referenced.

To assure alignment with these standards we have implicit and explicit control techniques to include: bi-annual program reviews, quarterly Accreditation Steering Committee meetings;

formal guidance (Curriculum Management CONOPS, Assessment Policy, Faculty/Staff Handbook), and recurring Self Assessments.

d. Quality Enhancement Plan (QEP): Using the annual review and the ICP processes, the ASC self assesses the institution annually and then identifies improvement opportunities and initiatives. These initiatives are designed to shore up the College approach to the criteria and assure continuous improvement throughout the enterprise. The initiatives are generally published in Jan/Feb and tracked by the Provost until resolved. An example of the QEP Template is provided below:

10. Procedures for Assessing Course Materials: Consistent with our accrediting standards, all NGC course materials will be built using a systematic set of instructional system development (ISD) procedures used to analyze, design, develop, implement, evaluate, and maintain training programs as outlined in the NGC Curriculum Management CONOPS. All NGC courses will be based on requirements for competencies tied to knowledge, skills, abilities, and/or attitudes needed for specific occupations within the NGA, the Department of Defense (DoD) or the Intelligence Community (IC). These requirements will drive the design of strategies used to teach, assess, and, ultimately, transfer learning to the workplace. Consistent with the philosophy of mastery learning, satisfactory attainment of course objectives will be the criterion for completing NGC courses.

An assigned Course Manager will manage each NGC course. Course Managers are responsible for ensuring their courses are valid, reliable, current, properly documented, taught consistent with NGC published guidance/our accrediting standards, and continuously improving based on the analysis of a variety of feedback mechanism to include: direct observation, customer engagement, student and post-graduate surveys, test results, instructor evaluations, and/or discussions with the students’ employers/stakeholders. Senior NGC faculty and staff will also exercise additional oversight responsibilities as required.

To ensure adequacy of course materials, the Schools will annually review their course documentation as part of the yearly self assessment process using the Course Manager Checklist (ANNEX 4). Additionally, the Curriculum Management Branch will manage/oversee these annual data calls and may conduct periodic reviews of high-interest NGC courses consistent with the NGC Curriculum Management CONOPS.

11. Procedures for Assessing Classroom Instruction:

a. For all NGC courses and all instructors at all locations:

(1). The Office of the Provost will:

Manage the overall approach to instructor assessment (see ANNEX 10).

Partner with the Faculty Development staff to integrate instructional standards into faculty/staff training and certification programs.

Develop a standards-based instructor assessment form for both formative and summative feedback that provides diagnostic information to faculty members on their teaching (see ANNEX 5 ).

Identify and train course managers, senior faculty, and staff as assessors. Only personnel selected by the schools, trained by the Provost, and ultimately scheduled by the schools may conduct evaluations of school faculty.

Communicate systemic deficiencies to the professional development staff and Schools for inclusion during future in-service workshops.

(2). The Schools will ensure the policies prescribed in Annex 10 are adhered to, some of which are highlighted below:

Only highly experienced and properly trained assessors approved by a

Dean, Commandant, or Division Chief may assess other faculty within their schools or departments

Evaluators use the most appropriate NGC Instructor Assessment Form, when conducting the instructor assessments.

Schools/Program leads will manage the overall performance and evaluation of their instructors. In doing so, they will establish a schedule by which all instructors are systematically assessed on at least an annual basis consistent with this CONOPS.

1. Purpose of Assessment: To provide feedback to the faculty so that they may become more reflective about their professional practice and, ultimately, more effective teachers and leaders in their classrooms

2. Team Approach: It is strongly recommended that assessors work in teams whenever possible (see Annex 10). Immediately following the presentation, teams should discuss their observations and create an integrated critique sheet that reflects their consensus of instructor performance relative to the standards.

3. Department Approach: Consistency between assessors is critical within a department and, ultimately across the College. It is recommended each Department identify and use their best assessors as a means to help assure alignment to the standards.

4. Frequency: At a minimum, every member of faculty must be assessed twice annually (i.e. a formative and a summative assessment). Some faculty may require more assessments, but never less.

12. Procedures for the Conduct of Level-One through Level-Three Evaluations: To augment the College’s customer engagement strategy, the NGC also follows up with its graduates through an end-of-course AND the post-graduate survey process. Responsibilities associated with each are described below:

a. End-of-Course (EOC) Level-One Evaluations (Student Reaction to Instruction):

Research, design, implement, and maintain the standardized level-one EOC survey in both hardcopy (SCANTRON/SPSS) and/or web-based formats (ANNEX 6).

Provide surveys to the Schools for dissemination to the students.

Aggregate student survey data, analyze the results, and provide custom reports for each course within two weeks of receipt.

Highlight to the Schools indicators of potential problem areas and/or courses that fail to participate in the Level-One Survey Program.

Aggregate course data and include such data in the written quarterly reports provided to the Schools and the Curriculum Management Branch.

Trend the data for each course and provide that data to the Schools, while allowing Department Heads a secure means to pull the data from a common Provost portal.

Post the level-one metrics on a common database available to the Schools and the Curriculum Management Branch

(2). The Schools will:

Employ the NGC end-of-course survey at the end of each course

Ensure only NGC Assessment Branch approved level-one EOC surveys are employed.

Permit on a case-by-case basis, a small number of new courses and/or significant blocks/modules within existing courses to deploy a non-standard level-one survey.

Coordinate any non-standard survey through the Assessment Branch and the Curriculum Management Branch prior to use.

Schedule at least 10 minutes of class time for students to complete the survey.

Strongly encourage all students to complete the survey.

Establish controls to ensure the confidentiality and security of collected surveys and data. To encourage participation and confidentiality, the Office of the Provost strongly recommends the following:

Instructors remind the students the value of feedback to NGC and strongly encourage participation.

Instructors provide directions on how to properly fill out blocks A-J.

Instructors then leave the room and ask the students to place the completed surveys in an envelope on his/her desk. A proxy may also be employed to administer the survey in lieu of the assigned instructor. Ultimately, an environment needs to be established where students feel free to offer constructive and anonymous feedback relevant to the class they just attended.

The instructor or the proxy then counts the surveys, annotates the tracking sheet (ANNEX 7), and seals the envelope for delivery to the Provost Office within two weeks of course completion. The information provided by the students is considered sensitive from the moment it is collected through processing, distribution, and storage. It is inappropriate to alter, copy, disseminate, and/or release raw survey data without prior approval from Course Manager, Department Head, Dean, or the Provost.

Complete a survey tracking sheet and return with the student surveys to the Assessment Office w/i 10-days of course completion

Review the end of course survey report and data provided by the Provost Office.

To protect the interests of both the students and the faculty the EOC Survey Report should never be released or forwarded to third parties outside of the College, without first being reviewed, sanitized, and/or aggregated. Any request for release of raw data or EOC reports to third parties must be approved by the Course Manager, Department Head, Dean, or Provost.

In partnership with the Curriculum Development Branch, analyze patterns, and implement course improvements as required.

Coordinate with the Provost Office on any requested changes to level-one surveys.

b. Level-Two Evaluation (Evaluation of Student Learning):

Provide guidance and conduct annual data calls to assure courses have a viable assessment strategy to include those courses currently under development.

Provide guidance to the Schools relative to formal testing/assessment (See Annex 11-12)

Maintain an assessment strategy for each course, based on instructionally sound objectives, that provides reasonable assurance of student proficiency.

Annually review all program/course objectives to assure currency and alignment with existing requirements and assessment tools/techniques

Work closely and continuously with the Curriculum Development or Management Branch to ensure that level-two assessments (tests, practical exercises, self assessment, observation etc.) accurately and reliably capture evidence of advancing knowledge, skills, or attitudes consistent with the course objectives.

Work to establish metrics at the course, program, and school level which provide evidence of student learning/proficiency

Work to assure compliance with the testing guidance contained within Annex 11 of this CONOPS

c. Level-Three Evaluations (Transference of Knowledge to the Job):

Deploy a systematic approach to level-3 evaluation that clearly describes the extent of learning transfer from the classroom to the job.

See L-3 Instrument at ANNEX 8.

Coordinate with the Schools to identify exceptional, new, revised, or otherwise high value courses that may require level-three evaluations 3-9 months following graduation.

Focus the L-3 program on students who completed a course 3-9 months earlier and/or the supervisors of those students.

o Completer Goal: 18 courses per year or 33% of NGC courses every 3-years o Supervisor Goal: 4 high-value courses per year

Sponsor meetings between the Schools and the client to discuss, and where possible resolve any school or work center problems that negatively impact KSA transference.

Assure all courses employ a learning transfer strategy.

Coordinate any non-standard level-3 evaluations through the Provost

Office prior to release to the proper customer/work segment.

Use the survey results to continuously improve the program/course.

1. Data Release: Any request for the release of raw survey data and/or specialized reports to third parties (outside the College) must be approved by the Course Manager, Department Head, Dean or Provost. The level of approval is based upon the breadth and scope of the data request. .

2. Data Usage: End-of-course and post-graduate surveys are designed to solicit student insights into course performance and should be primarily used for identifying both potential areas for improvement and the discovery of emerging best practices. .

3. Need for Complementary data: End-of-course and post graduate survey data represent three perspectives (current students, former students, and their supervisors) of course performance. Where ever possible these perspectives should be appropriately balanced by other complementary qualitative and quantitative indicators (i.e. insights from instructors, assessors, mentors, academic administrators, department heads, curriculum managers, and/or registrars).

13. Spot Checks and Staff Assistance Visits (SAV): The Provost may at any time conduct spot checks and staff assistance visits in partnership with the Schools and Teaching Departments.

These visits may entail a review of instructors (using the Instructor Assessment Critique Sheets);

a review of courses (through spot-checks of the Course Manager’s Checklist); and/or any number of other emerging special interest items.

14. Procedures for Soliciting Out of Cycle Feedback: The procedures described above provide an opportunity for the students to offer systematic feedback at the completion of each course and up to nine months following graduation. Other venues may also be used to ensure students always have a means to provide timely and anonymous feedback or suggestions to School officials on subjects ranging from educational programs to support processes. For all NGC courses at all locations:

a. The Office of the Provost will:

Establish a suggestion box or other means for students to provide immediate feedback and/or suggestions to the Schools and/or the Assessment Branch prior to course completion.

Design and provide to the Schools a standardized suggestion form that students can fill out prior to course completion should an issue arise that needs immediate attention. (see ANNEX 9)

Ensure blank suggestion forms are available on every campus.

Provide a centralized and secure location on each campus by which suggestions can be anonymously submitted.

Check the suggestion box and take action as required to address emerging issues.

b. The Schools will:

Establish and publicize the suggestion program as well as any additional means for students to raise concerns relevant to their training (visit division chief, Chief of Staff, etc).

Take action, as required, to address any relevant suggestions raised and coordinate such actions with the Curriculum Management Branch as required.

15. Academic Relationships outside of Non-NGC Declared Campuses: This CONOPS primarily focuses on the College as represented by its three declared campuses: a main campus at Ft Belvoir; a branch campus in St. Louis; and an extension campus in Bethesda. Occasionally, schools may enter into relationships with customers that require delivery of instruction at non- NGC campuses.

A common example is Mobile Training Teams (MTTs) led by NGC faculty. If an element of an NGC-sponsored program is taught outside the NGC footprint by NGC faculty, the responsible Department Head must ensure that the deployed training continues to align with the College’s accrediting standards and internal policies. Specifically, Department Heads must ensure that the instruction offered by NGC faculty regardless of location is current; is congruent with the occupational needs of those served; is of high quality; and is generally taught in a manner consistent with the delivery of the course at the main campus.

However, sometimes external organizations prefer not to have MTTs and instead may request to physically take possession of NGC courseware or they may request both the courseware and the training of their staff to teach the material for their own purposes (ie. Train the Trainer {T3}).

Additionally, some organizations may ask that their instructors be certified in either a T3 or Adjunct capacity. There are an endless number of possible relationships and combinations of requests which the Schools may encounter. It would not be practical to attempt to shape policy for each possible engagement or anticipated request, however, such engagements must conform to existing academic standards, published policies, and where applicable, the College’s functional management responsibilities. Consistent with such guidance as a general principle, the College and/or its Schools shall not:

formally or informally classify any non-NGC declared campus as a branch, extension, subordinate campus, extended learning site or classroom without the expressed written approval from the NGC Accreditation Liaison Officer, the Director NGC, and the Council on Occupational Education.

certify as Adjunct Instructors, Instructors, or Assistant Instructors individuals from external organizations who exclusively teach outside the three NGC-declared campuses and who lack recurring visitation and oversight from permanently assigned NGC faculty/staff.

export courses or certify instructors for teaching courses which are not active in the School’s Program (i.e. inactive courses which the College no longer teaches nor maintains).

require end-of-course surveys to be processed by the College for exported courses taught by non-NGC faculty at non-NGC locations.

register students who attend instruction at non-NGC campuses facilitated by non-NGC faculty nor consider them in annual student throughput data reported to COE and/or

ODNI.

These general principles frame the field of play and represent the initial starting point for each engagement. Because they can be shaped based on each unique circumstance, every external relationship (outside traditional MTT delivery modes) must be thoughtfully considered and pre-coordinated with the stakeholders within the College that may be required to execute, support, or defend elements of that emerging relationship (Provost {Faculty Development Chief, Accreditation Chief, End-of-Course and Post-Graduates Survey Program Managers; Registrar, Curriculum Management, Chief}; BX {potential resourcing to include either receiving or providing funding}; OCE {Outreach}) long before any external agreements are framed.

Because NGC’s curricula lies at the heart of many of these emerging relationships these engagements must be carefully considered and, ultimately, documented. As such, the School sponsoring the relationship should document and coordinate that relationship through any number of means to:

Establish the requirement for a relationship and the roles and responsibilities of each party in upholding the terms of that relationship.

Establish agreed upon internal controls needed to mitigate potential risks to consistency, congruency, currency and academic quality inherent in such a relationship.

Provide the stakeholders visibility and oversight into the relationship so as to assure that it aligns with accrediting standards; emerging assessment, curriculum management, and faculty development guidance; and, in some cases, emerging functional manager responsibilities.

The internal controls that may be exercised in support of these relationships are highly varied and entirely dependent upon each unique engagement and their proposed outcomes, but may be as simple as recurring visits to the main campus to talk with the NGC course manager about curriculum changes; participation in periodic curriculum reviews offered by the School or Department; and the collection of metric data showing the number of students who received that instruction per year. At the other end of the spectrum, deeper relationships may involve instructor certification and oversight, review of level-1-3 assessment data, student certification, annual requirement to complete course manager checklists, and/or spot checks of campus facilities. At the end of the day, common sense, judgment, and good communication are required in balancing the level of control with the level of perceived risk.

To assist in establishing such internal controls, a nine-item checklist is shown below and is also provided at ANNEX 13. The checklist should be used by Department Heads as a consistency check conducted prior to offering any NGC course delivered exclusively by Adjunct Faculty at a remote (non- NGC) location and then annually thereafter.

Ultimately the reputation of the College resides with its faculty and its curricula. Any time one is separated from the other, the risks and rewards must be carefully weighed.

16. Community Affairs: The Community Geospatial-Intelligence Training Council (CGTC) is a DoD and Intelligence Community forum chaired by a representative from the College’s Office of Community Engagement. The CGTC has the following responsibilities:

Validate overall geospatial-intelligence training requirements Develop and recommend geospatial-intelligence training requirements Propose, develop, and validate geospatial-intelligence training standards Implement geospatial intelligence training standards and related evaluation programs across the CGTC community Manage the creation of a network of informal Extended Learning Sites and their evolution to formal accredited status

Serving as one of several active listening posts for the NGC, the CGTC meets semiannually to follow up with and record the voice of one of our most important customer groups. It is a profoundly important community forum whose feedback directly shapes and improves the

Talking Paper on the NGC Evaluation Program

The purpose of the NGC Evaluation Program is to promote learning and enhance the value of the NGC academic experience.

- To achieve this goal, NGC systematically assesses its students, faculty, and training curricula.

Student Assessment: NGC will evaluate and, in turn, solicit feedback from our students.

- Using observation, exercises, and/or tests, NGC will determine if the intended learning objectives were achieved by our students.

- Using end-of-course surveys and/or focus groups, NGC will solicit feedback from students on their reaction to instruction immediately following course completion.

- Using post-graduate surveys, NGC may solicit feedback from students and/or their supervisors 3-9 months after instruction to determine the relevancy of their training to their jobs.

- Using suggestion forms students can provide anonymous feedback to the College at any point during their training experience.

- NGC has an open-door policy. Students are highly encouraged to discussion any issue with their instructor, department head, and/or other members of the College faculty and staff

Faculty Assessment: NGC internally evaluates the performance of assigned faculty and staff.

- To provide diagnostic information to faculty members on their teaching

- To enhance faculty professional growth and development

Curriculum Assessment: NGC continuously evaluates its curricula.

- To assess the effectiveness of the curriculum in accomplishing the NGC training objectives

- To provide feedback necessary to keep the curriculum dynamic and responsive to NGA, the DoD, the IC, Joint Forces, and Homeland Security needs

- To comply with our national accreditation standards granted through the Council on Occupational Education (COE)

The NGC Evaluation Program is a systematic approach to assessment based on internal and external feedback. At the end of this course, you’ll be asked to provide your reaction to the class you’re now taking, please take advantage of that opportunity to help us continuously improve.

DRAFT

DRAFT 1

Federal Standards and Criteria 2010 Edition

Council on Occupational Education

DRAFT 2

Institutional Mission

YES NO

1. The primary mission of the institution is to instruct students to such competency levels essential to success in their occupations, including job knowledge, job skills, work habits, and/or attitudes.

2. The institution’s mission is clearly and concisely stated in written form and represents the official statement of the institution.

3. The institution has an appropriate hard-copy and/or online publication which it uses to accurately present its mission statement and the educational courses and/or programs offered to achieve its mission.

4. The current mission statement is consistently used by the institution and is made available to the community that it serves (i.e. DoD, Navy, Army, USAF, USMC, Federal Agencies, etc).

5. The institution promotes its mission to the community that it serves.

Items in YELLOW represent changes from CY09

DRAFT 3

STANDARD 2--CHECK SHEETS

Educational Programs: Composite and Individual Program

Admissions/Recruiting YES NO N/A

1. Information about the institution’s academic courses/programs, prerequisites, and associated policies are clearly stated and made available to its students.

2. The institution clearly defines and publishes a policy on the transfer of students between programs within the institution and the transfer of students from other institutions.

3. Course and/or program requirements offer reasonable expectations for successful completion regardless of the delivery mode.

Programs YES NO N/A

4. Occupational education courses and/or programs offered by the institution are congruent with the mission of the institution and with the occupational needs of the people served by the institution.

5. The objectives for each educational course and/or program are evaluated annually.

6. Each occupational education course and/or program has clearly stated objectives, defined content relevant to these objectives and the current needs of business and industry, and a method of student evaluation that aligns to the course and/or program objectives.

7. A systematic process has been implemented to document that the objectives and content of courses and/or programs are current.

8. All courses and/or programs, regardless of location or mode of delivery must:

a. Be approved and administered under established institutional policies and procedures with oversight from an appropriate campus administrator.

b. Have appropriate involvement of on-campus administrators and faculty in planning, approval, and on-going evaluation.

c. Have individual student records, including period of enrollment, and educational course and/or program records, permanently maintained by the institution, parent headquarters, agency, and/or Service..

d. Be described in appropriate catalogs or brochures and include relevant requirements for enrollment, academics, and related information technology.

e. Provide for timely and meaningful interaction among faculty and students.

Instruction YES NO N/A

9. Academic competencies and occupational skills are integrated into the instructional courses and/or programs associated with each occupational area.

10. The instructional courses and/or programs provide education and training in the competencies essential to success in the occupation, including job knowledge, job skills, work habits, and attitudes.

11. The sequence of instruction required for course and/or program completion (lecture, lab, and work-based activities) is effectively organized in order to maximize the learning of competencies essential to success within the occupation.

12. Job-related health, safety, and fire-prevention are an integral part of instruction.

13. To develop skill proficiency, sufficient practice is provided with equipment and materials similar to those currently used in the occupation.

14. All instruction is effectively organized as evidenced by course outlines, lesson plans, competency tests, and other instructional materials.

15. The institution uses a systematic, objective, and equitable method of evaluating student achievement based on required competencies.

DRAFT 4

STANDARD 3-CHECK SHEETS

Program and Institutional Outcomes

YES NO N/A

1. Individual student progress data, including (a) appropriate evaluations of knowledge and skills required for occupation(s) studied and (b) notations of completion(s) of and/or withdrawal from programs, are maintained and, where appropriate, are made a part of his/her record.

2. The institution has a written plan to ensure that follow-up is systematic and continuous, and includes the following elements:

a. Identification of responsibility for coordination of all follow-up activities.

b. Collection of information from completers and/or from the organizations and units which employ them.

c. Information collected from completers and/or from the organizations and units which employ them is focused on course/program effectiveness and relevance to job requirements.

d. Follow-up information is used to evaluate and improve the quality of program outcomes.

e. Follow-up information is made available at least on an annual basis to appropriate instructional personnel and administrative staff.

DRAFT 5

STANDARD 4-CHECK SHEETS

Strategic Planning

YES NO

1. The institution has or aligns to a written strategic plan that includes, as a minimum, the following components:

a. Mission of the institution;

b. Vision of the institution;

c. Objectives for a minimum period of three years;

d. Strategies for achieving the objectives; and,

e. Strategies for evaluating progress toward achieving the objectives.

2. The strategic plan is reviewed by the faculty, administration, and institutional advisory committee and revised as necessary at least annually.

3. The results of the evaluation of progress toward achieving the objectives are documented annually.

DRAFT 6

STANDARD 5-CHECK SHEETS

Learning Resources

Media Services YES NO

1. The institution has, aligns to, or is supported by a media services plan and provider which is appropriate to the needs of the institution and addresses:

a. Availability and scope of services.

b. A variety of current and relevant educational materials, such as reference books;

periodicals and manuals of a business, professional, technical, and industrial nature;

audio-visual materials and equipment; internet access; and other materials to help fulfill the institution’s purposes and support its educational programs.

c. The staff person (administrative, supervisory, or instructional) responsible for the implementation and coordination of the media services.

d. Roles and responsibilities of designated staff member(s).

e. Orientation for user groups (i.e., instructors, students, and others).

f. Facilities essential for using media materials.

g. Annual budgetary support for the services.

h. Evaluation of the effectiveness of media services and utilization of the results to modify and improve media services.

2. A current inventory of media resources is maintained.

3. Provisions are made for necessary repair, maintenance, and/or replacement of media equipment and supplies.

4. Services for creating instructional media (both print and non-print) are adequate and appropriate to support all students and faculty in meeting the objectives of the education program(s).

5. Media services, facilities, and materials are accessible and available for use by students and instructors.

Instructional Equipment YES NO

6. The institution has a system of instructional equipment inventory.

7. Funds are budgeted or made available to provide instructional equipment at a level that assures quality education and training.

8. The institution has a system for emergency purchases to assure the acquisition and/or repair of equipment within a reasonable period of time to support continuous instruction.

9. Relevant and up-to-date equipment is available to support the instructional programs offered by the institution.

10. The institution has a written plan for maintaining equipment and for replacing or disposing of obsolete equipment.

11. All instructional equipment meets appropriate and required safety standards.

DRAFT 7

Instructional Supplies YES NO

12. Instructional supplies are available to support the instructional programs offered by the institution

13. The institution has a system for purchasing and storing instructional supplies.