36C77618Q9237-0001003.pdf

2017 HR Academy Annual Learning Evaluation Report

VA HR Academy (https://go.va.gov/bu24) Page |

Human Resources (HR) Academy

Annual Learning Evaluation

Fiscal Year 2017

Submitted to:

Submitted by:

September 15, 2017

VA HR Academy (https://go.va.gov/bu24) Page | i

TABLE OF CONTENTS

ES Executive Summary ............................................................................................................................ vii

ES.1 Scope of Assessment ................................................................................................................... vii

ES.2 Programmatic Assessment .......................................................................................................... vii

ES.2.1 FY16 Training Overview ................................................................................................... viii

ES.2.2 Program Changes in FY17 ................................................................................................. viii

ES.3 Learning Evaluation ................................................................................................................... viii

ES.3.1 Evaluation Framework - Kirkpatrick® Model ...................................................................... ix

ES.3.2 Training Evaluation for Level 1 ........................................................................................... ix

ES.3.3 Training Evaluation for Level 2 ............................................................................................ x

ES.3.4 Training Evaluation for Level 3 .......................................................................................... xi

ES.3.5 Training Evaluation for Level 4 ........................................................................................... xi

ES.4 Findings and Action Plan ........................................................................................................... xiii

ES.5 Detailed Results and Back-up Materials .................................................................................... xiii

1 Introduction

1.1 Program Overview

1.2 Report Organization

1.3 Scope

2 Programmatic Assessment

2.1 Understanding of HR Academy

Methodology Overview, including Evaluation Process

Offerings – Not Just Training

Support Across the Learning Spectrum

Training Decisions

Demographics of Virtual Instructor-Led Training Participants

Virtual Instructor-Led Training Completions

VA HR Academy (https://go.va.gov/bu24) Page | ii

2.2 Annual Customer Satisfaction Survey

Greater Job Satisfaction/Higher Worker Morale

Demographic Highlights

Request for Training

Improving Job Performance

Training Barriers

Key Influences

2.3 New Capabilities and Offering

Special Offerings

Using Automation from SharePoint

2.4 Program Changes from Previous Evaluations

3 Learning Evaluation

3.1 Evaluation Framework - Kirkpatrick® Model

Level 1—Reaction Evaluation

Level 2—Learning Evaluation

Level 3—Behavior Evaluation

Level 4—Results Evaluation

Pulling It All Together

3.2 Training Evaluation for Level 1

Level 1 Historical Trend Analysis

3.3 Training Evaluation for Level 2

Level 2 Historical Trend Analysis

3.4 Training Evaluation for Level 3

3.5 Training Evaluation for Level 4

Baseline Workforce Knowledge and Capabilities

Effectiveness of HR Services

VA HR Academy (https://go.va.gov/bu24) Page | iii

Increased Capabilities – Addressing Gaps

4 Findings and Action Plan

4.1 Programmatic Assessment

4.2 Learning Evaluation

Reactions – Customer satisfaction and Level 1 Surveys

Learning

Tools and Behaviors

Results – Relating Training & Competency Gaps

4.3 Actions

Aligning Competency Gaps with Offerings

Expanding HR-Metrics and Data Collections

Refining level 4 Metrics

5 Appendix A – Acronyms

VA HR Academy (https://go.va.gov/bu24) Page | iv

LIST OF FIGURES

Figure 1. FY13-17 Historical Trend Analysis for Level 1 ........................................................................... ix

Figure 2. Level 2. Historical Trend Analysis FY13-FY17 ........................................................................... x

Figure 3. Level 3: Business Results Impacts Comparison ........................................................................... xi

Figure 4. Leadership Competency .............................................................................................................. xii

Figure 5. HR Academy Methodology Process

Figure 6. HR Academy Learning Model

Figure 7. Support Across Learning Spectrum

Figure 8. FY17 Training Demographic Highlights

Figure 9. 2017 Customer Satisfaction Survey - Government Response Comparison

Figure 10. Results from Learning Culture and Shared Ownership of Career Development Questions

Figure 11. Customer Satisfaction Demographics - Job Series by Percentages

Figure 12. Customer Satisfaction Demographics - Administration/Office

Figure 13. Customer Satisfaction Survey - Training Requests

Figure 14. Customer Satisfaction Results – Improving Job Performance from Training

Figure 15. Customer Satisfaction Survey - Most Common Challenges

Figure 16. Customer Satisfaction Survey - Factors that Influence Training Registration

Figure 17. Communities of Practice (Novice Graduates as of 8/31/2017)

Figure 18. Automated Application Process for Learning Programs Offered by HR Academy and Others 15

Figure 19. Behind the Scenes of Kirkpatrick Model and Evaluation Methodology

Figure 20. FY17 Trend Analysis for Level 1 Evaluations

Figure 21. FY13-FY17 Historical Trend Analysis for Level 1

Figure 22. FY17 Level 2 Trend Analysis for Pre- and Post-Test Scores

Figure 23. FY13-FY17 Historical Trend Analysis for Level 2

Figure 24. Summary of Level 3 Participant/Manager Results by Assessment Category

VA HR Academy (https://go.va.gov/bu24) Page | v

Figure 25. Level 3 Summary of Job Impact Assessment

Figure 26. Level 3 Business Results Impacted – Participant/Manager View

Figure 27. Rates for Hiring and Return Certificates Since April 2016

Figure 28. FY17 Average Days from Announcement to Hire

Figure 29. Speed of Hire

Figure 30. Core & Leadership Competencies

Figure 31. Top 2 Reported Gap Areas

Figure 32. Highest Reported Gap Area

Figure 33. Training Barriers

Figure 34. Level 2 Average Test Score Statistics

Figure 35. Knowledge Gains to Competency Areas

Figure 36. 2016 vs 2017 Response Rate

Figure 37. Business Results Impact Comparison

VA HR Academy (https://go.va.gov/bu24) Page | vi

LIST OF TABLES

Table 1. Training Offerings and Number of Attendees ............................................................................. viii

Table 2. HR Academy’s Application of Kirkpatrick® Levels ..................................................................... ix

Table 3. Participant Feedback from Open-Ended Questions ........................................................................ x

Table 4. Sampling of HR Academy support of Technical Competency Gap areas .................................... xii

Table 5. Your Primary Motivation for Registering for a Course

Table 6. Where Have You Participated in Human Resources-Related Training

Table 7. Which of the Following Factors Would Influence you the Most When Deciding Whether to Register for a Training Course

Table 8. Training Offerings and Number of Attendees

Table 9. FY17 Comparison of Level 1 Evaluation Questions

Table 10. Participant Feedback from Open-Ended Questions

Table 11. FY17 Level 2 Pre- and Post-Test Statistics

Table 12. Level 3 Retrospective Questionnaire Ratings Summary

Table 13. Level 4 Business Outcomes and Metrics

Table 14. Top Seven Technical Competency Gaps of HR Professionals

Table 15. Top Ten Competency Strengths of HR Professionals

Table 16. Competency Strengths by Leadership vs Non-Leadership Profile Type

Table 17. Competency Gaps by Leadership Profile Type

Table 18. Technical Capabilities Competency/Completions

Table 19. Core Skills Competency/Completions

Table 20. Training & Competency Gaps

Table 21. Acronyms

VA HR Academy (https://go.va.gov/bu24) Page | vii

HR Academy Annual Learning Evaluation

ES Executive Summary This Annual Learning Evaluation Report summarizes the effectiveness of the Department of

Veterans Affairs’ (VA) Human Resources (HR) Academy’s learning and professional development program. The HR Academy supports the learning and professional development needs of approximately 5,000 HR professionals across all administrations and corporate offices. The focus is to support VA’s mission by developing a qualified and effective community of HR professionals to support the agency’s workforce needs; close competency gaps; and improve HR professional’s proficiency in order to provide better service to veterans and their families.

ES.1 Scope of Assessment

The HR Academy uses a two-pronged approach for evaluating its training program: 1) programmatic assessment and 2) learning evaluation. Program assessment metrics focus on areas such as offerings, usability, community satisfaction, and overall effectiveness. Learning evaluation includes assessing learning at four levels using the Kirkpatrick® Model. Levels 1 and 2 assess participant’s satisfaction and learning during training classes, while Level 3 determines the participant’s ability to implement on-the-job new skills or abilities learned in training. Level 4 evaluates the impact of HR Academy training on the VA HR community using outcome-based progress metrics, including how HR Academy has reduced competency gaps via its services.

The Annual Report is based primarily on data from training classes and the annual satisfaction survey. The reporting period covers 41 virtual instructor-led training (VILT) course offerings that were delivered from October 1, 2016 through August 31, 2017. During this period, the HR Academy had over 3,000 training completions by 1260 unique participants.

As of FY17, a few aspects of HR Academy have yet to be fully integrated into the evaluation process. For instance, Web-based Training is not included. Evaluation data is collected within TMS but not yet available for analysis at a detailed level. Other learning and professional development opportunities, including certificates and leadership-focused programs, are not reported at this time.

ES.2 Programmatic Assessment

From an operational perspective, HR Academy is a virtual learning and talent development organization that provides course curricula for HR practitioners at all levels of knowledge and experience from novice to advanced, including leaders. The curriculum consists of virtual instructor-led courses and self-paced, web-based training programs offered through a variety of vendors and modalities. The program is largely cost-effective, and demonstrably able to help meet VA’s training needs and close proficiency gaps. The VA Talent Management System (TMS) serves as the primary host for HR Academy courses to permit tracking and management of employee participation, class completion, and record keeping/management.

VA HR Academy (https://go.va.gov/bu24) Page | viii

ES.2.1 FY16 Training Overview Table 1 shows the total number of completions each month for the classes with 150 or more completions as well as totals across all course offerings. The HR Academy had the lowest number of completions in the first quarter of FY17. By using a different email distribution list, the new communications resulted in increased enrollments and completions throughout the rest of the year.

March set a high for the time frame with 661 completions across 18 training courses offered.

February (455) and June (404) were the next highest active months. The top three courses (Veterans’ Preference, Federal Staffing and Placement, and Qualifications Analysis) were also the most popular courses last year.

Table 1. Training Offerings and Number of Attendees Courses FY17 Quarter 1 FY17 Quarter 2 FY17 Quarter 3 FY17 Quarter 4

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Total Veterans' Preference 33 62 28 60 64 32 61 31 30 29 430 Federal Staffing and Placement 32 33 23 61 33 94 24 31 29

Qualifications Analysis 31 32 33 30 30 89 31 31 26 333 Best Practices in Initiating and Processing of Personnel Actions

11 27 29 63 29 31 190

Overview of Processing Personnel Actions 26 29 63 31 29 26 178 GS Pay Settings 29 30 31 26 30 27 173 Fundamentals of Legal Research 29 30 30 30 31 29 150

Total for ALL Courses Attended 150 211 236 351 455 661 305 278 404 287 301 3639

Similar to previous years, about 79% of participants are part of the Veterans Health Administration (VHA). Attendees range from Grades 5-12, with 45% being between GS-9 and GS-

12. Additional details about the universe of participants are available via the Customer Satisfaction Survey results.

ES.2.2 Program Changes in FY17 Each year HR Academy addresses issues identified in evaluations from prior years and the customer satisfaction survey to integrate appropriate changes and improvements to the program and curriculum. New capabilities developed this year include Intermediate certificates for Employee Benefits and Employee Relations. HR Academy also began delivering the Intermediate Certificate for Compensation that was developed last year. HR Academy also continued to support promotion and applications for cross-government programs (e.g., Emerging Leaders). HR Academy also began metrics pilots with HR Development (HRD) courses and worked with HRD counterparts to help develop the New Talent Development Program.

ES.3 Learning Evaluation

The HR Academy began delivering critical training during fourth quarter of FY 2010 and formally launched first quarter of FY 2011. Evaluation of training is one of the main components HR Academy leverages to ensure continuous improvement and assess the value of its learning courses, materials, instructors, and other related events. The learning evaluation includes an assessment of performance at four levels using the Kirkpatrick® Model. Using this model, each successive evaluation level is built on information provided by the earlier lower level and enables greater understanding of how to improve service offerings and effectiveness.

VA HR Academy (https://go.va.gov/bu24) Page | ix

Figure 1. FY13-17 Historical Trend Analysis for Level 1

ES.3.1 Evaluation Framework - Kirkpatrick® Model Monitoring and evaluation is an integral part of HR Academy’s overall program management, specifically intended to provide feedback on how training implementation is progressing. Data collection continues to evolve and mature. Originally, data was collected via paper forms and later through SurveyMonkey. Since February 2016, data has been collected using tests and surveys distributed through Metrics that MatterTM (MTM). To integrate MTM data with historical training data as well as data from other sources (e.g., TMS, USA Staffing), HR Academy uses an Excel-based dashboard tool call HR-Metric (Measuring Effectiveness of Training Reaching/Impacting Community). This section includes results generated from HR-Metric. The current data collection and utilization of data by Kirkpatrick Model level is summarized in Table 2.

Table 2. HR Academy’s Application of Kirkpatrick® Levels Level Measure Tools/Methods Utilization

Level 1:

Reaction

The degree to which participants react favorably to the learning event

End-of-course evaluations

Students received at the end of training and must be completed within 2 business days

Level 2:

Learning

The degree to which participants acquire the intended knowledge, skills, and attitudes through their participation in the learning event

Pre-training tests (pre-tests) and post-training tests (post-tests)

Pre-tests distributed prior to or at the beginning of a training class, and Post-tests distributed at the close of or end of a training class

Level 3:

Behavior

The degree to which participants apply what they learned during training when they are back on the job

Retrospective questionnaires to broader audience

Questionnaires sent to training participants and supervisors or managers approximately 60 days after training completion

Level 4:

Results

The degree to which targeted outcomes occur as a result of the learning events and subsequent reinforcement

Metrics that support achievement of performance-based outcomes

Related performance-based outcomes collected over the course of the training year for annual assessment

ES.3.2 Training Evaluation for Level 1 Level 1 captures the immediate reaction of the participants about the training. HR Academy collects the data through end-of-course evaluations, which participants, to fulfill completion requirements, must fill out the end-of-course evaluation within 2 business days of the class end date.

The evaluation gives the participants positive statements about the course across a 5-point Likert scale with responses ranging from “Strongly Disagree” (a score of 1, the lowest) to “Strongly Agree” (a score of 5, the highest).

The graphic illustrates how satisfaction has steadily grown over the last four fiscal years. This positive trend is supported by the increased demand in anecdotal feedback expressing great satisfaction with instructor/facilitators, relevance of content and overall experience.

The end-of-course evaluations also include qualitative data that they collect from open-ended questions used in the end-of-course evaluation. The HR

VA HR Academy (https://go.va.gov/bu24) Page | x

Academy solicits feedback specifically on instructors/facilitators, logistics and overall satisfaction.

Results were consistent between throughout the fiscal year, where instructor/facilitator satisfaction was the strongest. Approximately 90% of participants who provided feedback were impressed with the instructors and the facilitators used, an increase from 82% last year. Table 3 summarizes the qualitative feedback from students, providing insights into the quantitative scores. Overall course ratings were also high (78%). Unfortunately, logistics (e.g., availability, lengthy waitlist) continue to be an issue. The Virtual training environment is also reported as a challenge for some students.

Table 3. Participant Feedback from Open-Ended Questions Sampling of Level of Satisfaction Most Common Rationale within Open-Ended Questions

Do you have any other comments about the course instructor/facilitator?

90% High Level of Satisfaction

Knowledgeable instructors with strong understanding of VA and Federal sector Well-organized and patient training instructor and facilitator team Friendly, polite, and engaging; Able to solicit participating for all attendees

10% Low Level of Satisfaction

Unknowledgeable instructor, unable to respond to VA-specific questions Poor timing, rushing through questions or class material Lack of patience, inattentive, or potentially unfriendly/unengaging

Do you have any other comments about the course overall?

78% High Level of Satisfaction

Knowledgeable instructors with great understanding of VA and Federal sector Well-organized, highly professional and patient training instructor and facilitator team Relevance of course topics and material Well-paced class and overall punctuality and timeliness

22% Low Level of Satisfaction

Course too long/course too short Unknowledgeable instructor, unable to respond to VA-specific questions Preference for an alternative training modality (WBT or Face-to-Face) Not enough scheduled offerings

Do you have any other comments about the course logistics?

22% High Level of Satisfaction

Timely technical assistance, as needed Ease of use of tools (e.g., Blackboard and MTM participant interfaces)

73% Low Level of Satisfaction

Class availability; lack of sufficient scheduled offerings Lengthy waitlist times No place for virtual learners to escape noise of office

ES.3.3 Training Evaluation for Level 2 Level 2 evaluation is an examination of the actual cognitive development of participants as a result of their engagement with the program. The tests offer a more powerful indicator of learning that builds upon the self-reported information from Level 1.

HR Academy conducts pre- and post-tests to assess the degree to which participants acquired the intended knowledge and skills of the training. Course participants are prompted to complete the pre-test at the beginning of each class and the post-test at the conclusion of each class. Figure 2 shows improvement based on test scores.

Figure 2. Level 2. Historical Trend Analysis FY13-FY17

VA HR Academy (https://go.va.gov/bu24) Page | xi

The pre- and post-tests include identical questions, but the order of questions is randomized in the post-tests. The MTM platform integrates pre- and post-test data and enables the HR Academy to conduct comparative analysis and reporting. On average, students scored 11% higher in the post-test than on the pre-test. The graphic shows test scores and improvements over the past five years.

ES.3.4 Training Evaluation for Level 3 Level 3 helps determine how learning from the HR Academy curriculum has been applied to the workplace. HR Academy assesses Level 3 using a retrospective questionnaire distributed to students and managers approximately 60 days after training completion. HR Academy is using an updated existing Level 3 questionnaire from an earlier pilot. Figure 3 shows key areas both managers and training participants reported.

The program successes carried into the Level 3 (Behavior) evaluations which demonstrated a fairly high degree of knowledge transfer of critical, foundational skills learned at HR Academy to training participants’ on-the-job environment. Both training participants and managers included Return on Investment as their highest ratings with 4.2 and 4.4 respectively, on a scale of 1 to 5. HR Academy also achieved very high marks for Learning Effectiveness, being reported as 4.1 and 4.3 for participants and managers respectively. Even though Business Results was one of the lower areas (rating of 4.0 for participant and 3.9 for managers), participants noted the subcategories of “increasing quality” as well as employee and customer satisfaction very high.

ES.3.5 Training Evaluation for Level 4

Figure 3. Level 3: Business Results Impacts Comparison

VA HR Academy (https://go.va.gov/bu24) Page | xii

Evaluation at Level 4 helps to further understand how the learning impacted the participants’ work environment and includes analysis of some external factors that might contribute to the results. The intention is to explore the influence training has at the organizational level. Areas evaluated include, but are not limited to, increased technical capabilities. While information gathered is related to improved accuracy of HR services, higher productivity per HR employee, and improved timeliness of HR services, it is hard to link how training has or has not caused change in these factors. New this year – the team has begun linking how training can and has impacted the top competency gap areas identified by the HR Community.

Level 4 (Results) findings were the strongest in the areas of competency assessment efforts for HR professionals and how HR Academy can support them. For instance, the greatest gap within core competency was identified as Strategic Learning. For individuals who took Advanced Staffing Module 4, which targets this specific competency, participants left with approximately 13% better understanding after taking the course. To further reduce this gap across the community, additional offerings of this module may be beneficial. Similarly, there are courses in quarters 2 through 4 that helped with the leadership competency of Developing Others. Similar pre/post test results were found for this competency as shown in Figure 4.

For technical competencies, HR Academy also helped to support knowledge and skills in the top 5 gap areas as shown in Table 4.

Table 4. Sampling of HR Academy support of Technical Competency Gap areas Competency FY17 HR Academy Support

# Gaps % Gaps Courses Completions Labor Relations 1587 44% 2 115 Position Classification & Management 1450 39% 3 128 Recruitment and Staffing 1193 38% 11 1450 Employee Relations 1303 37% 5 342 Pay Administration 1205 36% 4 461

Figure 4. Leadership Competency

VA HR Academy (https://go.va.gov/bu24) Page | xiii

ES.4 Findings and Action Plan

The breadth and depth of both qualitative and quantitative data offers HR Academy an understanding of what has worked well and what can be improved. The Annual Report also offers the opportunity to reflect on how to improve services and offerings to the HR community.

As shown in the Proposed Actions text box, HR Academy is interested in exploring leadership assistance to address issues identified through the customer satisfaction survey and confirmed by Level 1 and 2 findings. Moreover, the three key actions that the team has identified for implementation in FY18 are:

1. Aligning competency gaps with offerings – this action was identified to build upon the initial analysis conducted at the end of FY17 and summarized in the Level 4 findings in the previous section.

2. Expanding HR-Metrics and Data Collection – this action was identified to build upon existing lessons learned, tools, and procedures to enhance access to qualitative and quantitative data for HR training delivered by groups beyond HR Academy.

3. Refining level 4 metrics – this action was identified to enable the Data Collection Plan to updated to focus on the highest-payoff metrics, consider changing those metrics that are providing minimal value, and potentially introducing new metrics that align with current government priorities and initiatives.

ES.5 Detailed Results and Back-up Materials

HR Academy continues to deliver a Semi-Annual and Annual Learning Evaluation each year as well as Monthly Data Reports. We also maintain HR-Metric, an interactive Excel-based dashboard that can be used to further analyze a course over years or a single instance of a course or set of courses. If you are interested in more details, please contact the HR Academy (vacohracadamy@va.gov).

Proposed Actions Opportunity for Executive Action:

Leadership may be able to assist the HR Academy by participating in final validation of new Level 4 metrics to ensure future reports meet leadership needs.

Another area may be stressing the importance of training to supervisors as this was identified as a barrier in both Level 1 feedback and the Customer Satisfaction Survey.

HR Academy is focusing on three key initiatives to continue to improve the metrics program as discussed in the text.

mailto:vacohracadamy@va.gov

VA HR Academy (https://go.va.gov/bu24) Page | 1

1 Introduction The Department of Veterans Affairs’ (VA) Human Resources (HR) Academy supports the learning and professional development needs of approximately 5,000 HR professionals across the agency. Originally a component of the Office of Human Resources Management (OHRM) and now part of Veterans Health Administration’s (VHA) Workforce Management Consulting (WMC), HR Academy supports the VA’s mission and guiding principles by developing a qualified and effective community of HR professionals to support the agency’s workforce needs close competency gaps and improve HR professional’s proficiency in order to provide better service to veterans and their families. Stakeholders include VA HR Professionals and HR Leadership across the Administrations:

VHA, Veterans Benefits Administration (VBA), National Cemetery Administration (NCA), and Staff Offices.

In FY2017, HR Academy continued to expand and fine-tune its approach and mechanisms to promote employee learning and professional development. Learning assessment continued to be a critical component and focal point of the program, which included implementing the Metrics that MatterTM (MTM) platform to further automate and centralize its learning and program evaluation process.

Over the course of the fiscal year, HR Academy has had over 3,600 training completions with a total of 1260 unique participants; approximately 79% of whom represented the VHA. HR Academy supplements its comprehensive training curriculum with other learning and professional development opportunities, including professional certificates across key HR disciplines, competencies, and leadership-focused programs.

1.1 Program Overview

Since its inception, HR Academy has been strongly aligned with the strategic direction outlined in the Office of Human Resources & Administration (HR&A) Strategic Plan FY2014-2020. The HR Academy directly supports:

Strategic Goal 1Drive VA transformation through strategic human capital engagement, development, and talent acquisition; specifically, the objective to “Develop leaders and improve workforce skills”

Strategic Goal 2Cultivate and sustain a culture of performance excellence within HR&A that embodies VA values and supports our customers

HR Academy’s own mission to cultivate a community of high-performing HR professionals who enable VA's workforce to serve our veterans with excellence is supported by its key goals and objectives:

Goal 1Deliver the best Federal HR training available to the VA to develop the knowledge and skills required to achieve VA’s strategic goals

Goal 2Offer innovative, easily accessible, and anywhere learning to reach the dispersed VA HR workforce

VA HR Academy (https://go.va.gov/bu24) Page | 2

Goal 3Provide a framework for “total development” of the HR community, leveraging the HR competency model and career maps to drive individual and organizational performance

Goal 4Establish a VA HR Certificate Program as the accepted means to demonstrate proficiency in the basic competencies required of all VA HR professionals

Goal 5Partner with our stakeholders to develop a long-term sustainable model for HR Academy support of the VA HR community

The HR Academy began delivering critical training during the fourth quarter of FY 2010 and formally launched the first quarter of FY 2011. The HR Academy uses a two-pronged, integrated approach for evaluating its training program: 1) programmatic assessment and 2) learning evaluation.

Program assessment metrics focus on areas such as timeliness, leadership engagement, stakeholder engagement, and overall effectiveness. The learning evaluation presents a framework to evaluate program learning at four levels using the Kirkpatrick® Model. Levels 1 and 2 assess the participant’s satisfaction and learning relative to the training attended, while Level 3 assesses the participant’s ability to implement on-the-job new skills or abilities learned in training. Level 4 gauges the impact of HR Academy training on the VA HR community using outcome-based progress metrics.

1.2 Report Organization

The Annual Report is composed of four (4) sections outlining the framework and approach for gathering and measuring HR learning data and information to assess learning effectiveness and support continuous improvement of the HR Academy learning program. The sections are organized as follows:

Section 1. Introduction and Scope of Assessmentprovides an overview of the HR Academy learning and professional development program and specifies the scope of this Annual Learning Evaluation Report

Section 2. Programmatic Assessmentprovides a view into the HR Academy offerings and customers’ view of those services. It also discusses improvements that have been undertaken and/or planned for the next fiscal year

Section 3. Learning Evaluationdefines the Kirkpatrick® Model learning evaluation framework and overall approach for analysis and assessment of the HR Academy training. This section presents the methodology and results of each assessment category, including those beyond the four levels (e.g., Customer Satisfaction Survey)

Section 4. Findings and Actionsoutlines the findings and conclusions of the overall evaluation

Appendix A. Acronyms lists the acronyms used in this document

1.3 Scope

The Annual Report is primarily comprised of data from training classes offered over the course of the fiscal year. It also includes analysis of HR employment data sources and information from the HR Academy’s annual customer satisfaction survey. The reporting period covers 67 virtual instructor-led training (VILT) course offerings that were delivered from October 1, 2016, through

VA HR Academy (https://go.va.gov/bu24) Page | 3

August 31, 2017. The learning evaluation encompasses all four levels of HR Academy’s learning evaluation framework, the Kirkpatrick® Model.

2 Programmatic Assessment From an operational perspective, HR Academy is a virtual learning and talent development organization that provides course curricula for HR practitioners at all levels of knowledge and experience from novice to advanced, including leaders. The curriculum consists of VILLT courses and Self-paced, Web-based Training (WBT) programs offered through a variety of vendors and modalities. The program is largely cost-effective and demonstrably able to help meet VA’s training needs and close proficiency gaps. The VA Talent Management System (TMS) serves as the primary host for HR Academy courses to permit tracking and management of employee participation, class completion, and record keeping/management. This section explains the highest-profile services delivered this year, how customers viewed HR Academy products and services, and continual improvement changes from previous years.

2.1 Understanding of HR Academy

This section provides an overview of HR Academy methodologies and offerings to help the reader gain perspective when reviewing other parts of this report.

Methodology Overview, including Evaluation Process This reporting period’s results are focused on the VILT courses, which have been fully integrated into the Kirkpatrick® model and MTM platform solution. MTM and the HR Academy’s Measuring Effectiveness of Training Reaching/Impact Community (HR Metric) tools were used to support data collection, analysis, and reporting.

Figure 5. HR Academy Methodology Process

VA HR Academy (https://go.va.gov/bu24) Page | 4

Offerings – Not Just Training HR Academy brings a comprehensive learning model. The main components are:

Training – Classes help candidates gain a greater appreciation of how to perform their jobs effectively and efficiently

Professional Development – Collaboration and networking is promoted to enable candidates to improve their skills not only during the program but for years to come

Action Learning – Using knowledge learned helps VA solve existing challenges and helps candidates understand how to apply it to their jobs

As shown in Figure 6, these components were selected based on their ability to support industry best practices for experiential learning:

• Sharing/Learning

• Active Listening

• Powerful Questions

• Reflection

• Career Development

• Action

Training provides the opportunity to share information from industry, existing VA programs, and other sources and enable learning across a broader audience. Members of the HR community can apply active listening during training classes, as well as other activities, to maximize what they gain from the materials. By asking powerful questions both during classes and professional development opportunities (e.g., communities of practice), candidates gain even more insights.

Professional development also includes on-the-job training where students reflect on the data and information gained from HR Academy, enabling not only them but also their peers and managers to gain and apply knowledge.

The first component of the overall certificate program is Training. It includes a set of classes to help candidates gain a greater appreciation of how to perform their jobs effectively and efficiently.

The HR Academy’s goal is to provide VA’s HR professionals with a resource of available learning and professional development opportunities that support VA’s HR professionals desire to acquire the knowledge, skills, and experience needed to succeed in their current position and to enhance their HR career. Examples include both Virtual Instructor-Led Courses and Web-based Training.

Another component is Action Learning. Higher learning institutions and educational professionals cite experiential “action learning” as a critical tool for training and problem-solving.

As part of the HR Academy’s offerings, action learning has been adopted differently based on the focus of the learning levels. For example, action learning components are embedded within individual classes within the training component, especially in the novice level courses. In other cases, monthly forums and interactive sessions are used to complete case studies and other similar

Training

Action Learning

Professional Development

Figure 6. HR Academy Learning Model

VA HR Academy (https://go.va.gov/bu24) Page | 5 exercises. Such exercises ask participants to apply fresh thinking to avoid responding to today’s problems with yesterday’s solutions while tomorrow’s challenges loom ahead.

The third component is Professional Development. Under this component, collaboration and networking are promoted to enable peers to improve their skills not only while in sessions but feeling free to reach out to each other or HR Academy when they confront something they have not seen before. We also provide a Website, SharePoint sites, and other tools to provide access to valuable information.

Moreover, we offer formal programs such as our Intermediate Certificates that bring all components together for similar purposes. HR Best is another formal program that helps with leadership skills. Our certificates focus on Recruitment and Placement, Compensation, Classification, Employee Benefits, and Employee Relations.

Support Across the Learning Spectrum Results from the 2017 Survey indicate that students attend HR Academy training predominantly to learn a new skill. Less than 20% are using HR Academy to simply fulfill training requirements.

Students attending HR Academy see more than just training while attending courses. Figure 7 shows reasons for taking different HR Academy offerings – from training through formal programs.

Figure 7. Support Across Learning Spectrum

Additional reasons include:

• For promotional needs and upward mobility

• Refresher and stay abreast of changes

• Support VA mission

Training Decisions The purpose of this section is to gain a better understanding of how communications and training should be delivered to gain maximum traction and return on effort. Respondents can provide comments to questions asked on the survey, the following tables represent the comments that some students provided in response to the question listed at the top of each table. This feedback is assessed and evaluated to improve the overall framework and benefit all disciplines of the certificate. Table 5 and 6 show why individuals registered for HR Academy courses and took

VA HR Academy (https://go.va.gov/bu24) Page | 6 courses through other sources, respectively. Table 7 explains what factors drove their decision of which training to take.

Table 5. Your Primary Motivation for Registering for a Course

HR Academy students can register for the courses when they have a need to fulfill training requirements but when they are unable to receive the classes, many turn to outside resources for classes that fit learning objectives, classes that work around their schedule, and provide the type or class environment they like most.

Table 6. Where Have You Participated in Human Resources-Related Training

When respondents decide to register for HR Academy training, the following factors influence their reasons. According to the survey, the main reason for registering for the classes is the course learning objectives.

Table 7. Which of the Following Factors Would Influence you the Most When Deciding Whether to Register for a

Training Course

17%

47% 20%

Your primary motivation for registering for a course through the HR Academy?

Fulfill annual or other training requirements Learn a new skill or expand my knowledge to more effectively perform my current responsibilities Learn a new skill expand my knowledge to pursue career goals

22% 13%

27%

13%

Course sponsored by local facility

Where have you participated in human resources-related training OUTSIDE of the HR Academy?

Have not participated in any human resources related training Have not participated in any human resource related training outside of the HR Academy The U.S. Office of Personnel Management (OPM) HR University (HRU)

Course learning objectives 46% Continued Education / Certification Credits 21% Format of course (e.g., web-based, classroom) 19%

Which of the following factors would influence you the most when deciding whether to register for a training course?

VA HR Academy (https://go.va.gov/bu24) Page | 7

Demographics of Virtual Instructor-Led Training Participants Figure 8 shows the demographic breakdown of trained participants in FY17. Consistent with previous years, about 79% of participants represented the VHA.

Figure 8. FY17 Training Demographic Highlights

Virtual Instructor-Led Training Completions Table 8 shows the total number of completions each month for VILT courses as well as totals across all course offerings. The HR Academy had the lowest number of completions in the first quarter of FY17. By using a different email distribution list, the new communications resulted in increased enrollments and completions throughout the rest of the year. March set a high for the time frame with 661 completions across 18 training classes offered. The months of February (455) and June (404) rounded out the top three highly attended months. The courses with the highest number of attendees during the reporting period were “Veterans’ Preference” (430), “Federal Staffing and Placement” (360) and “Qualifications Analysis” (333).

VA HR Academy (https://go.va.gov/bu24) Page | 8

Table 8. Training Offerings and Number of Attendees Courses FY17 Quarter 1 FY17 Quarter 2 FY17 Quarter 3 FY17 Quarter 4

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Total

Advanced Staffing 32 32 Advanced Staffing Module 1: Problem Solving and Decision Making 28 28

Advanced Staffing Module 2: Workforce Succession Planning 29 29

Advanced Staffing Module 3: Strategic Recruitment 23 23

Advanced Staffing Module 4:

Researching and Applying Regulations and Laws

25 25

Advanced Staffing Module 5:

Appointment Eligibility Determinations

24 24

Advanced Staffing Module 6: Complex Pay Determinations

19 19

Adverse Actions 32 30 62 Alternative Dispute Resolution (ADR) 17 17 Basic Staffing 23 23

Basic Title 38 Pay Setting 28 32 28 27 22 137 Best Practices in Initiating and Processing of Personnel Actions 11 27 29 63 29 31 190

Category Rating Workshop - Virtual Training 29 32 28 89

Central Personnel Data File Edits 1 1

Collective Bargaining Agreements (CBAs) 28 28

Consulting Skills for HRM Professionals 34 34 Determining Service Computation Dates - Virtual Training 31 31 32 26 25 145

EEO in the Federal Sector 64 64 Employee Benefits - Virtual Training 27 31 30 27 115

Employee Conduct and Performance 1 102 18 121

Employee Development 17 17 Employee/Labor Relations - Virtual Training 27 24 51

Federal Staffing and Placement 32 33 23 61 33 94 24 31 29 360 Fundamentals of Legal Research 29 30 30 30 31 29 179

GS Pay Setting 29 30 31 26 30 27 173

Interpersonal Communications 1 1 2

Oral Communications for HR Professionals 23 23

Overview of Processing Personnel Actions 26 29 63 31 29 26 204

Performance Management - Virtual Training 1 24 25

Performance-Based Actions 29 32 61

VA HR Academy (https://go.va.gov/bu24) Page | 9

Courses FY17 Quarter 1 FY17 Quarter 2 FY17 Quarter 3 FY17 Quarter 4 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Total

Position Classification and Position Management 31 28 59

Position Management - Virtual Training 10 28 38

Preparing to Negotiate 28 28 Proactive Succession Planning 31 31 Processing Personnel Actions - Foundations 30 30

Processing Personnel Actions - Virtual Training 14 33 30 32 109

Qualifications Analysis 31 32 33 30 30 89 31 31 26 333 Strategic Position Management 29 32 61

The Hiring and Retention Overview 29 30 30 89

VA Merit System Principles and Prohibited Personnel Practices Virtual Training

21 59 50 130

Veterans' Preference - Virtual Training 33 62 28 60 64 32 61 31 30 29 430 Grand Total 150 211 236 351 455 661 305 278 404 287 301 3639

2.2 Annual Customer Satisfaction Survey

HR Academy carries out an annual survey to better understand customer perception of training opportunities, tools, and services. The Customer Satisfaction Survey helps to improve learning and professional development services for VA HR professionals and provides the HR Academy with a survey that is independent of specific course offerings.

The HR Academy conducted a self-completion user survey employing a questionnaire through the MTM learning analytics platform. The questionnaire used for the 2017 survey was modified slightly from the versions used in previous years. Because of these differences, not all results presented are comparable with those collected prior to FY17.

The survey was conducted over a four-week period spanning Q2 and Q3 among HR professionals within the 0201, 0203 and 0343 job series, across all VA Administrations. The response rate was very positive; approximately 1047 respondents or 23% of participants responded. HR Academy received much higher response rates from participants of the Customer Satisfaction Survey in FY 17 as compared to FY16. In FY16 the response rate was 673 or approximately 14% of participants responded.

Figure 9. 2017 Customer Satisfaction Survey - Government Response Comparison

VA HR Academy (https://go.va.gov/bu24) Page | 10

VA HR Academy (https://go.va.gov/bu24) Page | 11

Greater Job Satisfaction/Higher Worker Morale HR Academy assesses job satisfaction and worker morale through the annual Customer

Satisfaction Survey. Examples of results include those depicted in Figure 10:

• 48% of responders confirmed that HR Academy effectively help perform current responsibilities in their career development

• 30% of responders asserted that HR Academy will help to pursue career goals in the VA

• 17% of responders affirmed that the quality of training of HR Academy information is a great tool and resource

Figure 10. Results from Learning Culture and Shared Ownership of Career Development Questions

Additional preferences that would help improve ability to perform HR jobs included:

• Modality Preferences:

o 29% In-person/Classroom o 28.4% Online/Web-based

• Top Training Requests:

o 15% HR Smart o 12% Recruitment and Placement o 12% Title 38/Hybrid Title 38

Demographic Highlights The survey demographics (see

Figures 11 and 12) are consistent with our training population: 71 % of responders were 201s – HR Specialist. The survey demographics are consistent with our training population, with 83% of the responders representing both the VHA and 6% representing the VBA and NCA, while 5% represent other staff offices.

Figure 11. Customer Satisfaction Demographics - Job Series by Percentages

VA HR Academy (https://go.va.gov/bu24) Page | 12

Figure 12. Customer Satisfaction Demographics - Administration/Office

Request for Training Figure 13 summarizes that areas in highest demand have shifted in comparison to 2016 (top 2 requests were recruitment & placement and employee relations). This graph indicates that employees within the VA HR community recognize the value of varied training options in helping improve their job performance.

Figure 13.

Customer Satisfaction Survey - Training Requests

VA HR Academy (https://go.va.gov/bu24) Page | 13

Improving Job Performance Figure 14 summarizes the responses to survey request to confirm the opportunities provided through HR Academy’s learning and professional development program. The opportunities provided through HR Academy led to a majority of training participants in improving their job performance and relevant knowledge.

Figure 14. Customer Satisfaction Results – Improving Job Performance from Training

Training Barriers Figure 15 summarizes the responses to the survey request regarding the most common challenges to participating in training courses. Classes were closed when trying to register and lack of time continues to be the greatest challenge.

Figure 15. Customer Satisfaction Survey - Most Common Challenges

VA HR Academy (https://go.va.gov/bu24) Page | 14

Key Influences Figure 16 summarizes responses to the survey request regarding the factors that most influence

HR professionals seeking out and registering for training. Course learning objectives, format, and length of the program combined are the most influential aspects with regard to training registration and enrollment.

Figure 16. Customer Satisfaction Survey - Factors that Influence Training Registration

2.3 New Capabilities and Offering

HR Academy continues to evolve and serve the breadth and depth of needs within the HR community. This section provides a sampling of some of the special capabilities.

Special Offerings HR Academy continues to have success with their special offerings. For instance, there are 847 unique candidates for the Novice certificates (some have enrolled in all 4 curriculums) as of August 31, 2017. HR Best was also launched to support development of leadership skills. As of the end of FY17, the Intermediate Certificates have also been drafted. The two certificates starred (*) below have also been started with pilot cohorts:

• Recruitment & placement

• Classification

• Compensation

• Employee benefits

• Employee relations

The novice certificate course (Figure 17) is self-paced and must be completed within 2 years. The candidates must have acceptance from supervisors and can enroll at any time. Moreover, HR Academy has launched a leadership training effort to close competency gaps with Figure 17. Communities of Practice (Novice Graduates as of 8/31/2017)

VA HR Academy (https://go.va.gov/bu24) Page | 15 training evaluations that do a comparison of actual performance with protentional performances which prepares HR professionals for leadership roles across the VA.

Using Automation from SharePoint The Training on Demand (TOD) Request is one of two forms available to request HR Academy services. Employees can access the form from the home page or the Training Officers page (drop down menu or page itself). Requestors fill out and submit the online form and the results are collected into a table that tracks training requests than downloaded to an MS Excel Spreadsheet.

The application process, illustrated in Figure 18, is used to…