36C77618Q9237-0001002.pdf

Annual Learning Evaluation Briefing

October 1, 2016 – August 31, 2017

HR Academy September 15, 2017

Scope of HR Academy Annual Assessment

Another Successful Year ̶ Completions ̶ Trend Analysis of Learning Improvements ̶ Behavior Change impacting Business Results

Relating Training & Competency Gaps

Action Plan

Opportunities for Executive Support

FY17 Annual Learning Evaluation Briefing

Scope of HR Academy Annual Assessment

• Purpose – Provide leadership and other key stakeholders the data and analysis needed to promote learning and professional development across VA’s HR workforce

• Assessment Results:

̶ What it does include:

Program assessment, including quantity and quality of training services

Evaluation of learning for FY17 Q1-Q4 at Levels 1-4 of the Kirkpatrick Model

̶ Focus of learning evaluation = Virtual Instructor Led courses

• Sources/Tools:

‒ Data collected within Metrics that Matter™ (MTM) :

‒ end-of-course evaluations

‒ pre-training and post-training tests

‒ retrospective questionnaire (60 days after training on 20% of classes)

‒ customer satisfaction/engagement survey

‒ HR Academy’s Measuring Effectiveness of Training Reaching/Impacting Community (HR- Metric) is used for more sophisticated analysis and reporting

‒ Other sources = VA Talent Management System, USA Staffing, Competency Assessment

Another Successful Year!

Six courses had 150 or more completions this year… over 3600 completions total

Learning continues across all courses, with post-test scores showing higher averages

60-days after the courses, managers report value – even higher than the participants

Relating Training & Competency Gaps

• Starting this fiscal year, the team has begun to link the metrics data to the primary competency area that courses support

• Annually, HR Academy conducts a competency assessment – employees identifies their proficiency level and supervisors validate

• Top 5 technical competencies with gaps are already supported by HR Academy courses… benefiting those that use our services

• As of the end of FY17, all are supported by Intermediate Certificates

• Novice certificates exist for all areas except for Position Classification

& Management

Competency FY17 HR Academy Support

# Gaps % Gaps Courses Completions Labor Relations 1587 44% 2 115 Position Classification & Management 1450 39% 3 128 Recruitment and Staffing 1193 38% 11 1450 Employee Relations 1303 37% 5 342 Pay Administration 1205 36% 4 461

Action Plan

#1 – Align Competency Gaps with Offerings FY17 Success! HR Academy has proven feasibility of linking competencies to courses + level 1 to 3

• Possible Change #1: report the results more periodically

• Possible Change #2: validate courses are aligned to the latest version of competencies

#2 – Expand HR-Metrics and Data Collection FY17 Success! HR Academy has been piloting metrics for other HR training providers within VHA

• Possible Change #1: finalize pilot project to validate how best to help those programs

• Possible Change #2: create more comprehensive view of training within future Annual Reports

#3 – Refine Level 4 Metrics FY17 Success! Improved tools and capabilities to analyze data… some data more useful than others

Suggested Roll-out and Timing:

• Conduct Workshop with HR Academy and other WMC stakeholder early in FY18

• Test feasibility of workshop ideas and report back recommended refinements

• Update Data Collection Plan and begin implementing accordingly

• Begin reporting based on the decisions based on the above discussions/workshop

Opportunities for Executive Support

• Assist with final validation of new Level 4 metrics to ensure future reports – we want to meet leadership needs

• Help stress the importance of training to supervisors (see barriers for not being able to use HR Academy – lack of time + supervisor not approving)… while HR Academy continues to look for ways to expand how often classes are offered

• Any questions or request for our team?

Briefing Overview

Programmatic Assessment ̶ Understanding HR Academy

• Evaluation Methodology

• Offerings and Community’s Usage

• Completions

̶ Annual Customer Engagement Survey Results ̶ New Capabilities and Offerings

Learning Evaluation ̶ Level 1 – Reaction ̶ Level 2 – Learning ̶ Level 3 – Behavior ̶ Level 4 – Results

Recommendations and Action Plan

Supporting Materials

Briefing Overview

• This Annual Learning Evaluation Briefing summarizes the effectiveness of the Department of Veterans Affairs’ (VA) HR Academy learning and professional development program using all four levels of the Kirkpatrick® Model.

• In FY17, the Academy had over 3,600 training completions with a total of 1260 unique participants; approximately 79% of whom represented the Veterans Health Administration.

The number of unique participants is based on an August 2017 VA Learning University (VALU) report covering FY 17 HR Academy Virtual Instructor-Led Training (VILT) statistics.

• Learning was evaluated at the following levels:

• Level 1 (Reaction) – Evaluations revealed favorable scores that appear to be a continuing stabilization of scores previously reported.

• Level 2 (Learning) – showed steady and consistent growth from pre- to post-test intervals for the

FY17 curriculum with an average of 11.65 percentage point improvement

• Level 3 (Behavior) – HR Academy surpassed the Metrics that MatterTM (MTM) platform’s government benchmark in all categories, and received the highest positive ratings for “increasing quality” and “customer satisfaction” from both training participants and their Managers. The “increasing productivity” category was the one which had the smallest difference of 3%; 48% for managers and 45% for participants.

• Level 4 (Results) – HR Academy demonstrates strengths and capabilities in helping meet VA’s current and emerging training needs and close proficiency gaps. It has positive contributions to business outcomes including helping to meet planned projection of hiring within 60 days and helping to improve employee job satisfaction. Findings were the strongest in the areas of competency assessment efforts for HR professionals and how HR Academy can support.

Programmatic Assessment

Understanding HR Academy

• Evaluation Methodology

• Offerings and Community’s Usage

• Completions Annual Customer Satisfaction Survey Results New Capabilities and Offerings

Section 1

The purpose of this section is to provides a view into the HR Academy delivery of services and customers’ view of those services. It also discusses improvements that have been undertaken and achieved in FY17. It includes the following sub-sections:

Understanding of HR Academy Evaluation Methodology

This reporting period’s results are focused on the virtual instructor-led courses, which have been fully integrated into the Kirkpatrick model and MTM platform solution.

HR Academy course delivery process is tightly coordinated with the evaluation process…. the complementary results make both sides stronger

Presenter Presentation Notes The Metrics that Matter™ (MTM) and the HR Academy’s Human Resources Measuring Effectiveness of Training Reaching/Impacting Community (HR-Metric) tools were used to support data collection, analysis and reporting.

Offerings – Not Just Training!

Training – Classes help candidates gain a greater appreciation of how to perform their jobs effectively and efficiently

Professional Development – Collaboration and networking is promoted to enable candidates to improve their skills not only during the program but for years to come

Action Learning – Using knowledge learned helps VA solve existing challenges and helps candidates understand how to apply it to their jobs

Offerings across the Learning Spectrum

Fulfill annual or other training requirements

• Learn a new skill or expand knowledge to more effectively perform current responsibilities

• Build competencies in my career model

• Learn a new skill or expand knowledge to pursue career goals

Formal Program

• Achieve certification

Results from the 2017 Survey indicate that students attend HR Academy training predominantly to learn a new skill.

As shown, less than 20% are using HR Academy to simply fulfill training requirements.

Additional reasons include:

• For promotional needs and upward mobility

• Refresher and stay abreast of changes

• Support VA mission

Professional Development

Training Action Learning Professional Development

Decisions to go to Other Sources

Your primary motivation for registering for a course through the HR Academy?

Fulfill annual or other training requirements 17%

Learn a new skill or expand my knowledge to more effectively perform my current responsibilities 47%

Learn a new skill or expand my knowledge to pursue career goals 20%

Where have you participated in human resources-related training OUTSIDE of the HR Academy?

Course sponsored by local facility 22%

Have not participated in any human resource-related training 13%

Have not participated in any human resource-related training outside of the HR Academy 27%

The U.S. Office of Personnel Management (OPM) HR University (HRU) 13%

HR Academy students are able to register for course when they have a need To fulfill training requirement but when they are unable to receive the classes many turn to outside resources for classes that fit learning objectives, classes work around their schedule and provide the type class environment that they like most.

Which of the following factors would influence you the most when deciding whether to register for a training course?

Course learning objectives 46%

Continued Education/Certification Credits 21%

Format of course (e.g., web-based, classroom) 19%

Completions – Demographic Highlights

Participant Demographic Highlights • Administrative/

Office

̶ 79% VHA

̶ 21% Other Staff

Offices

• Grade Level

̶ 27% GS 7

̶ 16% GS 11

̶ 15% GS 12

̶ 14% GS 9

̶ 14% GS 6

̶ 9% GS 5

Presenter

Insert a slide that shows which VISNs (maybe totals and/or %) and then just list of which VISNs (no % or quantity data)

VILT Offerings and Completions (Page 1 of 2)

During FY2017 the HR Academy had over 3,000 training completions with a total of 1260 unique participants. An increase of more than 20% over completions from the same period last year.

Courses

FY- 2017

FY- 2017 TotalQuarter 1 Quarter 2 Quarter 3 Quarter 4 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Advanced Staffing 32 32 Advanced Staffing Module 1: Problem Solving and Decision Making 28 28

Advanced Staffing Module 2: Workforce Succession Planning 29 29

Advanced Staffing Module 3: Strategic Recruitment 23 23

Advanced Staffing Module 4: Researching and Applying Regulations and Laws

25 25

Advanced Staffing Module 5: Appointment Eligibility Determinations 24 24

Advanced Staffing Module 6: Complex Pay Determinations 19 19

Adverse Actions 32 30 62 Alternative Dispute Resolution (ADR) 17 17 Basic Staffing 23 23 Basic Title 38 Pay Setting 28 32 28 27 22 137 Best Practices in Initiating and Processing of Personnel Actions 11 27 29 63 29 31 190

Category Rating Workshop - Virtual Training 29 32 28 89

Central Personnel Data File Edits 1 1

Collective Bargaining Agreements (CBAs) 28 28

Consulting Skills for HRM Professionals 34 34

Determining Service Computation Dates - Virtual Training

31 31 32 26 25 145

EEO in the Federal Sector 64 64 Employee Benefits - Virtual Training 27 31 30 27 115 Employee Conduct and Performance 1 102 18 121

Presenter

HR Academy had a high number of total completions in March (661), Feb (455) and Jun (404) while hitting lower numbers in Oct (150) and Nov (211). Lower completions were due in part to increased workloads for HR professionals due to seasonal influx of the workforce, which led to lower enrollments and class cancellations. Trainings with the highest number of completions during the reporting period— “Veterans’ Preference” (371), “Federal Staffing and Placement” (331) and “Qualifications Analysis” (276).

Courses

FY- 2017

FY- 2017 TotalQuarter 1 Quarter 2 Quarter 3 Quarter 4 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Employee Development 17 17

Employee/Labor Relations - Virtual Training 27 24 51

Federal Staffing and Placement 32 33 23 61 33 94 24 31 29 360 Fundamentals of Legal Research 29 30 30 30 31 29 179 GS Pay Setting 29 30 31 26 30 27 173 Interpersonal Communications 1 1 2

Oral Communications for HR Professionals 23 23

Overview of Processing Personnel Actions 26 29 63 31 29 26 204

Performance Management - Virtual Training 1 24 25

Performance-Based Actions 29 32 61

Position Classification and Position Management 31 28 59

Position Management - Virtual Training 10 28 38

Preparing to Negotiate 28 28 Proactive Succession Planning 31 31

Processing Personnel Actions - Foundations 30 30

Processing Personnel Actions - Virtual Training 14 33 30 32 109 Qualifications Analysis 31 32 33 30 30 89 31 31 26 333 Strategic Position Management 29 32 61

The Hiring and Retention Overview 29 30 30 89

VA Merit System Principles and Prohibited Personnel Practices Virtual Training

21 59 50 130

Veterans' Preference - Virtual Training 33 62 28 60 64 32 61 31 30 29 430 Grand Total 150 211 236 351 455 661 305 278 404 287 301 3639

Understanding of HR Academy VILT Offerings and Completions (Page 2 of 2)

Presenter Presentation Notes HR Academy had a high number of total completions in March (661), Feb (455) and Jun (404) while hitting lower numbers in Oct (150) and Nov (211). Lower completions were due in part to increased workloads for HR professionals due to seasonal influx of the workforce, which led to lower enrollments and class cancellations. Trainings with the highest number of completions during the reporting period— “Veterans’ Preference” (430), “Federal Staffing and Placement” (360) and “Qualifications Analysis” (333).

***Completion Numbers may vary due to individual participation. If an individual fails to complete all required sections for a course in TMS they will not receive a completion until those missed sections are complete and the dates of completion will reflect the actual date the individual fulfilled all the requirements not the last date of the scheduled offering.

Web-based Training Completions as of August 31, 2017 (Page 1 of 2)

Course ITEM ID Status August Completions

Overall Completions

Assessment Questionnaire Development 3940798 Active 32 550

Assessment Questionnaires for Hiring Managers 3940799 Active 1 18

Consulting Skills WBT 3867980 Active 5 276

Customer Service for HR Professionals 4191206 Active 11 305

Domestic Violence/Intimate Partner Violence (DV/IPV), Sexual Assault (SA), Stalking in the Workplace 3940800 Active 2 43

Job Analysis 3940805 Active 27 603

Job Analysis for Hiring Managers 3940803 Active 3 26

Merit System Protection Board (MSPB) 3858368 Active 4 267

Merit Systems and Prohibited Personnel Practices for Hiring Managers 3940806 Active 10 286

PAID Coding (4 courses) -GPPA and PAID Coding Manual -Processing HR Actions -Veterans’ Preference -Determining Service Computation Date (SCD)

3858492 Active 5 951

Processing Voluntary Retirement Applications for Civil Service Employees (CSRS) 3768170 Active 1 114

Processing Voluntary Retirement Applications for Federal Employees (FERS) 3768166 Active 10 367

Web-based Training Completions as of August 31, 2017 (Page 2 of 2)

Course ITEM ID Status August Completions

Overall Completions

Strategic Human Capital Planning 3858380 Active 1 59

Structured Interviews for Hiring Managers 3940807 Active 1 13

The Family Medical Leave Act (FMLA) 3940797 Active 6 101

Title 38 Discipline Overview 3867986 Active 3 156

Title 5 Discipline Overview 3867981 Active 5 169

USA Staffing Orientation 4190163 Active 22 638

VA Title 38 Professional Management Development Series: History and Introduction 4202942 Active 2 72

VA Title 38 Professional Management Development Series: Nurse Locality Pay System 4202943 Active 1 42

VA Title 38 Professional Management Development Series: Premium Pay 4202944 Active 2 35

VA Title 38 Professional Management Development Series: Probationary Period 4202945 Active 4 60

VA Title 38 Professional Management Development Series: Professional Standards Boards 4202947 Active 21 186

VA Title 38 Professional Management Development Series: The Appointment Process 4202941 Active 5 72

HR-Smart Awards and Incentives 4212190 Active 1 36

HR-Smart Manage Positions 4212185 Active 8 69

HR SMART – Employee Relations/Labor Relations Personnel Actions (Recorded Webinar) 4178858 Active 10 22

HR SMART – Benefits Personnel Actions (Recorded Webinar) 4178860 Active 6 18

Total WBT Completions 209 5554

IT Campus Training Completions as of August 31, 2017

*Due to the closure of the IT Campus, further individual reporting on this modality will only be for the print edition of the above.

Course ITEM ID Status August Completions

Overall Completions

HR Academy: Fair Employment Opportunities (Print) 3910965 Active 97 10155

Total IT Campus Completions 97 20871

CDN Training Completions as of August 31, 2017 (Page 1 of 3)

Course ITEM ID Status August Completions

Overall Completions

Alternative Dispute Resolution (CDN) 3949086 Active 13 231

Communicating across Cultures (CDN) 3937071 Active 3 91

Determining Tenure (CDN) 4176488 Active 1 3

Developing Your Political Savvy (CDN) 3937093 Active 1 46

Effective Listening (CDN) 3937091 Active 7 294

Effective Team Communication (CDN) 3937074 Active 19 685

Employee Conduct: Handling and Documenting Issues (CDN) 4176521 Active 0 16

Enhancing Your Attention to Detail (CDN) 3937096 Active 2 134

Giving and Receiving Feedback (CDN) 3937107 Active 3 108

HR Academy: Fair Employment Opportunities (CDN) 3911053 Active 84 8,260

Negotiation Skills (CDN) 3937108 Active 5 51

Organizational Agility (CDN) 3949101 Active 2 35

Overview of HR Competency Models (CDN) 4176518 Active 0 22

Overview of HR Competency Models (Recorded Webinar) 3894813 inactive 0 2,264

August 31, 2017 (Page 2 of 3)

Course ITEM ID Status August Completions

Overall Completions

Planning Your Professional Development (CDN) 3937121 Active 1 67

Policy Interpretation Tips and Tricks (CDN) 3949103 Active 0 36

Presenting with Your Audience in Mind (CDN) 3937085 Active 1 75

Recruitment Terminology (CDN) 3949107 Active 2 100

Reduction in Force Overview (CDN) 4176477 Active 4 9

Strategic Human Capital Management (CDN) 3949109 Active 0 28

Strategies for Decision Making (CDN) 3937112 Active 3 82

Structured Interviewing (CDN) 3949111 Active 0 46

Succession Planning (CDN) 3937105 Active 82 276

Supervisor Role in the HR Competency Self-Assessment and eIDP Process (CDN) 3949113 Active 3 105

VA Health and Wellness (CDN) 3949114 Active 2 57

Writing Position Descriptions Part 1 (CDN) 4176519 Active 2 127

Writing Position Descriptions Part 2 (CDN) 4176520 Active 3 100

HR BEST 1: Welcome and Introduction (CDN) 4193002 Active 2 22

HR BEST 2: Managing Organizational Performance (CDN) 4193003 Active 2 26

August 31, 2017 (Page 3 of 3)

Course ITEM ID Status August Completions

Overall Completions

HR BEST 3: Leadership Styles and Team Building (CDN) 4193004 Active 4 31

HR BEST 4: Building Partnerships for Success (CDN) 4193005 Active 1 20

HR BEST 5: Mentoring and Coaching (CDN) 4193006 Active 3 12

HR BEST 6: Supporting Your Team (CDN) 4193007 Active 4 21

VA Employee Health and Wellness (CDN) 4193208 Active 2 36 eOPF Folder Maintenance Part 1 (CDN) 4193209 Active 4 35 eOPF Folder Maintenance Part 2 (CDN) 4193210 Active 4 26 eOPF Working with Reports (CDN) 4193211 Active 4 20

HR SMART Manage Benefits (CDN) 4193212 Active 4 38

Regularizing Illegal Appointments (CDN) 4193213 Active 0 2

Staffing Variations Webinar (CDN) 4193214 Active 0 5

Understanding the Civil Service Retirement System (CSRS) Benefit Package:

Recording (BB) 4191563 Active 2 11

Understanding the Federal Employees Retirement System (FERS) Benefits Package: Recording (BB) 4191574 Active 3 51

Total CDN Completions 282 13,704

Feedback from VA HR Community on the services provided by HR Academy, whether they have attended training or not

Annual Customer Satisfaction Survey Results

Annual Customer Engagement Survey Government Response Comparison

HR Academy received much higher response rates from participants of the Customer Satisfaction Survey in FY 2017 as compared to FY 2016

In FY 2016 the Response rate was 673 or approximately 14% of participants responded

In FY 2017 the Response rate was 1047 or approximately 23% of participants responded

Demographic Highlights

Admin/Office

The survey demographics are consistent with our training population:

• 71% of responders were 201s – HR Specialists

• 83% of the responders representing VHA and 6% representing the

VBA and NCA while 5% represent other staff offices

*NCA, Staff Offices, & Organizations*

Job Series

20% of our HR Professionals have 5-10 years experience & more than 20 years of experience in the Human Resources sector. 25% of students have spent their time working for the VA in Human Resources for 2 to 10 years.

Annual Customer Engagement Survey Additional Demographic Highlights

Total number of years of experience in the human resources field is:

6 months to 1 year

Less than

6 months

1 year to 2 years

2 years to

5 years

5 years to

10 years

10 years to

15 years

15 years to

20 years

More than

20 years Grand Total

6 months to 1 year 29 2 1 32 3%

Less than 6 months 3 23 26 2%

1 year to 2 years 9 5 52 2 68 6%

2 years to 5 years 13 7 30 123 2 175 17%

5 years to 10 years 7 12 21 29 142 2 213 20%

10 years to 15 years 8 9 8 40 36 63 2 3 169 16%

15 years to 20 years 6 7 12 28 36 22 42 1 154 15%

More than 20 years 7 3 11 35 49 32 21 52 210 20%

Grand Total 82 68 135 257 265 119 65 56 1047

8% 6% 13% 25% 25% 11% 6% 5%

I have worked as a human resources professional within VA for:

Ability to Impact Job Satisfaction

HR Academy assesses job satisfaction and worker morale through the annual Customer Engagement Survey.

48% of responders confirmed Effectively Performing Current Responsibilities

30% of responders are Pursuing Career Goals within the VA

17% of responders affirmed that the quality of Training would create job satisfaction and worker morale.

** Based on respondents who Agree and Strongly Agree**

Training Requests

Areas in highest demand have shifted in comparison to 2016 (top 2 requests were recruitment & placement and employee relations).

This graph indicates that employees within the VA HR community recognize the value of varied training options in helping improve their job performance.

Improving Job Performance

Responses to the survey confirmed that the opportunities provided through HR Academy led to a majority of training participants in improving their job performance and relevant knowledge.

Training Barriers

Responses to the survey request regarding the most common challenges to participating in training courses. Classes were closed when trying to register and lack of time continues to be the greatest challenge.

Key Influences

Responses to the survey request regarding the factors that most influence HR professionals seeking out and registering for training. Course learning objectives, format, and length of program combined are the most influential aspects with regard to training registration and enrollment.

HR Academy continues to bring new capabilities to help the VA HR Community

New Capabilities and Offerings

Support for Training Officers Training on Demand (TOD) Request

• The TOD Request is one of two forms available to request HR Academy services

• Employees can access the form from the home page and the Training Officers page (drop down menu or page itself)

• Requestors fill out and submit the online form

• Results are collected into a table that tracks training requests

– Download to MS Excel

Spreadsheet

Similar capability exists to request new courses

Application Process (e.g., Emerging Leaders, CXO, Intermediate Certificates)

Solicit Applicants

• HRA sends link out to Senior Leadership

Identify Candidates

•Senior Leadership provides application link to select candidates

Submit Applications

•Candidates complete survey application

Coordinate submissions

• Any back and forth needed to complete applications and submissions

Review Applications

• HRA reviews applications

•Recommendations made for candidate selection

Make Selections

•Senior Leadership accepts/rejects applications

Update Applicant Status Status

•Mark applicants as accepted or rejected

Produce Final Report

•Applicants notified

LEGEND

• HRA

• Senior Leadership

• Shared

Communities of Practice Novice Graduates as of 8/31/2017

• Candidates can enroll at any time

• Acceptance requires supervisor approval

• Courses are self-paced and must be completed within 2 years

847 unique candidates (some have enrolled in all 4 curriculums)

Recruitment and Placement

Employee Relations Position Classification

Employee Benefits

24 23 25

Number Enrolled Number Completed

= enrollees = completions

New Formal Programs HR Best & Intermediate Certificates

HR Best = New program to help with HR leadership skills

And technical offerings

• Recruitment & placement

• Classification

• Compensation

• Employee benefits

• Employee relations

Learning Evaluation In the Kirkpatrick Model for learning evaluation, each successive evaluation level is built on information provided by lower level. This section includes results at each level:

• Level 1 assesses the participants’ satisfaction relative to the training attended

• Level 2 assesses the participants’ learning relative to the training attended

• Level 3 assesses the participants’ ability to implement the new skills or abilities learned in training while on-the-job

• Level 4 assesses the impact of HR Academy training on the VA HR community using outcome-based progress metrics

Section 2

Presenter Presentation Notes 1 – did you like it?

2 – did you learn something?

3 – did it help you in your job?

4 – has it made us better, cheaper, faster?

Kirkpatrick Model and Data Collection Approach

Measure = Degree to which: Tool Collection Method

Participants react favorably to the learning event

End-of-course evaluations

Distributed at the end of each training class and must be completed within 2 business days.

Participants acquire the intended knowledge, skills, and attitudes through their participation in the learning event

Pre-training tests (pre-tests) and post-training tests (post-tests)

Pre-tests are distributed prior to or at the beginning of a training class and Post-tests are distributed at the close of or end of a training class.

Participants apply what they learned during training when they are back on the job

Retrospective questionnaires for employees and supervisors

Questionnaires are distributed to training participants and supervisors or managers approximately 60 days after training completion.

Targeted outcomes occur as a result of the learning events and subsequent reinforcement

Metrics from other sources (e.g., USA Staffing, TMS)

Performance-based outcomes are collected over the course of the training year for annual assessment.

Ki rk pa tri ck M od el Reaction

Learning Behavior

Results

Participants’ satisfaction with training

Level 1 (Reaction) Yes, I had a great experience in this HR

Academy course!

Level 1: FY17 Monthly Evaluation Trend Analysis

Participant reaction and satisfaction with HR Academy’s course offerings has been steadily positive, as shown by the trendline, with an average score of 4.51 over the course of the fiscal year.

4.48 4.39

4.44

4.60 4.56 4.56 4.48

4.58

4.53 4.55

4.48

3.50 3.60 3.70 3.80 3.90 4.00 4.10 4.20 4.30 4.40 4.50 4.60 4.70 4.80 4.90 5.00

O ct ob er

N ov em be r

De ce m be r

Ja nu ar y

Fe br ua ry

M ar ch

Ap ril

M ay

Ju ne Ju ly

Au gu st

Q 1 Q 2 Q 3 Q 4

FY- 2017

Total

Linear (Total)

Presenter

The Level 1 evaluation gives the participants positive statements about the course across a 5-point Likert Scale, with responses ranging from “Strongly Disagree” (a score of 1, the lowest) to “Strongly Agree” (a score of 5, the highest).The highest average scores were received in the months of January (4.60), May (4.58), and February & March (4.56). These favorable scores appear to be a continuing stabilization of scores in prior years.

Annual Monthly Average Comparison

Questions Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Range

FY2017

Overall n=138 n=199 n=214 n=304 n=398 n=491 n=281 n=231 n=397 n=294 n=358 Average of This training will improve my job performance. 4.49 4.47 4.47 4.57 4.59 4.59 4.54 4.66 4.53 4.57 4.50 0.19

This delivery method was an effective way for me to learn the material. 4.30 4.30 4.39 4.45 4.48 4.52 4.42 4.46 4.42 4.50 4.44 0.22

I would recommend this course to others 4.49 4.39 4.48 4.64 4.58 4.60 4.52 4.65 4.58 4.56 4.46 0.26

The scope of the material was appropriate to my needs. 4.62 4.63 4.58 4.62 4.63 4.66 4.65 4.68 4.57 4.61 4.62 0.11

Overall, I was satisfied with this event. 4.48 4.39 4.44 4.60 4.56 4.56 4.48 4.58 4.53 4.55 4.48 0.21 Content/Instruction

The course met all of its stated objectives. 4.43 4.50 4.62 4.65 4.60 4.61 4.59 4.62 4.61 4.63 4.53 0.22

Showed respect and consideration for students. 4.59 4.74 4.77 4.80 4.75 4.74 4.00 4.75 4.76 4.78 4.70 0.80

The examples, hands-on labs or exercises enhanced my learning 4.41 4.40 4.47 4.54 4.51 4.55 4.48 4.54 4.49 4.55 4.42 0.15

The instructor(s) managed the class effectively (kept participants focused and moving forward).

4.43 4.50 4.62 4.71 4.63 4.67 4.59 4.65 4.65 4.65 4.57 0.28

The objectives clearly defined the content and/or skills to be addressed. 4.49 4.51 4.66 4.65 4.62 4.60 4.61 4.64 4.60 4.65 4.55 0.17

Logistics/Performance

The session was available when I needed it. 4.28 4.10 4.36 4.51 4.28 4.41 4.00 3.99 4.46 4.36 4.39 0.52

The registration and enrollment process was satisfactory. 4.39 4.41 4.61 4.52 4.45 4.55 4.33 4.41 4.57 4.47 4.52 0.28

The physical environment was conducive to learning. 4.33 4.30 4.45 4.53 4.47 4.55 4.32 4.36 4.45 4.57 4.49 0.27

I had the knowledge and/or skills required to take this session. 4.37 4.35 4.31 4.45 4.39 4.43 4.38 4.29 4.30 4.41 4.37 0.16

Average (All Questions) 4.43 4.43 4.52 4.59 4.54 4.57 4.42 4.52 4.54 4.56 4.50 0.17

Presenter Presentation Notes Participant scores fluctuated from scores as high as 4.80 to those as low as 4.10 in some questions. The highest scores were received in the Overall and Content/Instruction categories for the following questionnaire items:

Showed respect and consideration for students. (4.67 average score) The scope of the material was appropriate to my needs. (4.62 average score) The instructor(s) managed the class effectively. (4.61 average score) The objectives clearly defined the content and/or skills to be addressed. (4.60 average score) The lowest scores were received in the Logistics/Performance category, specifically:

The session was available when I needed it. (4.29 average score) I had the knowledge and/or skills required to take this session. (4.37 average score)

Open Ended Responses

FY17 Level of Satisfaction

Most Common Rationale (note: the ones in bold, italic or new for FY17; others remain unchanged from FY16)

FY16 Level of Satisfaction

Do you have any other comments about the course instructor/facilitator?

90% High Level of Satisfaction

Knowledgeable instructors with strong understanding of VA and Federal sector Well-organized and patient training instructor and facilitator team Friendly, polite, and engaging; Able to solicit participating for all attendees

82% High Level of Satisfaction

10% Low Level of Satisfaction

Unknowledgeable instructor, unable to respond to VA-specific questions Poor timing, rushing through questions or class material Lack of patience, inattentive, or potentially unfriendly/unengaging

18% Low Level of Satisfaction

Do you have any other comments about the course overall?

78% High Level of Satisfaction

Knowledgeable instructors with great understanding of VA and Federal sector Well-organized, highly professional and patient training instructor and facilitator team Relevance of course topics and material Well-paced class and overall punctuality and timeliness

58% High Level of Satisfaction

22% Low Level of Satisfaction

Course too long/Course too short Unknowledgeable instructor, unable to respond to VA-specific questions Preference for an alternative training modality (WBT or Face-to-Face) Not enough scheduled offerings

42% Low Level of Satisfaction

Do you have any other comments about the course logistics?

27% High Level of Satisfaction

Timely technical assistance, as needed Ease of use of tools (e.g., Blackboard and MTM participant interfaces)

28% High Level of Satisfaction

73% Low Level of Satisfaction

Class availability; lack of sufficient scheduled offerings Lengthy waitlist times No place for virtual learners to escape noise of office

72% Low Level of Satisfaction

Level of satisfaction for courses overall and instructors/facilitators increased this year over data collected the previous FY. Course logistics and comments across all areas remained about the same.

Presenter

HR Academy augments their quantitative analysis with qualitative data that they collect from open-ended questions used in the end-of-course evaluation. The Academy solicits feedback specifically on instructors/facilitators, logistics and overall satisfaction. Results were consistent throughout the fiscal year, where instructor/facilitator satisfaction was the strongest. Approximately 90% of participants who provided feedback were impressed with the instructors and the facilitators used.

Open Ended Question Results

Over 86% of employees have email as their preferred method of receiving information on future training sessions

While the preferences of how learners want to learn about opportunities have not significantly changed when compared to FY16 results, HR Academy is moving to better support the community based on increase in how they reported they found out about courses in FY16:

• Only 21% found out via emails

• 50% had to look on Website/TMS

How did you find out about the course?

How do you prefer to receive communications about future training opportunities?

All responses 2755 2755

E-mail/Outlook 1135 41.2 % 2384 86.5 %

Supervisor/Manager 648 23.5% 106 3.8 %

HR Academy Website/TMS 940 34.1 % 255 9.3 %

Not Listed/Other 32 1.2 % 10 0.4 %

Level 1: FY 2013 – FY 2017 Historical Trend Analysis

Presenter

Training participant satisfaction has steadily grown over the last five fiscal years (FY13-FY17). This positive trend is supported by the increased demand in anecdotal feedback expressing great satisfaction with instructor/facilitators, relevance of content and overall experience.

Level 1: Qualitative Feedback on other HR Academy Offerings

I had the following experience with respect to the annual Competency Assessment survey:

Waiting for supervisor to review it with me Completed but not sure how valuable yet Was not encouraged to complete it Completed my portion but received no response or next steps Completed but showing as not completed Technical issues

I had the following experience with my eIDP:

Complete lack of comprehensive instruction Don't know anything about eIDP Completed Supervisor has never reviewed with me Completed one last year....not utilized

Are there any barriers that have prevented you from participating in VA HR Academy training?

Approval from senior management.

Availability and time offered

Classes fill up too quickly and excessive waitlists Class time (e.g., 6:00 PST, need East Coast times)

Classroom training with an SME lead instructor Virtual training hard to complete at worksite I do better in a class room environment.

Participants’ learning during training

Level 2 (Learning) Thanks HR

Academy! I did a lot better on this post-test based on this class…

Level 2: FY17 Pre- and Post- Test Statistics

Q1 Q2 Q3 Q4 FY 17

Average

Pre-test Average 58.4 % 58.5 % 59.3 % 60.2 % 59.0 %

Post-test Average 69.2 % 72.5 % 71.2 % 71.3 % 71.3 %

Average Increase* 10.8 % 13.0 % 10.4 % 9.3 % 11.2 %

* Indicates average improvement for participants who took BOTH the pre-test and post-test.

HR Academy collected Level 2 data from classes taught from October 2016 to August 2017

̶ 3,639 participants took both a pre-test and post-test

̶ 11.2 percentage points was the average increase from pre- to post-test

Level 2: FY17 Monthly Averages Pre and Post Test Score Comparison

On average, students regularly increased their knowledge by attending HR Academy courses. The average increase was by 12.5 percentage points between pre-tests and post-tests over the past twelve months.

October November December January February March April May June July August

Presenter

Average improvements for each month of the current reporting period (October 1, 2016 through August 31, 2017) consistently showed improvements like those of previous years. The months of October (14.4%), January (14.5%), February (12.1%) and March (12.6%), had averages higher than the overall average of 11.2%. On the other hand, the months of November, December and July had the lowest average improvement. November and December tied with (9.4%) and July at (8.1%). 2017’s yearly improvement average increased by 0.6% over 2016’s. (2016=10.6% vs 2017=11.2%)

Level 2: FY13 – FY17 Historical Trend Analysis

57.7 58.7 58.8 60 58.7

74.5 74.8 75

72.9 71.3

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

FY2013 FY2014 FY2015 FY2016 FY2017

Presenter Presentation Notes HR Academy maintained consistency in pre- and post-test scores since the transition to virtual instructor-led (VILT) courses in FY13. Average test score statistics from FY13 - FY17 were as follows:

Pre-Test Average is 58.8 % Post-Test Average is 73.7 % Average Increase is 14.9 %

Level 3 (Behavior)

It’s helped me on the job… and my boss says so, too!

Participants’ ability to implement the new skills or abilities learned in training while on-the-job

Level 3: FY17 Retrospective Questionnaire Process Overview

• Questionnaires are distributed to training participants and their direct Manager/Supervisor approximately 60 days after training completion

• 20% of the scheduled class offerings during FY17 were selected

• The 16 classes chosen were:

̶ Fundamentals of Legal Research ̶ Federal Staffing & Placement ̶ Veterans’ Preference ̶ Category Rating ̶ Qualifications Analysis ̶ VA Adverse Actions ̶ VA Performance Based Actions ̶ Workforce Planning: Practice and Application ̶ VA Alternative Dispute Resolution ̶ VA Qualifications Standards for Non-Supervisory GS Positions ̶ GS Pay Setting ̶ Performance Management ̶ Employee Relations/Labor Relations ̶ Strategic Position Management ̶ Processing Personnel Actions – Foundations ̶ Overview of Processing Personnel Actions

Level 3: FY17 Retrospective Questionnaire Ratings Summary

Categories

Retrospective Questionnaire Responder

Training Participant Manager

# of Evaluations 177.0 73.0

Overall 3.9 4.1

Return on Investment 4.2 4.4

Support / Tools 3.6 4.3

Learning Effectiveness 4.1 4.3

Business Results 4.0 3.9

Job Impact 3.9 3.8

Presenter

The total number of training participant surveys received was 177; and the total number of Manager evaluations received was 73. The highest average ratings were received in the Return on Investment category for both participants and Managers.

Level 3: FY17 Business Results Impacts Participant View

• The highest rated impacts to business results were:

̶ Increasing quality ̶ Increasing productivity ̶ Increasing customer satisfaction

• These are the same top 3 results as last years survey results

Level 3: FY17 Business Results Impacts Manager View

• The highest rated impacts to business results were:

̶ Increasing Quality ̶ Increasing

Customer Satisfaction

̶ Increasing Employee Satisfaction

• Increasing Customer Satisfaction beat out Increasing Productivity for third which was first last year

Level 3: FY17 Business Results Impacts Comparison

When comparing the business impacts between the participants and managers, the managers are consistently seeing the benefits more than the participants.

With the two largest differences in Increasing Quality and Increasing Employee Satisfaction

48%

66%

62%

56%

45%

49%

37%

44%

0% 10% 20% 30% 40% 50% 60% 70%

Increasing Productivity

Increasing Quality

Increasing Employee Satisfaction

Increasing Customer Satisfaction

Business Results Impacts Comparison

Participant Manager

Level 3: FY16 vs FY17 Business Results Impacts Manager View

Impact of HR Academy training on the VA HR community using outcome-based progress metrics

Level 4 (Business Results)

Level 4: Reported Business Outcomes and Metrics

HR Academy worked with stakeholders to determine Level 4 metrics and outcomes that best indicate performance for learning and professional development.

Level 4 Business Outcome Metric

Baseline Workforce Knowledge and Capabilities

• Leadership and Core Competency Strengths of HR Professionals

• Aligning Competency Gaps to HR Academy Offerings

Effectiveness of HR Services

• Improved Accuracy of HR Services (e.g., Return Certificate Rate)

• Higher Productivity per HR Employee (e.g., Hiring Certificate Production)

• Improved Timeliness (e.g., Speed of Hire)

Increased Capabilities of HR Professionals

• Strengths of Leadership and Non-Leadership Employees

• Trend Analysis of Technical Gaps of HR Professionals by

Leadership Type

• Impacts of HR Academy – Learning Improvement related to FY17

Competency Gaps (Core, Leadership, & Technical)

Presenter

Evaluation at Level 4 helps to further understand how the learning impacted the participants’ work environment and includes analysis of some external factors that might contribute to the results. The intention is to explore the influence training has at the organizational level. To determine the Level 4 metrics and outcomes that best indicate HR Academy performance, four mini-workshops were held in December 2015 with key stakeholders (HR and related system subject matter experts) to set direction for data collection and analysis.

Level 4: Top 10 Competency Strengths of HR Professionals

The Work/Life Integration competency shows that approximately 1,869 employees (69%) are functioning at or above their target proficiency level (PL).

Strengths

Competency

FY17

# Strengths % Strengths Mode PL Work/Life Integration 2709 69% 2

Analytical Reasoning 2601 66% 2

Human Resources Information Systems 2577 65% 2

Strategic Thinking 2567 65% 1

Flexibility 2553 65% 2

Teamwork 2436 62% 3

Conflict Management 2428 62% 2

Organizational Awareness 2415 61% 2

Self-Management 2367 60% 2

Communications 2367 60% 2

Presenter

HR Academy continues to demonstrate its strengths and capabilities in helping meet VA’s current and emerging training needs and close proficiency gaps. All technical competency gaps have dropped since FY16. Overwhelmingly, FY17 results showed that HR staff has more strengths than gaps. The top technical competency strength is in Work/Life Integration, with approximately 2,709 employees (~69%) functioning at or above their target proficiency level. Each of the top ten technical competencies were above 50%.

Level 4: FY17 Top Seven Technical Competency Gaps of HR Professionals

Gaps

Competency FY17 HR Academy Support # Gaps % Gaps Mode PL Courses Completions

Labor Relations 1587 40% 2 2 115

Position Classification and Management

1450 37% 2 3 128

Performance Management 1315 34% 2 3 207

Employee Relations 1310 33% 1 5 342

Recruitment and Staffing 1257 32% 2 11 1450

Workforce and Succession Planning

1207 31% 1 2 51

Pay Administration 1205 31% 2 4 461

HR Academy is actively delivering VILT for the top 7 competency gaps.

Presenter

In FY17, the competency model used to assess gaps was enhanced and training is still in the progress of re-aligning with the new model. For instance, Employee Development was not identified as a unique competency area. The reality is that there may be Performance Management and/or Staffing courses that help support the Employee Development competency area. As demonstrated, HR Academy’s training is aligned with the technical needs of the VA HR community. Trending Key: Gap has increased Gap has decreased No Change

Level 4: FY17 Hiring vs. Return Certificate Rates

Minimal change in hiring process… likely due to HR community and hiring managers not able to take training or other factors outside of HR Academy’s control.

Apr-16

M ay-16

Jun-16

Jul-16

Aug-16

Sep-16

O ct-16

N ov-16

Dec-16

Jan-17

Feb-17

M ar-17

Apr-17

M ay-17

Jun-17

Jul-17

Hiring Certificate Production

Certificate w/ Selection Return w/out Selection

55 Speed of Hire

Presenter

The team has tracked speed of hire and certificate production to see what, if any, impact training may have the hiring process (shown to left). The percentage of selections, number of certificates, and speed of hire remain relatively consistent. Training is likely not impacting due to the ability for HR community and hiring managers to attend training rather than reflection of training.

Level 4: FY17 Actual vs Planned Percent of Hires

The VA meet and exceeded its planned projection of hiring within 60 days during FY17.

November December January February March April May June July

49 49

Da ys

Average Days from Announcement to Hire

Average # Days 60 Day Goal

Presenter

The speed-of-hire rate metric refers to the number of days from the hiring request that it takes for an employee to be hired to VA. The target is that at least 80% of hires are processed through the hiring lifecycle in 60 days or less.

Level 4: FY17 Competency Strengths Leadership Profile Type

Strengths

Competency

FY17

# Strengths %Strengths VA Core Competency Strengths

Integrity/Honesty 213 59% Flexibility 149 39% Teamwork 135 30%

VA Leadership Competency Strengths Change Management 192 42% Entrepreneurship 174 37% Financial Management 112 30%

Technical Competency Strengths Work/Life Integration 114 26%

Workforce and Succession Planning 80 22%

Human Resources Information Systems

67 18%

The highest competency strengths are

• Integrity/Honesty

(59%)

• Change Management

(42%)

• Entrepreneurship

(45%)

Figures based off of VHA & VBA Leadership.

Level 4: FY17 Competency Strengths Non-Leadership Profile Type

The highest competency strengths are

• Change

Management (74.5%)

• Tied for second Strategic Thinking

• , and Strategic Thinking (71.5%)

Figures based off of VHA & VBA Non-Leadership

Strengths

Competency

FY17

# Strengths % Strengths VA Core Competency Strengths

Strategic Thinking 2494 71.5% Analytical Reasoning 2486 71% Conflict Management 2219 70%

VA Leadership Competency Strengths Change Management 930 74.5% Entrepreneurship 747 57.5% Leveraging Diversity 650 52%

Technical Competency Strengths Work/Life Integration 2528 71.5% Human Resources Information Systems 2447 69%

Employee Development 1944 61%

Level 4: FY17 Competency Gaps Compared to FY16

Trending Key: Gap has increased Gap has decreased No Change Preliminary in FY16

Leadership Profile Gaps

Competency

FY17 FY16

# Gaps % Gaps % Gaps

VA Core Competency Gaps

Conflict Management 233 63% - Strategic Thinking 229 62% -

Information Assurance 219 59% -

VA Leadership Competency Gaps

Developing Others 248 67% -

Partnering 214 58% - Accountability 212 57% -

Technical Competency Gaps

Position Classification and Management 227 61% -

Employee Relations 221 61% -

Labor Relations 220 61% -

Non-Leadership Gaps

Competency

FY17 FY16

# Gaps % Gaps % Gaps

VA Core Competency Gaps

Information Assurance 684 20% 19% Integrity/Honesty 712 26.5% 27.5% Veterans Service Motivation 634 20% 13%

VA Leadership Competency Gaps

Human Capital Management 646 47% 38%

Political Savvy 622 47% 39% Financial Management 626 50.5% 40%

Technical Competency Gaps

Labor Relations 1289 37% 35%

Position Classification and Management 1158 31.5% 26%

Performance Management 1007 32% 23%

Presenter

When comparing Leadership and Non-leadership profiles, Information Assurance, Labor Relations, and Position Classification and Management score high in both profile assessments.

Level 4: Addressing Gaps Core & Leadership Competencies

Core Competency: Strategic Thinking

Supporting Courses:

• Advanced Staffing Module 4

Leadership Competency: Developing Others

Supporting Courses:

• Overview of Process Personnel Actions

• Processing Personnel Actions - Foundations

= Pre-Test = Post-Test

Level 4: Gaps Cont.

Technical – Top 2 Reported Gap Areas

Labor Relations

Supporting Courses:

• Employee/Labor Relations

• Preparing to Negotiate

Employee Relations

Supporting Courses:

• Alternative Dispute Resolution

• Fundamentals of Legal Research

• VA Merit System Principles and Prohibited

Personnel Practices

Level 4: Gaps Cont.

Technical – 3rd Highest Reported Gap Area

Supporting Courses:

• Position Classification and Position Management

• Position Management

Findings and Action Plan

Section 3

#1 – Aligning Competency Gaps with Offerings #2 – Expanding HR-Metrics to Additional Offerings #3 – Refining Level 4 Metrics

The purpose of this section is to summarize the findings presented in the previous sections and turn the results into actionable areas for change.

Recommended actions include:

Findings - Programmatic Assessment

HR Academy’s FY17 programs = highly successful overall:

• This year’s findings are consistent and corroborate with results from prior years

• HR Academy continues to meet the dynamic learning needs of the VA, focusing primarily this year on technical capabilities with largest number of completions related to:

• A new offering was introduced to help with HR-Smart based on mid-year requests for such training. This course was given several times reaching 190 students

• HR Academy also is not limited to these technical skills but continues to support the softer skills and helping HR professionals becoming more well-rounded employees. See courses below.

Competency Completions

Recruitment & Staffing 1450

Pay Administration 461

Employee Relations 342

Employee Benefits 260

Performance Management 207

Competency Completions

Developing Others 178

Strategic Thinking 86

Communications 59

Problem Solving 57

Findings – Reactions Customer Satisfaction and Level 1 Surveys

• Courses were most often selected based on the following three criteria:

– Course learning objectives

– Education/certification credits

– Format of course

• Most requested areas for new training include:

– HR Smart

– Recruitment & Placement

– Title 38/Hybrid Title 38

– Position Classification

– Employee Relations

• Individuals experienced the following barriers when trying to use HR Academy:

Findings – Learning

• VILT courses include a pre-test and post-test to validate that students gain knowledge and skills from all course objective areas.

• As shown through the Level 2 results, students leave courses with more knowledge than when they start. The average score increases this year are shown below:

• The evaluation team also began tracking not only course-by-course competencies, but also has begun to link test scores and knowledge gains to competency areas to see how HR Academy courses are (or are not) impacting the…