Training effectiveness has been the challenge for corporate training for long time. Kirkpatrick’s four levels have dominated the training evaluation discussion since it was first published in 1959. Recent findings show the bottleneck for training effectiveness is between Level Two and Level Three. Which means learner may not apply their learning even they have learning from the training. Without application, the training has no impact on business results, which are the fundamental goal for corporate’s investment in training.

Doctor Brinkhoff surveyed 100,000 people who attached training, and found out that less than 15% of them have applied their learning and have achieved concrete and valuable results; more than 85% of them either have not tried the learning at all, or have tried a bit, but met troubles, and went back to old methods.

The training transfer matrix by Broad & Newstrom shows the participant’s manager is the one that has the biggest impact on participant’s learning transfer after the training. This finding is different from what has been normally agreed that the trainer or the learner should be held responsible if there is no training transfer.  

To validate the findings from the Training Transfer Matrix are also valid in Saint-Gobain China, the company that I am working in, so that I can have the first hand data and information to promot manager’s involvement before and after the training to improve training application, so that the training’s contribution to final business result will be realized. One key management program has been chosen for this Action Research.

Through introducing to the participant’s manager what an important role that they play, the manager was asked to have the right communication with the participant before and after the training. Considering that they don’t know the training content most probably, they may be discouraged to have the pre- and post- communication with their participant; related tools have been designed to facilitate their work so that the expected results will be achieved.

To evaluate if the manager’s involvement before and after the training have reached the expected results, two rounds of surveys have been conducted respectively in 2009 and 2011. The surveys were sent to the participants and their managers. The survey is to find out if the manager has conducted the pre- and post- training talk, and if the participants have applied their learning and have achieved concrete and valuable results at work. The results of both rounds of surveys show much better results than the Dr. Brinkhoff’s data, which I have used as the benchmark.

This Action Research lasted 4 year, more than 330 participants and 85 managers were involved. The findings have validated that in Saint-Gobain China, the manager’s involvement before and after the training has positive impact on participant’s learning application after training, and hence contributes to the business results and realizes the training final effectiveness to business results.

KEY WORDS: Training Effectiveness, Training Effectiveness Evaluation; Training Application; Training Transfer Matrix

RESEARCH METHODOLOGY

1.1 The research focus and questions

With the literature review about different training effectiveness evaluations, the KirkPatrick’s 4 levels of evalution of training effectiveness has been found to be the most relevant to the author’s organization. Based on this training effectiveness evalution model, the author learnt that chanllenge for training effectiveness is from the transfer from the 2nd level to the 3rd level which is from the knowledge level to the application level. With this new finding, the author conducted further literature review about training transfer, and have found that there are three key stakeholders who affect the training transfer at three different stages of the learning: before, during and after; and the manager is the most impactful stakeholder for training transfer at both the pre-training and post-training stages.

The focus of this study is to validate that the manager does have the highest impact in participant’s training transfer in our company, so that managers of the participants understand this and involve themselves before and after the training with the right engagement actions to drive training at the application level (Level 3), and hence to leade to Level 4 business results and maxize overall training effectiveness.

This study will answer the following research questions:  

1. What can help managers to fulfill their roles in maximizing the training effectiveness, given the fact that they may not know the important role that they can play and they may not know the training content?

2. With the tools, after being communicated of their important roles, have the managers exert their important influences, have their subordinates applied the learning when they go back to work? If this works well in Saint-Gobain Environment?

I would like to use this training transfer matrix to validate if participant’s manager is the most influencial party to impact the training effectiveness in Saint-Gobain China, the company that I am working with, and also validate the tools and solutions for managers to exert their important role in maximizing the training contribution to business.

Action research methodology will be used to verify the findings with one key program from Saint-Gobain China. This program targets all our junior and middle level managers of about 800 of them. The programs started from May 2008.

1.2 Introduction to the company

Saint-Gobain Group is a company started as The Royal Glass Works by the French King Louise XIV in 1665. During the nearly 350 years, the company has become a Fortune 500 company, and the market leader of Sustainable Habitat with four sectors which cover 50 activities with over 1200 legal entities in 64 countries in the world. In 2011, the overall sales are over 42 billion euros with about 190,000 employees worldwide.

Saint-Gobain China belongs to Saint-Gobain Asia-Pacific Region, with over 10,000 employees, about 1000 managers in 57 legal entities, with its regional headoffice in Shanghai. Saint-Gobain has started its business in China since 1980s with over 1 billion euor of sales in 2011. Two of the four sectors of Saint-Gobain have been well-established in China. Asia and China has been defined as the emerging markets of Saint-Gobain of almost 350 years for its new developing engines.

1.3 Research design

1.3.1 Why action research?

1.3.1.1 Introduction to action research

The term action research was first used by Lewin in 1946. Though it has been interpreted by management researchers in different ways, four common themes can be found out within the literature. The first emphasises the purpose of the research: It is not research abut action, but research in actionso that, for example, the research is related to the decision of organisational issues such as the implications of change together with those who experience the issues directly.

The second is concerned with the involvement of practitioners in the research and, especially, a collaborative partnership between practitioners and researchers. The researcher is the employee or part of the organisation where the research and the change process are taking place rather than more typical research or consultancy where, for example, employees are subjects or objects of the study.

The third theme focuses upon the interative nature of the process of diagnosing, planning, taking action and evaluating (Figure below). The action research spiral starts with a specific context and with a clear purpose. This is normally to be expressed as an objective. Diagnosis, sometimes referred to as fact finding and analysis, is undertaken to enable action planning and a decision about the actions to be taken. After all these are taken, then the actions are evaluated (cycle 1). Subsequent cycles involve further diagnosis, taking into account previous evaluations, planning further actions, taking these actions and evaluating.

The forth and the final theme suggests that action research should have implications beyond the immediate project; in other words, it must indicate that the results could inform other contexts.

Thus action research differs from other research strategies because of its specific focus on action, in particular promoting change within the organisation. With this nature, it is particularly useful for answering‘how’questions. In addition, the person undertaking the research is involved in this action for change and subsequently application of the knowledge gained elsewhere. The strengths of an action research strategy are a focus on change, the recognition that time needs to be devoted to diagnosing, planning, taking action and evaluating, and the involvement of employees (practitioners) throughout the process (Mark Saunder et al, Research Methods for Business Students).

Once employees have identified a need for change and have widely shared this need, it becomes difficult to ignore, and the pressure for change comes from within the organisation. An action research strategy therefore combines both data gathering and facilitation of change.

Action research can have two distinct foci. The first of these aims is to fulfill the agenda of those undertaking the research rather than that of the sponsor. This does not, however, preclude the sponsor from also benefiting from the changes brought about by the research process. The second focus starts with the needs of the sponsor and involves those undertaking the research in the sponsor’s issues, rather than the sponsor in their issues.

These consultant activities are termed ‘process consultation’. The consultant, he argues, assists the client to perceive, understand and act upon the process events that occur within their environment in order to improve the situation as the client sees it. (Within this definition the term ‘client’ refers to the persons or person, often senior managers, who sponsor the research.) Using Schein’s analogy of a clinician and clinical enquiry, the consultant (researcher) is involved by the sponsor in the diagnosis (action research), which is driven by the sponsor’s needs. It therefore follows that subsequent interventions are jointly owned by the consultant and the sponsor, who is involved at all stages. The process consultant therefore helps the sponsor to gain the skills of diagnosis and fixing organisational problems so that the latter can develop autonomy in improving the organisation.

In this action research, the auther is both the consultant and the sponsor. First, the author is the sponsor, who has the real actual need to drive the manager’s behavior change from not so much to more involvement with their subordinate’s training before and after the training intervention; in order to drive the training transfer and effectiveness at the third and application level. Second, the auther is the DBA student, who would like to do research for the academic purpose, achieving findings for others to use through action research.

1.3.1.2 Why Action research in this study?

Training effectiveness has been an area and topic for the author as the head of the Training & Development function to explain further to the company management team and other key stakeholders. Driving training effectiveness also cannot be successful without management involvement and support. According to Training Transfer Matrix researched by Broad & Newstrom’s in 1992, managers are the most important parties in driving training effectiveness if they are involved in the training before and after the training intervention. How to engage management to be an active player with lasting management involvement behavior change is a key challenge to the author.

Working in an industrial company with majority management team members from engineering and business background, the author need find a good way to get their buy-in and commitment. Communication with data and results from real examples is important. If we can further engage them in the whole process, and find the effects from their involvement, they would be more willing to accept the results that they have been involved. Action Research is the perfect methodology in this case.

The diagram on the left shows how this research was designed and its different process. The diagram on the right shows how the research process is related to the Action Research Spinal. It started from identifying the problem in practice, which is the Context and Purpose in the Action Research Spinal. The corporate business and Human Development Context and business goals and HR goals need be considered. The purpose why to this Action Research needs to be defined and clarified.

After the Context and Purpose are clarified, there is the need to diagnose the status quo and to learn what theories have already been formed and tested, to avoid reinventing the wheel. Intensive literature review has been conducted to find out exiting good theories and practices that can be applied in this action research. KirkPatrick’s 4 levels of evalution of training effectiveness is most relevant to Saint-Gobain China where the action research is taken. Based on this training effectiveness evalution model, the author learnt that chanllenge for training effectiveness is from the transfer from the 2nd level to the 3rd level which is from the knowledge level to the application level. With this new finding, the author conducted further literature review about training transfer, and have found the Training Transfer Matrix by Broad & Newstrom, that there are three key stakeholders affects training transfer at different stages of the learning: before, during and after; and the manager is the most impactful stakeholder for training transfer at both the pre-training and post-training stages.

The next step is about planning the right interventions and tools to engage and enable managers to become an active player with their full involvement before and after the training. This is what to be focused at this action research. To validate that the manager does have higher impact in training transfer in our company, and to drive training effectiveness through the manager involvement at both the pre-training and post-training stages, the author decided to apply this Action Research study.

After the planning of the right interventions and tools, the Managerial Porgram Level 1 (MP L1) have been chosen as the program for this action research. MP L1 is a key program of three days, targeting all junior to middle level managers. There are about 800 managers of them. If their managers have involved in their training transfer process before and after the training, their application of the learning from MP L1 should be better than the learning application of other training programs that have no manager involvement.

At the Instrument application stage, which is the Taking Action stage according to the Action Research Spinal, a PPT presentation facilitationg why the manager’s roles are important have been prepared to communicate to the managers what an important role that they can plan, and relevant tools to play their roles well, considering that the managers normally neither know about their important role in the training transfer process, nor do they know about the training content, and are discouraged to conduct their pre-training talk and post-trianing follow up.

After the above steps have been conducted for about two years, an evaluation questionnaire was designed and sent to the participants and their managers to see if this action research has reached its targets at the first spinal, taking Dr. Robert O. Brinkerhoff’s research data as a benchmark.

By the time this dissertation was prepared; two rounds of evaluations of the participants’ application status have been conducted respectively in 2009 and 2011. Both have shown big improvements with the participants’ application after their managers’ intervention before and after the training. The result of 2011 is better than that of 2009.

1.4 Instrumentation

From the previous two chapters’ literature review, the author has two key findings:  1. The Kirkpatrick’s 4 levels of effectiveness are still the most commonly accepted training effectiveness evalution model.

According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

Sandy Almeida, a colleague of his son Jim Kirkpatrick, MD, MPH, had a research in 2009 which identified statistical correlation was significant between Level I and Level II, and between Level III and Level IV. Which means if learners have rated highly for Level I, their learning at Level II is normally high as well; if learners have rated highly for Level III, the Level IV result is also high. Sandy found that even providing excellent training does not lead to significant transfer of learning to behavior and subsequent results without a good deal of deliberate and consistent reinforcement. Application is a level that requires more efforts.

To improve the result at Application level, prepious study about Training Transfer has been reviewed, and key finding is the Training Transfer Matrix by Broad and Newstrom. According to this Matrix, there are three key stakeholders influencing the training transfer at three different stages. These three key stakeholders are the manager of the learner, the trainer who delivers the training, and the learner itself. And the three stages are differentiated according to the training delivery time: before the training, during the training and after the training.

many attention and training opportunities from the Group as their managers; on the other hand, they have large amount of subordinates from the workshop floor, hence they are the ones representing the company to the front line employees.

The next question is what competencies are more needed for them to play their present roles well. It is not necessary for them to be good at all 11 competencies. After a mixed-model research study, five of the eleven Managerial Core Competencies have been defined as very important for them to execute their present roles well, and they are: Team Leadership & People Development, Listening & Communication, Result-Orientation, Teamwork and Change Management.  

Based on these findings, the Singapore Institute of Management was engaged to design training programs with these 5 competencies, targeting middle to junior level managers. Finally, two programs have been designed, Managerial Program Level I and Level II, targeting to the respective 1st and 2nd level behavior description of these five competencies. The pilot sessions of these two programs in English have been launched in May, 2008.

We decided to internalize the Level I to Chinese version with internal trainers certified by the Singapore Institue of Management, targeting all managers that have not gone through any systematic management training; and keep the Level II in English with the original designer from Singapore as the trainer, targeting at least middle level managers and above.

During the process to internalize the Level I into Chinese version, the author as the program leader and trainer, has also made some changes with the delivering approaches, considering the adult learning attention will not last long, if we use the existing methodologies like lecturing, group discussion, exercise and video watching for all three days. The author and one of the other key trainers have added more methodologies like games, role play, quiz, movies, and paintings on top of existing methodologies. With these changes, the quality of the training program itself is guaranteed.

Since the launch of the internalized version, the program has received very good feedbacks, and it has been chosen by our different business activities as one of the key programs to train their managers. We have estimated that there are about 200 participants in total at the very beginning, but now the total amount that have received this program has reached 449 by the end of 2011. Following are the records of each session.

          

junior and middle level managers, many efforts have been spent on the training intervention itself. This has contributed to the good evaluation result at the first level, the reaction level.The author understands that if the efforts stop here, the program is no more than any other program from “a typical organization” called by Dr. Brent Peterson at Columbia University in his research in 2004, which invests 85% of its resources in the training event, while the event only contributes 24% to the learning effectiveness of the learners,

Normal situation of corporate training is as shown as below:

The Training Transfer Matrix by Broad and Newstrom has expressed that Transfer partnership requires a balanced distribution of concern for and adequate involvement of participants, their managers or supervisors, and trainers at all stages of the process-before, during and after the training program or course (Broad and Newstrom, 1992). More recent studies by DDI and Achieveglobal have also validated that impactfulness of different parties at different stages is different. Managers have the highest impact among the three parties at the Before and the After stage, while trainer has the biggest impact at the During stage. The Training Transfer Matrix further studied by DDI and Achieveglobal in a corporate training situation as following was used for this study.

With the satisfactory results of the Managerial Program Level 1 at the reaction level, to drive training effectiveness of this widely well received program, the author decided to apply the training transfer matrix, and focusing on the manager’s role before and after the training session. As the During stage has been well managed by the author. The author has defined the training needs, and has not only customized this selected program into a program that used all possible design methodologies, but also facilitates this aprogram as the key trainer. A pre-learning survey is also sent to the learners to gather their present status in relation to the training content. In general, the trainer has used all possible ways to maxize the trainer’s contribution at this during stage. This has been further confirmed by the average 93% of satisfaction rate of the first level evalution results. While the manager’s role at the Before and After stages have not caused any attention yet.

4.4.2.3 Powerpoint Presentation to Facilitate the Manager’s Important Role

For the manager, another document called How Training is Related to Business Results was prepared by the auther. This is for the author to facilitate the important roles the manager is to play for drving training application and transfer.

In this presentation, the author includes the Kirkpatrick four levels of training effectiveness evaluation, the training transfer matrix, the learning retention model, the benchmark data from Dr. Brinkerhoff’s survey result about training application from 100, 000 participants.

4.4.2.4 Pre- and Post- training tools

For the manager, to help them to be better prepared for their role before and after the training, two sets of tools were designed by the auther for them to conduct their pre- training talk and their post- training follow-ups, considering that the manager normally does not know about the details of the training content, and hence are discouraged to work with their subordinates who just finish the training and are familiar with the content.

The manager is not the one to apply the learning. If they can play the coach role, asking the right questions, so that the participants provide the solution proposals and implement them. So helping the managers asking the right questions is the solution. Questions are designed for both pre-training and post-training discussion between the manager and the participants.

1.4.2.4.1 Pre-training tool of Managerial Program Level I

Following are the pre-training tool, starting with a brief introduction of the important role that a manager can play in terms of supporting their subordinates in training transfer, then followed with five questions focusing the following:

1. the reason for sending the participant to the training

2. the learning objective for the participant

3. what are the manager’s expectation to the participant from the training

4. reminder to arrange work in advance so that the participant can focus on the training without disturbance. This is also a reminder to the manager to support the participant by not disturbing the participant during the training

5. ask the participant to share the learning to improve the learning retention.

1.4.2.4.2 Pre-training Talk of Managerial Program Level I

Research shows that the participant’s manager has a strong impact in the participant’s learning transfer. Actions like briefing the importance and expectation to the participants before training, and coaching & following-up after the training will drive the training application which relates training with business results.

Questions (“you” means training participant)

1. Why the plan for you to attend this training program? (The importance)

2. How is it related to your work? (The objectives)

3. What is my expectation to you (what you need focus) from this training?

4. Have you arranged your work in advance or delegated your work to others so that you are not distracted during training?

5. It is highly recommended that you can share your learning to your peers and team members when you come back. When? Where?

1.4.2.4.3 Post-training Talk of Managerial Program Level I

In the following post training Follow-up tool, the auther designed one coaching tool and three sub-tools in appendix 1, 2 & 3.

The first coaching tool is to coach the participant to work out their learning application action plan;

The first sub-tool has provided with the key contents of the five competencies with evalution criteria, so that they can record the participants present status with these five competencies before applying the learning from the training, then when they do the 2nd and 3rd follow-up sessions, they can see if there are any improvements or not and record the results in the form which will be filed by the parti- cipant and the manager.

The second sub-tool is for the participants to record their action plans with planned time to fulfill the plans.

The third sub-tool is for the manager to coach the participant to share their learning with others to improve the learning retention. The others can be peers and participants subordinates.  

1.4.2.4.4 Post-training Follow-up of Managerial Program Level I

Action:

i. Arrange a post-training meeting within one week with participant to understand what the participant has learnt from the training program. Following questions are for your reference for conducting this follow-up meeting.

ii. Evaluate the participant’s current level based on the criteria of competency level I and support the participant to complete the action plan. On-going support to the participant with several follow-up meetings so that the action plans are implemented and the expected results are reached.

iii. Ask participant to prepare to share their learning in their team meetings and call for their team members’ understanding

post-training meeting:

1. What impressed you most from this training session?

2. What can be applied in your work?

3. How are you going to apply what you have learnt in your work?

4. What barriers or issues may prevent your application of these skills on your work?

5. What methods, resources and possible solutions can be used to overcome these barriers?

6. What support do you need from me?

Appendix:

i. Participants Competency Evaluation:

"-" Below expectation

"=" Meet expectation

"+" Above expectation

"++"Highly exceeds expectation

ii. Participants Action Plan

iii. Participants sharing meeting:

1. What have you learned?

2. What can be applied in your work?

When you apply what you have learnt from training, understanding from your team members is important. You need specify what cooperation is needed from team members?

1.5 Instrument Application  

First level of effectiveness was managed by the auther, and evaluated with an evaluation survey which is sent to each participant right after the training session to gather their reaction level results.

Second level of evaluation was designed with a pre-training evaluation to the participants to understand their existing stauts in terms of the key learning points. Then the trainer checks the participants with the key learning points from a previous module at the beginning of a new module.

The above two levels have been mainly managed by the author, for the third level of effectiveness, as the manager is the key party to influence the results, the auther use the following approaches:

1. Communicate the PPT presentation to the China HR Council members and listen to their feedbacks. They are the HR Directors and managers from the key Saint-Gobain branch companies in China. They know about their business and employee situation. They will also be the author’s key partner to promote  and call the manager involvement in their respective companies.

2. Communicate the PPT presentation to the General Managers at China GM meetings, and ask for their support of management involvement through role modeling.

3. Gather the managers of the participants before the training session if possible and present the PPT presentation to them, and explain to them the reasons behind and how to use the pre- and post- training tools.

4. Communicate the PPT presentation to the participants at the end of the program, encouraging them to find their managers and initiate the meeting as well, considering the fact that not all managers have been communicated with the PPT presentation.

5. Use online survey tool to evaluate if the application has been improved or not.  

6. Check if the manager has played this role pre- and post- training through triangulation.

1.5.1 Evaluations about Level 1 & 2 Effectiveness

According to the Kirkpatrick’s 4 levels of evaluation model, training effectiveness starts from good result from level one. Without a good result from Level I, there should not be good results from Level II, III & IV. The author, who is the program manager and trainer of MP L1 program has done the following to gurantee the good result of Level I:

1. Training needs analysis to make for sure the training content is relavent to actual needs from the target audience.

2. Working with the training program designer to make for sure this program is for our specific needs.

3. Pre-training survey to engage participants and pre-evaluate how much they know about the training key points.

4. Customizing the training program with more work related cases and more interactions which are in line with this Learning Rntention Model from the National Training Laboratiories in Bethel, Maine, to drive for learing retention.  

5. During the training session, there are lecturing, reading, audio visual aids, demonstration, group discussion, learning games and practices

6. All participants are requested to fill out the 1st level evaluation sheet as below:

Tab1.8 1st level evaluation sheet

The first MP L1 was on May 12th -14th, 2008, and by end of Jul, 2009, there were 183 participants with 92% of First level satisfaction rate, which is a very high rate that has set a good start for further evaluation results.

The Level II of learning effectiveness has been designed at the training. The knowledge learning of a previous module has been tested at the beginning of the next module. Most participants have good learning of the knowledge. The auther also take this opportunity to find if there are some key learning points that have not been grasped, and will reinforce to make for sure the participants have a good understanding and learning of the content.

In 2009, Sandy Almeida, has further studied about the Kirkpatrick’s four level’s of evaluation, and have found statistical correlations between the four levels. There is good correlation between Levels 1 and 2. A good result from Level I leads to a higher degree of learning.

1.5.2 Evaluation about Level 3 Effectiveness

(As manager’s role is the most important among all three parties in both pre- and post-, so emphasis has been put on enhancing managers’ awareness of these two, and their relevant behavior changes.

Enhancing managers’ awareness: Communication the importance of manager’s involvement pre- and post- to the managers of this program

Tools designed based on the content to facilitate manager’s involvement easier, taking the fact that managers are busy and may not know the training content

a) pre-training talk guide

b) post-training follow-up guide

By July 2009, we have conducted the 1st survey with all participants by that time as investigation, and then with their managers as verification. Both results were very positive, compared to the result from Dr. Brinkhoff with 100,000 survey responses.  

With these results, we have continued promoting managers’ involvement before and after, and by July 2011, there are 183 more participants, and we have surveyed 163 of them, who are still working with Saint-Gobain. Similar results have been found through the second survey.  

1.6 Quality of the research

For the quality of the research, triangulation has been used.

For both evaluations from 2009 and 2011, the first roud of data collection was from the participants themselves. As their answers were very good, and there was doubt that they have been too positive to their after training application and related results, same questionnaires were sent to their managers for both 2009 and 2011 to get the view from their managers, which is normally more objective.  (Use the triangulation you told me that day). To get real inputs from their manages, there has been some efferts to find same managers for both the training time and the surbey time, which means that only managers who are still managers of the same subordinates have been included in the survey.

There have been doubts about if the manangers’ answers were objective, as the low application shows that they have not done their involvement part so well. Further info gathering has been conducted through one-on-one bases. The auther has taken the opportunity of internal meetings with HR and T&D people to ask them this question: Do you think the answers from the survey are valide? And “According to your observation, have the managers in your business units conduct the pre- and post- talk with their subordinates?” And the answers have been quite positive.（作者：王星懿）