Skip to main content

Advertisement

You are viewing the new article page. Let us know what you think. Return to old version

Research note | Open | Published:

Using Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff

Abstract

Objectives

This study is designed to evaluate the effect of a workshop about new teaching and learning methods on the response, knowledge, and behavior of healthcare staff working a large city healthcare center.

Results

Kirkpatrick’s program evaluation model showed that the workshop on new teaching and learning methods significantly improved the healthcare staff’s satisfaction about the teaching environment of workshops, their knowledge about new teaching and learning methods and their behavior in performing workshops for teaching people. It is recommended that this teaching and learning methods workshop should be considered in educational programs for healthcare staff.

Trial registration Trial registration number: IRCT20180619040150N1 approved by Iranian Registry of Clinical Trials at 2018-07-27

Introduction

Training is a useful investment and is one of the most important factors in human resource development. When done well, it can improve employee satisfaction, intended outcomes, and economical efficiency [1, 2]. Many factors can affect whether training programs achieve their desired outcomes [3, 4]. One of the most critical parts of implementing training programs is accurately assessing their impact; to assess programs requires using a suitable method [5]. Here, in addition to measuring learners’ satisfaction with self-assessment [6], meaningful assessment must measure input (learners and teachers), training process (educational programs, assessment methods, and facilities), and output (behavior of participants) [7]. One of the methods used to assess educational programs is Kirkpatrick’s model. The characteristics of this model include the simplicity of the process, measurement of a limited number of variables, ease of evaluation criteria, and lack of need to collect the basic data and learners’ previous performance, and independence of individual and environmental variables. This type of assessment is an appropriate model for evaluating educational programs [8]. Although all models have some deficiencies, according to the evaluation, this model has a suitable and acceptable performance for assessing the educational programs [9].

Kirkpatrick’s model assesses the effectiveness of training programs at four levels: (1) response of the trainee to the training experience (including training experience); (2) the learner’s learning outcomes and increases in knowledge, skill, and attitude towards the attendance experience (how much attendees learned the content after training). This level usually measured through using a pretest and posttest; (3) the students’ change in behavior and improvement (whether the learning transferred into practice in the workplace); and (4) results (the ultimate impact of training) [8, 10,11,12].

Since healthcare staff holds various courses for the people across all sections and levels, it is necessary for these staff to become familiar with new teaching and learning methods. Thus, this study used Kirkpatrick’s program evaluation model to evaluate the effect of workshops about new teaching and learning methods on the healthcare staff’s satisfaction, learning and behavior.

Main text

Method

The present study involved a pretest, educational intervention, and posttest.

$${\text{O}}_{ 1} \quad {\text{X}}\quad {\text{O}}_{ 2}$$

O1,2 is the Observation of the dependent variable (pretest and posttest), X is the Exposure to the educational intervention, the independent variable (workshop).

Participants

This study was a quasi-experimental study conducted on the healthcare staff working in the Shiraz healthcare center.

Educational intervention

A workshop entitled “New Teaching and Learning Methods” was held over 2 days (10 h total) for the healthcare staff. In this workshop, participants learned to teach and learn in small groups, role-playing, brainstorming, question and answer, interactive lecture, and team-based learning. The teachers of these workshops were medical education experts working at the Education Development Center of Shiraz University of Medical Sciences. The content of workshops was about how to deliver an interactive lecture, small group teaching and learning, classroom management, time management, group dynamic and etc.

Assessing workshop and data gathering tools

We evaluated the educational effect of this workshop according to the first three levels of Kirkpatrick’s model using two questionnaires and one checklist.

The first step of the Kirkpatrick program evaluation model is related to the participants’ reactions [8]. Thus, in this step, participants’ attitudes were evaluated regarding the effect of training on their educational ability via a researcher-administered questionnaire. Accordingly, the first level participants’ reaction toward the training course measured with a researcher-made questionnaire. The questionnaire had three categories including: instructor assessment, course content assessment and course support assessment that were scored based on a 5-point Likert scale from excellent to very weak. At the end of the questionnaire there were two questions about overall quality of workshops and overall satisfaction of participants. The validity of the questionnaire was determined by medical education experts and the reliability was confirmed after a pilot study (r = 0.83). A sample of questionnaire is in Additional file 1: Appendix S1.

In the second level of the Kirkpatrick program evaluation model, the effect of the program was evaluated on participants’ learning using pre- and post-test [11]. Accordingly, this questionnaire used before and after workshop which had ten questions including themes as educational goals, entrance exam, teaching time, and trainees’ preparation, the use of new teaching and learning methods, use of learning tools, teaching environment, the number of participants, teacher assessment, and teacher characteristics. The score of each item in the questionnaire was 1, and the total score of this questionnaire was 10. The validity of the questionnaire determined by medical education experts and the reliability determined after a pilot study (r = 0.81).

In the third level Kirkpatrick program evaluation model, we assessed whether the learning was transformed into practice in the workplace [8, 10, 11] by using an observational checklist. This checklist contains some parts about using new teaching and learning methods in the educational sections held by the healthcare staff. Before intervention and 2 months after intervention, two expert observers attended at all of the healthcare staff educational sections and assessed the sections based on this checklist. Every healthcare worker was conscious about the observers and for lowering the effect of the under-observation stress, the healthcare staff participated behavior in their own educational sessions assessed after participating in the workshop of new teaching methods and compared it with the results of one their educational sections before the educational intervention. Besides for lowering the effect of under observation stress, the observers attended at five educational sessions before and at five educational sessions after intervention and completed the checklist after at least half of the sessions to be sure about lowering the stress of healthcare workers.

Statistical analysis

The data were analyzed through SPSS 21, using paired t-test.

Results

This study was an interventional study conducted on 48 staff working in the Shiraz healthcare center. Among them, 12 (25%) were male, and 36 (75%) were female. The mean age of the participants was 40 ± 12 years and their mean work experience was 18 ± 8 years.

The results of the first level Kirkpatrick evaluation indicated that 30 subjects (62.5%) who participated in the new teaching and learning methods workshop declared that the quality of this workshop was excellent. Thirty-five participants (72.9%) were completely satisfied with the way of conducting workshops. In all other domains of the questionnaire more than 50% of participants declared that the quality was excellent.

The results of the second level Kirkpatrick’s evaluation about the participants learning revealed that there was a significant difference between the total scores, preparation of trainees, and use of new teaching and learning methods before and after the intervention (p < 0.05), but there were not any significant differences between other factors (Table 1).

Table 1 The scores of experts participating in the workshop before and after the intervention (second level)

To measure the third level of Kirkpatrick’s program evaluation method, the researchers observed the participants’ behavior during the workshops performed by them 2 months after their training. According to the results, the behavior change across all of the dimensions was significant (p ≤ 0.05) except in using pretests, defining educational goals, time management, defining key points of lecturing, using role play method, and speech structure (Table 2).

Table 2 Evaluating workshop held by participants before and after intervention according to the items related to third level Kirkpatric’s evaluations

Discussion

Each program evaluation model has strengths and weaknesses to measure training activities, but research has shown that the Kirkpatrick’s program evaluation model is more appropriate than other models [13, 14]. Accordingly, we used the Kirkpatrick’s model to evaluate the healthcare staff’s overall reaction to the workshop for new teaching and learning methods and its effects on their learning and behavior.

The results of the workshop evaluation in the reaction part (Level 1) showed that the participants were satisfied. A study conducted by Rabiee et al. on holding training courses for the staff working in Arak University of Medical Sciences suggested that half of the staff believed that the workshop was perfect while the other half thought that it was moderate or weak [15]. Other research conducted by Opoghi et al. to evaluate the librarians’ reaction toward a short-term training course using Kirkpatrick’s model revealed that there was a direct and positive relationship between the physical conditions of class and quality of learning [16].

Evaluation of the second level of Kirkpatrick’s model demonstrated a significant difference between the participants’ learning scores before and after the intervention. The overall results of this study suggested an increase in learning and satisfaction and changes in behavior scores. Dorri et al. also examined the effect of in-service training on cardiopulmonary resuscitation using Kirkpatrick’s model. They found it effective in increasing the participants’ learning and knowledge [17]. Poujahromi et al. conducted a study using Kirkpatrick’s model to evaluate a course on working with DC shock held for nurses working in Bushehr Hospital. The results were in line with this study. Another study performed by Anderson et al. in 2000 revealed educational success and organizational effectiveness in the use of this training method. They found a significant difference between educational assessment and educational achievement [18].

Mbagwu’s research in 2010, which was conducted on employees at the Federal University of Technology Library, showed that these training courses increased their capabilities, skills, and specific knowledge [19]. He emphasized that the training courses must be presented using a logical and appropriate method. Dorri’s research study dealt with the effectiveness of training courses using Kirkpatrick’s model. He observed that any changes in each of the four dimensions (reaction, learning, behavior, and results) were influential [17]. Abedini conducted a study to evaluate the effectiveness of in-service training courses from the perspective of employees working in Tehran Maskan Bank. He concluded that the participants assessed the behavioral changes at a reasonable level [20].

A brief review and overview of the process of investing in educational activities all over the world in recent years suggested that the amount of resources dedicated to this critical issue is growing compared to previous years. Nevertheless, some concerns still observed on the part of the officials and decision-makers about the lack of effectiveness of educational activities. The results of Farjad et al. indicated that although training courses can partially enhance skills and increase the scores of Kirkpatrick’s levels, a lack of attention to new and effective teaching and learning methods reduces their usefulness [21].

Integration of the health system and medical education in Iran prepare a unique environment that allows university educational experts to teach healthcare staff [22]. Good program evaluation models are needed to measure the effectiveness of these important educational activities.

Performing this kind of workshop familiarizes the healthcare staff with the organization’s new teaching and learning methods. Workshops can also be effective in motivating the staff. According to the results, holding workshops on new teaching and learning methods significantly improved the healthcare staff’s satisfaction and their function after the intervention.

Aside from curriculum and training delivery, results showed that the training about new teaching and learning methods is an essential factor in delivering effective teaching by healthcare staff. This staff is responsible for educating people in urban and suburban areas all over the country, so improving their teaching methods to deliver educational content effectively will lead to an increase in population health.

Also, health care staff help to promote knowledge, attitudes, and behaviors within the community on common diseases. The results of our study indicate that healthcare staff training can be used as an efficient and low-cost method for providing education to all of the community.

Limitations

The limitations of the present study were the short duration of workshops and the small number of participants. Another limitation was that measuring the fourth level of the Kirkpatrick model was not possible.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. 1.

    AlYahya MS, Norsiah B. Evaluation of effectiveness of training and development: the Kirkpatrick model. Asian J Bus Manag Sci. 2013;2(11):14–24.

  2. 2.

    Delavari S, Rezaee R, Hatam N, Delavari S. Technical efficiency of Shiraz school of medicine in research and education domains: a data envelopment analysis. J Adv Med Educ Prof. 2016;4(1):13.

  3. 3.

    Ghaffari R, Ranjbarzadeh FS, Azar EF, Hassanzadeh S, Safaei N, Golanbar P, et al. The analysis of learning styles and their relationship to academic achievement in medical students of basic sciences program. Res Dev Med Educ. 2013;2(2):73–6.

  4. 4.

    Rezaee R, Moadeb N. Team-based learning: a new approach toward improving education. Acta Med Iranica. 2016;54(10):678–82.

  5. 5.

    Kojuri J, Amini M, Karimian Z, Dehghani MR, Saber M, Bazrafcan L, et al. Needs assessment and evaluation of a short course to improve faculties teaching skills at a former World Health Organization regional teacher training center. J Adv Med Educ Prof. 2015;3(1):1.

  6. 6.

    Dennick R. Long-term retention of teaching skills after attending the Teaching Improvement Project: a longitudinal, self-evaluation study. Med Teach. 2003;25(3):314–8.

  7. 7.

    Wu Y-C, Hsieh L-F, Lu J-J. What’s the relationship between learning satisfaction and continuing learning intention? Procedia Soc Behav Sci. 2015;191:2849–54.

  8. 8.

    Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Eval Program Plan. 2004;27(3):341–7.

  9. 9.

    Hasani H, Bahrami M, Malekpour A, Dehghani M, Allahyary E, Amini M, et al. Evaluation of teaching methods in mass CPCR training in different groups of the society, an observational study. Medicine. 2015;94(21):e859.

  10. 10.

    Patel SR, Margolies PJ, Covell NH, Lipscomb C, Dixon LB. Using instructional design, Analyze, Design, Develop, Implement, and Evaluate (ADDIE), to develop e-Learning modules to disseminate Supported Employment for community behavioral health treatment programs in New York State. Front Public Health. 2018;6:113.

  11. 11.

    La Duke P. How to evaluate training: using the kirkpatrick model. Prof Saf. 2017;62(8):20.

  12. 12.

    Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34(5):e288–99.

  13. 13.

    Rouse DN. Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health information management courses and programs. Perspect Health Inf Manag. 2011;8(Spring):1c.

  14. 14.

    Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: a useful tool for evaluating training outcomes. J Intellect Dev Disabil. 2009;34(3):266–74.

  15. 15.

    Rabiee F, Moayedi S, Naderi Z, Aliabadi Farahani K, Shamsi M. Effect of in-service educational courses on human resources efficiency from university experts’ point of view. Educ Strategies Med Sci. 2011;4(2):85–9.

  16. 16.

    Ofoghi N, Sadeghi A, Babaei M. impact of class atmosphere on the quality of learning (QoL). Psychology. 2016;7(13):1645.

  17. 17.

    Dorri S, Akbari M, Sedeh MD. Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation. Iranian J Nurs Midwifery Res. 2016;21(5):493.

  18. 18.

    Pourjahromi N, Nezamian Z, Ghafarian Shirazi H, Ghaedi H, Momeninejad M, Mohamadi Baghmolaee M, et al. The effectiveness of training courses on “How to work with DC Shock device” for nurses, based on Kirkpatrick Model. Iranian J Med Educ. 2012;11(8):896–902.

  19. 19.

    Mbagwu F, Nwachukwu C. Staff training and development, enhancement of job performance: case of federal university of technology library, Owerri. Inf Technol (The). 2010. https://doi.org/10.4314/ict.v7i2.65644.

  20. 20.

    Abedini B, Kohandeli NP. a study on the effectiveness of employee training on the profitability of Maskan Banks (case study: Maskan Bank of Hormozgan Province). Int J Humanit Cult Stud (IJHCS) ISSN 2356–5926. 2016;1(1):1820–37.

  21. 21.

    Farjad S. The evaluation effectiveness of training courses in university by kirkpatrick model (case study: Islamshahr University). Procedia Soc Behav Sci. 2012;46:2837–41.

  22. 22.

    Salehi A, Harris N, Lotfi F, Hashemi N, Kojouri J, Amini M. Reform in medical and health sciences educational system: a Delphi study of faculty member’s views at Shiraz University of Medical Sciences. East Mediterr Health J. 2014;20(3):151–61.

Download references

Acknowledgements

This study is a part of the thesis that was done by the second author: Fatemeh Taghva for obtaining a master degree in medical education with IRC registration reference: IRCT20180619040150N1. This thesis was approved by the deputy research of Shiraz University of Medical Sciences. The author thanks all healthcare staff for participation in the study. We also thank Jennifer Wilson at Jefferson (Philadelphia University + Thomas Jefferson University) for editorial assistance.

Funding

All of the funds of the present study (including the funds for performing workshops and assessment cost were provided by the vice chancellor of research at Shiraz University of Medical Sciences. The vice chancellor provided all the necessary funds for designing the workshops.

Author information

All authors contributed to the commencement of the study, coordination, participated in the collection of data, and drafted the manuscript. MRH and FT participated in data collection and analysis. MA and MRH participated in the supervision of the study and analysis of data. MA, SD and MRH participate in drafting of the manuscript. All authors read and approved the final manuscript.

Correspondence to Mitra Amini.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Ethics committee of Shiraz University of Medical Sciences by ethical code number IR.SUMS.REC.1394.5574. Informed written consent to participate was obtained from all participants. Participants participated in the study voluntarily and their names were not mentioned in the scripts.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1. 1st level Kirkpatrick’s questionnaire.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Kirkpatrick’s model
  • Teaching methods
  • Workshop