Skip to main content

The concurrent validity of the Lund University Checklist for Incipient Exhaustion and the Karolinska Exhaustion Disorder Scale: a replication study

Abstract

Objective

As part of our research on Swedish school principals, we examined the concurrent validity between the Karolinska Exhaustion Disorder Scale (KEDS) and the Lund University Checklist for Incipient Exhaustion (LUCIE) in a cross-sectional study sample (N = 2670). Specifically, we examined: (a) to what extent LUCIE and KEDS identified the same individuals and their level of agreement, and (b) to what extent the present observations among school-principals agreed with previous observations made in a highly educated and healthy study sample drawn from the general population.

Results

Depending on established cut-points on LUCIE, the Kappa agreement (K) between LUCIE and KEDS varied between fair (K = 0.34 [95% Confidence Interval = 0.30–0.38]) and moderate (K = 0.54 [95% Confidence Interval = 0.51–0.58]). While the instruments did not always identify the same individuals, the most reasonable comparison between KEDS and LUCIE was achieved when the cut-off on LUCIE was made between step two and step three. The results essentially replicated our previous results observed in a highly educated and healthy study sample drawn from the general population. The level of agreement suggests that KEDS and LUCIE scores are supplementary rather than interchangeable. Thus, individual result from KEDS and LUCIE are probably best understood in dialogue with the person screened.

Peer Review reports

Introduction

Constructing instruments and making measurements are fundamental activities in empirical sciences. And as verified by the large number of screening instruments that are available for the screening of anxiety and depression symptoms, screening instruments are often developed for identical, or highly similar, purposes, and many times assumed to be interchangeable [1]. However, to interpret tests scores from screening instruments that assess the same, or highly similar, phenomena, it is essential to know how they relate to one another in various groups and/or contexts. A too large correspondence seem to propose that either of the instruments is superflous. Oppositely, a too low correspondence seem to propose that the instruments assess unrelated phenomena, or that one, or both, instruments lack sufficient precision.

In 2005, to contribute to the mitigation of work-related mental health problems (e.g., anxiety, burnout, depression and exhaustion disorder) that have troubled many governments during the last decades [2,3,4], the Swedish National Board of Health and Welfare (NBHW) implemented the diagnosis Exhaustion Disorder (ED: F43.8 A) in the tenth revision of the International Classification of Diseases (ICD-10-SE) [5,6,7]. In brief, to attain the ED diagnosis, the patient needs to experience significant suffering, and/or a reduced functioning at work or social situations, while reporting a symptomatology of bodily and mental complaints during a minimum of two weeks, related to excessive fatigue in response to one or more identifiable stressors of a duration of at least six months. The patient must also be devoid of other medical conditions that might explain fatigue (e.g., depression, diabetes, general anxiety, heart disease, thyroid disease, substance abuse etc.). Observably, the similarities between ED and the concepts of burnout [8], or “clinical burnout” [9], are obvious, and screening questionnaires for ED and burnout tend to exhibit a strong positive association [10, 11].

Following the implementation of the ED diagnosis, the Karolinska Exhaustion Disorder Scale (KEDS) [12] and the Lund University Checklist for Incipient Exhaustion (LUCIE) [10, 13, 14] were developed to assist in the detection, monitoring and/or treatment of ED. And it is not uncommon that these screening inventories are used concurrently in research or clinical practice [10, 15,16,17]. However, differences in their (a) emphasis (manifest signs [KEDS] versus incipient signs [LUCIE]), (b) response formats (a seven step Likert scale with alternating verbal anchors [KEDS] versus a four step Likert scale [LUCIE]), (c) time frames (two weeks [KEDS] versus four weeks [LUCIE]), and (d) scoring procedures, suggest that KEDS and LUCIE might be assumed to give results that are supplementary rather than interchangeable.

We have previously, in a study sample drawn from the general population, benchmarked KEDS and LUCIE with each other and other inventories relating to stress and burnout, [10, 11, 13, 14], showing that KEDS and LUCIE are systematically positively related and that the Kappa agreement is fair to moderate [10]. In the present study, we sought to replicate our previous observations by comparing KEDS and LUCIE in a new study sample entailing Swedish school principals. From previous analyses, we knew that school principals had circa a twice as high prevalence proportion of stress and exhaustion symptoms [17] than our previous population sample [10]. In addition, the school principals had clearly higher prevalence proportions than what has been observed among leaders and employees in a politically governed regional organisation [18] and among occupational therapists [19]. Although comparisons of prevalence proportions across study samples should be made with caution due to varying selection procedures, the ostensibly higher proportion of school principals with exhaustion signs generate a beneficial condition for examining the agreement between KEDS and LUCIE. The research questions were:

  • To what extent do LUCIE and KEDS identify the same individuals and what is the level of agreement between the instruments?

  • To what extent do the present observations among school principals agree with our previous observations of Kappa agreement in a highly educated and healthy population sample?

Methods

Participants and study design

The present study targeted Swedish principals at all school levels. In the absence of an accessible official register of occupationally active school principals, we recruited participants via an e-mail list that had a nationwide reach and covered principals who during the period 2008–2017 had participated in training programmes funded and arranged by the Swedish National Agency for Education and run by different universities in Sweden. In the present study, the first response from principals and assistant principals to an online survey issued in 2018 (n = 2274) and 2019 (n = 1959) [20] were used (n = 2670). Of these, 2219 responded in 2018 and 451 in 2019. The participants also had to work at least 50% of full-time in compulsory schools (46%), pre-schools (27%), upper secondary schools (15%), adult education (7%), or pre- and compulsory schools (5%). Their mean age was 49.4 years (SD = 7.3 years). Most participants were women (78%) and employed by a municipality (77%).

Measures

The LUCIE and KEDS inventories have been published [10, 12].

The Lund University Checklist for Incipient Exhaustion (LUCIE) entails 28 items covering six domains that build two supplementary scales: the Stress Warning Scale (SWS) and Exhaustion Warning Scale (EWS) [10]. Both scales ranges from 0 to 100 points. Cronbach’s alpha for the 28 LUCIE items was 0.94. And the median spearman rank order correlation between the SWS score and the individual LUCIE items was rho = 0.58 (Min = 0.44; Max = 0.70). For the EWS score, the median correlation with individual LUCIE items was rho = 0.47 (Min = 0.33; Max = 0.58). Following the manual, the SWS and EWS scores were categorized and combined. Specifically, SWS entails the three levels: Green (≤ 17.00), Yellow (17.01 to 38.50), and Red ≥ 38.51) whereas EWS entails two levels: Green (≤ 21.49) and Red (≥ 21.50). Using the pre-defined cut-off scores on each scale, the SWS and EWS are in practice combined into a four-step ladder of incremental stress symptomatology in which the combinations (a) SWS Green and EWS Red, and (b) SWS Yellow and EWS Red, are not possible to obtain or very rare, respectively. The four steps are:

Step 1-GG (SWS green zone and EWS green zone), indicating no signs of stress;

Step 2-YG (SWS yellow zone and EWS green zone), indicating possible, or weak signs, of stress;

Step 3-RG (SWS red zone and EWS green zone), indicating mild to moderate lasting stress symptoms, but less severe than ED;

Step 4-RR (SWS red zone and EWS red zone), indicating severe stress signs and possible ED.

The Karolinska Exhaustion Disorder Scale (KEDS) comprises nine items that refer to the past two weeks, covering the following nine domains: (a) ability to concentrate, (b) memory, (c) physical stamina, (d) mental stamina, (e) recovery, (f) sleep, (g) hypersensitivity to sensory impressions, (h) experience of demands, and (i) irritation and anger [12]. Each item is responded to on a 7-point scale (0–6). Descriptive verbal phrases, tailored for each separate item on a scale of increasing severity describe the corresponding domain-specific symptomatology, serving as anchors for the scale steps 0, 2, 4 and 6. Higher values reflect more severe symptoms, and the sum score (0 to 54) is used as an outcome, with a score ≥ 19 indicating possible ED [12, 21]. Cronbach’s alpha was 0.88. And the median spearman rank order correlation between the continuous total KEDS sum score and individual KEDS items was rho = 0.70 (Min = 0.67; Max = 0.80).

Data management and statistical analysis

Using the IBM SPSS software version 29, we applied non-parametric and parametric testing (two-tailed alpha level ≤ 0.05). Spearman rank correlations and Kappa statistics (both with bootstrap estimated 95% confidence intervals [95% CI] to compensate for ties) were used to estimate the degree of association and agreement between LUCIE and KEDS. For purpose of analysis, and to create symmetry between LUCIE and KEDS categories, we examined the changes in the Kappa statistics following the use of various cut-off scores for LUCIE. Sensitivity analyses were conducted by excluding individuals, by analysing data separately for men and women, and for subgroups defined by the year of the first response (i.e., 2018 or 2019). Landis and Koch criteria for judgment of the Kappa statistics were used: K 0–20 = poor; K ≤ 21–40 = fair; K ≤ 41–60 = moderate; K ≤ 61–80 = substantial; K > 80 = Good [22].

Results

Table 1 shows the proportion of individuals within a specific LUCIE category that reports above or below the exhaustion indication level in KEDS (Table 1). Descriptive data, and measures of the associations between the continuous LUCIE SWS and EWS scale scores and the KEDS sum scores, are presented in Additional file 1 (Table S1 and S2 and Figures S1 to S3).

Table 1 Descriptive cross-tabulation. The table shows the proportion of subjects within each Lund University Checklist of Incipient Exhaustion (LUCIE) category (Column %) that had either a normal score, or an exhaustion indication, on the Karolinska Exhaustion Disorder Scale (KEDS).

Table 2 shows the degree of Kappa agreement between LUCIE and KEDS. The lowest agreement was observed when making a cut-off between LUCIE step 3-RG and step 4-RR (K = 0.34). The highest agreement was observed when making a cut-off between LUCIE step 2-YG and step 3-RG (K = 0.54). At this cut-off point, the prevalence proportions for LUCIE and KEDS were 25.4% and 29.5%, respectively. Subsequent sensitivity analyses, in which individuals on step 2-YG and 3-RG were excluded, showed as expected a higher degree of agreement (i.e., K´s between 0.70 and 0.77) and revealed similar patterns of Kappa agreement for men and women and for subgroups of participants defined by their year of first response (see Additional file 2, Table S3 and S4).

Table 2 Estimated Kappa agreement between the Lund University Checklist of Incipient Exhaustion (LUCIE) and the Karolinska Exhaustion Disorder Scale (KEDS) at different LUCIE cutoff points and by excluding identifications in the middle range of LUCIE (sensitivity analysis)

Table 3 presents a comparison of the continuous mean and median KEDS sum scores across the four LUCIE steps of incremental signs of exhaustion. Descriptive sensitivity analyses for subgroups of men and women indicated that the pattern of mean and median KEDS sum scores were similar (see Additional file 3, Table S5).

Table 3 Descriptive Karolinska Exhaustion Disorder Scale (KEDS) scores (M), standard deviations (SD), median scores (Mdn), and accompanying 95% Confidence intervals (CI) across the four Lund University Checklist of Incipient Exhaustion (LUCIE) categories

Discussion

In essence, the pattern of Kappa estimates between LUCIE and KEDS in the present sample of Swedish principals and assistant principals can be said to replicate our previous observations in a study sample entailing highly educated and healthy individuals drawn from the general population [10]. However, and even if the Kappa estimates were consistently higher in the present study sample (see Table 2), the Kappa estimates had overlapping confidence intervals. Overall, the pattern of results from our present study and previous studies suggests that the concurrent validity of LUCIE and KEDS is a stable phenomenon across study samples, contexts, and gender.

That the Kappa agreement between LUCIE and KEDS at best amounts to “moderate” according to the Landis and Koch criteria [22] is not very surprising in view of the aforementioned differences in emphasis, item formats, covered time frames and that LUCIE makes a more fine-grained classification of early stress and exhaustion signs across four categories whereas KEDS uses two categories (i.e., indication/no-indication). Thus, to estimate the Kappa agreement, we dichotomized the LUCIE categories by applying various cut-off points. In so doing, we observed that the best agreement was achieved using a cut off between LUCIE step 2-YG and step 3-RG. Furthermore, as shown when computing means and median KEDS sum scores for each of the LUCIE categories, there is a clear escalating pattern. And that both the mean and median KEDS score falls just above the cut-off score for an exhaustion indication (≥ 19.0) at LUCIE step 3-RG, suggest that the most reasonable comparison between KEDS and LUCIE is achieved when merging step 3-RG and Step 4-RR in LUCIE. This observation is also in alignment with the general recommendation to use step 3-RG as the first significant indicator of stress symptoms of a clinical magnitude [15].

Noticeably, if excluding step 2-YG and/or step 3-RG the agreement between LUCIE and becomes substantial according to Landis and Kochs criteria [22]. Thus, apart from the reduction in categories it is also clear that the higher prevalence proportion of stress and exhaustion symptoms in the present study sample will influence the Kappa estimates. The higher prevalence proportion of exhaustion symptoms in the present study sample [17, 20] when compared to our previous population sample [10] is also a likely contributor to the consistently higher Kappa agreement in the present sample even if the 95% CI consistently overlap.

Conclusions

The present results essentially replicate our previous observations among highly educated and healthy individuals drawn from the general population. The fair to moderate Kappa agreement between KEDS and LUCIE suggests that these instruments might be supplementary rather than interchangeable. Thus, while the participants’ concurrent ratings in LUCIE and KEDS are systematically, logically, and consistently associated with each other, the instruments do not necessarily identify the same individuals as being stressed and/or exhausted. This underlines that on the individual level, the results from these screening instruments are best regarded as indicators of possible illness that needs to be understood in a broader context and in a confident dialogue with the person screened.

Limitations

All participants had a manager position, long education, and were in active full-time work to an extent of at least 50% of full-time. In the absence of a gold standard for assessing exhaustion disorder symptoms, it should be noted that our analyses only confirm the concurrent agreement between the two instruments. And because KEDS favours a dichotomized approach to identify possible exhaustion cases, comparisons with LUCIE, that favours a partitioning into four classes, become more difficult and less intuitively straightforward. In addition, the prevalence proportions of stress and exhaustion warnings in both LUCIE and KEDS were higher in this occupational sample than has been previously observed in the general population and in other occupational samples. Generalizations to other occupations and/or segments of the population (e.g., people with diagnosed ED) should be made with caution.

Data availability

Consistent with the study protocol approved by the Regional Ethical Review Board, anonymized data is stored locally at the Division of Occupational and Environmental Medicine, Lund University, Lund, Sweden. Because the participants (in accordance with the approved study protocol) were guaranteed that the crude data should not be published on the internet, access to data will only be granted to eligible researchers wanting to audit our research.

Abbreviations

CI:

Confidence interval

KEDS:

Karolinska Exhaustion Disorder Scale

LUCIE:

Lund University Checklist for Incipient Exhaustion

NBHW:

Swedish National Board of Health and Welfare

ED:

Exhaustion Disorder

EWS:

Exhaustion Warning Scale

Step 1-GG:

SWS Green zone and EWS Green zone

Step 2-YG:

SWS Yellow zone and EWS Green zone

Step 3-RG:

SWS Red zone and EWS Green zone

Step 4-RR:

SWS Red zone and EWS Red zone

SWS:

Stress Warning Scale

References

  1. Fried EI. The 52 symptoms of major depression: lack of content overlap among seven common depression scales. J Affect Disorders. 2017;208:191–7.

    Article  PubMed  Google Scholar 

  2. European Commission. Guidance on work-related stress - spice of life or kiss of death? Luxembourg: Office for Official publications of the European communities: Directorate-General Employment and Social affairs. Unit D. 2000;6:120.

    Google Scholar 

  3. Joyce S, Modini M, Christensen H, Mykletun A, Bryant R, Mitchell PB et al. Workplace interventions for common mental disorders: a systematic meta-review. Psychol Med. 2015:1–15.

  4. Harvey SB, Modini M, Joyce S, Milligan-Saville JS, Tan L, Mykletun A, et al. Can work make you mentally ill? A systematic meta-review of work-related risk factors for common mental health problems. Occup Environ Med. 2017;74(4):301–10.

    Article  PubMed  Google Scholar 

  5. WHO. The ICD-10 classification of Mental and behavioral disorders. Diagnostic criteria for research. Geneva, Switzerland: World Health Organization; 1993.

    Google Scholar 

  6. NBHW. Försäkringsmedicinskt beslutsstöd: Utmattningssyndrom. [Insurance medical decision support: exhaustion disorder]. Stockholm: The National Board of Health and Welfare (Socialstyrelsen); 2017.

    Google Scholar 

  7. NBHW, Utmattningssyndrom. Stressrelaterad psykisk ohälsa. [Exhaustion disorder. Stress related poor health]. Stockholm, Sweden: Bjurner & Bruno; 2003.

    Google Scholar 

  8. Maslach C, Jackson J, Leiter M. Maslach Burnout Inventory Manual. Palo Alto: Consulting Psychology Press; 1996.

    Google Scholar 

  9. van Dam A. A clinical perspective on burnout: diagnosis, classification, and treatment of clinical burnout. Eur J Work Organ Psy. 2021.

  10. Persson R, Osterberg K, Viborg N, Jonsson P, Tenenbaum A. The Lund University Checklist for Incipient Exhaustion-a cross-sectional comparison of a new instrument with similar contemporary tools. BMC Public Health. 2016;16(1):350.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Persson R, Osterberg K, Viborg N, Jonsson P, Tenenbaum A. Two Swedish screening instruments for exhaustion disorder: cross-sectional associations with burnout, work stress, private life stress, and personality traits. Scand J Public Health. 2017;45(4):381–8.

    Article  PubMed  Google Scholar 

  12. Beser A, Sorjonen K, Wahlberg K, Peterson U, Nygren A, Asberg M. Construction and evaluation of a self rating scale for stress-induced exhaustion disorder, the Karolinska exhaustion disorder scale. Scand J Psychol. 2014;55(1):72–82.

    Article  PubMed  Google Scholar 

  13. Österberg K, Persson R, Viborg N, Jonsson P, Tenenbaum A. The Lund University Checklist for incipient exhaustion: a prospective validation of the onset of sustained stress and exhaustion warnings. BMC Public Health. 2016;16(1):1025.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Persson R, Österberg K. Repeated assessment of work-related exhaustion: the temporal stability of ratings in the Lund University Checklist for incipient exhaustion. BMC Research Notes; 2020.

  15. Österberg K. LUQSUS-K. Manualpaket v 3.1. [LUQSUS-K. Manual - version 3.1]. Fhvmetodik.se; 2018.

  16. Tenenbaum A, Österberg K. Klinisk Handbok för LUQSUS-K. En praktisk guide för användning av LUQSUS-K i samband med utredning, sjukskrivning och rehabilitering av misstänkt stressrelaterad utmattning. [Clinical manual for LUQSUS-K. A practical guide for using the LUQSUS-K in connection with investigation, sick leave and rehabilitation of suspected stress-related exhaustion]. Fhvmetodik.se; 2019.

  17. Persson R, Leo U, Arvidsson I, Hakansson C, Nilsson K, Osterberg K. Prevalence of exhaustion symptoms and associations with school level, length of work experience and gender: a nationwide cross-sectional study of Swedish principals. BMC Public Health. 2021;21(1):331.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Björk L, Nilsson M, Kokko A, Nettervik Z, Paulsson S. Tillsammans för att främja hälsa och förebygga sjukfrånvaro. Slutrapport från Mobiliseringsinitiativet, Försäkringskassan Och Västra Götalandsregionen. [Together for promoting health and prevent sickness absence. Final report from Mobiliseringsinitiativet, Försäkringskassan Och Västra Götalandsregionen]. Försäkringskassan och Västra Götalandsregionen; 2017.

  19. Lexen A, Kahlin I, Erlandsson LK, Hakansson C. Occupational Health among Swedish occupational therapists: a cross-sectional study. Int J Environ Res Public Health. 2020;17(10).

  20. Persson R, Leo U, Arvidsson I, Nilsson K, Österberg K, Oudin A, et al. Svenska skolledares arbetsmiljö och hälsa: En lägesbeskrivning med förslag på vägar till förbättringar av arbetsmiljön. [Swedish school leaders’ work environment and health: a description with suggestions for ways to improve the work environment]. Lund: Arbets- och Miljömedicin Syd; 2021. Report No.: 4.

    Google Scholar 

  21. Kristiansen J, Friborg MK, Eller N, Brandt LPA, Glasscock DJ, Pihl-Thingvad J et al. Comparison of exhaustion symptoms in patients with stress-related and other psychiatric and somatic diagnoses. BMC Psychiatry. 2019;19.

  22. Landis JR, Koch GG. Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159–74.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

The authors wish to acknowledge the participants for their time and effort in completing the questionnaires.

Funding

The study was financially supported by AFA Insurances (dnr 170094 and 220072). The funder had no other role in the project.

Open access funding provided by Lund University.

Author information

Authors and Affiliations

Authors

Contributions

RP and KÖ designed the original study and were responsible for the data collection. The idea for this study was conceived by RP, which together with KÖ och JB drafted the manuscript and performed the statistical analyses. All authors (KÖ, RP, JB) made substantial intellectual contributions to the content and revised the manuscript accordingly. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Roger Persson.

Ethics declarations

Ethics approval and consent to participate

The research has been carried out in accordance with the “World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects” and national legislation in Sweden. The Regional Ethical Review Board in Lund, Sweden, approved the study protocol (reg. no. 2018/247). In accordance with the study protocol, the participants gave written informed consent when they entered the study by completing a short online survey prior to the main survey.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Supplementary Material 3

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Persson, R., Österberg, K. & Björk, J. The concurrent validity of the Lund University Checklist for Incipient Exhaustion and the Karolinska Exhaustion Disorder Scale: a replication study. BMC Res Notes 16, 325 (2023). https://doi.org/10.1186/s13104-023-06589-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13104-023-06589-4

Keywords