The concurrent validity of the Lund University Checklist for Incipient Exhaustion and the Karolinska Exhaustion Disorder Scale: a replication study

Abstract

Objective

As part of our research on Swedish school principals, we examined the concurrent validity between the Karolinska Exhaustion Disorder Scale (KEDS) and the Lund University Checklist for Incipient Exhaustion (LUCIE) in a cross-sectional study sample (N = 2670). Specifically, we examined: (a) to what extent LUCIE and KEDS identified the same individuals and their level of agreement, and (b) to what extent the present observations among school-principals agreed with previous observations made in a highly educated and healthy study sample drawn from the general population.

Results

Depending on established cut-points on LUCIE, the Kappa agreement (K) between LUCIE and KEDS varied between fair (K = 0.34 [95% Confidence Interval = 0.30–0.38]) and moderate (K = 0.54 [95% Confidence Interval = 0.51–0.58]). While the instruments did not always identify the same individuals, the most reasonable comparison between KEDS and LUCIE was achieved when the cut-off on LUCIE was made between step two and step three. The results essentially replicated our previous results observed in a highly educated and healthy study sample drawn from the general population. The level of agreement suggests that KEDS and LUCIE scores are supplementary rather than interchangeable. Thus, individual result from KEDS and LUCIE are probably best understood in dialogue with the person screened.

Introduction

Constructing instruments and making measurements are fundamental activities in empirical sciences. And as verified by the large number of screening instruments that are available for the screening of anxiety and depression symptoms, screening instruments are often developed for identical, or highly similar, purposes, and many times assumed to be interchangeable [1]. However, to interpret tests scores from screening instruments that assess the same, or highly similar, phenomena, it is essential to know how they relate to one another in various groups and/or contexts. A too large correspondence seem to propose that either of the instruments is superflous. Oppositely, a too low correspondence seem to propose that the instruments assess unrelated phenomena, or that one, or both, instruments lack sufficient precision.

In 2005, to contribute to the mitigation of work-related mental health problems (e.g., anxiety, burnout, depression and exhaustion disorder) that have troubled many governments during the last decades [2,3,4], the Swedish National Board of Health and Welfare (NBHW) implemented the diagnosis Exhaustion Disorder (ED: F43.8 A) in the tenth revision of the International Classification of Diseases (ICD-10-SE) [5,6,7]. In brief, to attain the ED diagnosis, the patient needs to experience significant suffering, and/or a reduced functioning at work or social situations, while reporting a symptomatology of bodily and mental complaints during a minimum of two weeks, related to excessive fatigue in response to one or more identifiable stressors of a duration of at least six months. The patient must also be devoid of other medical conditions that might explain fatigue (e.g., depression, diabetes, general anxiety, heart disease, thyroid disease, substance abuse etc.). Observably, the similarities between ED and the concepts of burnout [8], or “clinical burnout” [9], are obvious, and screening questionnaires for ED and burnout tend to exhibit a strong positive association [10, 11].

Following the implementation of the ED diagnosis, the Karolinska Exhaustion Disorder Scale (KEDS) [12] and the Lund University Checklist for Incipient Exhaustion (LUCIE) [10, 13, 14] were developed to assist in the detection, monitoring and/or treatment of ED. And it is not uncommon that these screening inventories are used concurrently in research or clinical practice [10, 15,16,17]. However, differences in their (a) emphasis (manifest signs [KEDS] versus incipient signs [LUCIE]), (b) response formats (a seven step Likert scale with alternating verbal anchors [KEDS] versus a four step Likert scale [LUCIE]), (c) time frames (two weeks [KEDS] versus four weeks [LUCIE]), and (d) scoring procedures, suggest that KEDS and LUCIE might be assumed to give results that are supplementary rather than interchangeable.

We have previously, in a study sample drawn from the general population, benchmarked KEDS and LUCIE with each other and other inventories relating to stress and burnout, [10, 11, 13, 14], showing that KEDS and LUCIE are systematically positively related and that the Kappa agreement is fair to moderate [10]. In the present study, we sought to replicate our previous observations by comparing KEDS and LUCIE in a new study sample entailing Swedish school principals. From previous analyses, we knew that school principals had circa a twice as high prevalence proportion of stress and exhaustion symptoms [17] than our previous population sample [10]. In addition, the school principals had clearly higher prevalence proportions than what has been observed among leaders and employees in a politically governed regional organisation [18] and among occupational therapists [19]. Although comparisons of prevalence proportions across study samples should be made with caution due to varying selection procedures, the ostensibly higher proportion of school principals with exhaustion signs generate a beneficial condition for examining the agreement between KEDS and LUCIE. The research questions were:

• To what extent do LUCIE and KEDS identify the same individuals and what is the level of agreement between the instruments?

• To what extent do the present observations among school principals agree with our previous observations of Kappa agreement in a highly educated and healthy population sample?

Methods

Participants and study design

The present study targeted Swedish principals at all school levels. In the absence of an accessible official register of occupationally active school principals, we recruited participants via an e-mail list that had a nationwide reach and covered principals who during the period 2008–2017 had participated in training programmes funded and arranged by the Swedish National Agency for Education and run by different universities in Sweden. In the present study, the first response from principals and assistant principals to an online survey issued in 2018 (n = 2274) and 2019 (n = 1959) [20] were used (n = 2670). Of these, 2219 responded in 2018 and 451 in 2019. The participants also had to work at least 50% of full-time in compulsory schools (46%), pre-schools (27%), upper secondary schools (15%), adult education (7%), or pre- and compulsory schools (5%). Their mean age was 49.4 years (SD = 7.3 years). Most participants were women (78%) and employed by a municipality (77%).

Measures

The LUCIE and KEDS inventories have been published [10, 12].

The Lund University Checklist for Incipient Exhaustion (LUCIE) entails 28 items covering six domains that build two supplementary scales: the Stress Warning Scale (SWS) and Exhaustion Warning Scale (EWS) [10]. Both scales ranges from 0 to 100 points. Cronbach’s alpha for the 28 LUCIE items was 0.94. And the median spearman rank order correlation between the SWS score and the individual LUCIE items was rho = 0.58 (Min = 0.44; Max = 0.70). For the EWS score, the median correlation with individual LUCIE items was rho = 0.47 (Min = 0.33; Max = 0.58). Following the manual, the SWS and EWS scores were categorized and combined. Specifically, SWS entails the three levels: Green (≤ 17.00), Yellow (17.01 to 38.50), and Red ≥ 38.51) whereas EWS entails two levels: Green (≤ 21.49) and Red (≥ 21.50). Using the pre-defined cut-off scores on each scale, the SWS and EWS are in practice combined into a four-step ladder of incremental stress symptomatology in which the combinations (a) SWS Green and EWS Red, and (b) SWS Yellow and EWS Red, are not possible to obtain or very rare, respectively. The four steps are:

Step 1-GG (SWS green zone and EWS green zone), indicating no signs of stress;

Step 2-YG (SWS yellow zone and EWS green zone), indicating possible, or weak signs, of stress;

Step 3-RG (SWS red zone and EWS green zone), indicating mild to moderate lasting stress symptoms, but less severe than ED;

Step 4-RR (SWS red zone and EWS red zone), indicating severe stress signs and possible ED.

The Karolinska Exhaustion Disorder Scale (KEDS) comprises nine items that refer to the past two weeks, covering the following nine domains: (a) ability to concentrate, (b) memory, (c) physical stamina, (d) mental stamina, (e) recovery, (f) sleep, (g) hypersensitivity to sensory impressions, (h) experience of demands, and (i) irritation and anger [12]. Each item is responded to on a 7-point scale (0–6). Descriptive verbal phrases, tailored for each separate item on a scale of increasing severity describe the corresponding domain-specific symptomatology, serving as anchors for the scale steps 0, 2, 4 and 6. Higher values reflect more severe symptoms, and the sum score (0 to 54) is used as an outcome, with a score ≥ 19 indicating possible ED [12, 21]. Cronbach’s alpha was 0.88. And the median spearman rank order correlation between the continuous total KEDS sum score and individual KEDS items was rho = 0.70 (Min = 0.67; Max = 0.80).

Data management and statistical analysis

Using the IBM SPSS software version 29, we applied non-parametric and parametric testing (two-tailed alpha level ≤ 0.05). Spearman rank correlations and Kappa statistics (both with bootstrap estimated 95% confidence intervals [95% CI] to compensate for ties) were used to estimate the degree of association and agreement between LUCIE and KEDS. For purpose of analysis, and to create symmetry between LUCIE and KEDS categories, we examined the changes in the Kappa statistics following the use of various cut-off scores for LUCIE. Sensitivity analyses were conducted by excluding individuals, by analysing data separately for men and women, and for subgroups defined by the year of the first response (i.e., 2018 or 2019). Landis and Koch criteria for judgment of the Kappa statistics were used: K 0–20 = poor; K ≤ 21–40 = fair; K ≤ 41–60 = moderate; K ≤ 61–80 = substantial; K > 80 = Good [22].

Results

Table 1 shows the proportion of individuals within a specific LUCIE category that reports above or below the exhaustion indication level in KEDS (Table 1). Descriptive data, and measures of the associations between the continuous LUCIE SWS and EWS scale scores and the KEDS sum scores, are presented in Additional file 1 (Table S1 and S2 and Figures S1 to S3).

Table 1 Descriptive cross-tabulation. The table shows the proportion of subjects within each Lund University Checklist of Incipient Exhaustion (LUCIE) category (Column %) that had either a normal score, or an exhaustion indication, on the Karolinska Exhaustion Disorder Scale (KEDS).

Table 2 shows the degree of Kappa agreement between LUCIE and KEDS. The lowest agreement was observed when making a cut-off between LUCIE step 3-RG and step 4-RR (K = 0.34). The highest agreement was observed when making a cut-off between LUCIE step 2-YG and step 3-RG (K = 0.54). At this cut-off point, the prevalence proportions for LUCIE and KEDS were 25.4% and 29.5%, respectively. Subsequent sensitivity analyses, in which individuals on step 2-YG and 3-RG were excluded, showed as expected a higher degree of agreement (i.e., K´s between 0.70 and 0.77) and revealed similar patterns of Kappa agreement for men and women and for subgroups of participants defined by their year of first response (see Additional file 2, Table S3 and S4).

Table 2 Estimated Kappa agreement between the Lund University Checklist of Incipient Exhaustion (LUCIE) and the Karolinska Exhaustion Disorder Scale (KEDS) at different LUCIE cutoff points and by excluding identifications in the middle range of LUCIE (sensitivity analysis)

Table 3 presents a comparison of the continuous mean and median KEDS sum scores across the four LUCIE steps of incremental signs of exhaustion. Descriptive sensitivity analyses for subgroups of men and women indicated that the pattern of mean and median KEDS sum scores were similar (see Additional file 3, Table S5).

Table 3 Descriptive Karolinska Exhaustion Disorder Scale (KEDS) scores (M), standard deviations (SD), median scores (Mdn), and accompanying 95% Confidence intervals (CI) across the four Lund University Checklist of Incipient Exhaustion (LUCIE) categories

Discussion

In essence, the pattern of Kappa estimates between LUCIE and KEDS in the present sample of Swedish principals and assistant principals can be said to replicate our previous observations in a study sample entailing highly educated and healthy individuals drawn from the general population [10]. However, and even if the Kappa estimates were consistently higher in the present study sample (see Table 2), the Kappa estimates had overlapping confidence intervals. Overall, the pattern of results from our present study and previous studies suggests that the concurrent validity of LUCIE and KEDS is a stable phenomenon across study samples, contexts, and gender.

That the Kappa agreement between LUCIE and KEDS at best amounts to “moderate” according to the Landis and Koch criteria [22] is not very surprising in view of the aforementioned differences in emphasis, item formats, covered time frames and that LUCIE makes a more fine-grained classification of early stress and exhaustion signs across four categories whereas KEDS uses two categories (i.e., indication/no-indication). Thus, to estimate the Kappa agreement, we dichotomized the LUCIE categories by applying various cut-off points. In so doing, we observed that the best agreement was achieved using a cut off between LUCIE step 2-YG and step 3-RG. Furthermore, as shown when computing means and median KEDS sum scores for each of the LUCIE categories, there is a clear escalating pattern. And that both the mean and median KEDS score falls just above the cut-off score for an exhaustion indication (≥ 19.0) at LUCIE step 3-RG, suggest that the most reasonable comparison between KEDS and LUCIE is achieved when merging step 3-RG and Step 4-RR in LUCIE. This observation is also in alignment with the general recommendation to use step 3-RG as the first significant indicator of stress symptoms of a clinical magnitude [15].

Noticeably, if excluding step 2-YG and/or step 3-RG the agreement between LUCIE and becomes substantial according to Landis and Kochs criteria [22]. Thus, apart from the reduction in categories it is also clear that the higher prevalence proportion of stress and exhaustion symptoms in the present study sample will influence the Kappa estimates. The higher prevalence proportion of exhaustion symptoms in the present study sample [17, 20] when compared to our previous population sample [10] is also a likely contributor to the consistently higher Kappa agreement in the present sample even if the 95% CI consistently overlap.

Conclusions

The present results essentially replicate our previous observations among highly educated and healthy individuals drawn from the general population. The fair to moderate Kappa agreement between KEDS and LUCIE suggests that these instruments might be supplementary rather than interchangeable. Thus, while the participants’ concurrent ratings in LUCIE and KEDS are systematically, logically, and consistently associated with each other, the instruments do not necessarily identify the same individuals as being stressed and/or exhausted. This underlines that on the individual level, the results from these screening instruments are best regarded as indicators of possible illness that needs to be understood in a broader context and in a confident dialogue with the person screened.

Limitations

All participants had a manager position, long education, and were in active full-time work to an extent of at least 50% of full-time. In the absence of a gold standard for assessing exhaustion disorder symptoms, it should be noted that our analyses only confirm the concurrent agreement between the two instruments. And because KEDS favours a dichotomized approach to identify possible exhaustion cases, comparisons with LUCIE, that favours a partitioning into four classes, become more difficult and less intuitively straightforward. In addition, the prevalence proportions of stress and exhaustion warnings in both LUCIE and KEDS were higher in this occupational sample than has been previously observed in the general population and in other occupational samples. Generalizations to other occupations and/or segments of the population (e.g., people with diagnosed ED) should be made with caution.