Validity and reliability of the Utrecht Work Engagement Scale-Student Version in Sri Lanka

Abstract

Objective

The present study was aimed at assessing the validity and the reliability of the Sinhala version of the Utrecht Work Engagement Scale-Student Version (UWES-S) among collegiate cycle students in Sri Lanka.

Results

The 17-item UWES-S was translated to Sinhala and the judgmental validity was assessed by a multi-disciplinary panel of experts. Construct validity of the UWES-S was appraised by using multi-trait scaling analysis and exploratory factor analysis (EFA) on data obtained from a sample of 194 grade thirteen students in the Kurunegala district, Sri Lanka. Reliability of the UWES-S was assessed by using internal consistency and test–retest reliability. Except for item 13, all other items showed good psychometric properties in judgemental validity, item-convergent validity and item-discriminant validity. EFA using principal component analysis with Oblimin rotation, suggested a three-factor solution (including vigor, dedication and absorption subscales) explaining 65.4% of the total variance for the 16-item UWES-S (with item 13 deleted). All three subscales show high internal consistency with Cronbach’s α coefficient values of 0.867, 0.819, and 0.903 and test–retest reliability was high (p < 0.001). Hence, the Sinhala version of the 16-item UWES-S is a valid and a reliable instrument to assess work engagement among collegiate cycle students in Sri Lanka.

Introduction

Keeping on par with the emerging trend towards a positive psychology focusing on optimal functioning rather than on malfunctioning, a growing enthusiasm is evident in student engagement research during the last few decades [1].

Work engagement is defined as, “a positive, fulfilling, work-related state of mind that is characterised by vigor (VI), dedication (DE), and absorption (AB). Rather than a momentary and specific state, engagement refers to a more persistent and pervasive affective-cognitive state that is not focused on any particular object, event, individual or behaviour” [2].

Engagement of students in their academic environment has been studied in numerous disciplines [3,4,5,6,7,8,9,10] and the global literature suggests that students’ engagement is positively linked with their academic performances [11, 12] and positive academic outcomes [13,14,15,16]. Hence, educators and policymakers are increasingly focusing on student engagement as a means of addressing problems of negative academic outcomes of varying student groups [17].

Amongst the different assessment tools of student work engagement, the most commonly used instrument is the Utrecht Work Engagement Scale-Student Version (UWES-S) [2, 18,19,20,21,22], which is a 17-item self-report measure assessing VI, DE, and AB subscales.

Even though a plethora of research has been conducted on work engagement among varying student populations across the globe, there is a paucity of literature in the South Asian context with no published literature on work engagement among Sri Lankan students, mainly owing to the lack of validated assessment tools. In the context of ever-increasing burden of mental health problems in the Sri Lankan collegiate cycle students [23,24,25,26], exploration of effective strategies for mental health and wellbeing promotion has become a timely need. Hence, the present study was designed to assess the validity and the reliability of a culturally adapted Sinhala version of the UWES-S among collegiate cycle students in Sri Lanka.

Main text

Methods

Structure of the UWES-S

The 17-item UWES-S is a self-administered questionnaire (SAQ) with six, five, and six items assessing VI, DE and AB subscales respectively, measured on a seven-point Likert scale anchored by the response options from 0 (never) to 6 (every day).

Translation and pre-testing of the UWES-S

The 17-item UWES-S was translated to Sinhala by using the forward–backward translation method [27,28,29], involving two independent bilingual translators, who are fluent in Sinhala and English.

The synthesised forward translation of UWES-S was pre-tested among a sample of 25 grade thirteen students outside the study setting. In a subsequent structured interview, the clarity in understanding, acceptability, and comprehension of items and feasibility of using the questionnaire were assessed. None of the items of the questionnaire was claimed to be difficult to understand.

Appraising the judgemental validity of the UWES-S

The face, content, and consensual validity were assessed. Using a modified Delphi technique, a multi-disciplinary panel of experts in the fields of psychiatry, psychology, public health, teaching, student counseling, and medical education has assessed each item on its relevance, appropriateness, and acceptability in the local context for assessing burnout among collegiate cycle students based on a rating scale from 0 (strong disagreement) to 10 (strong agreement). At the end of this iterative process, except for the item 13 stating, “When studying, I am very resilient, mentally”, all other items had a median score more than 7 for all the aspects. Thus, it was decided to include all 17 items for the assessment of construct validity.

Study design and setting

This school-based, cross-sectional validation study was conducted in the Kurunegala district, North Western province, Sri Lanka from May 2014 to April 2015. In Sri Lanka, the collegiate cycle in the education system consists of grade twelve and thirteen. Three Sinhala medium government schools in the Kurunegala district having all four collegiate cycle subject streams, viz., Science, Arts, Commerce and Technology were selected.

Study participants

From the selected schools, three grade thirteen classes each were selected representing both male and female students studying in all four subject streams. A total of 194 students participated in the study in their classrooms and each participant filled the SAQ independently. The response rate was 100.0% and 55.2% (n = 107) of the sample were females. The mean age was 18.3 years (SD = 0.43 years). The majority of students were studying in the Science stream (n = 78, 40.2%). The numbers of students in the Arts, Commerce and the Technology streams were 60 (30.9%), 41 (21.2%), and 15 (7.7%) respectively.

Data analysis

Data were analysed by using the Statistical Package for Social Sciences (SPSS) version 17.0. Preparatory data analysis showed that there were no violations of the assumptions related to the data analytic techniques. Given that the participants to variables ratio was 11.4, the sample size was adequate for factor analysis [30]. Even though there were few items that showed non-normal distribution of data, it is considered to be a ubiquitous phenomenon in psychological assessment research.

Multi-trait scaling analysis

Item-scale correlations were analysed and item-convergent and item-discriminant validity were assessed. A stringent criterion of correlation of 0.40 or greater between an item and its own subscale was considered as a success for assessing item-convergent validity. Furthermore, in assessing item-discriminant validity, items that correlated significantly higher (more than 1.96 standard errors) with its own subscale than with the other two subscales, were considered as scaling successes.

Exploratory factor analysis (EFA)

EFA was conducted by Principal Component Analysis (PCA) with Oblimin rotation. Kaiser’s criterion/eigenvalue, scree plot and parallel analysis were used to decide the number of factors to retain. The factors that lead to a meaningful interpretation and theoretical sense were ultimately selected.

Assessment of reliability

Reliability was assessed using internal consistency and test–retest reliability by re-administering the SAQ to a sub sample of 22 grade thirteen students after 2 weeks.

Results

Descriptive statistics of the UWES-S scores

Descriptive statistics of UWES-S subscales are given in Table 1. The mean item score was highest in the AB subscale (4.52, SD = 1.40) and was lowest in the DE subscale (3.70, SD = 1.23).

Table 1 Descriptive statistics of the UWES-S subscale scores among grade thirteen students (n = 194)

Multi-trait scaling analysis

Table 2 summarises the results of the multi-trait scaling analysis and as per the predetermined cut-off values, except for item 13, item-convergent validity and item-discriminant validity were confirmed for other 16 items in the UWES-S.

Table 2 Analysis of item-scale correlations to determine item-discriminant scaling successes for the UWES-S scores (n = 194)

Exploratory factor analysis

Item 13 was found to have poor psychometric properties in judgemental validity, item-convergent validity, and item-discriminant validity. Deletion of item 13 from the subscale also improved the internal consistency. Hence, both 17-item UWES-S and 16 items of the UWES-S (item 13 deleted) were subjected to PCA. The Kaiser–Meyer–Olkin Measure were 0.851 and 0.855 respectively and Bartlett’s test of sphericity reached statistical significance (p < 0.001) for both versions.

Only three factors had eigenvalues more than 1.0, which cumulatively explained 63.9% and 65.5% of the total variance for the 17-item and 16-item UWES-S versions respectively. For both versions, the parallel analysis showed only two factors with eigenvalues exceeding the corresponding criterion values and the screeplot showed a clear break after the second factor.

However, for both versions, the component matrix in PCA with unrotated loadings revealed a number of items of UWES-S loading on the third factor with values greater than 0.3. Furthermore, the pattern matrix generated in PCA using direct Oblimin rotation revealed that seven items had loading values more than 0.3. Based on this collective evidence, it was decided to ‘force’ a three-factor solution for further investigation.

Even though both Varimax rotation and Oblimin rotation was used, it was decided to report statistics related to Oblimin rotation as the factors were strongly correlated (> 0.3). The rotated three-factor solution revealed a simple structure with all three factors showing a number of strong loadings. In the 17-item UWES-S version, the three-factor solution explained a total of 63.9% of the variance, with factor one contributing 41.4%, factor two contributing 14.9%, and factor three contributing 7.6%, whereas, the corresponding values for the 16-item UWES-S version were 65.4, 44.0, 14.2, and 7.2%. Table 3 shows the pattern and structure matrix for PCA with Oblimin rotation of three-factor solution of 16-item UWES-S.

Table 3 Pattern and structure matrix for PCA with Oblimin rotation of three-factor solution of 16-item UWES-S

The interpretation of three-factor solution of the UWES-S with item 13 deleted, was consistent with previous research on the UWES-S, with subscale items of VI, DE, and AB loading strongly on three different factors. Hence, the three-factor solution with item 13 deleted was considered as the validated Sinhala version of the UWES-S.

Assessment of reliability

The impact of each item on the related subscale was assessed by computing Cronbach’s α when the respective item is deleted. None of the items in DE subscale and AB subscale improved the overall Cronbach’s α value; however, deletion of item 13 improved the Cronbach’s α value from 0.696 to 0.867. Hence, item 13 was removed from the UWES-S and the reliability was re-assessed for internal consistency and all three subscales show high internal consistency with Cronbach’s α coefficient values of 0.867, 0.819, and 0.903 for VI, DE, and AB subscales respectively.

There were strong, positive statistically significant (p < 0.001) correlations for each of the three subscales of the UWES-S in test–retest reliability assessment. The correlation coefficients were 0.831, 0.866 and 0.839 for the VI, DE, and AB subscales respectively.

Discussion

The present study was designed to validate the Sinhala version of the UWES-S among Sri Lankan collegiate cycle students addressing an important research vacuum on work engagement research in high school students in the South Asian context. A cross-sectional study design and the sample size of the study deemed appropriate with the study objective.

In assessing the judgemental validity, item 13 (“When studying, I am very resilient, mentally”) had the least median rating score and the main concern raised regarding the item was the difficulty in explaining the phrase “mentally resilience” in Sinhala language. The global literature suggests that modified versions of the UWES-S with removal of several items have been used in work engagement among high school, college, and university students [21, 22, 31].

Even though the three-factor structure of the Utrecht Work Engagement Scale (UWES) has been confirmed in a large number of studies [32,33,34,35,36], the limited number of studies that explored the factor validity of the UWES-S have failed to provide convincing evidence to support the three-factor structure of the UWES-S [37]. Furthermore, the findings of the study conducted by Sonnentag [38], raises concerns about the three-factor structure of UWES. This evidence coupled with the fact that UWES-S is a relatively novel measure in the South Asian context, EFA was employed to assess the construct validity, instead of using confirmatory factor analysis [39].

The three-factor solution of the original 17-item UWES-S, explained 63.9% of the variance, whereas the three-factor solution of the UWES-S with item 13 deleted, explained 65.4% of the variance. Furthermore, the interpretation of the three-factor solution of the UWES-S with item 13 deleted, was consistent with previous research on the UWES-S with subscale items of VI, DE, and AB loading strongly on three different factors [2, 22].

In assessing the internal consistency, deletion of item 13 improved the Cronbach’s α value from 0.696 to 0.867 and all three subscales showed high internal consistency. This finding is consistent with findings of studies using the Portuguese version [22], Spanish version [22], Dutch version [2, 22], Turkish version [21] and Romanian version [20] of the UWES-S.

The present study also revealed that there were statistically significant strong positive correlations for each of the three subscales of UWES-S in the test–retest reliability assessment. Though these findings are consistent with the previous study findings [21], the values were not as high as those reported in that study.

In conclusion, the study findings indicate that the Sinhala version of the 16-item UWES-S is a valid and a reliable instrument to assess work engagement among collegiate cycle grade thirteen students in Sri Lanka. Due to its brevity, ease of administration and sound psychometric properties, it could be used as an effective screening tool at the school level. The study findings pave the way to establish the initial evidence base for the relevance and the applicability of the concept of work engagement in the South Asian context.

Limitations

This study has some limitations. Given that the study sample was recruited not using a probability sampling technique in a selected district of Sri Lanka, the generalisability of the study findings to other populations should be done with caution, considering the variations in educational and cultural contexts. Furthermore, given that the three-factor structure of the UWES-S is established in the present study, confirmatory factor analysis procedures are recommended for future studies.