- Research article
- Open Access
A dissonant scale: stress recognition in the SAQ
BMC Research Notes volume 6, Article number: 302 (2013)
Our previous analyses using the Stress Recognition subscale of the Safety Attitudes Questionnaire (SAQ) resulted in significant effect estimates with equally opposing explanations. We suspected construct validity issues and investigated such using our own data and correlation matrices of previous published studies.
The correlation matrices for each of the SAQ subscales from two previous studies by Speroff and Taylor were replicated and compared. The SAS Proc Factor procedure and the PRIORS = SMC option were used to perform Common Factor Analysis.
The correlation matrices of both studies were very similar. Teamwork, Safety Climate, Job Satisfaction, Perceptions of Management and Working Conditions were well-correlated. The correlations ranged from 0.53 to 0.76. For Stress Recognition correlations ranged from -0.15 to 0.03. Common Factor Analysis confirmed the isolation of Stress Recognition. CFA returned a strong one-factor model that explained virtually all of the communal variance. Stress Recognition loaded poorly on this factor in both instances, and the CFA indicated that 96.4-100.0% of the variance associated with Stress Recognition was unique to that subscale, and not shared with the other 5 subscales.
We conclude that the Stress Recognition subscale does not fit into the overall safety climate construct the SAQ intended to reflect. We recommend that this domain be omitted from overall safety climate scale score calculations, and clearly identified as an important yet distinct organizational construct. We suggest that this subscale be investigated for its true meaning, characterized as such, and findings conveyed to SAQ end users. We make no argument against Stress Recognition as an important organizational metric, rather we suggest that as a stand-alone construct its current packaging within the SAQ may be misleading for those intent on intervention development and evaluation in healthcare settings if they interpret Stress Recognition results as emblematic of safety climate.
Our previous work with the SAQ raised validity concerns regarding its Stress Recognition subscale [1, 2]. This paper provides an update to our investigation of this domain. The Safety Attitudes Questionnaire (SAQ) is one of the most highly validated hospital climate assessment instruments in use today . The instrument elicits frontline healthcare workers’ perceptions of their organization’s safety culture at the level of the clinical area on which they work. Subscales within the SAQ include Teamwork, Safety Climate, Job Satisfaction, Perceptions of Management, Working Conditions, and Stress Recognition. The SAQ has been used to assess opportunities for quality improvement in obstetrical settings , intensive care units [5, 6], within single institutions [7–9], in multicentre studies [10–12], children’s hospitals , the Veteran’s Administration  and increasingly in international settings [8, 15]. The SAQ has been well-described [16–18].
In 2011, we published a paper showing a positive relationship between increasing scores on subscales of the Safety Attitudes Questionnaire and associations with decreasing nurse and patient injuries . In the results of that paper, we did not report one of the six subscales of the SAQ: Stress Recognition. We hypothesized that there might be construct validity concerns because Stress Recognition acted very differently from the other SAQ subscales. The results of our multilevel logistic regression showed a threefold increase in the odds of nurse injury with every 10 unit increase in stress recognition among nurses in a Level I Trauma Center. We saw similar relationships with medical errors, whereby increasing Stress Recognition was associated with a 1.5 to 3-fold increase in the odds of patient falls, medication errors, and decubitus ulcers [1, 2]. The stress recognition subscale of the SAQ was intended to measure frontline healthcare workers’ understanding that working in a highly stressful environment could put them in adverse conditions that might result in harm to their patients. Given our understanding of how the subscale was intended to work, we could not interpret our results conclusively and did not include them in our previous manuscript. We mentioned at the time that we were exploring this subscale for construct validity issues and present the results of such herein.
Analysis began with reconstruction of the correlation matrices for each of the SAQ subscales. Only two published studies reported correlations for the entire dataset: Speroff  and Taylor . Both matrices were reviewed, and the relationships amongst the subscales were examined. Common Factor Analysis (CFA) was then utilized to identify the number of latent traits underlying the six subscales, and to determine the relationship of the subscales with the identified trait(s). To execute the CFA, the SAS (version 9.2) Proc Factor procedure was used in conjunction with the PRIORS = SMC option (which utilizes squared multiple correlations to estimate shared communal variance). Given that a one factor (i.e. one dimensional) model was returned, the rotation of factor loadings could not be conducted.
Table 1 shows the items comprising the Stress Recognition subscale compared with those of the other 5 SAQ subscales. From a face validity perspective, the Stress Recognition items elicit an individual perspective about abilities (“I am more likely to make errors in tense or hostile situations” [emphasis added]) while the items on the other SAQ subscales elicit perspectives about their work area or broader organizational unit. (“The culture in this clinical area makes it easy to learn from the errors of others” [emphasis added]).
The similarities between the correlation matrices of Speroff and Taylor were immediately apparent. Teamwork, Safety Climate, Job Satisfaction, Perceptions of Management and Working Conditions were well-correlated. In Speroff, the correlations ranged from 0.66 to 0.76 (Table 2). For Taylor, the range was from 0.53 to 0.73 (Table 3). Stress Recognition was isolated in both results. For Speroff, Stress Recognition correlations ranged from -0.15 to -0.17, and for Taylor, they ranged from -0.06 to 0.03.
Exploratory Common Factor Analysis confirmed that 96% + of the variance in the Stress Recognition subscale was unique to that scale and not “in common” with the others. That fact demonstrates the isolation of the scale. The number of the factors used in our model was determined by the eigenvalues themselves as evidenced in the skree plot for both Taylor and Speroff (Figure 1). For both Speroff and Taylor, CFA returned a strong one-factor model that explained virtually all of the communal variance (Table 4). Stress Recognition loaded poorly on this factor in both instances, and the CFA indicated that 96.4% (Speroff) and 100.0% (Taylor) of the variance associated with Stress Recognition was unique to that subscale, and not shared with the other 5 components.
We were precluded from conducting a similar analysis with the Sexton data. Sexton reported between and within-area correlations, but not correlations for the entire dataset. Similarly, he reported descriptive statistics for geographical sub-samples, but not the sample as a whole. Thus we did not have access to the necessary data.
We focused on the psychometric validation of the SAQ to determine whether Stress Recognition is related to the overall safety climate construct. As evidenced by its weak fit with the other subscales of the SAQ, we conclude that the Stress Recognition subscale does not fit into the overall safety climate construct measured by the SAQ.
These results were confirmed by the work of Gallego et al. As explanation, Gallego suggest that “Stress Recognition items differ from those in other SAQ scales in that they assess self-behaviour while other SAQ scales focus mainly on behaviours of others in the respondent’s workplace …” and that “Stress Recognition scales … seem likely to remain relatively constant …” . Pettker et al. administered the SAQ before and after a obstetrics patient safety program, and observed that while the other scales changed in a statistically significant manner, Stress Recognition did not . While not definitive proof, such longitudinal testing would offer a way to evaluate the circumstances in which Stress Recognition changes relative to the other scales, and could either confirm or deny Gallego’s self-behavior versus others-workplace-behavior hypothesis above.
This finding is also supported by the previous work of Speroff who in reflection on their own correlations using the SAQ in 1,406 respondents stated that "the stress recognition items do not contribute positively towards the construct of safety climate as intended and should be excluded from the SAQ" . We believe that the inclusion of the Stress Recognition subscale in the SAQ does not affect its overall reliability. However, we have strong concerns that the stress recognition subscale of the SAQ affects its validity. We recommend that this domain be omitted from overall safety climate scale score calculations, and clearly identified as a distinct construct.
The Stress Recognition subscale of the SAQ was intended to measure frontline healthcare workers’ understanding that working in a highly stressful environment could put them in adverse conditions that might result in harm to their patients. Stress Recognition is defined as “the extent to which individuals acknowledge personal vulnerability to stressors such as fatigue, personal problems, and emergencies situations” . Furthermore, “Stress recognition may be enhanced or jeopardized by organizational practices such as scheduling, supervision and staffing levels” .
As constructed, this subscale is a measure of organizational buy-in: instead of blaming oneself, workers understand that working conditions create stressors which make them unable to do their jobs as well as they would normally expect. When we attempted to interpret our findings regarding the very significant increased odds of nurse and patient injury associated with increasing Stress Recognition, we came to two equally opposing explanations. The first was that the organization has been effective in its efforts to teach nurses that environmental stressors can lead to unsafe conditions, so nurses in turn report more occupational and patient injuries than nurses who have not. This type of reporting would be reflective of workers who have bought into the organizational effort to affirm that their concerns they would be heard and may garner additional resources to help ameliorate problems they face (e.g., low staffing levels, high turnover).
The alternative explanation was that when nurses were responding to these SAQ Stress Recognition items that they actually read them as indicative of measuring their stress level at the time. While this is not how the subscale was designed, it may in fact be how nurses are interpreting the items. In discussions with the SAQ developer, the Stress Recognition domain of the SAQ operates somewhat differently than the other domains. It represents the individual attitudes of the respondents rather than a consensus among those people working on the unit (Bryan Sexton, personal communication). Since the nurse injuries described in our 2012 publication were voluntary reports to the Hospital’s Occupational Health Department, the effect of increased stress recognition may represent an increased awareness of the importance of reporting injuries due either to heightened awareness of stress or because of a past stress-related experience. Our analysis herein did not address these competing hypotheses, but future research should investigate how respondents are interpreting the survey items.
We consider the poor fit of Stress Recognition within the SAQ as important because many hospitals and healthcare facilities use the SAQ as a baseline for improvement opportunities. We are concerned that the subscale is measuring something other than safety climate and therefore hospitals initiating interventions to improve stress recognition may not see a difference if they are using interventions specific to safety climate to make change. Conversely, hospitals with active safety climate interventions may not see any change in Stress Recognition on the SAQ because of its lack of association with safety climate.
We found that the Stress Recognition subscale of the Safety Attitudes Questionnaire does not fit with the overall heuristic of the instrument as designed. The other five subscales are highly correlated with one another, as evidenced by the results of the factor analysis. Stress Recognition is a standalone construct. The intent of the subscale was to capture attitudes that reflected an increased understanding of the role stress plays in the ability to do one's job safely. While the construct may indeed be capturing this exact perspective (and we offer no counterargument to the importance of understanding how stress works in one’s environment), Stress Recognition is a separate and distinct measure of organizational buy-in and is not reflective of safety climate. It may be indicative of fatigue as a precursor to burnout as evidenced in the work of Raftopoulos et. al.  Healthcare organizations seeking to improve their stress recognition measurements should be mindful that turning to proven safety climate interventions (e.g., executive walkarounds, teamwork training) may not produce expected results because stress recognition is not a part of safety climate and therefore not sensitive to interventions designed to improve it. Instead, they should review the fatigue and burnout literature to examine potential solutions to the perceptions the SAQ stress recognition domain is measuring in their workforce.
Taylor JA, Dominici F, Agnew J, Gerwin D, Morlock L, Miller MR: Do nurse and patient injuries share common antecedents? An analysis of associations with safety climate and working conditions. BMJ Qual Saf. 2012, 21 (2): 101-111. 10.1136/bmjqs-2011-000082.
Taylor JA: Utility of patient safety case finding methods and associations among organizational safety climate, nurse injuries, and errors, Dissertation. 2008, The Johns Hopkins University
Colla JB, Bracken AC, Kinney LM, Weeks WB: Measuring patient safety climate: a review of surveys. Qual Saf Health Care. 2005, 14 (5): 364-366. 10.1136/qshc.2005.014217.
Haller G, Garnerin P, Morales MA, Pfister R, Berner M, Irion O, Clergue F, Kern C: Effect of crew resource management training in a multidisciplinary obstetrical setting. Int J Qual Health C. 2008, 20 (4): 254-263. 10.1093/intqhc/mzn018.
France DJ, Greevy RA, Liu X, Burgess H, Dittus RS, Weinger MB, Speroff T: Measuring and comparing safety climate in intensive care units. Med Care. 2010, 48 (3): 279-284. 10.1097/MLR.0b013e3181c162d6.
Sexton JB, Berenholtz SM, Goeschel CA, Watson SR, Holzmueller CG, Thompson DA, Hyzy RC, Marsteller JA, Schumacher K, Pronovost PJ: Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. 2011, 39 (5): 934-939. 10.1097/CCM.0b013e318206d26c.
Huang DT, Clermont G, Sexton JB, Karlo CA, Miller RG, Weissfeld LA, Rowan KM, Angus DC: Perceptions of safety culture vary across the intensive care units of a single institution. Crit Care Med. 2007, 35 (1): 165-176. 10.1097/01.CCM.0000251505.76026.CF.
Deilkas E, Hofoss D: Patient safety culture lives in departments and wards: multilevel partitioning of variance in patient safety culture. BMC Health Serv Res. 2010, 10 (85): 85-
Paine LA, Rosenstein BJ, Sexton JB, Kent P, Holzmueller CG, Pronovost PJ: Assessing and improving safety culture throughout an academic medical centre: a prospective cohort study. Qual Saf Health Care. 2010, 19 (6): 547-554.
Pronovost PJ, Berenholtz SM, Goeschel C, Thom I, Watson SR, Holzmueller CG, Lyon JS, Lubomski LH, Thompson DA, Needham D: Improving patient safety in intensive care units in Michigan. J Crit Care. 2008, 23 (2): 207-221. 10.1016/j.jcrc.2007.09.002.
Huang DT, Clermont G, Kong L, Weissfeld LA, Sexton JB, Rowan KM, Angus DC: Intensive care unit safety culture and outcomes: a US multicenter study. Int J Qual Health C. 2010, 22 (3): 151-161. 10.1093/intqhc/mzq017.
Speroff T, Nwosu S, Greevy R, Weinger MB, Talbot TR, Wall RJ, Deshpande JK, France DJ, Ely EW, Burgess H: Organisational culture: variation across hospitals and connection to patient safety climate. Qual Saf Health Care. 2010, 19 (6): 592-596.
Grant MJ, Donaldson AE, Larsen GY: The safety culture in a children's hospital. J Nurs Care Quality. 2006, 21 (3): 223-229. 10.1097/00001786-200607000-00006.
Watts BV, Percarpio K, West P, Mills PD: Use of the safety attitudes questionnaire as a measure in patient safety improvement. J Patient Saf. 2010, 6: 206-2099. 10.1097/PTS.0b013e3181fbbe86.
Devriendt E, Van den Heede K, Coussement J, Dejaeger E, Surmont K, Heylen D, Schwendimann R, Sexton B, Wellens NI, Boonen S: Content validity and internal consistency of the Dutch translation of the Safety Attitudes Questionnaire: an observational study. Int J Nurs Stud. 2012, 49 (3): 327-337. 10.1016/j.ijnurstu.2011.10.002.
Sexton JB, Thomas EJ, Helmreich RL: Error, stress, and teamwork in medicine and aviation: cross sectional surveys. BMJ. 2000, 320 (7237): 745-749. 10.1136/bmj.320.7237.745.
Sexton JB: A mater of life or death: social psychological and organizational factors related to patient outcomes in the intensive care unit, Disertation. 2002, University of Texas at Austin
Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, Roberts PR, Thomas EJ: The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006, 6: 44-10.1186/1472-6963-6-44.
Gallego B, Westbrook MT, Dunn AG, Braithwaite J: Investigating patient safety culture across a health system: multilevel modelling of differences associated with service types and staff demographics. Int J Qual Health Care. 2012, 24 (4): 311-320. 10.1093/intqhc/mzs028.
Pettker CM, Thung SF, Raab CA, Donohue KP, Copel JA, Lockwood CJ, Funai EF: A comprehensive obstetrics patient safety program improves safety climate and culture. Am j Obstet Gynecology. 2011, 204 (3): e211-e216.
Sexton JB, Helmreich RL, Wilhelm JA, Merritt AC, Klinect JR: The Flight Management Attitudes Safety Survey (FMASS). The University of Texas Human Factors Research Project Technical Report 01–01. 2001, Austin, TX: The University of Texas at Austin, 8-
Helmreich RL: ACM: Culture at Work in Aviation and Medicine: National, Organizational and Professional Influences. 2001, UK: Ashgate, 28-2
Raftopoulos V, Charalambous A, Talias M: The factors associated with the burnout syndrome and fatigue in Cypriot nurses: a census report. BMC Public Health. 2012, 12: 457-10.1186/1471-2458-12-457.
This study did not have an external funding source. The authors would like to thank Dr. Arthur L. Frank, Chair of the Department of Environmental and Occupational Health, for his stewardship of an environment that encourages intellectual inquiry.
The authors declare they have no financial or non-financial competing interests.
JT and RP contributed to the study’s conception and design. JT acquired the data and created the correlation matrix. RP conducted the CFA analysis. JT and RP interpreted the data and contributed equally to writing and final approval of the manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.