- Short Report
- Open Access
Development and application of an indicator assessment tool for measuring health services accreditation programs
BMC Research Notes volume 8, Article number: 363 (2015)
Hospital accreditation programs are internationally widespread and consume increasingly scarce health resources. However, we lack tools to consistently identify suitable indicators to assess and monitor accreditation outcomes. We describe the development and validation of such a tool.
Using Australian accreditation standards as our reference point we: reviewed the research evidence for potential indicators; looked for links with existing external indicators; and assessed relevant state and federal policies. We allocated provisional scores, on a five point Likert scale, to the five accountability criteria in the tool: research; accuracy; proximity; no adverse effects; and specificity. An expert panel validated the use of the purpose designed indicator assessment tool. The panel identified hand hygiene compliance rates as a suitable process indicator, and hospital acquired Staphylococcus aureus infection (SAB) rates as an outcome indicator, with the hypothesis that improved hand hygiene compliance rates and lower SAB rates would correlate with accreditation performance.
This new tool can be used to identify, analyse, and compare accreditation indicators. Using infection control indicators such as hand hygiene compliance and SAB rates to measure accreditation effectiveness has merit, and their efficacy can be determined by comparing accreditation scores with indicator outcomes. To verify the tool as a robust instrument, testing is needed in other health service domains, both in Australia and internationally. This tool provides health policy makers with an important means for assessing the accreditation programs which form a critical part of the national patient safety and quality framework.
Health services accreditation programs are designed to strengthen quality and safety improvement efforts through compliance with clinical and organisational standards. Accreditation programs have been widely adopted with over 27 country specific hospital accreditation agencies active in a 2009 survey , and several international agencies providing cross-border accreditation services. However, the evidence for accreditation improving patient safety and quality is mixed [2–4]. An Australian study showed accreditation improved organisational measures , and a European study indicated that external assessment systems are associated with better clinical outcomes . Other work has shown a transient effect , or no effect on clinical outcomes . One of the difficulties in assessing the research and the range of findings is the lack of a comprehensive framework by which to analyse and compare the costs and benefits of accreditation programs. To meet this need and as part of the Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork (ACCREDIT) research partnership [9–11], we established a framework for evaluating the costs and benefits of hospital accreditation . Our purpose designed SIQNS framework comprises five discrete activities: (1) Scope and objectives; (2) Identify costs and benefits; (3) Quantify costs and benefits; (4) calculate Net social benefits; and (5) Sensitivity analysis.
There are a number of indicators that have been validated and accepted for measuring patient safety and quality in healthcare . However, one of the additional challenges in measuring the effectiveness of accreditation is in identifying indicators for the outcomes or benefits that can be linked specifically to the accreditation process. We therefore developed an indicator assessment tool to identify, compare and evaluate the suitability of either process or outcome indicators in hospital accreditation as part of our SIQNS framework. This tool incorporated the four accountability criteria used by The Joint Commission (the main United States health services accreditation agency) to measure improvements in the clinical processes of care . These were: level of research evidence underpinning the indicator; accuracy of the indicator in measuring implementation of the standard; proximity of the standard and indicator; and the potential for adverse effects or unintended consequences. We adapted these criteria to assess accreditation and added an additional criterion, specificity, to account for the difficulties of isolating the effects of accreditation from other quality and safety interventions (Table 1).
The aim of this study is to describe the development and validation of a purpose designed indicator assessment tool in identifying indicators for hospital accreditation performance. An expert panel assessed the tool using hand hygiene compliance rates and Staphylococcus aureus (SAB) rates to illustrate the selection process and assess the suitability of these indicators in assessing the impact of accreditation on patient safety and quality of care.
Study setting and process
This study takes the health service accreditation standards for Australian hospitals as its reference point. We reviewed the ten National Safety and Quality Health Service (NSQHS) Standards developed by the Australian Commission on Safety and Quality in Health Care (ACSQHC) and introduced in early 2013 . In addition, we reviewed the accreditation standards within the Evaluation and Quality Improvement Program (EQuIP), issued and maintained by the Australian Council for Healthcare Standards (ACHS) [16, 17]. We convened an expert panel to assess the tool, using two key indicators to demonstrate the selection process.
We focused on the first three NSQHS standards: Standard 1 Governance for safety and quality in health service organisations; Standard 2 Partnering with consumers; and Standard 3 Preventing and controlling healthcare associated infections . For each standard we reviewed the research evidence; looked for links with existing external indicators; and reviewed relevant state and federal policies in Australia. Given the recent introduction of the NSQHS Standards we also checked that the pre-existing EQuIP accreditation standards contained relevant content for retrospectively comparing indicators with accreditation outcomes [16, 17].
Evaluation of accreditation indicators
Four researchers from the ACCREDIT team assessed potential indicators against the five criteria in our indicator assessment tool during a series of four evaluation meetings. The researchers were experienced in a wide range of clinical and health services research expertise including: qualitative and quantitative methods; patient safety and quality; and organisational culture. Each criterion was allocated a score ranging from low to high on a five point Likert scale (Table 2). For example, high scores were allocated to the research criterion for evidence based on a systematic review, or acceptance by a national or international health body. A low score in research would be allocated where evidence was weak, for example from individual case studies. Since the criteria are measuring different aspects of each standard, these scores were considered separately and not used to create an overall score.
Review and assessment of indicator tool
We next convened an expert panel to validate the results of our prototype indicator assessment tool building on methods adopted for assessing quality indicators in primary health care . This panel comprised seven experienced, Australian based researchers working on accreditation and quality outcomes, as well as three high profile international healthcare quality and safety experts from Spain and the United Kingdom. The panel session comprised a 2 h meeting with all members present in person. Our panel members averaged 12.7 years in health services research and a total authorship of more than 1000 health research publications. We presented the panel with a description of the tool, the indicative scores allocated by the ACCREDIT team, the rationale for those scores, and the research evidence for the indicators. We used an evidence-based consensus approach to synthesise the results of the panel . The tool was additionally endorsed by an ACCREDIT Steering Committee comprising accreditation agency senior executives, health quality bodies, and health service managers.
We focused on Standard 3 relating to hospital acquired infection. This was partly due to the specificity of the interventions that could be mapped across the ACHS and NSQHS Standards. In addition, infection control processes have become increasingly important as infections acquired during a hospital stay are a major cause of morbidity and result in significant costs in terms of increased length of stay and additional treatment . Standard 3 comprises six main topics: governance and systems; patient management; infection prevention and control strategies; antimicrobial stewardship; cleaning, disinfecting and sterilisation; and communicating with patients and carers . Hand hygiene compliance rates and Staphylococcus aureus bacteraemia (SAB) incidence rates in the Australian acute care sector are routinely collected and published, and we selected these indicators to test our indicator assessment tool.
Hand hygiene programs in Australian hospitals, based on World Health Organization recommendations , take a multimodal approach, and are in line with best practice internationally . This approach incorporates a number of organisational and safety components including: staff education, appropriate infrastructure; compliance monitoring; and communicating with patients. This broad approach to quality improvement reflects the goals of the accreditation program within an overarching safety and quality framework .
The ACSQHC has recommended that hand hygiene programs in Australia are repeatedly monitored using both process indicators (compliance rates) and outcome indicators (infection rates) . Auditors trained by Hand Hygiene Australia (the agency tasked by ACSQHC to implement the program) monitor hand hygiene activity against the total number of potential “moments” for hand hygiene. These five “moments” comprise: before touching a patient; before a procedure; after a procedure or body fluid exposure risk; after touching a patient; and after touching a patient’s surroundings . The national target is a compliance rate of 70 % and results are publicly reported three times a year, providing readily accessible indicator data . SAB rates are reported on a monthly basis, with a national target incidence rates of <2 per 10,000 inpatient bed days .
The panel collectively reviewed the five criteria, assessed the evidence and interim evaluation provided, and awarded each a score ranging from low to high. The scores varied, with higher scores for research and proximity to lower scores for specificity. The results and rationale for the allocated scores for hand hygiene compliance rates are shown in Table 3, and those for SAB rates are shown in Table 4.
Tool and indicator validation
The expert panel endorsed the indicator assessment tool and its suitability for assessing process and outcome indicators for accreditation. The panel determined that hand hygiene would be suitable as a process indicator for hospital accreditation and should be assessed against accreditation outcomes. The higher scores for research and proximity were critical in this consideration and provide both face and content validity for the tool . The panel also recommended investigating SAB rates as an outcome indicator based on higher scores for research, proximity, and no adverse effect. The lower scores for specificity and accuracy were noted for both of these indicators. The panel recommended using the systematic approach embedded within the tool to widen the topics covered and include a range of indicators from across the accreditation standards.
Discussion and conclusions
Our validated indicator assessment tool can be used to systematically identify process and outcome indicators in order to assess their suitability as indicators of accreditation performance in Australian hospitals . This study focused on the infection control standard to illustrate the application of the tool. Further validation with a broader range of healthcare workers and across all ten NSQHS standards is merited. Identifying indicators of accreditation performance provides health policy makers with an important means for assessing a critical part of the patient safety and quality framework, both for monitoring the effect of current accreditation programs and in assessing their development. Testing is needed to ensure applicability of the tool in international and domestic accreditation programs across the acute, primary and aged care health domains.
accreditation collaborative for the conduct of research, evaluation and designated investigations through teamwork
Australian Council of Healthcare Standards
Australian Commission on Safety and Quality in Health Care
Evaluation and Quality Improvement Program NSQHS standards: national safety and quality health service standards
new south wales
a framework for evaluating the costs and benefits of health services accreditation (scope, identify, quantify, net social benefit calculation, and sensitivity)
World Health Organization
Shaw CD, Braithwaite J, Moldovan M, Nicklin W, Grgic I, Fortune T, Whittaker S. Profiling health-care accreditation organizations: an international survey. Int J Qual Health Care. 2013;25(3):222–31.
Mumford V, Forde K, Greenfield D, Hinchcliff R, Braithwaite J. Health services accreditation: what is the evidence that the benefits justify the costs? Int J Qual Health Care. 2013;25(5):62–6.
van Harten WH, Casparie TF, Fisscher OAM. Methodological considerations on the assessment of the implementation of quality management systems. Health Policy. 2000;54(3):187–200.
Flodgren G, Pomey M, Taber S, Eccles M. Effectiveness of external inspection of compliance with standards in improving healthcare organisation behaviour, healthcare professional behaviour or patient outcomes. Cochrane database of systematic reviews (Online). 2011;11.
Braithwaite J, Greenfield D, Westbrook J, Pawsey M, Westbrook M, Gibberd R, Naylor J, Nathan S, Robinson M, Runciman B, et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study. Qual Saf Health Care. 2010;19(1):14–21.
Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end points. BMJ. 2010;341:c4413.
Al Tehewy M, Salem B, Habil I, El Okda S. Evaluation of accreditation program in non-governmental organizations’ health units in Egypt: short-term outcomes. Int J Qual Health Care. 2009; 21(3):183–189.
Salmon J, Heavens J, Lombard C, Tavrow P. The impact of accreditation on quality of hospital care: Kwa-Zulu Natal Province, Republic of South Africa. Bethesda: USAID; 2003.
Braithwaite J, Westbrook J, Johnston B, Clark S, Brandon M, Banks M, Hughes C, Greenfield D, Pawsey M, Corbett A, et al. Strengthening organizational performance through accreditation research-a framework for twelve interrelated studies: the ACCREDIT project study protocol. BMC Res Notes. 2011;4(1):390.
Hinchcliff R, Greenfield D, Moldovan M, Pawsey M, Mumford V, Westbrook JI, Braithwaite J. Evaluation of current Australian health service accreditation processes (ACCREDIT-CAP): protocol for a mixed-method research project. BMJ Open. 2012;2(4):e001726.
Greenfield D, Hinchcliff R, Moldovan M, Mumford V, Pawsey M, Westbrook JI, Braithwaite J. A multimethod research investigation of consumer involvement in Australian health service accreditation programmes: the ACCREDIT-SCI study protocol. BMJ Open. 2012;2(5):e002024.
Mumford V, Greenfield D, Hinchcliff R, Moldovan M, Forde K, Westbrook JI, Braithwaite J. Economic evaluation of Australian acute care accreditation (ACCREDIT-CBA (Acute)): study protocol for a mixed-method research project. BMJ Open. 2013;3(2):e002381.
Agency for Healthcare Research and Quality. ARHQ quality indicators toolkit for hospitals: how to improve hospital quality and safety. Rockville: AHRQ; 2012.
Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures—using measurement to promote quality improvement. N Engl J Med. 2010;363(7):683–8.
Australian Commission on Safety and Quality in Health Care (ACSQHC). National safety and quality health service standards. Sydney: ACSQHC; 2012.
Australian Council on Healthcare Standards. The ACHS EQuIP 4 guide : part 2—standards. Sydney: ACHS; 2007.
Australian Council on Healthcare Standards (ACHS). EQuIP5 standards and criteria. Sydney: ACHS; 2010.
Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11(4):358–64.
Commission Productivity. Public and private hospitals: research report. Canberra: Productivity Commission; 2009.
Australian Commission on Safety and Quality in Health Care. Safety and quality improvement guide, standard 3: preventing and controlling healthcare associated infections. Sydney: AACSQHC; 2012.
World Health Organization. A guide to the implementation of the WHO multimodal hand hygiene improvement strategy. Geneva: WHO; 2009.
Gould DJ, Moralejo D, Drey N, Chudleigh JH. Interventions to improve hand hygiene compliance in patient care. Cochrane Database of Systematic Reviews (Online). 2010(9). Art. No. CD005186.
Grayson L, Hunt C, Johnson P, Maiwald M, Martin R, McLaws M-L. Hand hygiene. In: Cruickshank M, Ferguson J, editors. Reducing harm to patients from health care associated infection: the role of surveillance. Sydney: Australian Commission for Safety and Quality in Health Care; 2008.
Hand Hygiene Australia. Hand Hygiene Australia manual: 5 moments for hand hygiene. 3rd ed. Heidelberg: Commonwealth of Australia; 2013.
Australian Institute of Health and Welfare. Staphylococcus aureus bacteraemia in Australian public hospitals 2013–2014: Australian hospital statistics. Health services seres No. 59 Cat. No. HSE 155. Canberra: AIHW; 2014.
World Health Organization. WHO guidelines on hand hygiene in health care: a summary. First global patient safety challenge—clean care is safer care. Geneva, Switzerland: World Health Organisation; 2009.
Pfoh E, Dy S, Engineer C. Interventions to improve hand hygiene compliance: making health care safer II: an updated critical analysis of the evidence for patient safety practices. Rockville: Agency for Healthcare Research and Quality; 2013.
European Union Network for Patient Safety and Quality of Care. Safe clinical practice: multimodal intervention to increase hand hygiene compliance. 2013.
Pittet D, Allegranzi B, Sax H, Dharan S, Pessoa-Silva CL, Donaldson L, Boyce JM. Evidence-based model for hand transmission during patient care and the role of improved practices. Lancet Infect Dis. 2006;6(10):641–52.
Pincock T, Bernstein P, Warthman S, Holst E. Bundling hand hygiene interventions and measurement to decrease health care-associated infections. Am J Infect Control. 2012;40(4 Suppl):S18–27.
Duguid M, Cruickshank M. Antimicrobial Stewardship in Australian hospitals. Sydney: Australian Commision on Safety and Quality in Health Care; 2010.
NHMRC. Australian guidelines for the prevention and control of infection in healthcare. Canberra: Commonwealth of Australia; 2010.
Pantle A, Fitzpatrick K. Clean Hand save lives: final report of the NSW hand hygiene campaign. Sydney: Clinical Excellence Commission; 2007.
Clinical Excellence Commission. Infection Control Policy. Doc No PD2007_036. Sydney: NSW Health; 2007.
Mumford V, Greenfield D, Hogden A, Debono D, Gospodarevskaya E, Forde K, Westbrook J, Braithwaite J. Disentangling quality and safety indicator data: a longitudinal, comparative study of hand hygiene compliance and accreditation outcomes in 96 Australian hospitals. BMJ Open. 2014;4(9):e005284.
VM coordinated the study and drafted the manuscript. JB and JW initiated the study by proposing the original study and securing funding for the work. DG and JB contributed to the study design, assisted with drafting the manuscript, and organised the expert panel. KF, DD, and AH assisted with drafting the manuscript. All authors read and approved the final manuscript.
We would like to thank the ACCREDIT team and international advisory board for their participation in the expert panel.
Compliance with ethical guidelines
Competing interests The authors declare that they have no competing interests.