Compared to European countries and the US, the number of standardized instruments to evaluate healthcare services in Brazil and other developing countries is still small. However, to improve the quality of health care, tools able to measure its various dimensions accurately are necessary . The present article describes the construction of an instrument for the assessment of intensive care services adapted to Brazilian norms and inclusive of indicators sanctioned in the specialized literature. In addition, new indicators based on enacted Brazilian legislation were suggested.
Quality indicators comprise one of the pillars used to make improvements in healthcare services. The systematic use of such indicators allows the detection of opportunities for improvement and deviations from pre-established standards . A lack of instruments for the assessment of quality and a lack of governmental support to formulate such instruments hinder aspirations to improve the quality of health care .
The present study has some limitations: (a) the lack of a gold standard against which to compare the proposed instrument. (b) the application of one single approach to tool construction and data collection, whereby other techniques (field observations, reviews of clinical records, interviews with healthcare providers or users) and the participation of other actors of the health system in the tool construction are not performed. Nevertheless, valid and reliable information might also be produced when one single approach is used, and at an advantageously low cost . (c) The larger number of indicators corresponding to the structure dimension compared to the process dimension, which is a function of the strong presence of norms relative to the former in Brazilian legislation. This idiosyncrasy stems from a historical concern with the instrumental quality of services at the expense of the processes. In addition, there is a great difficulty to select indicators of process that truly represent work routines in the various Brazilian regions . (d) The difficulty of establishing cutoff points for the indicators that represent health care-associated infections (HAI), which is due to the lack of large-scale prevalence studies for Brazilian ICUs. (e) Possible interference in evaluation results due to interviewer or information bias. Finally, (f) “quality” is a construct, i.e. an unobservable theoretical concept and, therefore, cannot be measured directly. From this perspective, an indirect way to measure it (“proxy measure”) is the use of indicators.
Those limitations notwithstanding, the final result might be appraised positively for the following reasons: (a) the use of judicious methods in the elaboration of the instrument, including the participation of specialists in intensive care medicine and sole inclusion in the instrument of the indicators that attained 100% consensus only; (b) the inclusion of indicators already sanctioned in the international literature and the construction of others that represent the predominant norms in Brazil, thus resulting in an instrument particularly adapted to reflect the local healthcare practices; (c) the attribution of scores to the criteria, thus allowing comparison of the performance of any one service over time or to other services (benchmarking); (d) the inclusion of the criterion “below standard” broadening the scope of possible answers beyond mere presence or absence (“yes”, “no”) of the investigated attribute; (e) the division of the instruments into sections, which allows services to identify their weak points as well as opportunities for improvement relative to the assessed dimensions; and (f) its low cost, ease and short time required for application (30 min).
Use of interviews as an assessment instrument has been validated in low and middle-income countries. The sensitivity and specificity of that technique for assessing the quality of healthcare services are high compared to methods such as reviews of clinical records and direct observation, which are limited by missing data and interexaminer variability within this scenario. As a result, interviews provide an efficient means for assessment in countries whose healthcare system is not yet fully developed .
In a publication from 2008, Najjar-Pellet et al. described the construction and validation of an instrument to assess French ICUs, which included indicators of structure and process and use of a method similar to the one used for the instrument described in the present article . The instrument formulated by those authors allows scores to be attributed to the assessed services, as the one described here does, but it differs from the latter in that it does not include outcome indicators.
The present document differs from the ones constructed in the Netherlands, in Germany and by ESICM in the possibility of attributing scores to the assessed services, while the latter only allow establishing the presence or absence of a given attribute, without any value judgment of it or of the final result of the evaluation of the investigated ICU.
Along with the construction of the assessment instrument, the authors sought to include the largest possible number of structure and process indicators for which there is documented scientific evidence relative to their correlation with, and impact on, the results to be studied . This being the case, we call attention to the inclusion of some relevant indicators. One review performed in 2002 by Pronovost et al. showed that the mortality rate and length of stay decreased in the ICUs with 24-h availability of intensivists . According to some scientific evidence, a higher number of nursing associates have better provision of healthcare. That finding might be measured by some indicators, such as a lower rate of readmissions in hospitals with larger numbers of nurses compared to the ones with lower numbers of such professionals .
In a publication from 2003, Pronovost et al. reported the results of a study in which rigorous methods were used and which showed that failure to use standardized processes, such as appropriate sedation, ventilator-associated pneumonia (VAP) prevention, gastrointestinal bleeding prophylaxis, venous thromboembolism (VTE) prophylaxis and appropriate use of blood transfusions, was associated with poorer outcomes, as was increased ICU length of stay and mortality . Those findings give further support to the need for protocols specifically formulated for such clinical situations in ICUs, as well as to the relevance of appropriate adherence to them.
The inclusion of indicators representing a unit policy relative to the satisfaction of patients and permanence of relatives is based on some studies that reported an effect of those variables on the outcomes. Systematic assessment of satisfaction and increased presence of relatives in the ICUs indicates an improvement in the quality of care . Those indicators might also be used to assess the quality of care and communication for a given unit [19, 21].
With regard to the process dimension, the assessment of daily multidisciplinary rounds for case discussion was considered to be highly relevant. Kim et al. showed that performance of such rounds is associated with lower mortality rates. The results of that study, published in 2010, are highly significant because they show that such effects occur, even in units without available intensivists. As a function of the scarcity of that type of doctor, and being that implementation of that modality of process involves little or no additional cost, performance of daily multidisciplinary rounds for case discussion is a high-impact strategy that ought to be adopted in critical care services, the ones in developing countries in particular .
Some articles emphasize the relevance of including specific outcome indicators in assessment instruments. In a meta-analysis published in 1997, Ashton et al. reported that the ICU readmission rate is a satisfactory indicator of the quality of processes related to the care provided over the course of hospital stays. Reduction of the quality of such processes is associated with up to a 55% increase in the risk of readmission .
Gastmeier et al. showed that participation in HAI surveillance systems is associated with significant reduction of their occurrence . Thus, we might conclude that not only the HAI rates as such but also systematic collection of the corresponding data at participating ICUs and institutions are satisfactory quality indicators. Similarly, in a publication from 2008, Uçkay et al. asserted that the performance of indicators availability of protocols and surveillance of the prevalence of VAP in ICUs are satisfactory .
A constant challenge these authors all met, during the construction of the assessment instrument described here, was to keep a balance between the validity and reliability of the constructed indicators and the availability of and work load demanded by the collection of the corresponding data. One relevant issue that should be borne in mind is that in addition to aspects related to structure and process, the clinical condition of the patients admitted to the ICU exerts a strong influence on outcomes .
Many seemingly usefulness indicators in clinical practice were nonetheless excluded from the final version of the instrument. The reason for those exclusions was the rigorous methodological decision to include only the indicators that had achieved 100% consensus among the specialists. One further concern was to construct an instrument that would be easy to apply and take eventual regional differences into consideration.
The assessment instrument described here should be understood within a dynamic context. Consequently, we emphasize that this is a work still under construction. It is our view that, from the publication of this study, the scientific community and users would suggest new contributions that might be included in the instrument. In turn, such a move could improve its application in different scenarios. This is to say, indicators eventually shown to lose clinical relevance over time, or those no longer exhibiting variability, ought to be excluded. Conversely, new scientific evidence might come to indicate the need to include other indicators. Within that context, a systematic collection of quality indicators should not be understood as a goal unto itself but as a means to detect weak points in the system and opportunities for improvement . The data thus gathered allow for planning actions aimed at the correction of the weak points detected.