Skip to main content

Validation of the REduction of Atherothrombosis for Continued Health (REACH) prediction model for recurrent cardiovascular disease among United Arab Emirates Nationals

Abstract

Objectives

Prediction models assist health-care providers in making patient care decisions. This study aimed to externally validate the REduction of Atherothrombosis for Continued Health (REACH) prediction model for recurrent cardiovascular disease (CVD) among the Emirati nationals.

Results

There are 204 patients with established CVD, attending Tawam Hospital from April 1, 2008. The data retrieved from electronic medical records for these patients were used to externally validate the REACH prediction model. Baseline results showed the following: 77.0% were men, 69.6% were diagnosed with coronary artery disease, and 87.3% have a single vascular bed involvement. The risk prediction model for cardiovascular mortality performed moderately well [C-statistic 0.74 (standard error 0.11)] in identifying those at high risk for cardiovascular death, whereas for recurrent CVD events, it performed poorly in predicting the next CVD event [C-statistic 0.63 (standard error 0.06)], over a 20-month follow-up. The calibration curve showed poor agreement indicating that the REACH model underestimated both recurrent CVD risk and cardiovascular death. With recalibration, the REACH cardiovascular death prediction model could potentially be used to identify patients who would benefit from aggressive risk reduction.

Introduction

Prediction models assist health-care providers in making important daily patient care decisions by population screening, assessing responses to treatment, intensifying management, and motivating patients toward behavior change [1]. In addition, prediction models help risk-stratify patients into low, moderate, high, or very high-risk groups for appropriate prevention and management strategies [1, 2]. Those deemed high- or very-high-risk individuals need far more aggressive management and monitoring [2]. Having the diagnosis of cardiovascular disease (CVD) puts patients at an increased risk for a recurrent CVD event [3]. In a previous study, ischemic heart disease was projected to increase by 138% and 158% for women and men, respectively, from 1990 to 2020 in the Middle East [4]. Although alarming, these figures are likely to increase or decrease in the future as the pattern of CVD risk continues to change in most of the developing countries [4, 5].

Wilson et al. developed a risk prediction model for recurrent CVD events or cardiovascular death, with participants from the REduction of Atherothrombosis for Continued Health (REACH) Registry [6]. The researchers conducted and internally validated their risk prediction tool on 68,000 patients from 44 countries. These include patients from the Middle East [7], thus making it the most globally inclusive, ethnically diverse, and geographically extensive registry of CVD patients. The patient population from the Middle East made up 1.25% (850) of the total sample. It included countries such as Saudi Arabia [198 participants (0.29%)] and the United Arab Emirates (UAE) [149 participants (0.22%)] [7]. This study revealed that the risk for recurrent CVD was high among the Middle Eastern population [6]. Therefore, based on the geographical region and availability of similar data as defined in the REACH model, it played a significant role in our decision to externally validate this risk tool in our clinical setting. Empirical evidence shows that models tend to perform better when internally validated and not as well in other cohorts in which the model may be applied [8,9,10,11]. This illustrates the need for external validation (discrimination and calibration) among patients exclusively from the UAE in order to assess its fitness for recurrent CVD risk prediction. Thus, this study aimed to externally validate the REACH prediction tool among UAE nationals with established CVD.

Main text

Methods

Study design, data source, and study cohort

This is an external validation study. The study site was Tawam Hospital’s outpatient clinics. Tawam Hospital is the largest publicly funded teaching hospital in Al Ain providing health care exclusively to UAE nationals. It is affiliated with the College of Medicine and Health Sciences, UAE University. UAE nationals are defined as the citizens of the UAE.

All UAE nationals enrolled in this study were diagnosed with a CVD event, who were visiting the outpatient clinics over a 9-month period in 2008. The recruitment was done retrospectively using electronic medical records (EMRs). The inclusion criteria for this study were matched to that of the developmental REACH cohort [6]. It included patients aged 45 years and older with established CVD. Patients with CVD are defined as patients with cerebrovascular disease, coronary artery disease (CAD), or peripheral vascular disease (PVD), based on the documented diagnosis by a physician. There were 206 patients who met the inclusion criteria, but 2 patients with missing data at baseline were excluded.

Baseline characteristics

Baseline data were obtained from the EMRs. The sociodemographic and clinical variables were as follows: age, sex, body mass index (BMI), smoking status, number of vascular beds involved, geographic region (in this case, the UAE), and history of diabetes mellitus, congestive heart failure (CHF), atrial fibrillation (AF), a baseline cardiovascular event (defined as a documented diagnosis of myocardial infarction or cerebrovascular disease) occurring in the previous 24 months, antihypertensive and lipid-lowering medications, and acetylsalicylic acid therapy. CHF and AF were not defined as a CVD event by Wilson et al. [6] In this study, smokers had either a history of smoking or currently smoking, and BMI was stratified as < 20, 20–30, and > 30 kg/m2. Diabetes mellitus was established by a documented clinician’s diagnosis. Patients were classified as having hypertension, when they have a diastolic blood pressure ≥ 90 mmHg or systolic blood pressure ≥ 140 mmHg or are on antihypertensive medications.

Endpoints

Although participants were followed up for 24 months, the cutoff used was 20 months as defined by Wilson et al. [6] Cardiovascular death was defined as the occurrence of a fatal MI or fatal stroke, whereas a recurrent CVD event was the occurrence of cardiovascular death, a cerebrovascular disease, or a myocardial infarction (MI) event.

Data analysis

Baseline characteristics of the validation cohort were compared to the developmental cohort. Student’s t-test was used for continuous variables, while Fisher’s exact test or chi-square was used for categorical variables. The time between study enrolment and the first occurrence of a recurrent CVD event during the follow-up period was defined as the months at risk for developing the endpoints.

The probability of each participant developing each endpoint was calculated by applying the regression coefficients of the original Cox regression models for recurrent CVD and cardiovascular death in the developmental study. External validation is performed by evaluating the REACH model’s predictive performance by calibration and discrimination [12]. Calibration and discrimination were evaluated at 20 months of follow-up. Discrimination is defined as the capability to correctly differentiate between two participants based on a risk score, i.e., one who will not develop the endpoint and one who will over the follow-up duration [13]. On the other hand, calibration is defined as the degree of agreement between the risks predicted by the model and the observed frequencies of the outcomes under study. Discrimination was calculated using Harrell’s C-statistic with standard error (SE) [14]. Calibration was assessed graphically by comparing the predicted 20-month events with the observed events, grouped by quintiles of predicted probabilities. The R software version 3.5.2, using the packages “rms” and “riskRegression,” was used to analyze our data.

Results

Table 1 shows the baseline characteristics of both the developmental [6] and validation cohorts. The validation sample size included 204 participants, of which 77% were men, with a mean age of 67 years at baseline. Participants with CAD, cerebrovascular disease, and PVD together formed the CVD burden at 69.6, 38.2, and 6.4%, respectively. On the other hand, those with one vascular bed involvement were the most prevalent at 87.3%. Among the study cohort, 32% had experienced a CVD event in the previous year, whereas 10.3 and 6.4% had CHF and AF, respectively. Almost 26% were smokers, and approximately 63% were patients with diabetes mellitus. A very high percentage were on antihypertensive medications (93.6%), lipid-lowering medications (85.3%), and only 52.5% on acetylsalicylic acid (commonly known as aspirin).

Table 1 Baseline characteristics comparison between the REACH and external validation cohorts

Additional file 1: Table S1 outlines the number of recurrent CVD events and cardiovascular deaths observed with almost 4% having a non-fatal MI and 3% a fatal MI.

Discrimination

C-statistic was used to assess the association between the individuals from the validation sample and the developmental sample over 20 months. In the model performance analysis for recurrent CVD events, the C-statistic was 0.63 (SE 0.06) for a 20-month risk. Thus, the model performed poorly in predicting the next CVD event. On the other hand, for cardiovascular death, discrimination for a 20-month risk prediction was moderate, with a C-statistic of 0.74 (SE 0.11). Thus, the model performed moderately well to identify patients at the highest risk for cardiovascular mortality.

Calibration

The calibration curves for predicting the 20-month survival probability for recurrent CVD events and cardiovascular death are shown in Figs. 1 and 2, respectively. The calibration plot showed poor agreement between predicted recurrent CVD events and observed 20-month recurrent CVD event probabilities. In addition, the REACH model underestimated both the risk of recurrent CVD events and cardiovascular death.

Fig. 1
figure1

Calibration plot by quintiles of predicted and observed 20-month recurrent CVD events for the REACH prediction model. CVD cardiovascular disease, REACH REduction of Atherothrombosis for Continued Health

Fig. 2
figure2

Calibration plot by quintiles of predicted and observed 20-month cardiovascular death events for the REACH prediction model. CVD cardiovascular disease, REACH REduction of Atherothrombosis for Continued Health

Predicting death using the risk prediction model: an example

Developed by Wilson et al. [6], the recurrent risk prediction tool included the sum of numerical values assigned to the risk factors present in a patient to predict the next cardiovascular event. An arbitrary case in practice of a 45-year-old UAE man, with a history of diabetes mellitus, smoking, CHF, BMI > 20 kg/m2, and involvement of a single vascular bed, would have a 20-month risk of cardiovascular death of approximately 4.9%.

Discussion

This study represents the first external validation of the REACH prediction model for recurrent CVD exclusively for UAE nationals. The external validation results performed moderately well in discriminating those participants likely to die from a recurrent CVD event from those who were more likely to survive, whereas the calibration performed poorly by underestimating the cardiovascular mortality risk, over a 20-month period of follow-up. This outcome is not unusual as research has shown that prognostic tools developed to predict mortality are less than perfect [15]. Moreover, CVD risk prediction tools show a reduced accuracy when validated in new cohorts, mainly due to the differences in characteristics between the developmental and validation cohorts [16].

The notable differences between the developmental cohort and our study cohort could explain the underestimation of predicted risk of recurrent CVD and cardiovascular death seen in this study. In this study, patients were more likely to smoke and have diabetes mellitus, and only half were on aspirin therapy. Furthermore, the distribution of participants in the developmental model was across 44 countries. The types of included clinical practices were far more extensive, covering both urban and rural areas [7]. Despite this, with recalibration, this model could have the potential to help clinicians identify individuals at the greatest short-term risk and, therefore, benefit from intense follow-up and management.

Conclusion

Using the REACH model, recurrent CVD risk prediction was found to demonstrate a modest discriminative capacity for predicting death, with an underestimation of CVD risk in the short term. Although the current application may fail to identify high-risk individuals, with recalibration, it may still be of value for close monitoring, intensification of risk factor management, and tertiary prevention. Following international guidelines for CVD, management should be the ultimate goal of primary care physicians and specialists alike.

Limitations

The accuracy of documentation has to be accepted when using retrospective data. Differences in results may be attributed to a homogeneous case mix in our validation set. Also, the relatively low number of events may have affected the reliability of the analysis [17]. Despite the small sample size and recruitment of UAE nationals from a single center, the performance with the new data may be sufficient for the model to be potentially useful in our population until further validation studies are conducted. We encourage other researchers to also externally validate the REACH risk prediction tool, particularly with respect to recurrent CVD events. Nevertheless, this is the first study that externally validated a model for clinicians treating an increasing number of high-risk patients with CVD in this region.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

AF:

Atrial fibrillation

BMI:

Body mass index

BP:

Blood pressure

CVD:

Cardiovascular disease

CAD:

Coronary artery disease

CHF:

Congestive heart failure

CI:

Confidence intervals

EMRs:

Electronic medical records

PVD:

Peripheral vascular disease

SE:

Standard error

UAE:

United Arab Emirates

References

  1. 1.

    Smith GC, Seaman SR, Wood AM, Royston P, White IR. Correcting for optimistic prediction in small data sets. Am J Epidemiol. 2014;180:318–24.

    Article  Google Scholar 

  2. 2.

    Lin JS, Evans CV, Grossman DC, Tseng CW, Krist AH. Framework for using risk stratification to improve clinical preventive service guidelines. Am J Prev Med. 2018;54:S26-37.

    Article  Google Scholar 

  3. 3.

    Lin FJ, Tseng WK, Yin WH, Yeh H-I, Chen J-W, Wu C-C. Residual risk factors to predict major adverse cardiovascular events in atherosclerotic cardiovascular disease patients with and without diabetes mellitus. Sci Rep. 2017;7:9179.

    Article  Google Scholar 

  4. 4.

    Yusuf S, Reddy S, Ounpuu S, Anand S. Global burden of cardiovascular diseases: part 1: general considerations, the epidemiologic transition, risk factors, and impact of urbanization. Circulation. 2001;104:2746–53.

    CAS  Article  Google Scholar 

  5. 5.

    Gaziano TA, Bitton A, Anand S, Abrahams-Gessel S, Murphy A. Growing epidemic of coronary heart disease in low- and middle-income countries. Curr Probl Cardiol. 2010;35:72–115.

    Article  Google Scholar 

  6. 6.

    Wilson PW, D’Agostino R, Bhatt DL, Eagle K, Pencina MJ, Smith SC, Alberts MJ, Dallongeville J, Goto S, Hirsch AT, Liau CS, Ohman EM, Röther J, Reid C, Mas JL, Steg PG, REACH Registry. An international model to predict recurrent cardiovascular disease. Am J Med. 2012;125:695–703.

    Article  Google Scholar 

  7. 7.

    Bhatt DL, Steg PG, Ohman EM, Hirsch AT, Ikeda Y, Mas JL, Goto S, Liau CS, Richard AJ, Röther J, Wilson PW, REACH Registry Investigators. International prevalence, recognition, and treatment of cardiovascular risk factors in outpatients with atherothrombosis. JAMA. 2006;295:180–9.

    CAS  Article  Google Scholar 

  8. 8.

    Kaasenbrood L, Boekholdt SM, van der Graaf Y, Ray KK, Peters RJ, Kastelein JJ, Amarenco P, LaRosa JC, Cramer MJ, Westerink J, Kappelle LJ, de Borst GJ, Visseren FL. Distribution of estimated 10-year risk of recurrent vascular events and residual risk in a secondary prevention population. Circulation. 2016;134:1419–29.

    Article  Google Scholar 

  9. 9.

    Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KGM. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J ClinEpidemiol. 2015;68:279–89.

    Google Scholar 

  10. 10.

    Nieboer D, van der Ploeg T, Steyerberg EW. Assessing discriminative performance at external validation of clinical prediction models. PLoS ONE. 2016;11:e0148820.

    Article  Google Scholar 

  11. 11.

    van Eekelen R, McLernon DJ, van Wely M, Eijkemans MJ, Bhattacharya S, van der Veen F, van Geloven N. External validation of a dynamic prediction model for repeated predictions of natural conception over time. Hum Reprod. 2018;133:2268–75.

    Google Scholar 

  12. 12.

    Royston P, Altman DG. External validation of a cox prognostic model: principles and methods. BMC Med Res Methodol. 2013;13:33.

    Article  Google Scholar 

  13. 13.

    Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional andnovel measures. Epidemiology. 2010;21:128–38.

    Article  Google Scholar 

  14. 14.

    Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.

    Article  Google Scholar 

  15. 15.

    Siontis GCM, Tzoulaki I, Ioannidis JPA. Predicting death: An empirical evaluation of predictive tools for mortality. Arch Intern Med. 2011;171:1721–6.

    Article  Google Scholar 

  16. 16.

    Mansell H, Stewart SA, Shoker A. Validity of cardiovascular risk prediction models in kidney transplant recipients. Sci World J. 2014;2014:750579.

    Article  Google Scholar 

  17. 17.

    Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35:214–26.

    Article  Google Scholar 

Download references

Acknowledgements

We are indebted to Shehab Majud for his helpful comments on the language in the manuscript.

Funding

This study is funded by the UAE University (grant number: G00002956).

Author information

Affiliations

Authors

Contributions

RDG conceptualized the research topic. SAS designed and conducted all analyses. Both RDG and SAS drafted the initial manuscript and reviewed and revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Romona D. Govender.

Ethics declarations

Ethics approval and consent to participate

Ethical approval to review and use the data was obtained from the United Arab Emirates University and Tawam Hospital Review Boards (CRD 239/13). Patient information was anonymized prior to the analyses to protect the patient’s identity, and thus, the informed consent was waived.

Consent to publish

Not applicable.

Competing interests

The authors have no potential competing interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: Table S1.

Frequency of recurrent CVD eventsa and cardiovascular deathb.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Al-Shamsi, S., Govender, R.D. Validation of the REduction of Atherothrombosis for Continued Health (REACH) prediction model for recurrent cardiovascular disease among United Arab Emirates Nationals. BMC Res Notes 13, 484 (2020). https://doi.org/10.1186/s13104-020-05331-8

Download citation

Keywords

  • Calibration
  • Discrimination
  • External validation
  • Prediction model
  • Recurrent cardiovascular disease
  • United arab emirates