Skip to main content

A retrospective study on the usefulness of the JJ risk engine for predicting the incidence rate of coronary heart disease in type 2 diabetes patients

Abstract

Objective

In 2018, we conducted a retrospective survey using the medical records of 484 patients with type 2 diabetes. The observed value of coronary heart disease (CHD) incidence after 5 years and the predicted value by the JJ risk engine as of 2013 were compared and verified using the discrimination and calibration values.

Results

Among the total cases analyzed, the C-statistic was 0.588, and the calibration was p < 0.05; thus, the JJ risk engine could not correctly predict the risk of CHD. However, in the group expected to have a low frequency of hypoglycemia, the C-statistic was 0.646; the predictability of the JJ risk engine was relatively accurate. Therefore, it is difficult to accurately predict the complication rate of patients using the JJ risk engine based on the diabetes treatment policy after the Kumamoto Declaration 2013. The JJ risk engine has several input items (variables), and it is difficult to satisfy them all unless the environment is well-equipped with testing facilities, such as a university hospital. Therefore, it is necessary to create a new risk engine that requires fewer input items than the JJ risk engine and is applicable to several patients.

Introduction

Currently, a risk engine that calculates the probability of CHD in type 2 diabetes patients is a useful prediction method. The Japan Diabetes Complications Study (JDCS)/the Japanese Elderly Diabetes Intervention Trial (J-EDIT) risk engine (JJ risk engine) was developed in Japan in 2012. This risk engine was developed using Japanese patient data; therefore, it is expected that its prediction accuracy will be higher for Japanese subjects than for Western subjects [1,2,3]. The JJ Risk Engine accurately predicts macro- and micro-vascular complications and provides useful information for risk classification and health economic simulations. The algorithm of this risk engine uses the multi-state Cox regression model [4].

However, since 2012 when the JJ risk engine was developed, diabetes treatment policies have changed significantly. Before 2010, a high premium was placed on decreasing blood sugar levels while treating diabetes, making it difficult to avoid hypoglycemia. Moreover, since 2010, hypoglycemia, particularly severe hypoglycemia, has become a recognized risk factor for the development of coronary heart disease (CHD) [5]. With the announcement of the “Kumamoto Declaration 2013,” the diabetes treatment policy has changed to one that emphasizes the prevention of hypoglycemia [6,7,8]. Accordingly, the use of antidiabetic drugs that are less likely to cause hypoglycemia has increased.

Therefore, given these changes regarding the diabetes treatment policy and the use of antidiabetic drugs, it can be considered that the frequency of hypoglycemia is lower currently than at the time when the JJ risk engine was developed. Hence, the number of CHD cases predicted by the JJ risk engine and the actual number may be different because the JJ risk engine predicted the risk of CHD using the data of patients receiving treatments that are less likely to cause hypoglycemia.

Here, we aimed to compare the JJ risk engine-predicted values with the actual complication incidences to verify whether changes in the diabetes treatment policy affected the incidence of complications in type 2 diabetes patients.

Main text

Methods

Patients

Selection criteria: The subjects were patients with type 2 diabetes who visited the Kitasato University Kitasato Institute Hospital, Diabetes Center, from January 2013 to December 2013, and were on continuous treatment for the following 5 years, until 2018.

Exclusion criteria: Patients who refused to participate in the study or had any of the following histories were excluded: angina, myocardial infarction, stroke, peripheral arterial disease, familial hypercholesterolemia, familial type III hyperlipidemia, nephrotic syndrome, renal diseases other than diabetic nephropathy, microhematuria, pre-proliferative and proliferative retinopathy, and major ocular diseases (e.g., glaucoma, dense cataract, or history of cataract surgery).

Further, patients with test values of serum creatinine > 1.3 mg/dL and urinary albumin ≥ 150 mg/gCr were also excluded.

Other exclusion criteria included patients who lacked the input items of the JJ risk engine and patients with non-type 2 diabetes such as borderline diabetes, type 1 diabetes, and gestational diabetes.

This study was conducted in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects. The Kitasato University Kitasato Institute Hospital, Research Ethics Committee, approved the study (Control number: 18061) and provided permission to review patient records as well as the use of the corresponding data. The option to opt-out of the study was provided to the patients at the start of the study (2018).

Statistical analysis

The incidence of CHD for each patient was calculated by entering patient data into the JJ Risk Engine web application [1].

Some assessment indices of the JJ Risk Engine are explained below:

Discrimination: an index that evaluates how accurately the presence or absence of an event can be predicted by a prediction model; the C-statistic, calculated on the basis of the receiver operating characteristic (ROC) curve, is used as a criterion for measuring the predictive accuracy [9, 10].

Calibration: an index to measure the degree of agreement between the prediction by the model and the actual outcome; the significance probability calculated by the Hosmer–Lemeshow test is used as the criterion for predictability. The significance level was set at 0.05 (p < 0.05) [10].

Sensitivity: Proportion of patients with the target condition who have a positive test result.

Specificity: Proportion of patients without the target condition who have a negative test result..

R version 2.5.1 (http://www.r-project.org, library Design, Hmisc, ROCR) was used for determining discrimination, calibration, sensitivity, and specificity, whereas the ROC curve and Hosmer–Lemeshow test were used for calculation [10, 11].

Comparative analysis

Observed value of CHD and predicted value by the JJ risk engine: all patients (verification 1):

To measure the degree of agreement between the observed value, which is the number of individuals developing CHD after 5 years (in 2018), and the predicted value by the JJ risk engine (as of 2013), the observed value/predicted value (O/P) ratio and the discrimination value were validated using calibration.

Observed value of CHD and predicted value by the JJ risk engine: frequency of hypoglycemia (verification 2):

Focusing on the risk of hypoglycemia, patients were categorized into two groups:

Group : Patients who use antidiabetic drugs prone to cause hypoglycemia alone (glimepiride, gliclazide, insulin).

Group : Patients not applicable to Group

The predictive ability of the JJ risk engine for CHD onset in each group was verified in the same way as verification 1, and comparisons were made between the two groups.

Results

Patient baseline demographic and clinical characteristics are shown in Table 1.

Table 1 Baseline characteristics

As shown in Table 2, the observed value was 15 individuals, and the predicted value by the JJ risk engine was 6.99 individuals; thus, the observed value was > the predicted value. The O/P ratio was 2.14, the C-statistic was 0.588, and the calibration was p < 0.05.

Table 2 Observed value of CHD and predicted value by the JJ risk engine: all patients

Similarly, in group (Table 3),the observed value was > the predicted value. On the other hand, in group , the O/P ratio was 0.81; therefore, the measured value the predicted value (Table 3).

Table 3 Observed value of CHD and predicted value by the JJ risk engine: frequency of hypoglycemia

Discussion

No large large-scale studies have been published showing fewer hypoglycemia cases after 2013 compared to the period before the development of the JJ Risk Engine. However, there is a growing number of drugs, such as sodium-glucose cotransporter 2 (SGLT2) inhibitors [12] and long-acting insulin [13], that are less likely to cause hypoglycemia when used alone, and the primary treatment strategy is to prevent hypoglycemia [6, 7].

These factors suggest fewer hypoglycemia cases today than before the development of the JJ risk engine.

Our study shows that the JJ risk engine could not correctly predict the risk of developing CHD, as indicated from the O/P ratios, discrimination, and calibration values. The calibration value could be calculated only in the “all patients” group, but not for the specifically divided patient groups because of the small sample size of the observed value therein [14].

It is assumed that the JJ risk engine predicts the risk of CHD onset in type 2 diabetes patients by considering the related risk factors and influencing factors. However, in our study, its risk predictive ability for the “all patients” group was poor; therefore, the effects of these factors may not be fully reflected in the prediction. Moreover, apart from the input items of the JJ risk engine, there are several risk factors and influencing factors involved in the development of CHD, such as high low-density lipoprotein (LDL) cholesterolemia, hypertriglyceridemia, family history of dyslipidemia, and hypoglycemia. The input items of the JJ risk engine do not include items related to hypoglycemia. Therefore, it was necessary to verify whether hypoglycemia affects the ability of the JJ risk engine to predict the onset of CHD.

As shown in Table 3, in group comprising patients expected to have a high frequency of hypoglycemia, the JJ risk engine could not correctly predict the risk of developing CHD, as evident from the O/P ratio, the discrimination, and the calibration values. In contrast, in group comprising patients expected to have a low frequency of hypoglycemia, the risk of developing CHD as predicted by the JJ engine is considered relatively accurate. In the current treatment policy, the priority is to prevent hypoglycemia. According to the Japan Diabetes Clinical Data Management Study Group's 2019 basic data [15], since 2013, HbA1c levels have stopped declining and have been on a gradual upward trend. It has been suggested that the concern regarding the prevention of hypoglycemia has led to poor control of HbA1c levels, which in turn has increased CHD risk. Therefore, we consider the observed value to be larger than the predicted value.

However, the actual state of hypoglycemia in each patient is not sufficiently reflected from these results. By classifying patients according to the risk of hypoglycemia in association with the use of antidiabetic drugs, we could not find a relationship between the predictive ability of the JJ risk engine for CHD onset and the frequency of hypoglycemia. We can specifically explain this with the results shown in Table 3; the JJ risk engine could not correctly predict the risk of developing CHD in group patients who were on SU drugs and low-dose insulin. However, the JJ risk engine predictability was high when tested in patients using high-dose insulin. To get closer to the actual state of hypoglycemia, it is necessary to further investigate hypoglycemia associated with the use of antidiabetic drugs, as we did in this study. We had further segregated group patients expected to have a high frequency of hypoglycemia based on their use of antidiabetic drugs. However, we were not able to verify this in the present study. In the SU drug group, there were 189 glimepiride users, of which more than 90% were using low doses (2 mg or less per day) [16], and there were 7 gliclazide users. Therefore, it was impossible to classify the SU drug group into a low-dose and a high-dose group. The insulin user group consisted of 80 patients, and the sample size was too small to analyze.

It was neither possible to find a relationship between the inaccuracy in the predictability of the JJ risk engine for CHD risk and the frequency of hypoglycemia nor identify the cause of the inaccuracy, and it is difficult to accurately predict the complication rate of patients using the JJ risk engine based on the diabetes treatment policy after the Kumamoto Declaration 2013.

The JJ risk engine has several input items, and it is difficult to fulfill them all unless the environment is well-equipped with testing facilities such as university hospitals. In particular, in Japan, urine albumin levels are rarely tested, which cannot be calculated unless a diagnosis of diabetic nephropathy has been made for insurance purposes [17, 18]. Therefore, it is necessary to create a new risk engine that requires fewer input items and is easily accessible to many patients. Since there are factors involved in CHD development other than those entered in the JJ risk engine, it would be useful to examine other factors as well [1].

Conclusion

Our study shows that the JJ risk engine could not accurately predict the risk for CHD onset in patients with type 2 diabetes in our hospital. Moreover, it is necessary to create a new risk engine that can accurately predict the risk of developing CHD—with fewer input items and a simpler technique than the JJ risk engine—and aligns with the current diabetes treatment approach.

Limitations

The relatively low number of people who develop CHD may affect the reliability of the analysis [19].

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to the privacy of the research participants but are available from the corresponding author on reasonable request. However, in this case, permission from The Kitasato University Kitasato Institute Hospital Research Ethics Committee is required.

Abbreviations

CHD:

Coronary heart disease

ROC:

Receiver operating characteristic

HbA1c:

Glycated hemoglobin

NGSP:

National Glycohemoglobin Standardization Program

SU:

Sulfonylurea

LDL:

Low-density lipoprotein

References

  1. 1.

    Tanaka S, Tanaka S, Iimuro S, Yamashita H, Katayama S, Akanuma Y, et al. Predicting macro- and microvascular complications in type 2 diabetes: the Japan diabetes complications study/the Japanese elderly diabetes intervention trial risk engine. Diabetes Care. 2013;36:1193–9.

    Article  Google Scholar 

  2. 2.

    Sone H, Katagiri A, Ishibashi S, Abe R, Saito Y, Murase T, et al. Effects of lifestyle modifications on patients with type 2 diabetes: the Japan diabetes complications study (JDCS) study design, baseline analysis and three year-interim report. Horm Metab Res. 2002;34:509–15.

    CAS  Article  Google Scholar 

  3. 3.

    Araki A, Iimuro S, Sakurai T, Umegaki H, Iijima K, Nakano H, et al. Long-term multiple risk factor interventions in Japanese elderly diabetic patients: the Japanese elderly diabetes intervention trial–study design, baseline characteristics, and effects of intervention. Geriatr Gerontol Int. 2012;12:7–17.

    Article  Google Scholar 

  4. 4.

    Tanaka S, Tanaka S, Iimuro S, Yamashita H, Katayama S, Akanuma Y, et al. Supplementary data. 2013. http://care.diabetesjournals.org/lookup/suppl/doi:10.2337/dc12-0958/-/DC1. Accessed 8 Feb Jan 2021.

  5. 5.

    Goto A, Arah OA, Goto M, Terauchi Y, Noda M. Severe hypoglycaemia and cardiovascular disease: systematic review and meta-analysis with bias analysis. BMJ. 2013;347:f4533.

    Article  Google Scholar 

  6. 6.

    Japan Diabetes Society. Treatment Guide for Diabetes, BUNKODO; 2020–2021.

  7. 7.

    Nakanishi S, Hirukawa H, Shimoda M, Tatsumi F, Kohara K, Obata A, et al. Verification of Kumamoto Declaration 2013 and glycemic targets for elderly patients with diabetes in Japan for prevention of diabetic complications: a retrospective longitudinal study using outpatient clinical data. J Diabetes Investig. 2019;10:290–301.

    Article  Google Scholar 

  8. 8.

    Araki A, Haneda M, Kasuga M, Nishikawa T, Kondo T, Ueki K, et al. New glycemic targets for patients with diabetes from the Japan Diabetes Society. Diabetol Int. 2016;7:327–30.

    Article  Google Scholar 

  9. 9.

    Ayumi Shintani. Everyone's medical statistics Complete mastery of basic theory and EZR in 12 days. KODANSHA; 2016

  10. 10.

    Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.

    Article  Google Scholar 

  11. 11.

    Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. Amsterdam: Wiley; 2000.

    Book  Google Scholar 

  12. 12.

    Scheen A. SGLT2 inhibitors: benefit/risk balance. Curr Diab Rep. 2016;16:92.

    Article  Google Scholar 

  13. 13.

    Horvath K, Jeitler K, Berghold A, Ebrahim S, Gratzer T, Plank J et al. Long-acting insulin analogues versus NPH insulin (human isophane insulin) for type 2 diabetes mellitus. Cochrane Database Syst Rev. 2007;(2):CD005613.

  14. 14.

    Meurer WJ, Tolles J. Logistic regression diagnostics: understanding how well a model predicts outcomes. JAMA. 2017;317:1068–9.

    Article  Google Scholar 

  15. 15.

    Japan Diabetes Clinical Data Management Study Group’s basic data. http://jddm.jp/data/index-2019/#data_03. Accessed 8 Feb, Jan 2021.

  16. 16.

    Amaryl® package insert. https://www.info.pmda.go.jp/go/pdf/780069_3961008F1020_1_26. Accessed 23 Oct, Jan 2020.

  17. 17.

    Ikegami N, Yoo BK, Hashimoto H, Matsumoto M, Ogata H, Babazono A, et al. Japanese universal health coverage: evolution, achievements, and challenges. Lancet. 2011;378:1106–15.

    Article  Google Scholar 

  18. 18.

    Hashimoto H, Ikegami N, Shibuya K, Izumida N, Noguchi H, Yasunaga H, et al. Cost containment and quality of care in Japan: is there atrade-off ? Lancet. 2011;378:1174–82.

    Article  Google Scholar 

  19. 19.

    Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35:214–26.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author information

Affiliations

Authors

Contributions

YY designed the study and wrote the initial draft of the manuscript. RK contributed to data collection and interpretation. KM, GI, and HM critically reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yasunari Yamashita.

Ethics declarations

Ethics approval and consent to participate

This study was conducted in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects. The Kitasato University Kitasato Institute Hospital, Research Ethics Committee, approved the study (Control number: 18061). For this retrospective study, we disclosed the purpose of the study to participants and provided the option of opt-out before participating in the study. All participants provided written informed consent to participate in this study. In addition, opt-out notices, rather than informed consent, provided an opportunity to refuse research participation.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yamashita, Y., Kitajima, R., Matsubara, K. et al. A retrospective study on the usefulness of the JJ risk engine for predicting the incidence rate of coronary heart disease in type 2 diabetes patients. BMC Res Notes 14, 92 (2021). https://doi.org/10.1186/s13104-021-05508-9

Download citation

Keywords

  • JJ risk engine
  • Type 2 diabetes
  • Coronary heart disease
  • Discrimination
  • Calibration
  • Hypoglycemia
  • C-statistic