- Research Note
- Open access
- Published:
Evaluating understandability and actionability of online education materials for home-care patients with COVID-19 in Japan
BMC Research Notes volume 16, Article number: 291 (2023)
Abstract
Objective
In Japan, educational materials on the home care of coronavirus disease 2019 (COVID-19) were developed owing to limited access to medical care during the pandemic. This study quantitatively evaluated the understandability, actionability, natural flow, and readability of 87 materials published by local governments in Japan for patients with COVID-19. Their understandability and actionability were rated using the Japanese version of the Patient Education Material Evaluation Tool for Printed Materials (PEMAT-P). Natural flow and readability were rated using Global Quality Score (GQS) and jReadability, respectively.
Results
Of the 87 materials, 55 (62.1%) were understandable and 33 (37.9%) were actionable according to the PEMAT-P. Regarding understandability, the materials used medical terms without providing definitions and lacked summaries. Regarding actionability, the materials did not demonstrate explicit steps or utilize visual aids to help the readers take action. The mean (SD) of GQS was 3.44 (0.98), indicating a moderate level of naturalness and comprehensiveness of the materials. The mean (SD) score for readability was 2.4 (0.6), indicating a “lower advanced” level. However, challenges regarding the materials’ plain language remained, such as defining medical terms, summarizing the content for understandability, and using charts and tables that encourage patients to act.
Introduction
The coronavirus disease 2019 (COVID-19) outbreak caused a global public health crisis and was declared as a pandemic on March 11, 2020 by the World Health Organization (WHO). By April 1, 2023, approximately 26.5% of the population in Japan (33,462,859 people) had been diagnosed with COVID-19 [1]. According to the WHO, self-isolation and home care was suggested for asymptomatic or mild/moderate COVID-19 patients without risk factors [2]. However, considering the difficulty in accessing medical facilities and communicating directly with healthcare professionals, patients were at an increased risk of being exposed to misinformation. Therefore, information with understandable content on home care and recommendations of suitable actions was urgently required.
Several studies have evaluated the quality of websites associated with COVID-19. Kruse et al. [3] evaluated 141 COVID-19-related materials published by the United States’ academic hospitals and reported that their understandability was good but actionability was poor. Higashi et al. evaluated 50 websites on COVID-19 for patients with cancer in the United States and reported an overall understandability and actionability score of 82%. However, there were persistent difficulties, such as the use of medical terminology and absence of visual aids, which increases the ease of understanding the content [4]. These studies assessed materials explaining general information about COVID-19 from the initial days of the pandemic. Considering that three years have passed since the declaration of the pandemic, the lay public requires tailor-made information. Therefore, messages specific to COVID-19 educational materials include information on ways to manage symptoms without direct access to medical care, measures required in case the disease worsens at home, and ways to prevent the spread of infection in households. In Japan, these messeges distributed from local government were based on guidelines by Ministry of Health, Labor and Welfare. Additionally, many online educational materials were developed to assist patients who were unable to visit medical institutions during quarantine or self-isolation [5]. However, to the best of our knowledge, there was no comprehensive analysis of the quality of Japanese-language material on COVID-19. Hence, our study evaluated whether educational materials on COVID-19 home care in Japan were easily understandable and supported the health behavior of infected individuals.
Materials and methods
Study design
After revision
The COVID-19-related websites from the local governments (prefectures, ordinance-disignated cities and Tokyo metropolitan districts) were systematically and quantitatively analyzed using web accessibility evaluation tools. The Research Ethics Committee of the University of Tokyo Graduate School of Medicine and Faculty of Medicine waived informed consent for this study because educational materials on the website are publicly accessible and do not involve patient records.
Study sample
We evaluated educational materials on home care of COVID-19 patients available on local government websites on June 25, 2022. In Japan, COVID-19 patients registered themselves using the Ministry of Health, Labor and Welfare’s Health Center Real-time Information-Sharing System on COVID-19 (HER-SYS) via the website of the local government of their area. Local public health centers monitored patients’ conditions based on the information recorded in the HER-SYS [6]. In the present study, we searched for materials from COVID-19-related portal sites for all prefectures (n = 47), ordinance-designated cities (n = 20), and Tokyo Metropolitan Districts with populations of 500,000 or more (n = 8) to ensure comprehensiveness. The analyzed materials included webpages containing instructions for the home care of COVID-19 patients and leaflets in PDF format that were included on the webpage. All these materials were free to view or download.
Evaluation methods
Understandability and actionability
The Japanese version of the Patient Education Material Evaluation Tool for Printable Materials (PEMAT-P) was used to evaluate the materials’ understandability and actionability [7]. The PEMAT-P evaluates printable materials, i.e., brochures, booklets, and online materials in HTML or PDF format [8]. It is intended to be used by those who provide educational materials to patients or consumers, including healthcare providers, health librarians, and staff of governmental agencies. The PEMAT-P has two subdomains: (1) understandability, which measures how well the written material is understood by health consumers from diverse backgrounds, and (2) actionability, which measures how well consumers can identify what they need to do based on the information provided. It includes 23 items (16 items for understandability and seven for actionability) with a binary scale (agree = 1 or disagree = 0) (Additional file 1). The PEMAT-P understandability and actionability scores are calculated by adding all the points, dividing by the total possible points (excluding not applicable items), and multiplying by 100 to obtain a percentage. The threshold for both subdomains was set at 70% [9].
Natural flow and comprehensiveness
Each webpage was rated using the Global Quality Score (GQS) to evaluate the overall natural flow and comprehensiveness of the information on a five-point Likert scale (Additional file 2) [10]. It has been commonly used to evaluate health and medical information websites [11,12,13]. The GQS consists of a single item, while a score of one point indicates the poorest quality, a score of five points indicates excellent quality.
Readability
We additionally used readability, an objective measure to evaluate textual information. The plain text from each webpage was extracted, and any formatting elements that might interfere with readability assessment (headings, symbols, author information, and references) were removed. The text was then assessed using jReadability, an online readability evaluation system [14]. This tool automatically calculates readability based on the average length of sentences, difficulty level of words, proportion of grammatical parts of speech, and types of characters per sentence (Additional file 3).
Statistical analysis
Descriptive statistics were used to summarize the characteristics of the retrieved websites and calculate PEMAT-P, GQS, and readability scores. Since the PEMAT-P and GQS are subjective measures, we measured inter-rater reliability. Two physicians (EF and NS) with experience in creating patient education materials independently evaluated the material for a quarter of the entire webpages. These webpages were selected by a random number table created in Microsoft Excel. After the inter-rater reliability (Gwet’s AC1) was calculated, EF evaluated the rest of the materials. All statistical analyses were conducted using R (version 4.1.1; 2021-08-10).
Results
This study included 87 materials from 60 local governments. Table 1 shows the demographics of local governments. Overall, 55 (62.1%) materials were understandable and 33 (37.9%) were actionable according to the PEMAT-P. The mean understandability and actionability scores were 73.2% and 62.6%, respectively. PEMAT-P scores between groups were shown in Additional file 4.
Table 2 presents the scores for each PEMAT-P item. The highest percentage of the materials met the criteria for Item 1 on understandability: “The material made its purpose completely evident from the beginning (99%),” while the lowest percentage was for Item 10: “The material provided a summary (16%).” For actionability, most materials met the criteria for Item 19: “The material clearly identified at least one action that the user could perform (100%),” while the least materials met the criteria for Item 24: “The material explained how to use charts, graphs, tables, or diagrams to take actions (36%).”
The mean (SD) score of GQS was 3.44 (0.98), and range was 1 to 5. The mean score indicates a moderate level of naturalness and comprehensiveness of the materials. High inter-rater reliability was obtained for each of the PEMAT-P items (Gwet’s AC1 = 0.71-1.00) and GQS (Gwet’s AC1 = 0.76). The mean (SD) score for readability was 2.4 (0.6), indicating a “lower advanced” level. This score indicates that readers should have language skills to understand the complex structures found in the Japanese literature to comprehend the material.
Discussion
More than half of the included materials met the criteria for understandability; however, less than 40% of the materials were rated as actionable. The PEMAT-P scores in this study did not deviate much from previous studies [3, 4, 15,16,17].
This study confirmed the issues identified in previous studies which analyzed COVID-19 home care materials developed by governmental agencies [3, 4, 15,16,17,18]. First, concerning understandability, the materials used medical terms, such as “SpO2,” “specimen collection,” “risk factors,” “underlying disease,” and “mild/severe disease,” without including definitions or explanations. Additionally, most materials did not include summaries, making it difficult for readers to understand the main points at a glance [4]. Second, concerning actionability, the materials did not demonstrate explicit steps to help readers take action [4, 16]. Moreover, they did not use visual aids (i.e., tables, charts, illustrations, or diagrams) in situations that could promote the recommended actions [18]. Moreover, they did not use visual aids (i.e., tables, charts, illustrations, or diagrams) in situations that could promote the recommended actions [18]. This reflects a lack model [19], which states, “if the experts fill in the gaps in citizens’ knowledge, they will accept what the experts said”. However, we cannot expect the audience to act when they are only provided with knowledge. It is necessary to deliver messages about the behaviors in a way that increases self-efficacy for the audience, as in the actionability items listed in PEMAT.
The comprehensiveness and natural flow of the materials were moderate, and there were no substantial quality gaps between local governments. The readability level of the materials was lower at the advanced level. These findings were consistent with those of previous studies examining online information on COVID-19, which showed that the analyzed webpages mostly required a higher reading level than the recommended six-grade reading level [20,21,22,23,24]. This poor readability level is concerning because laypeople considered materials from local governments as a major source of health information during the COVID-19 pandemic.
In conclusion, the COVID-19 home care materials developed by the Japanese local government satisfied the criteria of understandability and actionability. The comprehensiveness and natural flow of the materials were moderate. However, as in previous studies, challenges were identified in defining medical terms, summarizing the content for understandability, and the use of charts and tables that could encourage patients to take action. The materials were somewhat difficult for non-native Japanese speakers or those without higher education to comprehend. While access to medical care was limited during the COVID-19 pandemic, COVID-19 home care information should be disseminated to the general public in an understandable and actionable manner.
Limitations
While patients with COVID-19 may use resources outside public institutions, this study only evaluated materials from local governments. Additionally, the understandability and actionability of educational materials were measured at one distinct timepoint in this study. Therefore, it does not reflect the most recent data. Hence, further studies are required to evaluate the latest data and conduct a longitudinal comparative study to determine improvements. The Japanese version of PEMAT has been verified for predictive validity through surveys of the general public. However, the items of the PEMAT do not fully reflect the patient’s perspective. Therefore, the opinions of the audience should be evaluated elsewhere. Future research is needed to qualitatively assess how audience feel about the materials, or how they likely to take actions recommended in the materials.
Data Availability
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Abbreviations
- WHO:
-
The World Health Organization
- COVID-19:
-
The coronavirus disease 2019
- HER-SYS:
-
The Ministry of Health, Labor and Welfare’s Health Center Real-time Information-Sharing System on COVID-19
- PEMAT-P:
-
Patient Education Material Evaluation Tool for Printed Materials
- GQS:
-
The Global Quality Score
References
Ministry of Health, Labour and Welfare. Eleven things you need to know NOW about COVID-19. 2023. Available from: https://www.mhlw.go.jp/content/000927280.pdf
WHO. Considerations for quarantine of contacts of COVID-19 cases. Emergencies Preparedness, WHO Headquarters (HQ); 2021 25 June. Report No.: WHO-2019-nCoV-IHR-Quarantine-2021.1.
Kruse J, Toledo P, Belton TB, Testani EJ, Evans CT, Grobman WA, et al. Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States. Am J Infect Control. 2021;49(6):690–3.
Higashi RT, Sweetenham JW, Israel AD, Tiro JA. COVID-19 Communication from Seven Health Care Institutions in North Texas for English- and spanish-speaking Cancer patients: mixed method website study. JMIR Cancer. 2021;7(3):e30492.
Self-Isolation Handbook for COVID-19 Patient. In: Bureau of Social Welfare and Public Health TMG, editor. 2021.
Ministry of Health., Labour and Welfare. Health Center Real-time Information-sharing System on COVID-19 (HER-SYS). 2022. Available from: https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/0000121431_00181.html
Furukawa E, Okuhara T, Okada H, Shirabe R, Yokota R, Iye R, et al. Translation, cross-cultural adaptation, and validation of the Japanese Version of the Patient Education materials Assessment Tool (PEMAT). Int J Environ Res Public Health. 2022;19(23):15763.
The Patient Education Materials Assessment Tool (PEMAT). and User’s Guide. 2014. Available from: https://www.ahrq.gov/health-literacy/patient-education/pemat.html
Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96(3):395–403.
Bernard A, Langille M, Hughes S, Rose C, Leddin D, Veldhuyzen van Zanten S. A systematic review of patient inflammatory bowel Disease information resources on the world wide web. Am J Gastroenterol. 2007;102(9):2070–7.
Li M, Yan S, Yang D, Li B, Cui W. YouTube™ as a source of information on food Poisoning. BMC Public Health. 2019;19(1).
Chen Z, Pan S, Zuo S. TikTok and YouTube as sources of information on anal fissure: a comparative analysis. Front Public Health. 2022;10:1000338.
Jung MJ, Seo MS. Assessment of reliability and information quality of YouTube videos about root canal treatment after 2016. BMC Oral Health. 2022;22(1):494.
jReadability P. [cited 2021 October 1]. Available from: https://jreadability.net/
Georgsson S, Carlsson T. Readability, understandability and language accessibility of Swedish websites about the coronavirus Disease 2019: a cross-sectional study. BMC Med Inform Decis Mak. 2022;22(1).
Bakker MM, Luttikhuis T, Putrik P, Jansen I, Rademakers J, De Wit M et al. Consideration of health literacy in patient information: a mixed-methods study of COVID-19 crisis communication in Dutch rheumatology. BMC Rheumatol. 2022;6(1).
Fulmer AA, Abboud GA 2nd, Wallace LS. Health literacy characteristics of over-the-counter rapid antigen COVID-19 test materials. Res Social Adm Pharm. 2022;18(12):4124–8.
Mani NS, Ottosen T, Fratta M, Yu F. A health literacy analysis of the consumer-oriented COVID-19 information produced by ten state health departments. J Med Libr Assoc. 2021;109(3).
House of Lords Select Committee on Science and Technology. House of Lords - Science and Technology - Third Report 2000 [cited 2023 July 15]. Available from: https://publications.parliament.uk/pa/ld199900/ldselect/ldsctech/38/3801.htm
Serry T, Stebbins T, Martchenko A, Araujo N, McCarthy B. Improving access to COVID-19 information by ensuring the readability of government websites. Health Promot J Austr. 2022.
Worrall AP, Connolly MJ, O’Neill A, O’Doherty M, Thornton KP, McNally C, et al. Readability of online COVID-19 health information: a comparison between four English speaking countries. BMC Public Health. 2020;20(1):1635.
Valizadeh-Haghi S, Khazaal Y, Rahmatizadeh S. Health websites on COVID-19: are they readable and credible enough to help public self-care? J Med Libr Assoc. 2021;109(1).
Mac OA, Muscat DM, Ayre J, Patel P, McCaffery KJ. The readability of official public health information on COVID-19. Med J Aust. 2021;215(8):373–5.
Basch CH, Mohlman J, Hillyer GC, Garcia P. Public Health Communication in Time of Crisis: readability of On-Line COVID-19 information. Disaster Med Public Health Prep. 2020;14(5):635–7.
Acknowledgements
We would like to thank Editage (www.editage.com) for English language editing.
Funding
This work was supported by the Japan Society for the Promotion of Science KAKENHI (grant number 20K10397).
Author information
Authors and Affiliations
Contributions
Conceptualization, E.F. and T.O.; methodology, E.F., H.O., and T.O.; software, E.F.; validation, E.F., H.O., and T.O.; formal analysis, E.F.; investigation, E.F.; resources, E.F.; data curation, E.F. and N.S.; writing—original draft preparation, E.F.; writing—review and editing, E.F. and T.O.; visualization, E.F.; supervision, T.O.; project administration, T.K; funding acquisition, H.O. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
This study was exempted from the Research Ethics Committee of The University of Tokyo Graduate School of Medicine and Faculty of Medicine because the educational material on the website is publicly accessible and does not involve patient records.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Furukawa, E., Okuhara, T., Okada, H. et al. Evaluating understandability and actionability of online education materials for home-care patients with COVID-19 in Japan. BMC Res Notes 16, 291 (2023). https://doi.org/10.1186/s13104-023-06570-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13104-023-06570-1