Evaluating understandability and actionability of online education materials for home-care patients with COVID-19 in Japan

Abstract

Objective

In Japan, educational materials on the home care of coronavirus disease 2019 (COVID-19) were developed owing to limited access to medical care during the pandemic. This study quantitatively evaluated the understandability, actionability, natural flow, and readability of 87 materials published by local governments in Japan for patients with COVID-19. Their understandability and actionability were rated using the Japanese version of the Patient Education Material Evaluation Tool for Printed Materials (PEMAT-P). Natural flow and readability were rated using Global Quality Score (GQS) and jReadability, respectively.

Results

Of the 87 materials, 55 (62.1%) were understandable and 33 (37.9%) were actionable according to the PEMAT-P. Regarding understandability, the materials used medical terms without providing definitions and lacked summaries. Regarding actionability, the materials did not demonstrate explicit steps or utilize visual aids to help the readers take action. The mean (SD) of GQS was 3.44 (0.98), indicating a moderate level of naturalness and comprehensiveness of the materials. The mean (SD) score for readability was 2.4 (0.6), indicating a “lower advanced” level. However, challenges regarding the materials’ plain language remained, such as defining medical terms, summarizing the content for understandability, and using charts and tables that encourage patients to act.

Introduction

The coronavirus disease 2019 (COVID-19) outbreak caused a global public health crisis and was declared as a pandemic on March 11, 2020 by the World Health Organization (WHO). By April 1, 2023, approximately 26.5% of the population in Japan (33,462,859 people) had been diagnosed with COVID-19 [1]. According to the WHO, self-isolation and home care was suggested for asymptomatic or mild/moderate COVID-19 patients without risk factors [2]. However, considering the difficulty in accessing medical facilities and communicating directly with healthcare professionals, patients were at an increased risk of being exposed to misinformation. Therefore, information with understandable content on home care and recommendations of suitable actions was urgently required.

Several studies have evaluated the quality of websites associated with COVID-19. Kruse et al. [3] evaluated 141 COVID-19-related materials published by the United States’ academic hospitals and reported that their understandability was good but actionability was poor. Higashi et al. evaluated 50 websites on COVID-19 for patients with cancer in the United States and reported an overall understandability and actionability score of 82%. However, there were persistent difficulties, such as the use of medical terminology and absence of visual aids, which increases the ease of understanding the content [4]. These studies assessed materials explaining general information about COVID-19 from the initial days of the pandemic. Considering that three years have passed since the declaration of the pandemic, the lay public requires tailor-made information. Therefore, messages specific to COVID-19 educational materials include information on ways to manage symptoms without direct access to medical care, measures required in case the disease worsens at home, and ways to prevent the spread of infection in households. In Japan, these messeges distributed from local government were based on guidelines by Ministry of Health, Labor and Welfare. Additionally, many online educational materials were developed to assist patients who were unable to visit medical institutions during quarantine or self-isolation [5]. However, to the best of our knowledge, there was no comprehensive analysis of the quality of Japanese-language material on COVID-19. Hence, our study evaluated whether educational materials on COVID-19 home care in Japan were easily understandable and supported the health behavior of infected individuals.

Materials and methods

Study design

After revision

The COVID-19-related websites from the local governments (prefectures, ordinance-disignated cities and Tokyo metropolitan districts) were systematically and quantitatively analyzed using web accessibility evaluation tools. The Research Ethics Committee of the University of Tokyo Graduate School of Medicine and Faculty of Medicine waived informed consent for this study because educational materials on the website are publicly accessible and do not involve patient records.

Study sample

We evaluated educational materials on home care of COVID-19 patients available on local government websites on June 25, 2022. In Japan, COVID-19 patients registered themselves using the Ministry of Health, Labor and Welfare’s Health Center Real-time Information-Sharing System on COVID-19 (HER-SYS) via the website of the local government of their area. Local public health centers monitored patients’ conditions based on the information recorded in the HER-SYS [6]. In the present study, we searched for materials from COVID-19-related portal sites for all prefectures (n = 47), ordinance-designated cities (n = 20), and Tokyo Metropolitan Districts with populations of 500,000 or more (n = 8) to ensure comprehensiveness. The analyzed materials included webpages containing instructions for the home care of COVID-19 patients and leaflets in PDF format that were included on the webpage. All these materials were free to view or download.

Evaluation methods

Understandability and actionability

The Japanese version of the Patient Education Material Evaluation Tool for Printable Materials (PEMAT-P) was used to evaluate the materials’ understandability and actionability [7]. The PEMAT-P evaluates printable materials, i.e., brochures, booklets, and online materials in HTML or PDF format [8]. It is intended to be used by those who provide educational materials to patients or consumers, including healthcare providers, health librarians, and staff of governmental agencies. The PEMAT-P has two subdomains: (1) understandability, which measures how well the written material is understood by health consumers from diverse backgrounds, and (2) actionability, which measures how well consumers can identify what they need to do based on the information provided. It includes 23 items (16 items for understandability and seven for actionability) with a binary scale (agree = 1 or disagree = 0) (Additional file 1). The PEMAT-P understandability and actionability scores are calculated by adding all the points, dividing by the total possible points (excluding not applicable items), and multiplying by 100 to obtain a percentage. The threshold for both subdomains was set at 70% [9].

Natural flow and comprehensiveness

Each webpage was rated using the Global Quality Score (GQS) to evaluate the overall natural flow and comprehensiveness of the information on a five-point Likert scale (Additional file 2) [10]. It has been commonly used to evaluate health and medical information websites [11,12,13]. The GQS consists of a single item, while a score of one point indicates the poorest quality, a score of five points indicates excellent quality.

Readability

We additionally used readability, an objective measure to evaluate textual information. The plain text from each webpage was extracted, and any formatting elements that might interfere with readability assessment (headings, symbols, author information, and references) were removed. The text was then assessed using jReadability, an online readability evaluation system [14]. This tool automatically calculates readability based on the average length of sentences, difficulty level of words, proportion of grammatical parts of speech, and types of characters per sentence (Additional file 3).

Statistical analysis

Descriptive statistics were used to summarize the characteristics of the retrieved websites and calculate PEMAT-P, GQS, and readability scores. Since the PEMAT-P and GQS are subjective measures, we measured inter-rater reliability. Two physicians (EF and NS) with experience in creating patient education materials independently evaluated the material for a quarter of the entire webpages. These webpages were selected by a random number table created in Microsoft Excel. After the inter-rater reliability (Gwet’s AC1) was calculated, EF evaluated the rest of the materials. All statistical analyses were conducted using R (version 4.1.1; 2021-08-10).

Results

This study included 87 materials from 60 local governments. Table 1 shows the demographics of local governments. Overall, 55 (62.1%) materials were understandable and 33 (37.9%) were actionable according to the PEMAT-P. The mean understandability and actionability scores were 73.2% and 62.6%, respectively. PEMAT-P scores between groups were shown in Additional file 4.

Table 1 Characteristics of the included local governments

Table 2 presents the scores for each PEMAT-P item. The highest percentage of the materials met the criteria for Item 1 on understandability: “The material made its purpose completely evident from the beginning (99%),” while the lowest percentage was for Item 10: “The material provided a summary (16%).” For actionability, most materials met the criteria for Item 19: “The material clearly identified at least one action that the user could perform (100%),” while the least materials met the criteria for Item 24: “The material explained how to use charts, graphs, tables, or diagrams to take actions (36%).”

Table 2 The mean score for each of the PEMAT-P item

The mean (SD) score of GQS was 3.44 (0.98), and range was 1 to 5. The mean score indicates a moderate level of naturalness and comprehensiveness of the materials. High inter-rater reliability was obtained for each of the PEMAT-P items (Gwet’s AC1 = 0.71-1.00) and GQS (Gwet’s AC1 = 0.76). The mean (SD) score for readability was 2.4 (0.6), indicating a “lower advanced” level. This score indicates that readers should have language skills to understand the complex structures found in the Japanese literature to comprehend the material.

Discussion

More than half of the included materials met the criteria for understandability; however, less than 40% of the materials were rated as actionable. The PEMAT-P scores in this study did not deviate much from previous studies [3, 4, 15,16,17].

This study confirmed the issues identified in previous studies which analyzed COVID-19 home care materials developed by governmental agencies [3, 4, 15,16,17,18]. First, concerning understandability, the materials used medical terms, such as “SpO₂,” “specimen collection,” “risk factors,” “underlying disease,” and “mild/severe disease,” without including definitions or explanations. Additionally, most materials did not include summaries, making it difficult for readers to understand the main points at a glance [4]. Second, concerning actionability, the materials did not demonstrate explicit steps to help readers take action [4, 16]. Moreover, they did not use visual aids (i.e., tables, charts, illustrations, or diagrams) in situations that could promote the recommended actions [18]. Moreover, they did not use visual aids (i.e., tables, charts, illustrations, or diagrams) in situations that could promote the recommended actions [18]. This reflects a lack model [19], which states, “if the experts fill in the gaps in citizens’ knowledge, they will accept what the experts said”. However, we cannot expect the audience to act when they are only provided with knowledge. It is necessary to deliver messages about the behaviors in a way that increases self-efficacy for the audience, as in the actionability items listed in PEMAT.

The comprehensiveness and natural flow of the materials were moderate, and there were no substantial quality gaps between local governments. The readability level of the materials was lower at the advanced level. These findings were consistent with those of previous studies examining online information on COVID-19, which showed that the analyzed webpages mostly required a higher reading level than the recommended six-grade reading level [20,21,22,23,24]. This poor readability level is concerning because laypeople considered materials from local governments as a major source of health information during the COVID-19 pandemic.

In conclusion, the COVID-19 home care materials developed by the Japanese local government satisfied the criteria of understandability and actionability. The comprehensiveness and natural flow of the materials were moderate. However, as in previous studies, challenges were identified in defining medical terms, summarizing the content for understandability, and the use of charts and tables that could encourage patients to take action. The materials were somewhat difficult for non-native Japanese speakers or those without higher education to comprehend. While access to medical care was limited during the COVID-19 pandemic, COVID-19 home care information should be disseminated to the general public in an understandable and actionable manner.

Limitations

While patients with COVID-19 may use resources outside public institutions, this study only evaluated materials from local governments. Additionally, the understandability and actionability of educational materials were measured at one distinct timepoint in this study. Therefore, it does not reflect the most recent data. Hence, further studies are required to evaluate the latest data and conduct a longitudinal comparative study to determine improvements. The Japanese version of PEMAT has been verified for predictive validity through surveys of the general public. However, the items of the PEMAT do not fully reflect the patient’s perspective. Therefore, the opinions of the audience should be evaluated elsewhere. Future research is needed to qualitatively assess how audience feel about the materials, or how they likely to take actions recommended in the materials.