What do letters to the editor publish about randomized controlled trials? A cross-sectional study
© Kastner et al.; licensee BioMed Central Ltd. 2013
Received: 28 January 2013
Accepted: 3 October 2013
Published: 14 October 2013
To identify published letters to the editor (LTE) written in response to randomized controlled trials (RCTs), determine the topics addressed in the letters, and to examine if these topics were affected by the characteristics and results of the RCTs.
Comparative cross-sectional study of a representative sample of RCTs from a set of high-impact medical journals (BMJ, Lancet, NEJM, JAMA, and Annals of Internal Medicine). RCTs and their published LTE were searched from these 5 journals in 2007. Data were collected on RCTs and their characteristics (author affiliation, funding source, intervention, and effect on the primary outcome) and the topics addressed in published LTE related to these RCTs. Analysis included chi-square and regression analysis (RCT characteristics) and thematic analysis (LTE topics).
Of 334 identified RCTs, 175 trials had at least one LTE. Of these, 381 published LTE were identified. Most RCTs, tested drug interventions (68%), were funded by government (54%) or industry (33%), and described an intervention that had a positive impact on the primary outcome (62%). RCT authors were primarily affiliated with an academic centre (78%). Ninety percent of the 623 LTE topics concerned methodological issues regarding the analysis, intervention, and population in the RCT. There was a significant association between funding source and impact on outcomes (p = 0.002) or type of intervention tested (p = 0.001) in these trials. Clinical and “Other” LTE topics were more likely to be published in response to a government funded RCT (p = 0.005 and p = 0.033, respectively); no other comparisons were significant.
This study showed that most LTE are about methodological topics, but found little evidence to support that these topics are affected by the characteristics or results of the RCTs. The lack of association may be explained by editorial censorship as a small proportion of LTE that are submitted are actually published.
Published letters to the editor (LTE) are short communications that report study data or more often provide viewpoints that support or challenge a published paper. LTE have also been used to teach critical appraisal skills for medical students  and to enhance the educational value of journal clubs . They are considered a form of post-publication peer review and an essential part of scientific debate [3–7]. LTE provide a platform to comment on, clarify and correct published research by alerting readers about aspects of a paper that may have been overlooked by authors, peer-reviewers and editors [5–8]. Some authors have argued that an article is not fully peer-reviewed until after publication [9, 10] and that authors of LTE may have more credibility than pre-publication peer reviewers because their opinions are signed, published and thus transparent .
However there is some skepticism regarding the representativeness of opinions expressed in published LTE [4–6] (for example, do the statements reflect a general opinion, a clinical practice experience in a given therapeutic area or a suggestion pertaining to the study methods?). Little is known about the topics commonly addressed in LTE and whether these topics are influenced by characteristics of the randomized controlled trial (RCT) (e.g. positive/negative outcomes, presence of serious harm, type of intervention, author affiliation, funding source, etc.). For example, LTE topics may represent opinions that are partially based on whether the RCT reported positive or negative results for the primary outcome. LTE topics may also be influenced by whether an RCT was financially supported by a government body, an academic institution or drug industry. For example, a negative trial of a new drug funded by a peer national granting agency may lead those with a vested interest in the outcome (e.g. company employees, academics whose careers are vested in the drug) to write negative comments about the methodology of the trial, which they might not have done had the trial results been positive. The primary objectives of this empirical study were 1) to identify the types of topics addressed in published LTE for a sample of RCTs, and 2) to examine if these topics were affected by characteristics of the RCTs.
Using a quantitative cross-sectional design, we used a representative sample of 5 high impact general medical journals that publish methodologically rigorous RCTs (BMJ, Lancet, NEJM, JAMA, and Annals of Internal Medicine), to consecutively select RCTs with published LTEs from each journal’s on-line websites (except for rapid responses published by BMJ) for a period of one year beginning with the first issue in January 2006. It was not possible to perform a formal sample size calculation, as there is insufficient evidence to estimate a magnitude of the expected difference in our outcome. However, based on a study with similar methods to investigate the content of LTE  (and given that our study was exploratory), we estimated that a sample size of approximately 200 published RCTs with at least one LTE would identify the majority of reasons published LTE are written, and allow for a statistical estimation of the relative frequency of these reasons. We considered only online-published LTE in our sample. To ensure that we captured all published LTE related to their trial to which they pertained, we estimated the publication time period for these letters by analyzing a sample set of 10 RCTs (2 from each journal) and their time to published LTE. We found that responses appeared 3-8 months after the publication of the original RCT, so we selected RCTs published at least one year prior to the beginning of our study period of 2007 (i.e., we searched for RCTs published in 2006). We performed the search for RCTs and their corresponding LTE in PubMed by “AND-ing” each of the 5 journals with the publication year of 2006 (e.g., “Lancet[Jour] AND 2006[pdat]). These yields were then filtered by AND-ing them with the term: “randomized controlled trial”. One reviewer scanned search yields for RCTs with at least one LTE. If no LTE was listed, we entering the RCT citation information in the “single citation matcher” feature of PubMed interface (http://www.ncbi.nlm.nih.gov/pubmed/citmatch) to verify if any correspondence related to the original RCT was available and accessible via hyperlinks.
Data collection and outcomes
RCTs: The following data were extracted from RCTs: the first author and their affiliation, funding source, the characteristics of the intervention (drug, other), and the results (effect upon the primary outcome and harms). Two researchers (MK & AM) abstracted data and classified the trial results independently using a combination of assessment of the benefits and harms and the interpretation of the results as provided and interpreted by the authors of the RCT. RCTs were divided into the following categories: positive impact upon primary outcome (i.e., intervention significantly better than control according to thresholds set by authors) and serious harms or no serious harms; negative impact upon the primary outcome (i.e., intervention is significantly worse than control); no impact upon the primary outcome (i.e., intervention neither better nor worse than control) and serious harms or no serious harms. Harms were recorded based on what authors reported in the results and discussion section of their paper. The purpose of this investigation was to identify any possible associations between the outcomes of the RCT and the topic(s) of the LTE that was written in response.
Letters to the Editor: We extracted data from published LTE including their topics and objectives, and how these themes related to the design and results of the RCT. Three broad topic categories were identified a priori by the research team: Methodological (topics related to the RCT’s study design, outcomes, population, intervention, and analysis); clinical (topics describing biological mechanisms, diagnosis, and treatment of the clinical area); and “other” (any aspect of the RCT not related to methodological or clinical topics).
Descriptive statistics were used to determine 5 RCT characteristics: impact on primary outcomes (positive, negative, none), presence of serious harms (harm, no harm), type of intervention tested (drug, other), author affiliation (i.e., academic centre, hospital, research institute, community practice/clinic, government, and industry), and funding source (government, academic, industry, other, not indicated). We also investigated the proportion of LTE by topic category within each of the 5 journals. RCT-level comparisons were analyzed using chi-square statistics to test associations between the impact on outcomes reported in the RCT (positive/negative/none) and its funding source; and the type of intervention tested in the RCT (drug/other) and its funding source using SPSS, version 17.0. All 5 RCT characteristics were considered of similar importance in the analysis. To account for RCTs with different numbers of LTE, we used Poisson regression analysis to test LTE-level comparisons. We tested associations between LTE topic and the following RCT characteristics: impact of the intervention on primary outcomes, presence of serious harm, type of intervention, funding source, and author affiliation. LTE topics were analyzed qualitatively using thematic analysis, which involved two researchers (MK and AM) independently classifying topics into 3 broad categories (methodological, clinical, other), and then iteratively identifying sub-categories and themes within these core categories. Two researchers (AL, SS) were consulted to resolve conflicts, and to verify the appropriateness and relevance of the final topic classifications and their labels.
Proportion of RCTs, LTE, and categories of LTE themes by journal*
Number of RCTs (%)
Number of LTEs (%)
Number of topics (%)
Categories of LTE topics (N = 623) (%)
Ninety percent of the topics discussed in LTE were methodological; most addressed the analysis (40%), intervention (21%), and population (13%) (Table 1). Sixty-one percent of letters about clinical themes were about clinical opinions (e.g., “A double standard clearly exists among the screening community, who seem to be in denial - the problem of over-diagnosis that was finally confirmed by irrefutable evidence in the study can no longer be denied”). In the “Other” category, 41% of LTE topics were related to ethical issues (e.g., “Study patients should not pay for trial medications and we are surprised that the local ethics committee ignored this fact”); 29% were about conflict of interest (e.g., “One of the principal contributors fails to declare a given conflict of interest which can be found in a Cochrane review”); and 29% were about a criticism of an accompanying editorial (e.g., “…The statement in the accompanying editorial that the results of the author are not likely to influence current clinical practice is perplexing”).
A significant association between funding source and the impact of the intervention on outcomes in the RCTs (p = 0.002) was found. Of 108 RCTs with a positive impact on outcomes, 44% were funded by government, 40% by industry, 7% by academic, and 4% by other (5% did not indicate); “no impact” RCTs (N = 58) were primarily funded by government (78%), while about half of the negative impact trials (N = 9) were funded by industry (56%). There was also a statistically significant association between funding source and the type of intervention (p = 0.001). Most drug intervention RCTs (N = 119) were funded by government (49%) or industry (42%), while “other” intervention RCTs (N = 56) was funded mostly by government (66%). Industry-funded RCTs (N = 56) tested more drug interventions than government-funded RCTs (N = 95) (89% vs 61%; p = 0.001 [Pearson Chi-square 19.86]). No statistically significant associations were found for other RCT-level comparisons such as impact of primary outcome or type of intervention tested and author affiliation. Please see Additional file 1: Table S1 for additional data on the proportion of RCTs according to impact on primary outcome and type of intervention by funding source.
RCT characteristics and LTE-level comparisons*
Number of Topics (%)
P-value†RCT characteristic compared with LTE topic: Methodological, Clinical or Other
(Number of RCTs; N = 175)
Method (N = 564)
Clinical (N = 42)
Other (N = 17)
Impact on outcomes reported in the RCT
Positive (N = 108)
Negative (N = 9)
None (N = 58)
Presence of serious harm in outcomes of the RCT
Harm (N = 161)
No harm (N = 4)
Type of intervention tested in the RCT
Drug (N = 119)
Other (N = 56)
Author affiliation of the RCT
Academic (N = 137)
Hospital (N = 26)
Research Institute (N = 6)
Community practice (N = 3)
Government (N = 2)
Industry (N = 1)
Funding source of the RCT
Government (N = 95)
Academic (N = 9)
Industry (N = 56)
Other (N =7)
Not indicated (N = 8)
Number of letters according to sub-categories of methodological themes by journal
To our knowledge, this is the first study to investigate the types of topics addressed in published LTE for a sample of RCTs across 5 high impact journals, and to test the association between the outcomes of the RCT and the topic(s) of the LTE that was written in response to the RCT. There are some limitations to this study. We searched only 5 medical journals, so our sample size may have been too small to detect significant differences. However, we included the journals that publish a large proportion of methodologically rigorous randomized trials, which likely maximized the number of LTE and the breadth (reaching researchers as well as clinicians) and depth (more focused and detailed discussions) of topics. Our focus on high-impact journals may have also biased the ‘type’ of scrutiny provided by their letters compared with those published in lower impact journals—an important consideration if LTE are used for the purposes of post-publication peer review. Indeed, LTE addressing methodological topics might be more often reported in lower impact journals. In addition, not all letters submitted to these journals get published, and we did not account for the balance of RCTs published across journals (i.e., some publish more trials per issue than others), so the frequency of topics we identified may not be an accurate representation of all the issues raised by LTE. Furthermore, we did not compare RCT characteristics between studies that generated a letter with those that did not – this information may have strengthened our findings and is a potential topic for a future study. Another limitation is that we may have also missed some letters that could have been published beyond our predicted 1-year period. However, it is unlikely that a small number of additional letters would have influenced our findings. Finally, analysis of all letters submitted to journals (not just the ones published by journals or those accessible electronically) may have also increased our power to detect differences.
The study found little evidence that LTE topics are affected by the characteristics or results of their corresponding RCTs. However, some letters that are submitted do not get published, so these findings may in part reflect editorial censorship as journal editors make the final decision on LTE publications. Our findings also showed that most LTEs from 5 high-impact medical journals are about methodological topics, which may indicate a potential for LTE to be used as a form of post-publication peer review.
This study had no external funding.
- Edwards R, White M, Gray J, Fischbacher C: Use of journal club and letter-writing exercise to teach critical appraisal to medical undergraduates. Med Educ. 2001, 35: 691-694. 10.1046/j.1365-2923.2001.00972.x.PubMedView ArticleGoogle Scholar
- Kallen AJ, Wilson CT, Russell MA, et al: Group writing of letters to the editor as the goal of journal club. JAMA. 2006, 296 (9): 1053-1054.PubMedView ArticleGoogle Scholar
- Kirsch M: Letters to the editor: competition or search for truth?. Am J Gastroenterol. 2008, 103: 1596-1597. 10.1111/j.1572-0241.2007.01685.x.PubMedView ArticleGoogle Scholar
- Goetzsche PC, Delamothe T, Godlee F, Lundh A: Adequacy of authors’ replies to criticism raised in electronic letters to the editor: a cohort study. BMJ. 2010, 341: c3926-10.1136/bmj.c3926.View ArticleGoogle Scholar
- Altman DG: Poor-quality medical research: what can journals do?. JAMA. 2002, 287: 2765-2767. 10.1001/jama.287.21.2765.PubMedView ArticleGoogle Scholar
- Horton R: Postpublication criticism and the shaping of clinical knowledge. JAMA. 2002, 287: 2843-2846. 10.1001/jama.287.21.2843.PubMedView ArticleGoogle Scholar
- von Elm E, Wandel S, Juni P: The role of correspondence sections in post-publication peer review: a bibliometric study of general and internal medicine journals. Scientometrics. 2009, 81 (3): 747-755. 10.1007/s11192-009-2236-0.View ArticleGoogle Scholar
- Altman DG: Unjustified restrictions on letters to the editor. PLoS Med. 2005, 2 (5): 0457-View ArticleGoogle Scholar
- Bhopal RS, Tonks A: The role of letters in reviewing research. BMJ. 1994, 208: 1582-1583.View ArticleGoogle Scholar
- Adam D: Publish and be damned…. Nature. 2002, 419: 772-776. 10.1038/419772a.PubMedView ArticleGoogle Scholar
- Mahesh S, Kabos M, Walvoort HC, Overbeke AJ: Significance of letters published in the Dutch journal of medicine. 1997/98. Ned Tijdschr Geneesk. 2001, 145 (11): 531-535.Google Scholar
- Baethge C, Seger G: Letters to the editor are an important component of the discussion of scientific articles. Deutsch Arztebl Int. 2009, 106 (12): 207-209.Google Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.