- Research article
- Open Access
Visibility of retractions: a cross-sectional one-year study
© Decullier et al.; licensee BioMed Central Ltd. 2013
- Received: 6 November 2012
- Accepted: 11 June 2013
- Published: 19 June 2013
Retraction in Medline medical literature experienced a tenfold increase between 1999 and 2009, however retraction remains a rare event since it represents 0.02% of publications. Retractions used to be handled following informal practices until they were formalized in 2009 by the Committee on Publication Ethics (COPE). The objective of our study was to describe the compliance to these guidelines.
All retractions published in 2008 were identified using the Medline publication type “retraction of publication”. The notices of retraction and the original articles were retrieved. For each retraction, we identified the reason for retraction, the country of affiliation of the first author, the time to retraction, the impact factor of the journal and the mention of retraction on the original article.
Overall, 244 retractions were considered for analysis. Formal retraction notices could not be retrieved for 9. Of the 235 retractions available (96%), the reason was not detailed for 21 articles (9%). The most cited reasons were mistakes (28%), plagiarism (20%), fraud (14%) and overlap (11%). The original paper or its location was found for 233 retractions (95%). Of these, 22% were available with no mention of the retraction.
A standard retraction form could be helpful, with a check list of major reason, leaving the editor free to provide the reader with any further information. Original articles should remain available with a clear mention of the retraction.
- Retraction of publication
- Scientific misconduct
The process of scientific communication relies on trust: the researcher is supposed to conduct his research according to Good Practice, they should report the results properly  and declare any conflict of interest . It is difficult for reviewers to detect errors or suspected fraud . Their task is to improve the paper. Various types of inappropriate behavior can be encountered during the writing of a paper and the review process, including misconduct, faked data, falsification, ethical misconduct, plagiarism, etc. . Mistakes, i.e. misinformation without intent, can also diminish the quality of scientific evidence . In cases of scientific misconduct or mistakes, it is necessary that readers are informed, which is why the editor later publishes a correction (correcting a mistake by substituting correct information), expression of concern (issued in case of suspected misconduct, not yet proven) or retraction (published in case of proven misconduct) [6, 7].
Following the Joachim Boldt case with 88 retractions  and the Scott Reuben case with 21 falsified papers , the subject of retraction was studied in 4 dedicated publications in 2011 [5, 10–12]. The number of retractions in journals covered by the Science Citation Index Expanded has increased 20 times i.e. a tenfold increase since there has been a twofold increase in articles production between 1990 and 2008 . A similar tenfold increase was found when focusing on Medline only (1999–2009), although retraction remains a rare event since it only represents 0.02% of publications .
The Committee on Publication Ethics (COPE) has established guidelines on dealing with retractions in 2009 . COPE recommends that retractions should be issued in case of unreliable findings (misconduct or error), plagiarism or unethical research.
To better understand the reasons for retractions, 5 studies investigated cohorts of retraction notices (sometimes leading to more than one publication) [10–12, 15–18]. All these studies covered different time periods, different durations of time, and provided the rate of retractions related to mistakes (research error and inability to reproduce the results). The rates of retractions related to mistakes tended to decrease according to the last year investigated: 55% for 1997 , 62% in 2002 , 42% in 2004 , 39% in 2008  and 32% in 2010 .
COPE’s guidelines also state that notices of retraction should clearly identify the retracted article (title and authors) and be linked with the retracted article. The notices should be available freely and state the reason for retraction (without being defamatory) and who is retracting. Steen found that 45% of retracted papers were either available without any mention or deleted .
We analysed retractions published in Medline over a single year period to describe the conformity with retraction guidelines as well as the reasons for retraction and their distribution across countries.
Data extraction (August, 22th 2011)
Reasons used to classify retractions: proposed definitions
Falsified data, fabricated data
Confirmed doubt over data raised by others
Mistakes concerning data found in the paper raised by the author(s)
Publication of data or text already published by others
Multiple publication of same data or self-plagiarism
Property or legal concerns
Publication of elements without obtaining permission
Concerns on the ethical validation of the research
Production or administrative error
Data were extracted by GS, then LH and ED independently reviewed the reasons for retractions. Discrepancies were resolved by consensus between the two reviewers.
We described the frequency of each reason, and cross-tabulated it with the country to obtain the ranking of countries for each reason.
Among the 9 reasons, we combined fraud, mistakes and inconsistent data since retracted articles led to the circulation of false information (labelled as “false information”) and were compared to the 6 other reasons combined.
To assess the conformity of visibility of retraction to good practices, we decided to choose the COPE guidelines. Even if published in 2009, COPE guidelines were the formalization of common sense and informal practices of some journals. We therefore determined whether the retraction was mentioned on the pdf of the original article or not. Visibility of retractions was defined by three possibilities: 1) deletion of the article from the journal’s website; 2) presence of the article with a mention of the retraction on the paper itself (either a comment at the beginning and/or at the end; or a clear indication for example printed diagonally across the paper) and/or mention on the journal’s website; and 3) presence of the article with no mention.
Time to retractions and impact factors were compared using the Mann–Whitney or Kruskal-Wallis test. Visibility of retractions was compared using Chi2 test (or Fisher exact test where Chi2 test conditions were not fulfilled).
All analyses were performed using SAS v9.2 (Cary, North Carolina).
Overall, 241 notices of retraction were retrieved. Concerning access, 209 were available by open access or accessible through our institution’s subscription, 23 were found outside of our institution and 9 were impossible to locate.
These 241 notices of retraction represented 253 retractions (10 retracting 2 articles and 1 retracting 3 articles). After deleting 3 duplicate notices, 250 retractions remained available. Lastly, one “retracted retraction” and 5 partial retractions were discarded. Overall, 244 retractions were considered for analysis.
Reason for retraction (n = 235)
As previously stated, formal retraction notices could not be retrieved for 9 of the retractions. Of the 235 retractions available, the reason was not given for 21 articles (9%). For example, the only information given by some journals is that the article is being retracted. Sometimes, it is also specified “retracted by the authors” or “retracted to be consistent with the publisher policy on article withdrawal” or “retracted by the authors with the agreement of the journal editors and the publisher” without any other details.
Reasons for retraction ranked according to countries with at least 10 retractions in 2008
Property or legal concerns
The reasons are also presented according to countries in Table 2. In India and Japan, the most frequent reason was fraud, representing 43% for India (15/35) and 48% for Japan (12/25). In China, it was plagiarism with 34% (10/29); whereas in the USA the main reason was mistakes at 60% (31/52) and plagiarism at 36% for United Kingdom (4/11).
Comparison of the 3 main reasons for retraction by impact factor, time to retraction and visibility of retractions
Impact factor (n = 137)
Time to retraction (n = 244), years
Publication year of retraction vs original article (n = 244), n (%)
1 to 5 years
Visibility (n = 233), n (%)
Mention of retraction on the original article (n = 233)
Mention of retraction on the original article
Original article (n = 233):
both website and article
If mention available on article (n = 115), details:
At least a clear indication†
present without notes
present with notes at the end
present with notes at the beginning
at the end
at the beginning
both at the beginning and the end
Visibility of retraction on original articles was significantly higher for false information (p = 0.003). Among the 106 articles with false information, 75 (71%) had a mention compared to 64 / 127 articles (50%) retracted for other reasons; the rate of deleted articles was lower for false information compared to other reasons (11 (10%) vs 31 (24%)); finally, the rate of articles available with no mention was similar (20 (19%) vs 32 (25%)).
The most frequent reasons for retraction in 2008 were mistakes (28%), followed by plagiarism (20%) and fraud (14%), and journals insufficiently followed the retraction good practices formalized by the 2009 COPE guidelines.
All investigations into Medline retractions have highlighted the fact that it is difficult to find formal retraction notices containing the reason for retraction (studies found that formal explanation was unavailable for 5% , 11% , 12% , 18% ). Here, the content of the retraction was not explicit for 21 retractions (9%). As recommended in previous studies, editors should always provide the precise reason for retraction. Sometimes obtaining the formal retraction notice is subject to a pay-per-view fee, and is therefore linked to the institution’s membership. One study specified that the search was restricted to articles accessible within the institution , another specified that 46 notices (5.8%) could not be retrieved , the other studies did not specify these cases were excluded or counted them as “reason unclear”. Overall, in our study, 9 retractions were not available even with payment. The fact that retraction notices are not freely available and when available, are not explicit, contravenes the COPE guidelines  and should be a priority for journals as they are essential for the reader’s understanding of whether the results still hold.
However, editors are not always willing to retract articles . It is therefore necessary that all journals implement a retraction policy, indeed in a 2004 review it was shown that only 18% had a dedicated policy . The way to handle mistakes is sometimes controversial: some recommend to publish corrections , whereas others state that they should be full retractions . Practices are not homogeneous , and retractions for mistakes are regularly published.
It has already been demonstrated that the retracted literature continues to be cited [18, 22]. One of the explanations might be that retractions are not correctly reported on original article. We found that the mention of the retraction was incorrect for half of the retractions: the article was either completely deleted, or was still available with no mention. This is similar to the 45% found by Steen . The retraction good practices formalized by the COPE guidelines  are probably not well known or respected. Clear rules should be applied to ensure that this literature is identified as retracted. To be consistent with full transparency of publication, it is inappropriate to completely remove a retracted article from the website of a journal. As long as a clear mention of the retraction is available on both the journal website and the article, scientists must still have access. Some publishers are probably afraid of the circulation of false information, particularly of fraudulent data, and retracted articles continue to be removed from printed or online versions of some journals.
We also decided for the purposes of this study to combine fraud, inconsistent data and mistakes to estimate what proportion of retracted articles might have damaged the body of knowledge. Other reasons include, of course, misconduct however the information in these papers remains true. Steen has also shown that many patients are put at risk since approximately 2 472.6 subjects per retracted paper were enrolled in subsequent studies using this retracted research . Recently, the CrossMark system has been developed following the collaboration between several publishers to standardize the way of providing readers how to locate the current version of an article . If one journal applied the CrossMark icon on every pdf, it is to try to maintain the content of published articles and, by clicking on it, to warn readers if changes have occurred. In the future, this system may reduce citations to retracted articles.
We cross-tabulated reasons for retraction with countries. However, retraction rates by country should be interpreted relative to publication rate: the USA represents 22% of retractions but also represents the highest number of publication. The aim of this study was to provide a global description of retraction, and due to the low numbers in each category we did not provide any statistical comparison of countries. Good practices should be better disseminated to authors and editors in order to educate them and raise their awareness to better prevent misconduct; however fraud will not be deterred by this dissemination. We found similar rates as Wager : 28% for mistakes vs. 28%, 20% for plagiarism vs. 16% and 14% for fraud vs. 11%.
We noticed an association between impact factor and reasons for retraction with mistakes being published in higher impact factor journals. This could be explained by the fact that some authors might hurry to publish in a prestigious journal without taking enough time to check their data. When a paper deals with a hot topic, journals tend to publish special issues very rapidly. Moreover, mistakes are also very frequent in the USA where researchers can be subject to greater pressure and are more inclined to submit to prestigious international journals. The publications by Steen and Fang also found that higher impact factor journals were more prone to retraction [12, 25].
It would be useful to use a standard retraction form with for example a check list of major reason, which would then leave the editor free to provide the reader with any further information. Original articles should remain available with a clear mention of the retraction, and not only a mention on the journal website or in notes at the beginning or end of the article. This is probably on the COPE agenda, and our work can contribute to proposing definitions.
ED is a senior researcher (PhD), LH is a senior researcher (PharmD, PhD), GS is a junior researcher (PharmD), HM is a senior researcher (MD).
We would like to thank Florence Bouriot from the documentation center of Hospices Civils de Lyon for retrieving articles and notices of retraction, Joan Marsh (Wiley, London, UK), Paola De Castro (Istituto Superiore di Sanità, Roma, Italy), Barbara Gastel (Texas A&M Health Science Center, USA), Shou-chu Qian (Chinese Medical Association, China) who helped us to retrieve articles; and Kim Barrett for proofreading.
- Uniform Requirements for Manuscripts Submitted to Biomedical Journals. [http://www.icmje.org]
- Drazen JM, Van Der Weyden MB, Sahni P, Rosenberg J, Marusic A, Laine C, Kotzin S, Horton R, Hebert PC, Haug C, et al: Uniform format for disclosure of competing interests in ICMJE journals. Lancet. 2009, 374: 1395-1396. 10.1016/S0140-6736(09)61796-7.PubMedView ArticleGoogle Scholar
- Can peer review police fraud?. Nat Neurosci. 2006, 9: 149-Google Scholar
- Martinson BC, Anderson MS, De Vries R: Scientists behaving badly. Nature. 2005, 435: 737-738. 10.1038/435737a.PubMedView ArticleGoogle Scholar
- Steen RG: Misinformation in the medical literature: What role do error and fraud play?. J Med Ethics. 2011, 37: 498-503. 10.1136/jme.2010.041830.PubMedView ArticleGoogle Scholar
- Sox HC, Rennie D: Research misconduct, retraction, and cleansing the medical literature: lessons from the Poehlman case. Ann Intern Med. 2006, 144: 609-613. 10.7326/0003-4819-144-8-200604180-00123.PubMedView ArticleGoogle Scholar
- Smith R: When to retract?. BMJ. 2003, 327: 883-884. 10.1136/bmj.327.7420.883.PubMedPubMed CentralView ArticleGoogle Scholar
- Marcus A: Retractions Come in Boldt Case, Likely Most for Single Author. Probe finds 88 papers lacked ethics approval. Anesthesiology News. 2011, 37:Google Scholar
- Rittner HL, Kranke P, Schafer M, Roewer N, Brack A: [What can we learn from the Scott Reuben case? Scientific misconduct in anaesthesiology]. Anaesthesist. 2009, 58: 1199-1209. 10.1007/s00101-009-1637-6.PubMedView ArticleGoogle Scholar
- Wager E, Williams P: Why and how do journals retract articles? An analysis of Medline retractions 1988–2008. J Med Ethics. 2011, 37: 567-570. 10.1136/jme.2010.040964.PubMedView ArticleGoogle Scholar
- Steen RG: Retractions in the scientific literature: is the incidence of research fraud increasing?. J Med Ethics. 2011, 37: 249-253. 10.1136/jme.2010.040923.PubMedView ArticleGoogle Scholar
- Steen RG: Retractions in the scientific literature: do authors deliberately commit research fraud?. J Med Ethics. 2011, 37: 113-117. 10.1136/jme.2010.038125.PubMedView ArticleGoogle Scholar
- Corbyn Z: Retractions up tenfold. 2009, Higher Education: TimesGoogle Scholar
- COPE: Retraction guidelines. 2009, [http://www.publicationethics.org/files/retraction%20guidelines.pdf]Google Scholar
- Nath SB, Marcus SC, Druss BG: Retractions in the research literature: misconduct or mistakes?. Med J Aust. 2006, 185: 152-154.PubMedGoogle Scholar
- Redman BK, Yarandi HN, Merz JF: Empirical developments in retraction. J Med Ethics. 2008, 34: 807-809. 10.1136/jme.2007.023069.PubMedView ArticleGoogle Scholar
- Budd JM, Sievert M, Schultz TR: Phenomena of retraction: reasons for retraction and citations to the publications. JAMA. 1998, 280: 296-297. 10.1001/jama.280.3.296.PubMedView ArticleGoogle Scholar
- Budd JM, Sievert M, Schultz TR, Scoville C: Effects of article retraction on citation and practice in medicine. Bull Med Libr Assoc. 1999, 87: 437-443.PubMedPubMed CentralGoogle Scholar
- Newman M: The rules of retraction. BMJ. 341: c6985-Google Scholar
- Atlas MC: Retraction policies of high-impact biomedical journals. J Med Libr Assoc. 2004, 92: 242-250.PubMedPubMed CentralGoogle Scholar
- Williams P, Wager E: Exploring why and how journal editors retract articles: findings from a qualitative study. Sci Eng Ethics. 2013, 19: 1-11. 10.1007/s11948-011-9292-0.PubMedView ArticleGoogle Scholar
- Pfeifer MP, Snodgrass GL: The continued use of retracted, invalid scientific literature. JAMA. 1990, 263: 1420-1423. 10.1001/jama.1990.03440100140020.PubMedView ArticleGoogle Scholar
- Steen RG: Retractions in the medical literature: how many patients are put at risk by flawed research?. J Med Ethics. 2011, 37: 688-692. 10.1136/jme.2011.043133.PubMedView ArticleGoogle Scholar
- CrossMark Support Site. [http://crossmarksupport.crossref.org/]
- Fang FC, Casadevall A: Retracted Science and the Retraction Index. Infect Immun. 2011, 79: 3855-3859. 10.1128/IAI.05661-11.PubMedPubMed CentralView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.