- Research note
- Open Access
Unpublished systematic reviews and financial support: a meta-epidemiological study
BMC Research Notesvolume 10, Article number: 703 (2017)
PROSPERO, an international prospective register of systematic reviews, was launched in February 2011 to reduce publication bias of systematic reviews (SRs). A questionnaire survey of SR researchers conducted in 2005 indicated the existence of unpublished SRs and the potential influence of lack of funding as a reason for non-publication. Here, we investigated the publication status of registered SRs in the 1st year that PROSPERO launched and assessed the association between publication and the existence of funding or conflicts of interest (COIs).
We identified 326 SRs registered in PROSPERO from February 2011 through February 2012. Among them, 85 SRs (26%) remained unpublished at least 65 months after registration. We found 241 published reports, including four conference abstracts and one poster presentation. Median time to publication from protocol registration was 16.3 months. Funding for SRs was associated with publication [odds ratio (OR) = 2.10; 95% confidence interval (CI) = 1.26 to 3.50]. We found no significant association of author-reported COIs with publication (OR = 2.35; 95% CI = 0.67 to 8.20). Twenty SRs were not published despite the authors reporting completion of the reviews in PROSPERO.
Publication bias occurs when the publication of study results is influenced by the strength of the research findings. It includes two fundamental concepts: study findings and non-publication . Many studies have reported bias in the dissemination of research findings other than systemic reviews (SRs) [2,3,4]. PROSPERO, an international prospective register of SRs, was launched in February 2011 to reduce publication bias of SRs . Before PROSPERO, there was no specific international registration system for SRs; therefore, the issue of unpublished SRs could not be assessed directly. A questionnaire survey of SR researchers conducted in 2005 indicated the existence of unpublished SRs and the potential influence of lack of funding as a reason for non-publication . As of June 2017, PROSPERO contains over 23,000 entries. Here, we investigated the publication status of SR protocols registered in PROSPERO and assessed the relationship of financial support for these SRs with their publication.
We investigated the publication status of registered SRs as of June 2017. The time frame for SR sampling was limited to the 1st year that PROSPERO launched in order to allow a lead time for publication. Cochrane has reported that the median time from protocol to SR publication is approximately 30 months . We searched Google (https://www.google.com) and Google Scholar (https://scholar.google.com) in June 2017 for published SRs whose publication status was not reflected in the PROSPERO records. We defined publication as either author-reported publication status in PROSPERO, or dissemination of the results in a publicly available forum or in any journals indexed in Google or Google Scholar. We manually screened the search results first by titles and the URLs of all search results provided by Google on the results page and by titles of all search results provided by Google Scholar. If the website seemed to contain publication report, we entered the website and looked for full publication report. SRs that were not reported as published in PROSPERO or whose status was unavailable by searching Google or Google Scholar were considered unpublished. The search terms used for each SR were the title of the protocol, the PROSPERO ID, and the authors’ names. For example, we first searched Google Scholar using the PROSPERO ID. If we could not find the published report, we searched Google and Google Scholar using the title of the protocol enclosed in quotation marks. If this was unsuccessful, we searched Google and Google Scholar using the names of all listed authors of the protocol without quotation marks and screened the first page of search results. The time (in months) from protocol registration to publication of SRs was defined as the number of days from registration to publication divided by 30. The date of publication was selected for each review according to the following hierarchy: (1) the acceptance date, if available; (2) the online publication date, if available; (3) the earlier of the journal publication date or the date of author-reported “published” status in PROSPERO; (4) the earlier of the journal publication date of a conference abstract or the documented date of public poster dissemination detected by searching Google or Google Scholar. If only the publication month was reported, the midpoint of that month (day 15) was set as the publication day.
The association between publication and the existence of funding or conflicts of interest (COIs) was investigated using multivariable logistic regression analysis. Adjusted variables were funding and COIs because a previous study indicated the potential influence of lack of funding as a reason for non-publication . We also conducted a post hoc analysis for the SRs registered in PROSPERO from September 2011 to February 2012 (posterior half period). Statistical analysis was performed using StataSE version 13 (StataCorp, College Station, TX).
We identified 326 SRs registered in PROSPERO from February 2011 through February 2012. As Cochrane began registering protocols in October 2013, no Cochrane protocols were included . The details of investigation of publication status are shown in Fig. 1. Among the identified SRs, 85 (26%) had not been published by June 2017, at least 65 months after protocol registration (the median time passed at the time of investigation was 68.8 months). We found 241 published reports, including 4 conference abstracts and 1 poster presentation. Of them, 126 SRs (52%) remained with unpublished status in PROSPERO. For SRs (N = 115) with published status, we could find published reports (100%). For 165 SRs (68%) we could find the exact accepted date for publication. For 25 SRs (10%), they reported the accepted date only by months; we used the mid-point of the month (day 15). For 51 (21%) SRs we could not find the accepted date, so we used the surrogate date according to the hierarchy described above. Median time to publication was 16.3 months (Fig. 2). Funding for SRs was associated with publication [odds ratio (OR) = 2.10; 95% confidence interval (CI) = 1.26 to 3.50; Table 1]. We found no significant association of author-reported COI with publication (OR = 2.35; 95% CI = 0.67 to 8.21; Table 1). The association was similar for the SRs registered in PROSPERO from September 2011 to February 2012 (Additional file 1: Table S1). Twenty SRs were not published despite the authors reporting completion of the reviews in PROSPERO.
This is the first reported direct assessment of SRs remaining unpublished after protocol registration in PROSPERO and the relationship between publication and funding source. We found a considerable proportion (26%) of unpublished SRs even at more than 65 months after protocol registration. Tricco et al. reported that the publication of SRs may be affected by whether the results are informative . We did not investigate the potential influence of the strength of research findings on publication (i.e., the effect of the clinical significance of SR results) because we could only access the published results retrospectively; however, these unpublished SRs might potentially impacted by the direction or strength of their findings.
Decullier et al. reported that funding was the determining factor for project initiation in clinical research, but once the project was initiated, funding had no significant influence on study completion or publication . Conversely, we found that funding for SRs was related to their publication. One likely reason for these conflicting findings is the difference in funding sources between SRs and clinical trials: most SRs have non-profit funding , whereas clinical trials tend to have for-profit funding.
Non-profit funding for registered SRs may mitigate the issue of non-publication after protocol registration. Although we did not find a significant association between author-reported COI and publication, COIs reported by authors are actually a mixture of financial- and academic-related issues. Further research is needed regarding the association of financial conflicts of interest with SR publication.
The low proportion of SRs with registered protocols is a possible limitation of this study. Page et al. reported that only 16% of SRs published in 2014 have publicly available protocols . Our sample included only SRs with protocols, which may not be representative of all SRs. However, the impact on assessment of non-publication of high-quality SRs was probably minimal, because the quality of reporting of published SRs in 2014 was still low despite the fact that reporting guidelines recommend protocol registration [10, 11].
Qualitative research by Tricco et al. revealed that the main reasons reported for non-publication were lack of time, overly broad SR scope, and few studies eligible for SRs as well as rejection . Because data regarding these factors are unavailable in PROSPERO records, we could not adjust for them in our analysis, which is a limitation of this study.
In addition, while this survey covered only the 1st year after PROSPERO was launched, the proportion of protocol-registered SRs appearing in high-impact journals is increasing . Future research should extend the search period to several years after launch in order to more fully investigate the characteristics of SR publications.
The international prospective register of systematic reviews
conflicts of interest
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14:1–220.
Dickersin K, Min Y, Meinert C. Factors influencing publication of research results. JAMA. 1992;267:374–8.
Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009;1:5–8.
Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986;4:1529–41.
Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev. 2012;1:2.
Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, Hopewell S, et al. An international survey indicated that unpublished systematic reviews exist. J Clin Epidemiol. 2009;62(617–623):e5.
Cochrane. Cochrane organisational dashboard 2016. In: 2016 Annual Cochrane Dashboard. 2016. http://community.cochrane.org/sites/default/files/uploads/inlinefiles/2016%20Dashboard%20open%20access.pdf. Accessed 2 Dec 2017.
Cochrane Community. Editorial and Publishing Policy Resource. In: PROSPERO. 2016. http://community.cochrane.org/editorial-and-publishing-policy-resource/overview-cochrane-library-andrelated-content/prospero. Accessed 2 Dec 2017.
Decullier E, Huot L, Chapuis FR. Fate of protocols submitted to a French national funding scheme: a cohort study. PLoS ONE. 2014;9:1–5.
Page MJ, Shamseer L, Altman DG, Tetzlaff JM, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13:e1002028.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.
Tsujimoto Y, Tsujimoto H, Kataoka Y, Kimachi M, Shimizu S, Ikenoue T, et al. Majority of systematic reviews published in high-impact journals neglected to register the protocols: a meta-epidemiological study. J Clin Epidemiol. 2017;84:54–60.
HT, YT, and YK: study concept and design, analysis and interpretation of data and critical revision of the manuscript for important intellectual content. HT: acquisition of data and drafting of the manuscript. All authors read and approve the final manuscript.
The authors gratefully acknowledged Centre for Reviews and Dissemination, University of York for the free access data of the PROSPERO (http://www.crd.york.ac.uk/PROSPERO/).
The authors declare that they have no competing interests.
Availability of data and materials
The authors only used publicly available data. The datasets analyzed during the current study available from Open Science Framework (osf.io/qmh4e).
Consent for publication
Ethics approval and consent to participate
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.