The COIVD-19 pandemic has produced an unprecedented volume of research, with over 125,000 articles published or released in the first 10 months of the pandemic.
1In addition, there has been an unusual number of article retractions and withdrawals, with papers being retracted far earlier than the usual time period, which is typically 2 to 3 years.
- Fraser N.
- Brierley L.
- Dey G.
- et al.
The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape.
PLOS Biol. 2021; 19e3000959
2This raises concerns about the publishing process and research quality
- Steen R.G.
- Casadevall A.
- Fang F.C.
Why has the number of scientific retractions increased?.
PLoS One. 2013; 8: e68397
3and asks whether these retractions reflect problematic research or are a by-product of a rush to publish during a pandemic. Potential causes for article retraction include the popularity of pre-print servers, increased scrutiny of COVID-19 research, and expedited publisher reviews. These retractions have occurred in a range of journals, including top-tier journals,
- Bramstedt K.A.
The carnage of substandard research during the COVID-19 pandemic: a call for quality.
J Med Ethics. 2020; 46: 803-807
4indicating that these retractions cannot be attributed solely to lesser-known, lower quality, or predatory journals. While previous studies have examined COVID-19 publications, some studies included articles from a variety of sources (including pre-print servers) and searched multiple databases.
- Anderson C.
- Nugent K.
- Peterson C.
Academic journal retractions and the COVID-19 pandemic.
J Prim Care Commun Health. 2021; 12 (21501327211015592)
5Furthermore, given the unique nature of the pandemic, retraction rates should ideally be compared to previous health crises, especially epidemics and pandemics. Since PubMed is a major journal database, is used as an indicator of journal quality, and is arguably the premier repository of biomedical articles, we exclusively searched PubMed indexed journals for retracted COVID-19 articles and for retracted articles from other recent infectious disease outbreaks.
- El-Menyar A.
- Mekkodathil A.
- Asim M.
- et al.
Publications and retracted articles of COVID-19 pharmacotherapy-related research: a systematic review.
Sci Prog. 2021; 104 (00368504211016936)
We searched the PubMed database (https://pubmed.ncbi.nlm.nih.gov/) for articles published between 01 November 2019 and 01 August 2021 using the term “COVID-19.” Filters for “Retract Publication” and “Retraction of Publication” were applied and duplicates were omitted. We found 43 COVID-19 retractions or withdrawals from 35 journals (“retraction” here denotes both outcomes). Six journals had multiple retractions. Journal impact factors were obtained from Clarivate Journal Reports™ (https://clarivate.com/webofsciencegroup/solutions/journal-citation-reports/) for the year 2020. These journals had mean and median impact factors (excluding duplicate journals from the list) of 10.8 ± 22.0 and 3.1, respectively; the impact factors ranged from 0.9 to 91.3 (seven journals had no impact factor listed) (Table 1). Mean, median, and range of time to retraction were 139.1 ± 107.9 days, 120 days, and 3 - 433 days, respectively. Publication and retraction dates were based on the earliest and most specific date available from PubMed and publishers. Retracted articles had a total of 3470 citations (mean 80.7 ± 240.1; median 6) by 02 Aug 2021. The H-Indices for these journals were obtained from Scimago (https://www.scimagojr.com/) and had mean, median, and ranges of 136.5 ± 221.0, 81.5 (excluding duplicate journals from the list), and 3–1030, respectively (one journal had no H-Index). Welch's t-test was used to compare differences in retraction time between articles retracted in 2020 and 2021.
Table 1Characteristics of retracted COVID-19 articles and associated journals.
|10.8 ± 22.0||3.13||0.9-91.3||-|
|136.5 ± 221.0||81.5||3-1030||-|
|Time to Retraction, days|
|139.1 ± 107.9||120||3-433||-|
|80.7 ± 240.1||6||0-1210||3470|
We also compared COVID-19 article retractions to articles retracted during previous outbreaks. Searches for “H1N1”, “Zika”, “SARS”, and “Ebola” were performed using the same PubMed filters and for the first 19 months of each epidemic/pandemic: 01 November 2002 – 01 June 2004 (SARS), 01 April 2009 – 01 November 2010 (H1N1), 01 December 2013 – 01 July 2015 (Ebola), and 01 May 2015 – 01 December 2016 (Zika). Timelines for each outbreak were obtained from the World Health Organization (WHO) and the Centers for Disease Control (CDC). Given the unusually high volume of COVID-19 publications, we also compared outbreak publications “to date” (01 Aug 2021)[except for SARS, which was searched 01 November 2002 to 01 November 2019 to avoid overlap with SARS-CoV-2]. This second period is referred to as “Time frame 2”. Percent retractions from previous outbreaks ranged from 0.00 to 0.04% (Table 2). The number of articles published (154,562) and retracted (43) during the COVID-19 pandemic was much higher than those in previous outbreaks for both the first 19 months of each outbreak and to date (i.e., 01 Aug-2021). However, the retraction rate of COVID-19 articles (0.03%) was similar to that observed in the H1N1 pandemic for both time frames (0.04%) and is lower than a previously estimated universal rate of retraction (0.04%).
6Other studies have also observed disproportionately higher publications for COVID-19 compared to other outbreaks and scientific topics
- Brainard J.Y.J.
What a massive database of retracted papers reveals about science publishing's ‘death penalty.
Science. 2018; https://doi.org/10.1126/science.aav8384
- Di Girolamo N.
- Meursinge Reynders R.
Characteristics of scientific articles on COVID-19 published during the initial 3 months of the pandemic.
Scientometrics. 2020; : 1-18
8; our study is the most recent analysis of the retraction rate. For example, an earlier study also searched PubMed and observed a higher retraction rate (0.097%) by 08 June 2020.
- Palayew A.
- Norgaard O.
- Safreed-Harmon K.
- Andersen T.H.
- Rasmussen L.N.
- Lazarus J.V.
Pandemic publishing poses a new COVID-19 challenge.
Nat Hum Behav. 2020; 4: 666-669
9This higher rate of retraction may be due to an influx of COVID-19 publications in the following year (as of this writing, COIVD-19 publications for 2021 have exceeded the total for 2020). Even though the majority of retractions occurred in 2021 (25; 58.1%), these data suggest retraction rates may normalize over time given increased total rates of publication. Interestingly, there was a statistically significant difference in retraction time between articles retracted in 2020 (n = 18, mean 87.9 ± 63.5 days) and 2021 (n = 25, mean 176.0 ± 118.9 days) (p = 0.003). This may be due to decreased interest in COVID-19 research quality, lower individual article visibility from increased total publications, or articles with less serious or obvious errors being retracted more slowly. Finally, some articles received a high number of citations despite relatively rapid rates of retraction. This observation supports a previous study on the citation of COVID-19 articles post-retraction,
- Yeo-Teh N.S.L.
- Tang B.L.
An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID-19).
Acc Res. 2021; 28: 47-53
10can complicate the use of citations as a measure of article quality, and potentially prolongs the impact of problematic research.
- Piller C.
Disgraced COVID-19 studies are still routinely cited.
Science. 2021; 371: 331-332
Table 2Comparison of retraction rate between COVID-19 and other outbreaks.
|Disease||Time Frame 1|
|Time Frame 2|
First 19 months of outbreak- 01 November 2002 – 01 June 2004 (SARS), 01 April 2009 – 01 November 2010 (H1N1), 01 December 2013 – 01 July 2015 (Ebola), 01 May 2015 – 01 December 2016 (Zika), and 01 November 2019 and 01 August 2021 (SARS-CoV-2)
Beginning of outbreak to present- 01 November 2002 – 01 November 2019 (SARS), 01 April 2009 – 01 August 2021 (H1N1), 01 December 2013 – 01 August 2021 (Ebola), 01 May 2015 – 01 August 2021 (Zika)
Article dually listed as an Ebola retraction
In conclusion, many COVID-19 retractions have occurred in PubMed indexed journals. This demonstrates that journals listed in major databases are still susceptible to retractions. There is also a remarkable influx in COVID-19 publications and retractions compared to previous outbreaks, possibly due to global interest, expedited review processes, and journal interest in COVID-19. However, similar rates of retraction to the H1N1 pandemic and universal retraction rates suggest that research quality may be similar to previous outbreaks. While early studies expressed concern about the unusually large number of retractions, the retraction rate may eventually normalize over time. Future studies should assess retraction rates after a more traditional time frame for retractions has passed.
Declaration of Competing Interest
- The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape.PLOS Biol. 2021; 19e3000959
- Why has the number of scientific retractions increased?.PLoS One. 2013; 8: e68397
- The carnage of substandard research during the COVID-19 pandemic: a call for quality.J Med Ethics. 2020; 46: 803-807
- Academic journal retractions and the COVID-19 pandemic.J Prim Care Commun Health. 2021; 12 (21501327211015592)
- Publications and retracted articles of COVID-19 pharmacotherapy-related research: a systematic review.Sci Prog. 2021; 104 (00368504211016936)
- What a massive database of retracted papers reveals about science publishing's ‘death penalty.Science. 2018; https://doi.org/10.1126/science.aav8384
- Characteristics of scientific articles on COVID-19 published during the initial 3 months of the pandemic.Scientometrics. 2020; : 1-18
- Pandemic publishing poses a new COVID-19 challenge.Nat Hum Behav. 2020; 4: 666-669
- An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID-19).Acc Res. 2021; 28: 47-53
- Disgraced COVID-19 studies are still routinely cited.Science. 2021; 371: 331-332
Published online: February 02, 2022
Accepted: January 26, 2022
Received: September 18, 2021
© 2022 Southern Society for Clinical Investigation. Published by Elsevier Inc. All rights reserved.