Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue?
© Hasenboehler et al; licensee BioMed Central Ltd. 2007
Received: 17 October 2007
Accepted: 27 November 2007
Published: 27 November 2007
Research articles reporting positive findings in the fields of orthopedic and general surgery appear to be represented at a considerably higher prevalence in the peer-reviewed literature, compared to published studies on negative or neutral data. This "publication bias" may alter the balance of the available evidence-based literature and may affect patient safety in surgery by depriving important information from unpublished negative studies.
A comprehensive review of all published articles in a defined 7-year period was performed in 12 representative journals in the fields of orthopedic and general surgery. Every article published in all volumes of these journals between January 2000 and December 2006 was reviewed and rated by three investigators. Rating of articles was performed according to a uniform, standardized algorithm. All original articles were stratified into "positive", "negative" or "neutral", depending on the reported results. All non-original papers were excluded from analysis.
A total of 30,197 publications were reviewed over a 7-year time-period. After excluding all non-original articles, a total of 16,397 original papers were included in the final analysis. Of these, 12,251 (74%) articles were found to report positive findings, 2,709 (17%) reported negative results, and 1,437 (9%) were neutral. A similar publication pattern was found among all years and all journals analyzed. Altogether, 91% of all original papers reported significant data (positive or negative), whereas only 9% were neutral studies that did not report any significant findings.
There is a disproportionately high number of articles reporting positive results published in the surgical literature. A bias towards publishing positive data will systematically overestimate the clinical relevance of treatment effects by disregarding important information derived from unpublished negative studies. This "publication bias" remains an area of concern and may affect the quality of care of patients undergoing surgical procedures.
Peer-reviewed biomedical journals are more likely to publish original papers reporting positive results than studies with negative data [1–4]. This "publication bias," also termed "positive-outcome-bias," has been recognized and described in the internal medicine literature [5–8]. In addition, commercial funding of clinical studies has been described as an independent variable associated with the frequency of publication of positive articles [9–11]. Evidence-based decision making processes on therapeutic modalities rely on the availability of unbiased, balanced, and objective data from published studies, independent of the reported outcome . Particularly, high-quality systematic meta-analyses are corrupted by the "positive-outcome-bias" of individual studies, rendering clinical recommendations flawed towards a positive effect of specific treatment strategies [13, 14]. In the era of evidence-based medicine, this prevalent, often unrecognized, publication bias poses a severe challenge to patient safety by promoting unjustified therapeutic concepts [13, 15–17]. This notion is particularly true in surgical disciplines, since specific surgical techniques are frequently adapted or abandoned depending on the current status of evidence-based recommendations in the scientific literature [18–20].
Despite its potentially devastating clinical impact, the prevalence of a "publication bias" remains largely unexplored in the field of surgery and surgical subspecialties [9, 21–23]. The present study was designed to analyze a potential "positive outcome bias" in 12 representative orthopedic and general surgery journals over a time-period of 7 years. We hypothesized that studies reporting positive outcomes are represented in the surgical literature at considerably higher prevalence than those which report negative or neutral findings.
Overview on the 12 selected peer-reviewed journals analyzed in this study. Journal ranking1 and journal impact factors2 are derived from the Institute for Scientific Information (ISI) database from 2006. Journal ranking was determined by the specific subject categories of Surgery* and Orthopedics§. The numbers of articles analyzed in this study are shown as the overall publications3 (including non-original articles) and as original articles exclusively4. Articles were assessed in all individual journals' volumes from 01/2000 to 12/2006.
Total number of articles analyzed3
Number of original papers analyzed4
Annals of Surgery
Journal of Bone and Joint Surgery (American)
British Journal of Surgery
Journal of Bone and Joint Surgery (British)
Journal of Orthopaedic Trauma
Archives of Surgery
Journal of the American College of Surgeons
Journal of Trauma
World Journal of Surgery
This study demonstrates for the first time, to our knowledge, a high prevalence of "positive outcome-bias" in the preeminent orthopedic and surgical literature. The disproportion of published positive studies, compared to negative or neutral papers, remained relatively constant over the seven years (2000–2006) and among the twelve journals reviewed in this study. Even more striking was the discrepancy between published original papers with significant findings (positive or negative) compared to purely "neutral" studies which did not report any significant findings (91% vs. 9%).
Our results are consistent with prior investigations which revealed a high prevalence of a "positive-outcome" publication bias in medical journals [7, 15, 16]. Other groups have examined the likelihood of a subsequent publication of positive versus negative studies from submitted manuscripts, unpublished manuscripts, or manuscripts derived from abstracts presented at scientific meetings [6, 8, 23–25]. Easterbrook et al. retrospectively reviewed 487 projects submitted for publication and found a high odds-ratio in favor of publishing articles with a statistically significant outcome compared to those manuscripts which reported no difference between the study groups . Dickersin and Min performed an analysis of 198 NIH-funded trials . They reported that those trials with "significant" results were more likely to be published than studies with "non-significant" data, by an adjusted odds-ratio of 12.30 .
In the orthopedic literature, Harris and coworkers reviewed all abstracts presented at the annual meeting American Academy of Orthopaedic Surgeons (AAOS) meeting in 1999 . They found that articles with a positive outcome and significant results were more likely to be published within the following years . Callaham and colleagues studied the publication pattern of abstracts presented at a 1991 major annual US research meeting in the field of emergency medicine . The authors found that articles with a positive outcome were more likely to be published in peer-reviewed journals than negative studies .
Study funding patterns have furthermore been identified as independent factors that influence the publication bias. In particular, the presence of corporate/industrial funding was shown to be associated with a higher prevalence of published studies with positive findings. In the orthopedic literature, Leopold et al. found a significant association between commercially funded studies being more likely to have positive outcomes in publications, compared to unfunded studies . This industrial funding driven bias has been confirmed in other publications [5, 8, 10]. Lynch et al. recently challenged this hypothesis by examining articles submitted to the American edition of the Journal of Bone and Joint Surgery and concluded that publications from commercially funded studies were not more likely to present positive data than non-funded studies . However, this study was limited in design in that analysis was restricted to articles related to hip and knee arthroplasty during a seventeen-month period .
Finally, there appears to be a significant bias against negative studies in newspaper reports of medical research . Thus, as for the scientist and clinician, the lay reader is also exposed to filtered information regarding the outcome of clinical studies.
The present study has a number of strengths and weaknesses. To our knowledge, it is the largest analysis of its kind in the medical literature. The 12 selected journals represent relevant sources of clinical knowledge, and are widely used as evidence-based decision-making tools in the fields of general, trauma, and orthopedic surgery. The algorithm used for stratifying articles is widely inclusive for all published original articles in the screened journals. The results were cross-checked and revisited in regular intervals by three different investigators and approved by the senior author. This allowed for a reliable, objective, and reproducible method of scoring.
Some drawbacks and limitations of this study must be addressed. Despite the uniform algorithm used for screening and classification of articles, there remains some degree of inherent inter-observer variability in the assessment of publications. The algorithm was developed by the authors for the present study and has not been externally validated by other groups. Furthermore, study analysis was limited to the last seven years in a limited number of journals in orthopedics and general surgery. Our results are neither representative for other surgical subspecialties and other fields of medicine, nor for papers published prior to the year 2,000.
A further drawback of this study is that we did not stratify between in vitro and experimental studies vs. clinical trials. A bias in the latter group is certainly associated with a higher risk for patients, since clinical treatment trials constitute the main basis for decision making in a clinical setting.
Another limitation is that published articles were assessed exclusively, without accounting for submitted and unpublished manuscripts. Thus, our study design does not determine whether the positive publication bias occurred at the level of manuscript submission or at the editorial decision-making level. The former notion would imply a bias towards a preferential submission of manuscripts with positive outcomes and statistically significant results; the latter would suggest a bias towards preferential publication of positive articles by peer-reviewers and editors. In this regard, Olson and colleagues revealed that the "positive-outcome-bias" does not occur at the editorial decision making stage, based on assessment of manuscripts submitted to the Journal of the American Medical Association (JAMA) between 1996 and 1999 .
Based on the large and encompassing analysis presented here, it is apparent that research reporting negative outcomes and/or statistically insignificant results is underrepresented in the surgical literature. The clinical implications of this trend are of concern due to the potential impact on patient care. The "positive-publication-bias" may alter the balance of the available evidence-based literature and may negatively affect recommendations and guidelines derived from systematic meta-analyses [13, 20]. Recently, a new open-access online journal was launched which is devoted exclusively to publishing negative results in biomedicine . In our opinion, it is imperative to promote the submission and publication of studies with negative outcomes and insignificant results, in order to ensure a balanced availability of evidence-based data for clinical decision-making regarding the best treatment modality for our patients.
- Olson CM: Publication bias. Acad Emerg Med. 1994, 1 (3): 207-209.View ArticlePubMedGoogle Scholar
- Dickersin K: The existence of publication bias and risk factors for its occurrence. Jama. 1990, 263 (10): 1385-1389. 10.1001/jama.263.10.1385.View ArticlePubMedGoogle Scholar
- Angell M: Negative studies. N Engl J Med. 1989, 321 (7): 464-466.View ArticlePubMedGoogle Scholar
- Callaham M, Wears RL, Weber E: Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Jama. 2002, 287 (21): 2847-2850. 10.1001/jama.287.21.2847.View ArticlePubMedGoogle Scholar
- Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H: Publication bias and clinical trials. Control Clin Trials. 1987, 8 (4): 343-353. 10.1016/0197-2456(87)90155-3.View ArticlePubMedGoogle Scholar
- Dickersin K, Min YI: Publication bias: the problem that won't go away. Ann N Y Acad Sci. 1993, 703: 135-46; discussion 146-8. 10.1111/j.1749-6632.1993.tb26343.x.View ArticlePubMedGoogle Scholar
- Moscati R, Jehle D, Ellis D, Fiorello A, Landi M: Positive-outcome bias: comparison of emergency medicine and general medicine literatures. Acad Emerg Med. 1994, 1 (3): 267-271.View ArticlePubMedGoogle Scholar
- Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337 (8746): 867-872. 10.1016/0140-6736(91)90201-Y.View ArticlePubMedGoogle Scholar
- Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN: Industry support and correlation to study outcome for papers published in Spine. Spine. 2005, 30 (9): 1099-104; discussion 1105. 10.1097/01.brs.0000161004.15308.b4.View ArticlePubMedGoogle Scholar
- Ezzet KA: The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003, 18 (7 Suppl 1): 138-145. 10.1016/S0883-5403(03)00289-4.View ArticlePubMedGoogle Scholar
- Kjaergard LL, Als-Nielsen B: Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ. Bmj. 2002, 325 (7358): 249-10.1136/bmj.325.7358.249.PubMed CentralView ArticlePubMedGoogle Scholar
- Copas J, Jackson D: A bound for publication bias based on the fraction of unpublished studies. Biometrics. 2004, 60 (1): 146-153. 10.1111/j.0006-341X.2004.00161.x.View ArticlePubMedGoogle Scholar
- Jackson D: The implications of publication bias for meta-analysis' other parameter. Stat Med. 2006, 25 (17): 2911-2921. 10.1002/sim.2293.View ArticlePubMedGoogle Scholar
- Henmi M, Copas JB, Eguchi S: Confidence intervals and P-values for meta-analysis with publication bias. Biometrics. 2007, 63 (2): 475-482. 10.1111/j.1541-0420.2006.00705.x.View ArticlePubMedGoogle Scholar
- Johnson RT, Dickersin K: Publication bias against negative results from clinical trials: three of the seven deadly sins. Nat Clin Pract Neurol. 2007Google Scholar
- Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Bmj. 1997, 315 (7109): 640-645.PubMed CentralView ArticlePubMedGoogle Scholar
- Koren G, Klein N: Bias against negative studies in newspaper reports of medical research. Jama. 1991, 266 (13): 1824-1826. 10.1001/jama.266.13.1824.View ArticlePubMedGoogle Scholar
- Scott IA, Lodge RS, Russell DM: Evidence-based guide to perioperative medicine. Intern Med J. 2007, 37 (6): 389-401. 10.1111/j.1445-5994.2007.01344.x.View ArticlePubMedGoogle Scholar
- Siparsky P, Ryzewicz M, Peterson B, Bartz R: Arthroscopic treatment of osteoarthritis of the knee: are there any evidence-based indications?. Clin Orthop Relat Res. 2007, 455: 107-112. 10.1097/BLO.0b013e31802fc18c.View ArticlePubMedGoogle Scholar
- Spahn DR, Cerny V, Coats TJ, Duranteau J, Fernandez-Mondejar E, Gordini G, Stahel PF, Hunt BJ, Komadina R, Neugebauer E, Ozier Y, Riddez L, Schultz A, Vincent JL, Rossaint R: Management of bleeding following major trauma: a European guideline. Crit Care. 2007, 11 (1): R17-10.1186/cc5686.PubMed CentralView ArticlePubMedGoogle Scholar
- Lynch JR, Cunningham MR, Warme WJ, Schaad DC, Wolf FM, Leopold SS: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007, 89 (5): 1010-1018. 10.2106/JBJS.F.01152.View ArticlePubMedGoogle Scholar
- Leopold SS, Warme WJ, Fritz Braunlich E, Shott S: Association between funding source and study outcome in orthopaedic research. Clin Orthop Relat Res. 2003, 293-301. 10.1097/01.blo.0000093888.12372.d9.Google Scholar
- Hamlet WP, Fletcher A, Meals RA: Publication patterns of papers presented at the Annual Meeting of The American Academy of Orthopaedic Surgeons. J Bone Joint Surg Am. 1997, 79 (8): 1138-1143.PubMedGoogle Scholar
- Callaham ML, Wears RL, Weber EJ, Barton C, Young G: Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Jama. 1998, 280 (3): 254-257. 10.1001/jama.280.3.254.View ArticlePubMedGoogle Scholar
- Harris IA, Mourad M, Kadir A, Solomon MJ, Young JM: Publication bias in abstracts presented to the annual meeting of the American Academy of Orthopaedic Surgeons. J Orthop Surg (Hong Kong). 2007, 15 (1): 62-66.Google Scholar
- Dickersin K, Min YI: NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993, Doc No 50:Google Scholar
- Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. Jama. 2002, 287 (21): 2825-2828. 10.1001/jama.287.21.2825.View ArticlePubMedGoogle Scholar
- Journal of Negative Results in Biomedicine. http://www.jnrbm.com
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.