- Open Access
Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue?
Patient Safety in Surgeryvolume 1, Article number: 4 (2007)
Research articles reporting positive findings in the fields of orthopedic and general surgery appear to be represented at a considerably higher prevalence in the peer-reviewed literature, compared to published studies on negative or neutral data. This "publication bias" may alter the balance of the available evidence-based literature and may affect patient safety in surgery by depriving important information from unpublished negative studies.
A comprehensive review of all published articles in a defined 7-year period was performed in 12 representative journals in the fields of orthopedic and general surgery. Every article published in all volumes of these journals between January 2000 and December 2006 was reviewed and rated by three investigators. Rating of articles was performed according to a uniform, standardized algorithm. All original articles were stratified into "positive", "negative" or "neutral", depending on the reported results. All non-original papers were excluded from analysis.
A total of 30,197 publications were reviewed over a 7-year time-period. After excluding all non-original articles, a total of 16,397 original papers were included in the final analysis. Of these, 12,251 (74%) articles were found to report positive findings, 2,709 (17%) reported negative results, and 1,437 (9%) were neutral. A similar publication pattern was found among all years and all journals analyzed. Altogether, 91% of all original papers reported significant data (positive or negative), whereas only 9% were neutral studies that did not report any significant findings.
There is a disproportionately high number of articles reporting positive results published in the surgical literature. A bias towards publishing positive data will systematically overestimate the clinical relevance of treatment effects by disregarding important information derived from unpublished negative studies. This "publication bias" remains an area of concern and may affect the quality of care of patients undergoing surgical procedures.
Peer-reviewed biomedical journals are more likely to publish original papers reporting positive results than studies with negative data [1–4]. This "publication bias," also termed "positive-outcome-bias," has been recognized and described in the internal medicine literature [5–8]. In addition, commercial funding of clinical studies has been described as an independent variable associated with the frequency of publication of positive articles [9–11]. Evidence-based decision making processes on therapeutic modalities rely on the availability of unbiased, balanced, and objective data from published studies, independent of the reported outcome . Particularly, high-quality systematic meta-analyses are corrupted by the "positive-outcome-bias" of individual studies, rendering clinical recommendations flawed towards a positive effect of specific treatment strategies [13, 14]. In the era of evidence-based medicine, this prevalent, often unrecognized, publication bias poses a severe challenge to patient safety by promoting unjustified therapeutic concepts [13, 15–17]. This notion is particularly true in surgical disciplines, since specific surgical techniques are frequently adapted or abandoned depending on the current status of evidence-based recommendations in the scientific literature [18–20].
Despite its potentially devastating clinical impact, the prevalence of a "publication bias" remains largely unexplored in the field of surgery and surgical subspecialties [9, 21–23]. The present study was designed to analyze a potential "positive outcome bias" in 12 representative orthopedic and general surgery journals over a time-period of 7 years. We hypothesized that studies reporting positive outcomes are represented in the surgical literature at considerably higher prevalence than those which report negative or neutral findings.
A comprehensive review of 12 selected journals from the disciplines of orthopedic and general surgery was systematically performed in all published volumes between January 2000 and December 2006. The representative journals within these disciplines were selected by the senior author. Selection was based on the appraisal that these journals represent relevant sources of clinical knowledge in general and orthopedic surgery, with a wide distribution of official journal rankings and impact factors (Table 1). All co-authors concurred with the selection of these journals at the time when the study design was established.
The published volumes of these 12 journals from 01/2000 through 12/2006 were screened online at the individual journals' homepage. Screening and stratification of articles was performed by three investigators (EAH, IC, and JTN) according to a defined algorithm (Figure 1). Based on these criteria, each published item was counted individually and evaluated with regard to the inclusion and exclusion criteria. All non-original articles were excluded from final analysis.
Of the included original papers, the abstract was analyzed first, and categorized into either "positive," "negative" or "neutral," based on the reported results. If the abstract was unable to clearly divulge the study results, the full paper was downloaded and the manuscript's entire text was analyzed to elicit the appropriate category. Ambiguous articles were re-assessed by the senior author (PFS) who made a final decision on classification. Results were compiled for each journal and each year individually. From these numbers, the percentage of positive, negative and neutral articles was calculated as:
A total of 30,197 manuscripts published in 12 journals between January 2000 and December 2006 were reviewed. Based on the screening algorithm, 13,800 articles were classified as non-original papers and thus excluded from further analysis. The remaining 16,397 original articles were classified according the definitions outlined in the methods section (Figure 1) into "positive" studies (n = 12,251; 74%), "negative" studies (n = 2,709; 17%), and "neutral" studies (n = 1,437; 9%; Figure 2A). A total of 14,960 papers (91%) reported significant findings (positive and negative combined), whereas only 9% of all studies were based on data without significant differences between the groups (Figure 2B).
There was no difference in this publication pattern over the years, from 2000 to 2006 (Figure 3). The subgroup analysis by individual journal revealed a similar publication pattern among all 12 journals, with a trend toward a higher prevalence of negative studies reported in trauma and orthopedic journals, compared to general surgery journals (Figure 4).
This study demonstrates for the first time, to our knowledge, a high prevalence of "positive outcome-bias" in the preeminent orthopedic and surgical literature. The disproportion of published positive studies, compared to negative or neutral papers, remained relatively constant over the seven years (2000–2006) and among the twelve journals reviewed in this study. Even more striking was the discrepancy between published original papers with significant findings (positive or negative) compared to purely "neutral" studies which did not report any significant findings (91% vs. 9%).
Our results are consistent with prior investigations which revealed a high prevalence of a "positive-outcome" publication bias in medical journals [7, 15, 16]. Other groups have examined the likelihood of a subsequent publication of positive versus negative studies from submitted manuscripts, unpublished manuscripts, or manuscripts derived from abstracts presented at scientific meetings [6, 8, 23–25]. Easterbrook et al. retrospectively reviewed 487 projects submitted for publication and found a high odds-ratio in favor of publishing articles with a statistically significant outcome compared to those manuscripts which reported no difference between the study groups . Dickersin and Min performed an analysis of 198 NIH-funded trials . They reported that those trials with "significant" results were more likely to be published than studies with "non-significant" data, by an adjusted odds-ratio of 12.30 .
In the orthopedic literature, Harris and coworkers reviewed all abstracts presented at the annual meeting American Academy of Orthopaedic Surgeons (AAOS) meeting in 1999 . They found that articles with a positive outcome and significant results were more likely to be published within the following years . Callaham and colleagues studied the publication pattern of abstracts presented at a 1991 major annual US research meeting in the field of emergency medicine . The authors found that articles with a positive outcome were more likely to be published in peer-reviewed journals than negative studies .
Study funding patterns have furthermore been identified as independent factors that influence the publication bias. In particular, the presence of corporate/industrial funding was shown to be associated with a higher prevalence of published studies with positive findings. In the orthopedic literature, Leopold et al. found a significant association between commercially funded studies being more likely to have positive outcomes in publications, compared to unfunded studies . This industrial funding driven bias has been confirmed in other publications [5, 8, 10]. Lynch et al. recently challenged this hypothesis by examining articles submitted to the American edition of the Journal of Bone and Joint Surgery and concluded that publications from commercially funded studies were not more likely to present positive data than non-funded studies . However, this study was limited in design in that analysis was restricted to articles related to hip and knee arthroplasty during a seventeen-month period .
Finally, there appears to be a significant bias against negative studies in newspaper reports of medical research . Thus, as for the scientist and clinician, the lay reader is also exposed to filtered information regarding the outcome of clinical studies.
The present study has a number of strengths and weaknesses. To our knowledge, it is the largest analysis of its kind in the medical literature. The 12 selected journals represent relevant sources of clinical knowledge, and are widely used as evidence-based decision-making tools in the fields of general, trauma, and orthopedic surgery. The algorithm used for stratifying articles is widely inclusive for all published original articles in the screened journals. The results were cross-checked and revisited in regular intervals by three different investigators and approved by the senior author. This allowed for a reliable, objective, and reproducible method of scoring.
Some drawbacks and limitations of this study must be addressed. Despite the uniform algorithm used for screening and classification of articles, there remains some degree of inherent inter-observer variability in the assessment of publications. The algorithm was developed by the authors for the present study and has not been externally validated by other groups. Furthermore, study analysis was limited to the last seven years in a limited number of journals in orthopedics and general surgery. Our results are neither representative for other surgical subspecialties and other fields of medicine, nor for papers published prior to the year 2,000.
A further drawback of this study is that we did not stratify between in vitro and experimental studies vs. clinical trials. A bias in the latter group is certainly associated with a higher risk for patients, since clinical treatment trials constitute the main basis for decision making in a clinical setting.
Another limitation is that published articles were assessed exclusively, without accounting for submitted and unpublished manuscripts. Thus, our study design does not determine whether the positive publication bias occurred at the level of manuscript submission or at the editorial decision-making level. The former notion would imply a bias towards a preferential submission of manuscripts with positive outcomes and statistically significant results; the latter would suggest a bias towards preferential publication of positive articles by peer-reviewers and editors. In this regard, Olson and colleagues revealed that the "positive-outcome-bias" does not occur at the editorial decision making stage, based on assessment of manuscripts submitted to the Journal of the American Medical Association (JAMA) between 1996 and 1999 .
Based on the large and encompassing analysis presented here, it is apparent that research reporting negative outcomes and/or statistically insignificant results is underrepresented in the surgical literature. The clinical implications of this trend are of concern due to the potential impact on patient care. The "positive-publication-bias" may alter the balance of the available evidence-based literature and may negatively affect recommendations and guidelines derived from systematic meta-analyses [13, 20]. Recently, a new open-access online journal was launched which is devoted exclusively to publishing negative results in biomedicine . In our opinion, it is imperative to promote the submission and publication of studies with negative outcomes and insignificant results, in order to ensure a balanced availability of evidence-based data for clinical decision-making regarding the best treatment modality for our patients.
Olson CM: Publication bias. Acad Emerg Med. 1994, 1 (3): 207-209.
Dickersin K: The existence of publication bias and risk factors for its occurrence. Jama. 1990, 263 (10): 1385-1389. 10.1001/jama.263.10.1385.
Angell M: Negative studies. N Engl J Med. 1989, 321 (7): 464-466.
Callaham M, Wears RL, Weber E: Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Jama. 2002, 287 (21): 2847-2850. 10.1001/jama.287.21.2847.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H: Publication bias and clinical trials. Control Clin Trials. 1987, 8 (4): 343-353. 10.1016/0197-2456(87)90155-3.
Dickersin K, Min YI: Publication bias: the problem that won't go away. Ann N Y Acad Sci. 1993, 703: 135-46; discussion 146-8. 10.1111/j.1749-6632.1993.tb26343.x.
Moscati R, Jehle D, Ellis D, Fiorello A, Landi M: Positive-outcome bias: comparison of emergency medicine and general medicine literatures. Acad Emerg Med. 1994, 1 (3): 267-271.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337 (8746): 867-872. 10.1016/0140-6736(91)90201-Y.
Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN: Industry support and correlation to study outcome for papers published in Spine. Spine. 2005, 30 (9): 1099-104; discussion 1105. 10.1097/01.brs.0000161004.15308.b4.
Ezzet KA: The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003, 18 (7 Suppl 1): 138-145. 10.1016/S0883-5403(03)00289-4.
Kjaergard LL, Als-Nielsen B: Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ. Bmj. 2002, 325 (7358): 249-10.1136/bmj.325.7358.249.
Copas J, Jackson D: A bound for publication bias based on the fraction of unpublished studies. Biometrics. 2004, 60 (1): 146-153. 10.1111/j.0006-341X.2004.00161.x.
Jackson D: The implications of publication bias for meta-analysis' other parameter. Stat Med. 2006, 25 (17): 2911-2921. 10.1002/sim.2293.
Henmi M, Copas JB, Eguchi S: Confidence intervals and P-values for meta-analysis with publication bias. Biometrics. 2007, 63 (2): 475-482. 10.1111/j.1541-0420.2006.00705.x.
Johnson RT, Dickersin K: Publication bias against negative results from clinical trials: three of the seven deadly sins. Nat Clin Pract Neurol. 2007
Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Bmj. 1997, 315 (7109): 640-645.
Koren G, Klein N: Bias against negative studies in newspaper reports of medical research. Jama. 1991, 266 (13): 1824-1826. 10.1001/jama.266.13.1824.
Scott IA, Lodge RS, Russell DM: Evidence-based guide to perioperative medicine. Intern Med J. 2007, 37 (6): 389-401. 10.1111/j.1445-5994.2007.01344.x.
Siparsky P, Ryzewicz M, Peterson B, Bartz R: Arthroscopic treatment of osteoarthritis of the knee: are there any evidence-based indications?. Clin Orthop Relat Res. 2007, 455: 107-112. 10.1097/BLO.0b013e31802fc18c.
Spahn DR, Cerny V, Coats TJ, Duranteau J, Fernandez-Mondejar E, Gordini G, Stahel PF, Hunt BJ, Komadina R, Neugebauer E, Ozier Y, Riddez L, Schultz A, Vincent JL, Rossaint R: Management of bleeding following major trauma: a European guideline. Crit Care. 2007, 11 (1): R17-10.1186/cc5686.
Lynch JR, Cunningham MR, Warme WJ, Schaad DC, Wolf FM, Leopold SS: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007, 89 (5): 1010-1018. 10.2106/JBJS.F.01152.
Leopold SS, Warme WJ, Fritz Braunlich E, Shott S: Association between funding source and study outcome in orthopaedic research. Clin Orthop Relat Res. 2003, 293-301. 10.1097/01.blo.0000093888.12372.d9.
Hamlet WP, Fletcher A, Meals RA: Publication patterns of papers presented at the Annual Meeting of The American Academy of Orthopaedic Surgeons. J Bone Joint Surg Am. 1997, 79 (8): 1138-1143.
Callaham ML, Wears RL, Weber EJ, Barton C, Young G: Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Jama. 1998, 280 (3): 254-257. 10.1001/jama.280.3.254.
Harris IA, Mourad M, Kadir A, Solomon MJ, Young JM: Publication bias in abstracts presented to the annual meeting of the American Academy of Orthopaedic Surgeons. J Orthop Surg (Hong Kong). 2007, 15 (1): 62-66.
Dickersin K, Min YI: NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993, Doc No 50:
Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. Jama. 2002, 287 (21): 2825-2828. 10.1001/jama.287.21.2825.
Journal of Negative Results in Biomedicine. http://www.jnrbm.com
PFS is a reviewer for Injury and J Trauma. WRS and BHZ are reviewers for JBJS-A, J Orthop Trauma, Injury, and J Trauma. The authors declare no other competing interests with regard to this manuscript.
WRS and PFS designed the study. EAH, IKC, JTN, and PFS wrote the manuscript. EAH, IKC, and JTN performed the screening and classification of published articles. PFS cross-checked the classifications. WRS and BHZ helped with final analysis of the data and editing of the manuscript. All authors read and approved the final version of the manuscript.