Assessing the level of evidence in the orthopaedic literature, 2013–2018: a review of 3449 articles in leading orthopaedic journals
Patient Safety in Surgery volume 14, Article number: 23 (2020)
In biomedical research, level of evidence (LOE) indicates the quality of a study. Recent studies evaluating orthopaedic trauma literature between 1998 and 2013 have indicated that LOE in this field has improved. The objective of this study was to determine the validity of one such study by 1) comparing our results and how they relate to more recent years of publications; and 2) assessing how our findings may be used to estimate future changes.
A total of 3449 articles published from 2013 to 2018 in The Journal of Orthopaedic Trauma (JOT); Journal of Bone and Joint Surgery, American Volume (JBJS-Am); and Clinical Orthopaedics and Related Research (CORR) were evaluated for their LOE. Articles published in JBJS-Am or CORR were classified as trauma or nontrauma studies; articles published in JOT were considered trauma studies. Articles were assigned a LOE using guidance published by JBJS-Am in 2015.
The percentage of total high-level (level I or II) trauma and nontrauma articles published in JOT, JBJS-Am, and CORR decreased from 2013 to 2018 (trauma 23.1 to 19.2%, p = 0.190; nontrauma 28.8 to 24.9%, p = 0.037). JBJS-Am published the highest percentage of level-I trauma studies, and CORR published the lowest percentage of level-IV studies. JBJS-Am and CORR published higher percentages of level-I trauma studies and lower percentages of level-IV nontrauma studies than all trauma studies.
Based on our results we cannot validate the findings of previous studies as we found the overall LOE of both trauma and nontrauma orthopaedic literature has decreased in recent years. JBJS-Am published a greater percentage of high-level studies than did JOT and CORR. Although the number and percentage of high-level studies published in JOT increased during the study period, it still lagged behind JBJS-Am and CORR.
Evidence-based medicine (EBM) is medical practice intended to optimize decision-making, treatment, and diagnosis based on well-designed research. The concept of level of evidence (LOE) was introduced in 1979 by the Canadian Task Force on the Periodic Health Examination  and was designed to support EBM, stratifying research designs according to their validity. The Oxford Centre for Evidence-Based Medicine (OCEBM) published the guidelines for LOE for prognostic, diagnostic, and therapeutic studies in 2009 . Level-I research is the strongest evidence available, with data acquired through randomized controlled trials or meta-analysis of randomized controlled trials, whereas level-V evidence is the weakest evidence, representing cadaveric studies, case reports, or expert opinion.
Implementation of the LOE system encourages authors to improve the strength of their studies by avoiding bias, including control groups, and using methodical steps throughout the study . The Journal of Bone and Joint Surgery, American Volume (JBJS-Am) was the first orthopaedic journal to apply the classification system and has assigned LOE for all studies it has published since 2003 .
In 2015, JBJS-Am updated their LOE criteria by modifying slightly the updated recommendations published by the OCEBM . This change helped emphasize the clinical implications of research findings while implementing a more comprehensive assessment of the publication’s LOE. This update gave authors the flexibility to assign a grade of level I through IV, with a higher level indicating higher quality and a lower level indicating poorer quality or inconsiderable clinical effect.
Studies examining the LOE of scholarly orthopaedic articles have reported that journal impact factor is positively correlated with the proportion of “high-level” (level I or II) articles a journal publishes, and that journals with lower impact factors publish articles with an inconsistent LOE [6,7,8,9]. Prior studies such as the one conducted by Scheschuk et al. assessing LOE in orthopaedic literature between 1998 and 2013 have demonstrated a decrease in the number of low-level studies, an increase in high-level studies, and that orthopaedic trauma journals still publish a higher proportion of low-level articles [10,11,12]. The purpose of this study was to assess the validity of a previous published article  by 1) comparing our results and how they relate to more recent years of publications; and 2) assess how our findings may be used to estimate future changes.
Articles from the 3 most commonly used English-language journals publishing orthopaedic trauma and nontrauma research were evaluated for the LOE of their articles: the Journal of Orthopaedic Trauma (JOT) (trauma journal) and Journal of Bone and Joint Surgery, American Volume (JBJS-Am) and Clinical Orthopaedics and Related Research (CORR) (nontrauma journals). All articles published by these journals from 2013 through 2018 were evaluated. In accordance with the updated JBJS-Am LOE criteria  we defined high-level articles as level I or II and low-level articles as Level III or IV.
We excluded level-V studies (i.e., anatomical, animal, cadaveric, basic science, biomechanical, simulation, and educational studies; case reports; technical notes; expert opinions; surveys; publication analyses; and review articles) . We also excluded studies with unclear methods (LOE not listed or study methods not clearly defined), those for which full text was unavailable, and those published in supplementary issues of the journals.
Trauma vs. nontrauma
We categorized the articles in JBJS-Am and CORR as either trauma or nontrauma studies. Nontrauma studies were those concerning congenital or developmental abnormalities, arthroplasty, tumors, sports medicine, nontraumatic spine disorders, hand surgery, or nontraumatic foot and ankle surgery. Trauma studies were those concerning trauma to the upper or lower extremities, spine, pelvis, acetabulum, foot, or ankle.
Each study was reviewed by 1 of 2 authors. We assessed interobserver agreement by randomly selecting and assigning articles that met our inclusion criteria to reviewer 1 or reviewer 2, who assigned a LOE. In case of disagreement between reviewers, a third author (reviewer 3) was asked to judge the LOE. The percentage of agreement was 95% for study type (κ = 0.91) and 82% for LOE (κ = 0.73).
The proportions of articles in each LOE were compared using chi-squared tests. The Cochran-Mantel-Haenszel test was used to adjust research years as a stratification factor. P < 0.05 was considered statistically significant.
After classifying the LOE of each article, we analyzed the following: 1) LOE of all orthopaedic trauma articles compared with all nontrauma articles; 2) LOE of JOT articles versus all trauma and nontrauma articles in JBJS-Am and CORR; 3) LOE of JOT articles versus JBJS-Am and CORR trauma articles only; 4) changes in LOE during the study period for JOT and for trauma articles in JBJS-Am and CORR by journal (Fig. 1); and 5) changes in LOE for all trauma and non-trauma articles over the duration of the study period. R, version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria), was used for statistical analysis.
During the study period, 4664 articles were published by the 3 journals, of which 3449 (74%) met our study criteria. The sample comprised 1378 articles (40%) from CORR, 1272 (37%) from JBJS-Am, and 799 (23%) from JOT. Only 1217 articles (35%) were classified as trauma studies across all 3 journals.
LOE of all reviewed studies for all journals
The most common LOE was level III (1362 articles, 39%), followed by level IV (1232 articles, 36%), level II (464 articles, 13%) and level I (391 articles, 11%) (Fig. 2). JBJS-Am published the highest percentage of level-I studies among the 3 journals (P < 0.001). CORR published the lowest percentage of level-IV studies (P < 0.001; Fig. 2).
LOE of trauma vs. nontrauma studies
Trauma literature (470 articles; 38.6%) had a significantly higher percentage of level IV articles than nontrauma (762 articles; 34.1%) (P = 0.01). The percentage of level I articles in nontrauma literature (269 articles; 12.1%) was higher than that of the trauma literature (122 articles; 10.0%), although this finding was not significant (P = 0.08) (Fig. 3). The total percentage of high-level articles (level I and II) over the study period for trauma and nontrauma literature was 25.6% (571) and 23.3% (284), respectively.
LOE of trauma vs. nontrauma studies in JBJS-am and CORR
In JBJS-Am, we found a significant difference in the LOE between trauma and nontrauma studies (P = 0.029), with trauma (54 articles; 22.2%) demonstrating a higher percentage of level-I articles than nontrauma (166 articles; 16.1%). We found no significant difference in LOE between trauma and nontrauma articles in CORR (P = 0.937) (Fig. 4).
High-level vs low-level studies by journal
A significantly higher percentage of level-I and level-II studies were published in JBJS-Am (31%) compared with JOT (22%) and CORR (21%) (P < 0.001). There was no such difference when comparing JOT and CORR (P = 0.77; Fig. 5).
When analyzing the percentages of high-level studies by year, we found significant differences between JBJS-Am vs. JOT (P = 0.005) and JBJS-Am vs. CORR (P < 0.001). No such difference was found between JOT vs. CORR (P = 0.674). JOT published a smaller percentage of high-level studies (22%) compared with the percentage of high-level trauma studies published in JBJS-Am (32%) (P = 0.001). No difference was found between the percentages of high-level trauma studies published in JOT (22%) and CORR (19%) (P = 0.609). We also found no significant difference in LOE between trauma vs. nontrauma studies published by JBJS-Am (P = 0.695) or CORR (P = 0.608).
When analyzing LOE in each journal over time, JOT had a significant decrease in level-I (7.3% in 2013 to 1.9% in 2018) and level-IV articles (49% in 2013 to 37% in 2018) and an increase in level-III articles (34% in 2013 to 45% in 2018). JBJS-Am had a significant decrease in level-II articles (21% in 2013 to 6.6% in 2018), and minimal changes in level-I, level-III, and level-IV articles. The LOE in CORR improved, with a significant decrease in level-IV articles (40% in 2013 to 11% in 2018), a significant increase in level-III articles (37% in 2013 to 62% in 2018), but minimal changes in level-I and level-II articles (Fig. 6a-b).
High-level vs. low level studies over time (trauma vs non-trauma)
After combining all studies from the three included journals, we found that between 2013 and 2018, there was a non-significant downward trend in the percentage of high-level trauma articles (23% in 2013 to 19% in 2018; P = 0.190) and a significant downward trend in percentage of high-level non-trauma articles over time (29% in 2013 to 25% in 2018; P = 0.037) (Fig. 6c).
The purpose of this study was to assess the validity of the findings of the study conducted by Scheschuk et al.  by evaluating the LOE of trauma and nontrauma research published in 3 major orthopaedic journals, comparing it to their findings and examining it has changed in recent years, and assessing how our findings may be used to estimate future changes. From 2013 through 2018, we found that most articles published by these journals (63%) were low-level (level III or IV) studies, and most (63%) focused on treatment (as opposed to diagnosis, prognosis, or economics). Only 30.4% of trauma and 34.4% of non-trauma articles were classified as level I or level II, with downward trends for both (although only the non-trauma trend was significant). JBJS-Am published the highest percentages of level-I and level-II studies, and CORR published the highest percentage of level-III studies and the lowest percentage of level-IV studies. JOT published the lowest percentage of level-I studies of all 3 journals.
Our finding regarding the decrease in high-level studies deviates from previous literature assessing levels of evidence in orthopaedics over time. Scheschuk et al. examined JBJS-Am, CORR, and JOT articles for LOE in 1998, 2003, 2008, and 2013, reporting an upward trend in level-I and -II studies over their study period . Cunningham et al. examined eight orthopaedic subspecialty journals for articles published in 2000, 2005, and 2010 and reported an increase the proportion of level-I and -II studies over time . The current study provides an update and new look at the state of orthopaedic literature, both trauma and non-trauma. One potential explanation for the difference in studies is the increased use of national databases in orthopaedic research, particularly since 2013 [14, 15]. These databases can provide large volumes of retrospective data but are still considered level-III evidence. Another reason could be year-to-year variation; we saw a rise of high-level trauma articles and a decrease in high-level non-trauma articles in the year 2015. The selective sampling of certain years may have caused the prior studies to miss important data in the years between. Our findings suggest that the increased implementation of the LOE system has not been effective in improving the quality of orthopaedic literature in recent years; thus, we were unable to validate the findings of Scheschuk et al. .
Across all 3 journals, high-level trauma studies were published more often by JBJS-Am than by JOT or CORR. No significant differences in LOE were found between JBJS-Am and CORR when comparing all articles. Okike et al. showed that LOE was the only factor significantly associated with journals’ acceptance of submitted manuscripts, and that level-III and level-IV studies were least likely to be accepted by JBJS-Am . Because of the nature of orthopaedic trauma, in which patients are often treated on an emergency basis, designing high-level studies, such as randomized controlled trials, is difficult. Level-III and level-IV studies are the most frequently published clinical orthopaedic trauma studies, likely because of their greater feasibility and lower cost compared with high-level studies. Blinding can be difficult to implement, and the use of placebo controls would be unethical in orthopaedic trauma studies. Further, obtaining a large enough sample size to achieve adequate statistical power in orthopaedic trauma is difficult. Bhandari et al. reported that the quality of randomized trials in JBJS-Am could be improved if blinding, randomization concealment, and patient inclusion/exclusion criteria were made consistent across studies . Retrospective studies and case series, when performed with rigorous methods, were also considered valuable studies because of design methods.
Our study has several strengths. First, we analyzed all published articles during a 6-year period from 3 major orthopaedic journals that publish orthopaedic trauma research. This contrasts with previous publications, which evaluated articles in an inconsistent manner, reviewing only certain years across a time period and comparing them to articles that were published before the implementation of the LOE system established by JBJS-Am in 2003 [10,11,12]. Second, our comprehensive review eliminates selection bias that might arise from using an inconsistent review period. Third, we confirmed good interobserver agreement in LOE assessment across all 3 journals for the entire study period.
A limitation of our study is our use of the LOE grading system updated by JBJS-Am 2015. Although the system is simple and reproducible, it allows reviewers to grade LOE downward or upward based on the study quality and methodology used, which can be considered as a confounding bias to properly effect results. Furthermore, this may also create differences in LOE when graded by different reviewers.
We found a decreasing proportion of high-level evidence across our study period, both in trauma and non-trauma articles. Of the 3 journals analyzed, JBJS-Am published the greatest percentage of high-level studies. JOT published a lower percentage of high-level articles than did JBJS-Am and CORR. A higher percentage of level-I articles were published in JBJS-Am and CORR, whereas no significant difference was found among the 3 journals.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
level of evidence
Oxford Centre for Evidence-Based Medicine
Journal of Bone and Joint Surgery, American Volume
The Journal of Orthopaedic Trauma
Clinical Orthopaedics and Related Research
The periodic health examination. Canadian task force on the periodic health examination. Can Med Assoc J. 1979;121(9):1193–254.
Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes B, et al. Oxford Centre for evidence-based medicine-levels of evidence (march 2009). 2009.
Slobogean G, Bhandari M. Introducing levels of evidence to the Journal of Orthopaedic Trauma: implementation and future directions. J Orthop Trauma. 2012;26(3):127–8.
Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal: LWW; 2003.
Marx RG, Wilson SM, Swiontkowski MF. Updating the assignment of levels of evidence. J Bone Joint Surg Am. 2015;97(1):1–2.
Cvetanovich GL, Fillingham YA, Harris JD, Erickson BJ, Verma NN, Bach BR Jr. Publication and level of evidence trends in the American journal of sports medicine from 1996 to 2011. Am J Sports Med. 2015;43(1):220–5.
Murphy RF, Cibulas AM, Sawyer JR, Spence DD, Kelly DM. Levels of evidence in the journal of pediatric Orthopaedics: update and comparison to the journal of bone and joint surgery. J Pediatr Orthop. 2015;35(7):779–81.
Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P III, Bhandari M. Level of evidence in orthopaedic journals. JBJS. 2005;87(12):2632–8.
Wupperman R, Davis R, Obremskey WT. Level of evidence in spine compared to other orthopedic journals. Spine. 2007;32(3):388–93.
Cunningham BP, Harmsen S, Kweon C, Patterson J, Waldrop R, McLaren A, et al. Have levels of evidence improved the quality of orthopaedic research? Clin Orthopaedics Related Res. 2013;471(11):3679–86.
Little Z, Newman S, Dodds A, Spicer D. Increase in quality and quantity of orthopaedic studies from 2002 to 2012. J Orthop Surg. 2015;23(3):375–8.
Scheschuk JP, Mostello AJ, Lombardi NJ, Maltenfort MG, Freedman KB, Tjoumakaris FP. Levels of evidence in orthopaedic trauma literature. J Orthop Trauma. 2016;30(7):362–6.
Fayaz HC, Haas N, Kellam J, Bavonratanavech S, Parvizi J, Dyer G, et al. Improvement of research quality in the fields of orthopaedics and trauma—a global perspective. Int Orthop. 2013;37(7):1205–12.
Karlson NW, Nezwek TA, Menendez ME, Tybor D, Salzler MJ. Increased utilization of American administrative databases and large-scale clinical registries in orthopaedic research, 1996 to 2016. JAAOS Global Res Rev. 2018;2(11).
Sivasundaram L, Pannell W, Heckmann N, Alluri RK, Omid R, George F, et al. Database studies: an increasing trend in the United States orthopaedic literature. Curr Orthopaedic Pract. 2016;27(6):673–9.
Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the journal of bone and joint surgery (American volume). JBJS. 2008;90(3):595–601.
Bhandari M, Richards RR, Sprague S, Schemitsch EH. The quality of reporting of randomized trials in the journal of bone and joint surgery from 1988 through 2000. JBJS. 2002;84(3):388–96.
Erik Hasenboehler receives a yearly grant from DePuy Synthes Trauma for a research fellowship. No additional funding was received for publication. He is also a paid consultant for DePuy Synthes Trauma, a paid lecturer and faculty for AO North American Trauma, and has stock ownership in Summit Med Ventures. For the remaining authors, no conflicts were declared.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Luksameearunothai, K., Chaudhry, Y., Thamyongkit, S. et al. Assessing the level of evidence in the orthopaedic literature, 2013–2018: a review of 3449 articles in leading orthopaedic journals. Patient Saf Surg 14, 23 (2020). https://doi.org/10.1186/s13037-020-00246-6