Has the Level of Evidence in Orthopaedic Trauma and Nontrauma Publications Improved? A Publication Review of 3 Major Orthopaedic Journals

Background In biomedical research, level of evidence (LOE) indicates the quality of a study. Recent studies evaluating orthopaedic trauma literature between 1998-2013 have indicated that LOE in this �eld has improved. The objective of this study was to determine the validity of one such study by 1) comparing our results and how they relate to more recent years of publications; and 2) assessing how our �ndings may be used to estimate future changes. Methods A total of 3,449 articles published from 2013 to 2018 in The Journal of Orthopaedic Trauma (JOT); Journal of Bone and Joint Surgery, American Volume (JBJS-Am); and Clinical Orthopaedics and Related Research (CORR) were evaluated for their LOE. Articles published in JBJS-Am or CORR were classi�ed as trauma or nontrauma studies; articles published in JOT were considered trauma studies. Articles were assigned a LOE using guidance published by JBJS-Am in 2015. Results The percentage of total high-level (level I or II) trauma and nontrauma articles published in JOT, JBJS-Am, and CORR decreased from 2013 to 2018 (trauma 23.1% to 19.2%, p = 0.190; nontrauma 28.8% to 24.9%, p = 0.037). JBJS-Am published the highest percentage of level-I trauma studies, and CORR published the lowest percentage of level-IV studies. JBJS-Am and CORR published higher percentages of level-I trauma studies and lower percentages of level-IV nontrauma studies than all trauma studies. Conclusions Based on our results we cannot validate the �ndings of previous studies as we found the overall LOE of both trauma and nontrauma orthopaedic literature has decreased in recent years. JBJS-Am published a greater percentage of high-level studies than did JOT and CORR. Although the number and percentage of high-level studies published in JOT increased during the study period, it still lagged behind JBJS-Am and CORR.


Background
Evidence-based medicine (EBM) is medical practice intended to optimize decision-making, treatment, and diagnosis based on well-designed research.The concept of level of evidence (LOE) was introduced in 1979 by the Canadian Task Force on the Periodic Health Examination (1) and was designed to support EBM, stratifying research designs according to their validity.The Oxford Centre for Evidence-Based Medicine (OCEBM) published the guidelines for LOE for prognostic, diagnostic, and therapeutic studies in 2009 (2).Level-I research is the strongest evidence available, with data acquired through randomized controlled trials or meta-analysis of randomized controlled trials, whereas level-V evidence is the weakest evidence, representing cadaveric studies, case reports, or expert opinion.
Implementation of the LOE system encourages authors to improve the strength of their studies by avoiding bias, including control groups, and using methodical steps throughout the study (3).The Journal of Bone and Joint Surgery, American Volume (JBJS) was the rst orthopaedic journal to apply the classi cation system and has assigned LOE for all studies it has published since 2003 (4).
In 2015, JBJS updated their LOE criteria by modifying slightly the updated recommendations published by the OCEBM (5).This change helped emphasize the clinical implications of research ndings while implementing a more comprehensive assessment of the publication's LOE.This update gave authors the exibility to assign a grade of level I through IV, with a higher level indicating higher quality and a lower level indicating poorer quality or inconsiderable clinical effect.
Studies examining the LOE of scholarly orthopaedic articles have reported that journal impact factor is positively correlated with the proportion of "high-level" (level I or II) articles a journal publishes, and that journals with lower impact factors publish articles with an inconsistent LOE (6-9).Prior studies such as the one conducted by Scheschuk et.al. assessing LOE in orthopaedic literature between 1998-2013 have demonstrated a decrease in the number of low-level studies, an increase in high-level studies, and that orthopaedic trauma journals still publish a higher proportion of low-level articles (10)(11)(12).The purpose of this study was to assess the validity of a previous published article (12) by 1) comparing our results and how they relate to more recent years of publications; and 2) assess how our ndings may be used to estimate future changes.

Journal Selection
Articles from the 3 most commonly used English-language journals publishing orthopaedic trauma and nontrauma research were evaluated for the LOE of their articles: the Journal of Orthopaedic Trauma (JOT) (trauma journal) and Journal of Bone and Joint Surgery, American Volume (JBJS) and Clinical Orthopaedics and Related Research (CORR) (nontrauma journals).All articles published by these journals from 2013 through 2018 were evaluated.In accordance with the updated JBJS LOE criteria (5) we de ned high-level articles as level I or II and low-level articles as Level III or IV.

Exclusion Criteria
We excluded level-V studies (i.e., anatomical, animal, cadaveric, basic science, biomechanical, simulation, and educational studies; case reports; technical notes; expert opinions; surveys; publication analyses; and review articles) (13).We also excluded studies with unclear methods (LOE not listed or study methods not clearly de ned), those for which full text was unavailable, and those published in supplementary issues of the journals.

Article Categorization Trauma vs. Nontrauma
We categorized the articles in JBJS and CORR as either trauma or nontrauma studies.Nontrauma studies were those concerning congenital or developmental abnormalities, arthroplasty, tumors, sports medicine, nontraumatic spine disorders, hand surgery, or nontraumatic foot and ankle surgery.Trauma studies were those concerning trauma to the upper or lower extremities, spine, pelvis, acetabulum, foot, or ankle.

Study Evaluation
Each study was reviewed by 1 of 2 authors.We assessed interobserver agreement by randomly selecting and assigning articles that met our inclusion criteria to reviewer 1 or reviewer 2, who assigned a LOE.In case of disagreement between reviewers, a third author (reviewer 3) was asked to judge the LOE.The percentage of agreement was 95% for study type (κ = 0.91) and 82% for LOE (κ = 0.73).

Statistical Analysis
The proportions of articles in each LOE were compared using chi-squared tests.The Cochran-Mantel-Haenszel test was used to adjust research years as a strati cation factor.P < 0.05 was considered statistically signi cant.
After classifying the LOE of each article, we analyzed the following: 1) LOE of all orthopaedic trauma articles compared with all nontrauma articles; 2) LOE of JOT articles versus all trauma and nontrauma articles in JBJS and CORR; 3) LOE of JOT articles versus JBJS and CORR trauma articles only; 4) changes in LOE during the study period for JOT and for trauma articles in JBJS and CORR by journal (Fig. 1); and 5) changes in LOE for all trauma and non-trauma articles over the duration of the study period.R, version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria), was used for statistical analysis.

LOE of Trauma vs. Nontrauma Studies in JBJS and CORR
In JBJS, we found a signi cant difference in the LOE between trauma and nontrauma studies (P = 0.029), with trauma (54 articles; 22.2%) demonstrating a higher percentage of level-I articles than nontrauma (166 articles; 16.1%).We found no signi cant difference in LOE between trauma and nontrauma articles in CORR (P = 0.937) (Fig. 4).

High-Level vs Low-Level Studies by Journal
A signi cantly higher percentage of level-I and level-II studies were published in JBJS (31%) compared with JOT (22%) and CORR (21%) (P < 0.001).There was no such difference when comparing JOT and CORR (P = 0.77; Fig. 5).
When analyzing the percentages of high-level studies by year, we found signi cant differences between JBJS vs. JOT (P = 0.005) and JBJS vs. CORR (P < 0.001).No such difference was found between JOT vs. CORR (P = 0.674).JOT published a smaller percentage of high-level studies (22%) compared with the percentage of high-level trauma studies published in JBJS (32%) (P = 0.001).No difference was found between the percentages of high-level trauma studies published in JOT (22%) and CORR (19%) (P = 0.609).We also found no signi cant difference in LOE between trauma vs. nontrauma studies published by JBJS (P = 0.695) or CORR (P = 0.608).
When analyzing LOE in each journal over time, JOT had a signi cant decrease in level-I (7.3% in 2013 to 1.9% in 2018) and level-IV articles (49% in 2013 to 37% in 2018) and an increase in level-III articles (34% in 2013 to 45% in 2018).JBJS had a signi cant decrease in level-II articles (21% in 2013 to 6.6% in 2018), and minimal changes in level-I, level-III, and level-IV articles.The LOE in CORR improved, with a signi cant decrease in level-IV articles (40% in 2013 to 11% in 2018), a signi cant increase in level-III articles (37% in 2013 to 62% in 2018), but minimal changes in level-I and level-II articles (Fig. 6a-b).

High-Level vs. Low Level Studies over Time (Trauma vs Non-trauma)
After combining all studies from the three included journals, we found that between 2013 and 2018, there was a non-signi cant downward trend in the percentage of high-level trauma articles (23% in 2013 to 19% in 2018; P = 0.190) and a signi cant downward trend in percentage of high-level non-trauma articles over time (29% in 2013 to 25% in 2018; P = 0.037) (Fig. 6c).

Discussion
The purpose of this study was to assess the validity of the ndings of the study conducted by Scheschuk et.al. (12) by evaluating the LOE of trauma and nontrauma research published in 3 major orthopaedic journals, comparing it to their ndings and examining it has changed in recent years, and assessing how our ndings may be used to estimate future changes.From 2013 through 2018, we found that most articles published by these journals (63%) were low-level (level III or IV) studies, and most (63%) focused on treatment (as opposed to diagnosis, prognosis, or economics).Only 30.4% of trauma and 34.4% of non-trauma articles were classi ed as level I or level II, with downward trends for both (although only the non-trauma trend was signi cant).JBJS published the highest percentages of level-I and level-II studies, and CORR published the highest percentage of level-III studies and the lowest percentage of level-IV studies.JOT published the lowest percentage of level-I studies of all 3 journals.Our nding regarding the decrease in high-level studies deviates from previous literature assessing levels of evidence in orthopaedics over time.Scheschuk et.al. examined JBJS, CORR, and JOT articles for LOE in 1998LOE in , 2003LOE in , 2008LOE in , and 2013, reporting an upward trend in level-I and -II studies over their study period (12).Cunningham et.al. examined eight orthopaedic subspecialty journals for articles published in 2000, 2005, and 2010 and reported an increase the proportion of level-I and -II studies over time (10).The current study provides an update and new look at the state of orthopaedic literature, both trauma and non-trauma.One potential explanation for the difference in studies is the increased use of national databases in orthopaedic research, particularly since 2013 (14,15).These databases can provide large volumes of retrospective data but are still considered level-III evidence.Another reason could be year-toyear variation; we saw a rise of high-level trauma articles and a decrease in high-level non-trauma articles in the year 2015.The selective sampling of certain years may have caused the prior studies to miss important data in the years between.Our ndings suggest that the increased implementation of the LOE system has not been effective in improving the quality of orthopaedic literature in recent years.Across all 3 journals, high-level trauma studies were published more often by JBJS than by JOT or CORR.No signi cant differences in LOE were found between JBJS and CORR when comparing all articles.Okike et al showed that LOE was the only factor signi cantly associated with journals' acceptance of submitted manuscripts, and that level-III and level-IV studies were least likely to be accepted by JBJS (16).Because of the nature of orthopaedic trauma, in which patients are often treated on an emergency basis, designing high-level studies, such as randomized controlled trials, is di cult.Level-III and level-IV studies are the most frequently published clinical orthopaedic trauma studies, likely because of their greater feasibility and lower cost compared with high-level studies.Blinding can be di cult to implement, and the use of placebo controls would be unethical in orthopaedic trauma studies.Further, obtaining a large enough sample size to achieve adequate statistical power in orthopaedic trauma is di cult.Bhandari et al reported that the quality of randomized trials in JBJS could be improved if blinding, randomization concealment, and patient inclusion/exclusion criteria were made consistent across studies (17).
Retrospective studies and case series, when performed with rigorous methods, were also considered valuable studies because of design methods.
Our study has several strengths.First, we analyzed all published articles during a 6-year period from 3 major orthopaedic journals that publish orthopaedic trauma research.This contrasts with previous publications, which evaluated articles in an inconsistent manner, reviewing only certain years across a time period and comparing them to articles that were published before the implementation of the LOE system established by JBJS in 2003 (10)(11)(12).Second, our comprehensive review eliminates selection bias that might arise from using an inconsistent review period.Third, we con rmed good interobserver agreement in LOE assessment across all 3 journals for the entire study period.
A limitation of our study is our use of the LOE grading system updated by JBJS 2015.Although the system is simple and reproducible, it allows reviewers to grade LOE downward or upward based on the study quality and methodology used, which can be considered as a confounding bias to properly effect results.Furthermore, this may also create differences in LOE when graded by different reviewers.

Conclusion
Based on the results of this study, we cannot validate the ndings of Scheschuk et.al (12).We found a decreasing proportion of high-level evidence across our study period, both in trauma and non-trauma articles.Of the 3 journals analyzed, JBJS published the greatest percentage of high-level studies.JOT published a lower percentage of high-level articles than did JBJS and CORR.A higher percentage of level-I articles were published in JBJS and CORR, whereas no signi cant difference was found among the 3 journals.

Figure 2 Numbers
Figure 2

Figure 3 Numbers
Figure 3

Figure 4 Numbers
Figure 4