Impact factors of orthopaedic journals between 2010 and 2016: trends and comparisons with other surgical specialties

Introduction

Increasing legislative efforts to improve quality of care has led to a rejuvenated emphasis on utilizing evidence-based medicine as a guide for clinical practice (1,2). These efforts have spurred an increased demand for the academic community to publish high-quality studies. To meet this demand, there has been a significant rise in the number of scientific medical journals. According to Journal Citation Report (JCR)—Thomson Reuters, the number of “clinical medicine journals” published between the years 1999 and 2010 grew from 1,291 to 1,986 (3). However, despite the increase in the total number of scientific journals, investigators still seek to publish in journals that garner the largest audiences and potential impact. Particularly for orthopaedics, in which new research findings often dictate healthcare policy and reimbursement, the pressure to publish in journals with the largest impact is of paramount importance.

In 1951, Eugene Garfield developed an algorithm, designated “impact factor” (IF), to quantify and rank journal quality (4). IF is calculated by measuring the frequency with which articles from a journal are cited over the total number of published articles from the same journal in a particular year (Figure 1). This algorithm was devised with the intent of controlling for journals with a larger number of publications. Despite some objection against the use IF as a means to measure journal quality, IF remains the most standardized method for journal appraisal and is commonly used by medical librarians in selecting journals for their institutions (5-9).

Figure 1 Impact factor equation.

Moverley et al. (10) investigated IF trends among orthopaedic journals between the years 2000 and 2010 and reported an increase in the mean IF for orthopaedic journals. We aimed to update their study by assessing IF trends in orthopaedic journals between the years 2010 to 2016. More specifically, we assessed trends for IF trends among orthopaedic journals and aimed to compare these trends to that of medicine and general surgery journals.

Methods

Journal IFs were obtained directly from JCR for the years 2010–2016 and analyzed for trends. Two authors (NS Mohamed and CU Gwam) screened and categorized each journal into either general, or subspecialty journals, based on whether the journal covered broad topics in orthopaedics or contained focused deeper knowledge (Table 1). Only journals that were considered primarily orthopaedic journals were included, as with previous studies (11,12). We also included the top 10 journals by IF in both internal medicine and surgery for comparison (Table 2).

Table 1 Journals in 2016 primarily focusing on orthopaedics, subdivided into general and specialist journal (accepted JCR abbreviations were used)
Full table

Table 2 Journals focusing on orthopaedics added and removed between 2010 and 2016 (accepted JCR abbreviations were used)
Full table

Each journal was analyzed by IF, and trends across time were evaluated. The differences in mean IF between orthopaedic specialty groups, geographical regions, and language of publication were analyzed using an independent samples t-test. The trends in mean IF for internal medicine and surgery were analyzed using a one-way Analysis of Variance (ANOVA). Significance was set at a value of P<0.05. All analysis was conducted using SPSS version 24 (IBM corporation, Armonk, New York, USA).

Results

In 2010, the number of orthopaedic journals with IF totaled 48, and increased to 55 in 2016 (Table 1). There were 27 general orthopaedic journals published, and 28 specialist journals published. The mean IF increased from 1.4 (range, 0.0–3.9) in 2010 to 1.9 (range, 0.5–5.7) in 2016. In 2010, English was the most common language of publication (n=37, 77.1%), followed by German (n=4, 8.3%). The majority of journals were published in the United States (n=23, 47.9%). In 2016, the percentage of English journals increased to 87.3% (n=48), while the percentage of journals published in the United States remained about the same (n=26, 47.3%). A significant difference found between the IF of journals published in English and those published in other languages (P=0.004). IFs for 2016 ranged from 5.7 for the American Journal of Sports Medicine to 0.5 for the Acta Ortopedica Brasileira (Figures 2,3).

Figure 2 2016 impact factors of the journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.

Figure 3 2010 impact factors of the journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.

The mean IF of both general and specialized orthopaedic journals increased from 2010 to 2016 (Figure 4). There was no significant difference found between the mean IF of these groups in any of the years included (2010: P=0.563; 2011: P=0.945; 2012: P=0.807; 2013: P=0.689; 2014: P=0.807; 2015: P=0.812; 2016: P=0.889).

Figure 4 Change in mean impact factor of orthopaedic journals between 2010 and 2016. Orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.

The majority of journals published in both 2010 and 2016 experienced an increase in IF (range, 0.04–1.9), while 5 journals had a decrease in IF: Spine (−0.011); Spine Journal (−0.062); BMC Musculoskeletal Disorders (−0.148); Connective Tissue Research (−0.261); and Journal of Orthopaedic Research (−0.284) (Figure 5).

Figure 5 Change in impact factor from 2010 to 2016 for orthopaedic journals.

The mean IF of the top 10 journals in both surgery and internal medicine increased from 2010 to 2016 (Figure 6). However, the increase within each specialty was found to be nonsignificant [internal medicine (IM): P=0.202; surgery: P=0.573] (Figure 7). The surgical journal with the highest IF, Annals of Surgery, experienced an increase in IF similar to the top orthopaedic journals (1.506). The internal medicine journals with the highest IF had major increases in IF from 2010 to 2016: Journal of the American Medical Association (JAMA) (14.394); Lancet (14.198), and New England Journal of Medicine (18.92) (Figure 8).

Figure 6 Change in mean impact factor of orthopaedic, internal medicine, and surgery journals from 2010 to 2016.

Figure 7 Change in mean impact factor of internal medicine and surgery journals from 2010 to 2016 (IM: P=0.202; surgery: P=0.573). IM, internal medicine.

Figure 8 Change in impact factor from 2010 to 2016 for internal medicine and surgery journals.

The field of orthopaedics has added at least 14 new journals since 2010, but concomitantly, approximately 7 journals ceased publication and no longer had any publication data (Table 3).

Table 3 Top 10 journals in 2010, primarily focusing on internal medicine and surgery (accepted JCR abbreviations were used)
Full table

Discussion

Legislative efforts to improve healthcare quality have led to an increased demand for high quality research as a means of directing clinical practice (1,2). For orthopaedists, the importance of publishing the latest research in journals that garner large audiences is crucial, as both private and government insurance institutions look to improve care value by utilizing data from high impact journals (13,14). Currently, the most widespread method for assessing a journal’s impact on the scientific community is through Eugene Garfield’s “IF” algorithm. Therefore, this study was a continuation of Moverley et al.’s 2013 study (10), which assessed the IF trends in orthopaedics and provided comparisons to trends in major internal medicine and general surgery journals. Our findings revealed a mean increase in IF among orthopaedic journals, similar to the increase demonstrated among internal medicine and general surgery journals. Additionally, our study revealed an increase in the total number of orthopaedic journals published between 2010 and 2016, with a significant portion of journals published in English and in the United States.

This study was not without limitations. By utilizing an amalgamated database created by the Journal of Citation reports website (3), our results are subject to the same documentation and data collection errors that may have persisted on the website. Furthermore, this study is retrospective and does not denote causality, nor does it offer conclusive rationale to explain the phenomenon. At best, we are able to offer a hypothesis based on evidence in order to explain our findings. Despite these limitations, our study is a useful addition to the current literature and demonstrates an increase in orthopaedic journal IF between 2010 and 2016.

Similar to our study, Moverley et al. (10) reported an increase in the number of orthopaedic journals (+17) and mean IF between the years 2000 and 2010. However, this increase may be the result of an increased number of inter-citations and self-citations between orthopaedic journals. Hakkalamani et al. (11) evaluated seven general orthopaedic journals {Journal of Bone and Joint Surgery [JBJS (Am)], the British volume of the Journal of Bone and Joint Surgery [JBJS (Br)], Clinical Orthopaedics and Related Research (CORR), Acta Orthopedica Scandinavica (Acta Orthop Scand), International Orthopaedics (Int Orthop), Orthopaedic Clinics of North America (OCNA) and the Archives of Orthopaedics and Trauma Surgery (Arch Orthop Trauma Surg)} and reported a self-citation rate greater than 20%. Additionally, the authors reported a strong correlation between journal self-citation and reported IF (r=0.613; P<0.001). This may indicate that, while the amount of orthopaedic literature may be increasing, this literature may not be permeating non-orthopaedic audiences.

Despite the demonstrated increase in mean IF, its use as a measure of journal quality has been highly contested. Kodumuri et al. (15) evaluated publications from the top thirteen IF ranked journals in trauma and orthopaedics between the years 2007 and 2008, and assessed the number of citations at 2 and 5 years after publication. The authors revealed lower rates of mean citation among surgical papers at 2 years when compared to basic science papers (2.18 vs. 4.68, respectively). However, by 5 years post-publication, the citation rates were similar (26.57 for surgery, 30.35 for basic science/medical). The authors concluded that it may be necessary to adjust IF to measure citation rate at 5 years post-publication in order to be a more appropriate indicator of journal impact. The limitations housed by IF were further exemplified by a 2008 publication in Acta Crystallographica in which the authors explicitly asked readers to cite their publication. This resulted in over 10,000 citations, thus inflating the journal’s IF from 2.051 (in 2008) to 49.926 (in 2009) and 54.333 (in 2010) (16,17).

Despite these objections, in the absence of a better system, IF is still largely used as a surrogate marker for excellence among universities and research funding institutes (15). Unfortunately, this places orthopaedic literature at a marked disadvantage when compared to medicine and surgery journals that also report on outcomes following orthopaedic intervention. Delanois et al. (13) evaluated all publications from JAMA, NEJM, and Lancet between January 1976 and December 2016 and found that 56% of orthopaedic publications reported on complications. Furthermore, the authors reported potential policy impact from nearly half (46%) of the publications that reported conclusively about the efficacy and/or advisability of orthopedic intervention. This effect may be augmented due to rise in IF among major IM journals, as demonstrated by our study.

Conclusions

Despite concerns, a journal’s IF remains the most commonly used measure for evaluation of journal quality and impact. Between the years 2010 and 2016, there has been a significant increase in the number of peer-reviewed orthopaedic journals, and in mean orthopaedic journal IF. Furthermore, English journals from the United States continue to have the largest impact when compared to non-English journals and journals from outside the United States. However, this increased IF does not seem to translate to increased non-orthopaedic audiences, as health-care policy makers, and private institutions still utilize higher impact internal medicine journals to shape health-care. Future studies should aim to better qualify journal impact, while limiting confounders such as self-citation.

Acknowledgements

None.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Bourdeau M, Winter R, Marshall R. Opportunities for improving legislative public health policy in Rhode Island through evidence-based education. R I Med J (2013) 2013;96:29-31. [PubMed]
2. Harolds JA, Merrill JK. Evidence-based medicine and health care reform legislation. Clin Nucl Med 2010;35:65. [Crossref] [PubMed]
3. . https://jcr-incites-thomsonreuters-com.ccmain.ohionet.org/JCRJournalHomeAction.action?SID=A2-Rl8GAi7nxxxxqBkOjWm65NXocNM4ov16Xv-18x2duSF8x2B93x2BHM0ILyGpmx2Fh9wgx3Dx3DoovcjO4gKcRPt9SoKFhwTwx3Dx3D-9vvmzcndpRgQCGPd1c2qPQx3Dx3D-wx2BJQh9GKVmtdJw3700KssQx3Dx3DInCites.
4. Garfield E. Citation indexes for science; a new dimension in documentation through association of ideas. Science 1955;122:108-11. [Crossref] [PubMed]
5. Isohanni M. Peer review--still the well-functioning quality control and enhancer in scientific research. Acta Psychiatr Scand 2005;112:165-6. [Crossref] [PubMed]
6. Baethge C. Impact factor--a useful tool, but not for all purposes. Dtsch Arztebl Int 2012;109:267-9. [PubMed]
7. Ironside PM. Rethinking the meaning and significance of journal impact factors. J Gerontol Nurs 2007;33:3-4. [PubMed]
8. Seglen PO. Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthop Scand 1998;69:224-9. [Crossref] [PubMed]
9. Smith R. Beware the tyranny of impact factors. J Bone Joint Surg Br 2008;90:125-6. [Crossref] [PubMed]
10. Moverley R, Rankin KS, McNamara I, et al. Impact factors of orthopaedic journals between 2000 and 2010: trends and comparisons with other surgical specialties. Int Orthop 2013;37:561-7. [Crossref] [PubMed]
11. Hakkalamani S, Rawal A, Hennessy MS, et al. The impact factor of seven orthopaedic journals: factors influencing it. J Bone Joint Surg Br 2006;88:159-62. [Crossref] [PubMed]
12. Siebelt M, Siebelt T, Pilot P, et al. Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskelet Disord 2010;11:4. [Crossref] [PubMed]
13. Delanois RE, Gwam CU, Piuzzi NS, et al. Hip and Knee Arthroplasty Orthopedic Literature in Medical Journals—Is It Negatively Biased? J Arthroplasty 2018;33:615-9. [Crossref] [PubMed]
14. De Oliveira GS Jr, Chang R, Kendall MC, et al. Publication bias in the anesthesiology literature. Anesth Analg 2012;114:1042-8. [Crossref] [PubMed]
15. Kodumuri P, Ollivere B, Holley J, et al. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J 2014;96-B:414-9. [Crossref] [PubMed]
16. Van Noorden R. Scientists join journal editors to fight impact-factor abuse: News blog. Available online: http://blogs.nature.com/news/2013/05/scientists-join-journal-editors-to-fight-impact-factor-abuse.html
17. Sheldrick GM. A short history of SHELX. Acta Crystallogr A 2008;64:112-22. [Crossref] [PubMed]