Application of a feature extraction and normalization method to improve research evaluation across clinical disciplines

Introduction

Official administration organizations tend to prefer to allocate more funding to those deemed to be “excellent performers” as research groups that perform better in the evaluation process are thought to be more likely to achieve better scientific results in the near future. Thus, research evaluation analyses conducted at the individual or institutional level provide useful information that can be used not only for the evaluation of the scientific performance of individuals (e.g., professors) and research institutions (e.g., hospitals, colleges, and even official scientific research institutions) but also for detailed future planning of funding allocation. For more than 30 years, this trend has resulted in debate regarding how to conduct these evaluations more scientifically (1-5). Impact factors (IF) are widely used in the evaluation process, as it fits well with the opinion we have in each field of the best journals in our specialty (6). However, even the creator of IFs, Eugene Garfield, stated that this metric is not appropriate for assessing the importance or significance of individual works (7).

In the field of medicine, scientific research performance has become a crucial index for measuring the comprehensive strength of hospitals: how should the comprehensive scientific research strength of hospitals be measured, especially for hospitals that are affiliated with universities? China has provided its own answer. Nanjing University took the lead in using the Science Citation Index (SCI) as the core index for scientific research evaluation at the end of the 20th century. Since then, the number of SCI papers and their IFs has gradually become the only or most heavily weighted index for evaluating the scientific research level of hospitals and even individuals, which promoting doctors to write more effective and efficient SCI papers to benefit their own promotion and personal acknowledgment. In Chinese hospitals, all departments, including internal medicine, obstetrics and gynecology, surgery, and pediatrics, share the same criteria: the number of SCI papers and their corresponding IFs.

Although IFs are not an optimal measure of the quality of an individual’s scientific articles, they are used as a tool for the comparative evaluation of different departments in more than 90% of Chinese hospitals. There are 3 main problems with using only the IFs to compare scientific research performance among different disciplines and departments. First, this approach fails to take into account that some disciplines are in a weaker position and that the scientific researchers in these disciplines are at a disadvantage in the evaluation practice. For example, the Journal Citations Report 2018 states that there were 129,352 SCI articles related to general and internal medicine, but only 13,474 articles related to critical care medicine (https://jcr.clarivate.com/JCRHomePageAction.action?, accessed 08/20/2019). There were nearly 10 times more articles in the former discipline than in the latter. Second, most of the collections in SCI prioritize journals in English, but a substantial number of secondary disciplines, such as family planning, have distinct Chinese characteristics and include little research at the international level. Consequently, this kind of evaluation obviously falls short. Third, for less popular subjects, such as health engineering, local epidemiology, tropical medicine, and other disciplines, related research is very limited, so there are fewer matching SCI journals, compared with other disciplines, leading to limited choices for researchers seeking to publish their scientific results. In the long term, this puts these researchers at a disadvantage in scientific evaluations and discourages research on less popular subjects.

However, relatively few studies have focused on evaluation across disciplines in hospitals, with a few exceptions, such as a study conducted by Bordons et al. that analyzed the structure and research performance of teams in 2 biomedical subfields: pharmacology and pharmacy and the cardiovascular system (8). Recently, using references to measure “interdisciplinarity” has been the focus of attention. Mishra and Torvik suggested that, by using a set of Medical Subject Headings (known as MeSH), it was easier to capture complex trends in how the phenomenon of publishing novel, interdisciplinary articles changes across an author’s career (9). Wang also argued that using references made it possible to evaluate the effect of cooperation intensity on knowledge creation (10). Leydesdorff et al. proposed a new indicator, DIV, which assesses the variety, balance, and disparity of disciplines independently (11). For this indicator, the Gini coefficient is used to achieve a balance. Results indicate that, compared with earlier indicators, DIV is an improved method for distinguishing between interdisciplinary knowledge integration versus knowledge diffusion. However, to date, researchers have rarely measured the specific differences between disciplines, and limited empirical evidence has been examined in hospitals. Leydesdorff et al. also noted that the boundaries between disciplines are fluid and difficult to define (11). Defining a discipline concretely seems to be particularly complicated in a changeable environment that advocates global collaboration and interdisciplinary partnership.

We believe that both individuals and scientific organizations are in urgent need of such definitions. This is particularly the case for hospitals, which increasingly focus on scientific performance and tend to allocate different levels of funding to departments using these indexes. Therefore, to address this complex phenomenon, in the present study we used normalization method, referring to the calculation of the normalized coefficient of a specific discipline, to enable a relatively clear evaluation across different disciplines based on computer science technology. We sought to answer the following questions: (I) how should the characteristics of medical disciplines corresponding to different departments be defined? (II) What are the characteristics of the distribution of SCI journals in various disciplines, and what are the implications of this for scientific researchers’ contributions? (III) How can innovations be made in theoretical methods to balance the disciplinary differences among departments?

Methods

Discipline selection and definition of characteristics

In the retrieval analysis, we comprehensively considered the 18 subdisciplines of clinical medicine based on the Web of Science classification of research areas and the National Standard Disciplines of China, which include general internal medicine; pediatrics; geriatrics and gerontology; neurosciences and neurology; psychiatry; dermatology; radiology, nuclear medicine, and medical imaging; clinical laboratory; nursing; surgery; obstetrics and gynecology; ophthalmology; otorhinolaryngology; oncology; rehabilitation medicine; sport sciences; anesthesiology; and emergency medicine. Included articles had to satisfy the following two criteria: (I) the journal where the article was published was included in the Journal Citation Reports and the Web of Science Core Collection; and (II) because we considered a period of 3 years (from 2016 to 2018), each journal had to be included in the Journal Citation Reports for these 3 consecutive years.

Information on the final articles retrieved after excluding journals with missing values is presented in Table 1. The IFs were taken from the Journal Citation Reports database (2016, 2017, and 2018 editions). To clarify disciplinary characteristics, we first defined each of the subdisciplines of clinical medicine on the basis of the research content, research methods, and applications. We then analyzed the key elements of each discipline through a meta-analysis of the literature. Finally, we selected the characteristic fields for each clinical discipline, creating the keyword database. The process included the following steps: (I) reviewing the classic international textbooks as a foundation of the keyword database with the aim of determining the basic research content of each discipline; (II) conducting definition extension by defining core keywords through the definition given by experts of this discipline as a supplement; and (III) identifying international core journals to screen for “hot” keywords in LetPub and the Web of Science database in order to keep up with the research hotspot.

The keyword database was then reviewed by experts with more than 10 years of scientific experience in their fields. After 3 rounds of review, a revised database was established to provide a basis for using feature extraction to match each discipline’s journal catalog afterwards.

Feature extraction process

The aim of the present study was to improve the traditional evaluation system used in Chinese hospitals and provide references for researchers to get further paper submission smoother, with a narrow focus on the clinical medicine subdisciplines that are included among the departments in all Chinese hospitals. To respond to the question of how to define the characteristics of medical disciplines, a variety of methods for analyzing interdisciplinary differences have been reported in the literature, including in a number of studies examining observations of disparities across a wide variety of specific domains (economics, medical subject categories, library and information science, and science and technology studies) (12-14). The authors of these previously published articles called for the use of more detailed classifications at the subject level. However, none of these existing studies has tried to cover all the journals in a given discipline in defining the discipline’s characteristics, and most research has remained at the stage of qualitative or semiquantitative analysis. Creating an adequate delineation of disciplines to provide a framework within which not only articles but also personal performance can be thoroughly compared is a less discussed, but equally important, issue.

In this study, we present the novel method of feature extraction. This method is based on the sorting algorithm of an internet search engine. It refers to evaluating and sorting the correlations between relevant sentences in the retrieved text and keywords according to their degree of match, determined by, for example, the location and frequency of keywords in the keyword database appearing in the text. After crawling each page, the search engine extracts the features of the page and builds an index. In the ranking of search results, results with a more relevant feature have a higher ranking. The present study also set a weight index for the occurrence position of the keyword field to provide another reference for sorting. Articles in which the keyword field was more important have a higher ranking. We used this method to establish the correlation ranking of each journal and the above disciplines, and we used the IF as the weight to be included in the evaluation criteria of the discipline-related journal ranking.

The feature extraction process can be divided into the following 3 procedures: the gripping layer, the preprocessing layer, and the computing layer. These 3 layers are explained in more detail later and visualized schematically in Figure 1. The first step is to create a gripping system, which is built in Python language, and all data are stored in MongoDB. The purpose of establishing the gripping layer is to input the keywords of the characteristics of each discipline from the keyword database, unify the input format, and de-duplicate the keywords. The gripping layer adopts the Spider class and the Item class of the Scrapy framework to complete the construction, and the Web of Science is used as the crawling source. Spider is the primary class responsible for crawling and can be customized to point to different sites. Item is used to structure the captured items. The fields required for this research included article titles, keywords, abstracts, the position of keywords in the text, the journal where the article was published, and the year the article was published. In the preprocessing layer, we used SQL statements, the Scrapy framework, and the NLTK (Natural Language Toolkit) for natural language processing to complete the preprocessing and calculation of the fetching data. For all articles in the database, we used the term frequency-inverse document frequency algorithm to calculate the weight of keywords—that is, to obtain the weight dictionary of the corresponding keywords for each article. Finally, in the computing layer, the journal correlation coefficient was calculated. The algorithm was as follows: the number of articles captured in a certain journal in a given year in the preprocessing layer was used as the denominator, x_ia, and the number of all articles captured in that journal in that year, x_ib, was taken as the numerator and then divided by the weighted average of all the articles captured in the year, β_i. The equation for the journal correlation coefficient, Y_i, was as follows:

Yi=∑xia×βixia×xiaxib=∑xia×βixib[1]

Figure 1 Key procedures and details of feature extraction. NLTK, Natural Language Toolkit.

We defined the integrated impact coefficient, γ_i, of journal i as:

γi=Yi×Ii[2]

Here, I_i is the IF of journal i in a given year.

Normalized evaluation

The whole range of IFs differed not only over different time periods but also across different research fields (15). For example, according to the CD-ROM version of the science and social science editions of the Journal Citation Reports for 2018, in the field of management, the highest IF was 12.289 for the Academy of Management Annals, whereas the highest IF in nanoscience and nanotechnology was 74.499 for Nature Reviews Materials. There was also a large gap between the average IFs in these fields. The average IF in nanoscience and nanotechnology was 6.795, nearly 3 times higher than the average IF in management (2.983). To minimize the gap in citation numbers across different fields, Rehn et al. proposed the item-oriented field-normalized citation score, which was later referred to as the normalized citation score (NCS) (16). This well-received field-normalized indicator is calculated by dividing the number of citations to a specific paper by the average citation rate of the papers published in the same discipline category and year. The equation for the NCS of a given paper is as follows:

μi=1n∑i=1nCi[3]

NCSi=Ciμi[4]

Here, C_i denotes the overall number of citations to paper i, and n refers to the number of papers in the same discipline category as paper i. Therefore, µ_i is the total average of citation numbers in a specific discipline category. The NCS is defined by the specific values of C_i and μ_i.

Using the feature extraction process described earlier, we included all journals in all fields in an efficient, systematic, and low-cost method, ultimately obtaining the journal correlation coefficient. Because of the wide use of the NCS, we then explored how to convert the standardized scores of papers in various disciplines. Through the process of transforming a dimensional expression into a dimensionless scalar, we obtained a normalized weighted system for each discipline, enabling the construction of an interdisciplinary and comprehensive evaluation system of papers and the realization of uniform and fair evaluation across different clinical disciplines. The calculation procedures were as follows.

The mean weight of a specific discipline was obtained by calculating the sum of the influence factors and the journal correlation coefficient of the selected journals. Y_i was the journal correlation coefficient of journal i, and I_i was the IF of journal i (journal i was included when it belonged to the selected discipline). We defined the weight of a given discipline i as:

pi=1n∑i=1n(Yi⋅Ii)=1n∑i=1nγi[5]

The sum of all the weights of every discipline was defined as:

T0=∑i=1nρi[6]

The calculation of the normalized coefficient of a specific discipline (discipline-normalized coefficient, DNC_i) was defined as:

DNCi=ρiT0[7]

Statistical analysis

Statistical analyses were performed using Python 3.7 (Python Software Foundation, Wilmington, Delaware, USA). SQL statements, Scrapy framework, and NLTK (Natural Language Toolkit) for natural language processing were used to tackle the feature extraction process.

Results

Overview of retrieved journals in 3 consecutive years

Table 2 provides an overview of all journals retrieved using the method discussed earlier over 3 consecutive years (2016, 2017, and 2018). Using feature extraction, we can see that in 2016, at least 1,144 journals were retrieved among 18 disciplines, with a maximum number of 5,870 (more than 5 times higher than the minimum). Using our keyword database, we found the papers that were highly related to a certain discipline, with 5,520 papers related to oncology and only 390 classified into the field of nursing. Referring to the results for 2017 and 2018, the total numbers of journals retrieved using our method changed slightly. Neurosciences and neurology remained the top discipline, with the largest total number of journals in both years, and there was a sharp difference between this discipline and the discipline with the lowest total number of journals in 2016 and 2017, sport sciences, which ranked last in 2018, with only 1,070 journals found. The same trend was observed in the total number of papers retrieved in 2017 and 2018. However, the number of papers in the field of nursing decreased slightly in 2017 (n=348), and then slightly increased in 2018 (n=386). The number of papers in the discipline of oncology increased stably, from 5,592 in 2017 to 5,691 in 2018.

To generate an approximation of a discipline that is sufficient for analysis, we aimed to concentrate on the top 50 journals, including the journals ranked in the top 50 by IFs and journal correlation coefficients. We adopted this simplification because of the aim of the method, which was not only to exhaustively display all related publications but to generate an overview of a particular research policy context and guide further evaluations and contributions for researchers in different fields. In 2016, oncology had the highest average IF (mean: 8.717, range, 4.041–33.900), and the field of general internal medicine had the second highest average IF (mean: 8.359, range, 1.847–72.406). In contrast, nursing was identified as the discipline with the lowest average IF (mean: 1.797), including journals with IFs from 1.261 to 3.755. In 2018, general internal medicine took the place of oncology as the discipline with the highest average IF, with the mean IF for general internal medicine increasing from 9.081 in 2017 to 9.461 in 2018. Nursing remained in last place, but saw a slight increase in the average IF, from 1.841 in 2017 to 1.974 in 2018. Examining the fluctuations in the journal correlation coefficients over these 3 consecutive years, it can be seen that 9 disciplines showed a slight increase during this timeframe, and no discipline steadily decreased throughout the analyzed period. The average journal correlation coefficient fluctuated from 0.025 to 1. From 2016 to 2018, surgery had the highest average journal correlation coefficients (range, 0.750–0.754), and the lowest average journal correlation coefficients belonged to clinical laboratory (range, 0.215–0.217).

Analysis of differences across disciplines

Table 3 compares the top 3 journals ranked by IFs and journal correlation coefficients. Considering that changes over 2 years might not be obvious, we constructed a cross-year comparison; that is, 2016 and 2018 data were selected for analysis. We also clearly displayed the corresponding journal correlation coefficient for each journal in a given discipline. For example, our process of feature extraction identified the New England Journal of Medicine as having the highest IF, but its related journal correlation coefficient was much lower compared with the journals ranked in the top 3. In obstetrics and gynecology, the New England Journal of Medicine ranked first based on IFs, but its journal correlation coefficient, 0.043, was relatively low. In contrast, the highest journal correlation coefficient in this discipline was 0.826 and belonged to Gynecologic Oncology; this was nearly 20 times higher than the correlation coefficient of the New England Journal of Medicine. A similar phenomenon was observed in many disciplines: Journals with high IFs did not have proportionately high journal correlation coefficients.

Comparing differences across the 18 disciplines, each discipline had its own featured journals, and their ranges of correlation coefficients varied substantially. Ophthalmology had the highest journal correlation coefficient in both 2016 and 2018, with the Annual Review of Vision Science ranking first; this journal’s correlation coefficient reached 1. However, in the disciplines of clinical laboratory and sport sciences, the journal correlation coefficients were lower (0.561 and 0.541, respectively). When considering 2 years, the list of the top 3 journals ranked by IFs remained almost unchanged. However, slight differences were seen in the list ranked by journal correlation coefficients from 2016 to 2018. In 2018, taking the discipline of psychiatry as an example, the Journal of Affective Disorders, Depression and Anxiety, and Bipolar Disorders appeared first in the list, with correlation coefficients of 0.843, 0.824, and 0.821, respectively.

Wagner stated that “interdisciplinary” areas of scientific practice have not been well studied, although these areas are growing rapidly (17). The underlying concept of “disciplines” are social baselines used to allocate the privileges and responsibilities of expertise, as well as resources. Our comparison of the characteristics of different disciplines in 3 recent years suggests a clear distinction between the coverage of different disciplines, which provides evidence for questioning the fairness of judging productivity using IF alone. Furthermore, it can be concluded that, in the analysis of the variation tendencies of IFs and journal correlation coefficients, some disciplines exhibit a steady publication feature, which may inform further contributions for researchers to find a more suitable journal to publish their original articles.

Results of the normalized evaluation

For the fair evaluation of papers across different disciplines in clinical medicine, the normalized coefficient of each secondary discipline was calculated using the normalized evaluation method, represented by DNC_i. Here, the 18 disciplines are numbered from 1 to 18, and the specific discipline numbers are as follows: (I) general internal medicine; (II) pediatrics; (III) geriatrics and gerontology; (IV) neurosciences and neurology; (V) psychiatry; (VI) dermatology; (VII) radiology, nuclear medicine, and medical imaging; (VIII) clinical laboratory; (IX) nursing; (X) surgery; (XI) obstetrics and gynecology; (XII) ophthalmology; (XIII) otorhinolaryngology; (XIV) oncology; (XV) rehabilitation medicine; (XVI) sport sciences; (XVII) anesthesiology; and (XVIII) emergency medicine. As mentioned earlier, to achieve an approximate outline of a discipline that is sufficient for analysis, we focused on only the top 50 journals, including journals ranked in the top 50 using the multiple of each journal’s IF and correlation coefficient. We used the product of the IF and the journal correlation coefficient for this sorting for 2 main reasons. First, some journals had extremely high IFs, but through extraction, we observed that these journals were relatively less correlated with the specific disciplines. Consequently, it would be difficult for scientific researchers to publish in such journals, which only published a few articles related to their disciplines. To minimize the gap between the IF and the journal’s correlation with a given discipline, we used the normalized evaluation. The foundation of this method is using the multiples of 2 indexes for assessing the journals’ overall characteristics.

Table 4 shows 2 indexes from 2016 to 2018: the weight of a given discipline, ρ_i, and the normalized coefficient of a specific discipline, DNC_i. The normalized coefficients of the disciplines differ from each other in the horizontal comparison, and these coefficients vary with time in the vertical comparison. In these 3 years, oncology had the highest normalized coefficient, which reflects the highest correlation between the characteristic journals of the discipline. Additionally, it is easier for oncology scholars to publish high-quality SCI articles. In contrast, the normalized coefficients of otorhinolaryngology and emergency medicine are very low. It is difficult for scholars in these disciplines to publish articles with high IFs. Evidence of this can be found in the number of journals retrieved that are related to these disciplines, as mentioned earlier. During these 3 years, the normalized coefficient of each discipline generally showed a steady increase, whereas there was a U-shaped change for a low number of disciplines, reflecting the gradual increase of the specificity of these disciplines, where journals gradually began to focus on the characteristic research of the discipline.

Discussion

The results of our empirical study show that the disciplines of clinical medicine are not represented equally, and that the characteristics of the journals of each discipline vary substantially. As the findings of the present study only relate directly to the disciplines of clinical medicine, we cannot assume that they apply to other disciplines. However, although our analysis was restricted to these disciplines, the results obtained explore the variation of discipline features and indicate broader implications.

Advantages of the method’s focus on clinical medicine

Compared with traditional IF evaluation, the application illustrated in the present study illustrates our method’s results, and the focus on clinical medicine can facilitate the evaluation of scientific researchers in Chinese hospitals. The application of the combined methods proposed here has provided numerous types of information on each discipline, with a total of 702,996 journals retrieved in 2016 (although there is some overlap in journals across different disciplines). Firstly, the advantage of the innovative feature extraction method used in this study is that the most relevant journals in each discipline are obtained through computer dynamic recognition and retrieval. Among the 18 examined disciplines of clinical medicine, there are many that intersect with public health and basic medicine, and the examined disciplines also intersect with each other. The feature matching method can quickly select the journals that are directly related to a particular discipline from the vast number of SCI journals. Secondly, through further analysis of the selected journals, this method can identify the discipline’s characteristics “hot” research areas and new research progress. Thirdly, also most importantly, through discipline selection and characteristics retrieval, we found the citation and research popularity differences among disciplines, then we performed the normalization method to further balance those disparities in disciplines during evaluations.

Potential application in the near future

The proposed method can be characterized as a widely applicable approach that, beyond being used to assess the scientific performance of researchers in hospitals, could be expanded for use in other disciplines, institutions, and databases. For example, this method is also applicable to universities, which also face multidisciplinary academic evaluations. The current university evaluation is dominated by IF, which is obviously unfair for less popular subjects as well. From the perspective of researchers in a particular specialty, the adjusted evaluation system may at first seem a bit small because the adjustments it makes to the original IFs may be seen as a minor change. However, in the reality of the evaluation of individual scientists, substantial gaps do exist between different disciplines, and narrowing these gaps is of great value. The rapid growth of “big data” provides new opportunities and challenges for the field of bibliometrics (18). More logical and applicable approaches are needed to improve the evaluation of researchers, institutions, and publishing agencies, particularly in China.

Potential improvements for designing a more precise method

In the present study, we presented a theoretical model of feature extraction and normalization for the research performance of 18 disciplines for a theoretical examination, see Waltman et al. (19); for an empirical study, see Waltman et al. (20). The method design can be further refined at each phase to better fit the demands of future investigations for wider disciplines, such as management or economics. Furthermore, more advanced techniques could be used to increase the accuracy of the delineation. For example, how can journals with minimal relevance be optimally filtered out to conserve resources? Such questions are worthy of further investigation.

Refinements that could be made to the method design to better fit the demands of wider acceptance and further application in other fields include the following.

Source selection

In the present study, we selected only journals from the Web of Science, because almost all clinical medical research is included in this database, which is becoming the core data source for assessment in the evaluation of researchers in clinical medicine. However, to construct an overview of other disciplines, the database selection needs to be changeable to include 1 or more appropriate databases to exhaust the disciplinary characteristics. As Frandsen and Nicolaisen noted, the results of bibliometric research based on data retrieved from databases are affected by differences in the database coverage (21). In the field of management, for example, it would be reasonable to change the focus to more related databases, such as the Social Sciences Citation Index (founded by the American Institute for Scientific Information) and the Chinese Social Sciences Citation Index (founded by the Chinese Social Science Research Evaluation Institute, covering the fields of law, management, economics, history, and political science).

Keyword selection

Selected articles could be included in a more refined way by excluding discipline-specific stop words in extracting information from abstracts or full text (22). Additionally, as proposed by Milojevic et al., key phrases could be extracted from titles and transferred into a common list of key title words (23). Furthermore, although the full text of articles or abstracts is generally available for analyzing the characteristics of a discipline, more detailed criteria could be used instead of, or in addition to, the keyword list based on title words. For example, Waltman and van Eck proposed using natural language processing techniques to derive sets of terms characterizing fine-grained clusters of publications (24).

Field weight setting

For the selection and subsequent processing of the field database, in the present study we chose a simple approach where all data fields for a certain discipline were treated equally. However, future work could explore whether fields with more dynamic differentiation might produce better results. In summary, different ways of setting field weights are some of the factors to consider in developing a more reasonable evaluation system.

Acknowledgments

We thank Jennifer Barrett, PhD, from Liwen Bianji, Edanz Editing China (www.liwenbianji.cn/ac), for editing the English text of a draft of this manuscript.

Funding: This study was supported by the National Natural Science Foundation of China (Nos. 71804128, 71774116, 71904145, 71603182), Chinese State Scholarship Fund (No. 201806265001), National Key Research and Development Program of China (No. SQ2018YFC130057), the Shanghai Municipal Planning Commission of Science and Research Fund (No. 201740202), and Shanghai Jiao Tong University China Hospital Development Institute 2019 Local High-level University Hospital Management Special Project (No. CHDI-2019-C-01), the Personnel Development Plan of Shanghai Tenth People’s Hospital of Tongji University (No. 2021SYPDRC014).

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/atm-21-5046). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Costas R, van Leeuwen TN, van Raan AF. The "Mendel syndrome" in science: durability of scientific literature and its effects on bibliometric analysis of individual scientists. Scientometrics 2011;89:177-205. [Crossref] [PubMed]
2. Leydesdorff L. Alternatives to the journal impact factor: I3 and the top-10% (or top-25%?) of the most-highly cited papers. Scientometrics 2012;92:355-65. [Crossref] [PubMed]
3. Fraley RC, Vazire S. The N-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One 2014;9:e109019 [Crossref] [PubMed]
4. Comins JA, Hussey TW. Detecting seminal research contributions to the development and use of the global positioning system by reference publication year spectroscopy. Scientometrics 2015;104:575-80. [Crossref]
5. Budd JM. Measuring Research: What Everyone Needs to Know. College & Research Libraries 2018;79:853-4. [Crossref]
6. Hoeffel C. Journal impact factors. Allergy 1998;53:1225. [Crossref] [PubMed]
7. Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295:90-3. [Crossref] [PubMed]
8. Bordons M, Zulueta MA. Comparison of research team activity in two biomedical fields. Scientometrics 1997;40:423-36. [Crossref]
9. Mishra S, Torvik VI. Quantifying Conceptual Novelty in the Biomedical Literature. Dlib Mag 2016; [Crossref] [PubMed]
10. Wang J. Knowledge creation in collaboration networks: Effects of tie configuration. Research Policy 2016;45:68-80. [Crossref]
11. Leydesdorff L, Wagner CS, Bornmann L. Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient. Journal of Informetrics 2019;13:255-69. [Crossref]
12. van Leeuwen TN, Medina CC. Redefining the field of economics: Improving field normalization for the application of bibliometric techniques in the field of economics. Research Evaluation 2012;21:61-70. [Crossref]
13. Schmidt M, Sirtes D. Transforming the Heterogeneity of Subject Categories into a Stability Interval of the MNCS. Proceedings of Issi 2015 Istanbul: 15th International Society of Scientometrics and Informetrics Conference; 2015:365-71.
14. Leydesdorff L, Bornmann L. The operationalization of "fields" as WoS subject categories (WCs) in evaluative bibliometrics: The cases of "library and information science" and "science & technology studies". J Assoc Inf Sci Technol 2016;67:707-14. [Crossref]
15. Althouse BM, West JD, Bergstrom CT, et al. Differences in Impact Factor Across Fields and Over Time. J Am Soc Inf Sci Technol 2009;60:27-34. [Crossref]
16. Rehn C, Kronman U, Wadskog D. Bibliometric indicators – Definitions and usage at Karolinska Institutet. Stickholm, Sweden: Karolinska Institutet University Library; 2014.
17. Wagner H, Finkenzeller T, Würth S, et al. Individual and team performance in team-handball: a review. J Sports Sci Med 2014;13:808-16. [PubMed]
18. Ding Y, Zhang G, Chambers T, et al. Content-based citation analysis: The next generation of citation analysis. J Assoc Inf Sci Technol 2014;65:1820-33. [Crossref]
19. Waltman L, van Eck NJ, van Leeuwen TN, et al. Towards a new crown indicator: Some theoretical considerations. J Informetr 2011;5:37-47. [Crossref]
20. Waltman L, van Eck NJ, van Leeuwen TN, et al. Towards a new crown indicator: an empirical analysis. Scientometrics 2011;87:467-81. [Crossref] [PubMed]
21. Frandsen TF, Nicolaisen J. Intradisciplinary differences in database coverage and the consequences for bibliometric research. J Am Soc Inf Sci Technol 2008;59:1570-81. [Crossref]
22. Makrehchi M, Kamel MS. Automatic extraction of domain-specific stopwords from labeled documents. Advances in Information Retrieval 2008;4956:222-33. [Crossref]
23. Milojevic S, Sugimoto CR, Yan EJ, et al. The Cognitive Structure of Library and Information Science: Analysis of Article Title Words. J Am Soc Inf Sci Technol 2011;62:1933-53. [Crossref]
24. Waltman L, van Eck NJ. A new methodology for constructing a publication-level classification system of science. J Am Soc Inf Sci Technol 2012;63:2378-92. [Crossref]

(English Language Editor: R. Scott)