{ "type": "Article", "authors": [ { "type": "Person", "familyNames": [ "Moslehian" ], "givenNames": [ "Mohammad", "Sal" ] } ], "description": "Our aim is twofold. On the one hand we discuss the limitations of the impact\nfactor as a criterion for assessing mathematical journals, and suggest\nsubstituting a set of different types of indicators including the SCImago\nJournal Rank. On the other, we state that scientometrics such as the impact\nfactor cannot be used alone in evaluating researchers’ work: one must have both\na package of metrics as an objective measure and peer review by human beings as\na subjective judgement.", "identifiers": [], "references": [ { "type": "Article", "id": "bib-bib1", "authors": [], "title": " M. R. Elkins, C. G. Maher, R. D. Herbert, A. M. Moseley, and C.\nSherrington, Correlation between the Journal Impact Factor and three other\njournal citation indices. Scientometrics85, 81–93 (2010)\n" }, { "type": "Article", "id": "bib-bib2", "authors": [], "title": " A. Ferrer-Sapena, E. A. Sánchez-Pérez, L. M.\nGonzález, F. Peset, and R. Aleixandre-Benavent, Mathematical properties of\nweighted impact factors based on measures of prestige of the citing journals.\nScientometrics105, 2089–2108 (2015)\n" }, { "type": "Article", "id": "bib-bib3", "authors": [], "title": " E. García-Pachón and R. Arencibia-Jorge, A comparison of\nthe impact factor and the SCImago journal rank index in respiratory system\njournals. Archivos de Bronconeumología50, 308–309 (2014)\n" }, { "type": "Article", "id": "bib-bib4", "authors": [], "title": " B. González-Pereira, V. P. Guerrero-Bote, and F.\nMoya-Anegón, A new approach to the metric of journals’ scientific prestige:\nThe SJR indicator. J. Informetrics4, 379–391 (2010)\n" }, { "type": "Article", "id": "bib-bib5", "authors": [], "title": " T. Lando and L. Bertoli-Barsotti, Measuring the citation impact\nof journals with generalized Lorenz curves. J. Informetrics11, 689–703 (2017)\n" }, { "type": "Article", "id": "bib-bib6", "authors": [], "title": " H. F. Moed, Measuring contextual citation impact of scientific\njournals. J. Informetrics4, 265–277 (2010)\n" }, { "type": "Article", "id": "bib-bib7", "authors": [], "title": " H. F. Moed, From Journal Impact Factor to SJR, Eigenfactor,\nSNIP, CiteScore and Usage Factor. In Applied Evaluative Informetrics.\nQualitative and Quantitative Analysis of Scientific and Scholarly\nCommunication. Springer, Cham, 229–244 (2017)\n" }, { "type": "Article", "id": "bib-bib8", "authors": [], "title": " K. Moustafa, The disaster of the impact factor. Science\nand Engineering Ethics21, 139–142 (2015)\n" }, { "type": "Article", "id": "bib-bib9", "authors": [], "title": " D. I. Stern, Uncertainty measures for economics journal impact\nfactors. J. Econ. Lit. 51, 173–189 (2013)\n" }, { "type": "Article", "id": "bib-bib10", "authors": [], "title": " J. K. Vanclay, Impact factor: Outdated artefact or stepping-stone\nto journal certification? Scientometrics92, 211–238 (2012)\n" } ], "title": "Impact factor, an inadequate yardstick", "meta": {}, "content": [ { "type": "Paragraph", "id": "p1", "content": [ "In the 1960s, the notion of impact factor was introduced to assist libraries in\ndeciding which journals to purchase. Since the late 1990s, it has been employed\nas a metric for measuring the quality of scholarly journals." ] }, { "type": "Paragraph", "content": [ "The Web of Science (WOS), a bibliographical database created by Clarivate\nAnalytics, computes the journal impact factor (JIF) to recognize the relative\nimportance of each journal. To be assigned a JIF, a journal first needs to\nsatisfy certain quality criteria in order to be included in the Journal\nCitation Report (JCR). The JCR is a selective list consisting of more than\n11,000 journals. The (2-year-) impact factor of a journal in a specific year\nmeasures the average number of citations from that year of the papers published\nin that journal during the previous 2 years. More precisely, the 2-year-impact\nfactor of a journal in a year ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "n", "meta": { "altText": "n" } }, " is computed by the formula" ] }, { "type": "MathBlock", "id": "S0.Ex1", "mathLanguage": "mathml", "text": "J⁢I⁢Fn=CnPn-1+Pn-2,", "meta": { "altText": "JIF_{n}=\\frac{C_{n}}{P_{n-1}+P_{n-2}}," } }, { "type": "Paragraph", "content": [ "where ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "Cn", "meta": { "altText": "C_{n}" } }, " denotes the number of citations in the year ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "n", "meta": { "altText": "n" } }, " of papers published\nin the journal in the years ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "n-1", "meta": { "altText": "n-1" } }, " and ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "n-2", "meta": { "altText": "n-2" } }, ", and ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "Pm", "meta": { "altText": "P_{m}" } }, " stands for the number of\npapers published in the journal in the year ", { "type": "MathFragment", "mathLanguage": "mathml", "text": "m", "meta": { "altText": "m" } }, ". A citation of a paper given\nby the author(s) of the paper is called a self-citation." ] }, { "type": "Paragraph", "id": "p3", "content": [ "The SCImago Journal Rank (SJR) of a journal is a 3-year-impact factor\nreflecting the influence of the journal supported by Scopus. It depends not\nonly on the number of citations of its published papers but also on the\nprestige of the journals in which the citations appeared; see [", { "type": "Cite", "target": "bib-bib4", "content": [ "4" ] }, "]. A\ndrawback, however, has been reported regarding Scopus: namely, the database of\nScopus journals with assigned SJR includes about 30,000 journals, which is a very\nlarge number of journals of varying quality." ] }, { "type": "Paragraph", "id": "p4", "content": [ "Furthermore, WOS provides the indicator Eigenfactor (EF) that ranks journals in\nthe same manner as that used by Google to rank websites. Based on 5-year\ncitation data, it adjusts for citation differences through various disciplines.\nThus the SJR and EF seem to be well-suited for evaluation of the quality of a\njournal; see [", { "type": "Cite", "target": "bib-bib7", "content": [ "7" ] }, "]." ] }, { "type": "Paragraph", "id": "p5", "content": [ "Each subject category of JCR journals is divided into four quartiles: Q1, Q2,\nQ3, and Q4, where Q1 denotes the top 25 percent of all journals in terms of\ntheir JIF. There are analogous quartiles for the journals in Scopus according\nto their SJRs." ] }, { "type": "Table", "id": "S0-T1", "caption": [ { "type": "Paragraph", "content": [ "Scientometrics indices as found in databases in the year 2020" ] } ], "rows": [ { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [] }, { "type": "TableCell", "content": [ "JIF" ] }, { "type": "TableCell", "content": [ "SJR" ] }, { "type": "TableCell", "content": [ "MCQ" ] }, { "type": "TableCell", "content": [ "EF" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "Acta Math." ] } ] }, { "type": "TableCell", "content": [ "2.458" ] }, { "type": "TableCell", "content": [ "5.77" ] }, { "type": "TableCell", "content": [ "3.95" ] }, { "type": "TableCell", "content": [ "0.007" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "Iran. J. Fuzzy Syst." ] } ] }, { "type": "TableCell", "content": [ "2.276" ] }, { "type": "TableCell", "content": [ "0.51" ] }, { "type": "TableCell", "content": [ "0.11" ] }, { "type": "TableCell", "content": [ { "type": "MathFragment", "mathLanguage": "mathml", "text": "<", "meta": { "altText": "<" } }, "0.001" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "J. Funct. Anal." ] } ] }, { "type": "TableCell", "content": [ "1.496" ] }, { "type": "TableCell", "content": [ "2.42" ] }, { "type": "TableCell", "content": [ "1.61" ] }, { "type": "TableCell", "content": [ "0.035" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "J. Funct. Spaces" ] } ] }, { "type": "TableCell", "content": [ "1.896" ] }, { "type": "TableCell", "content": [ "0.46" ] }, { "type": "TableCell", "content": [ "0.43" ] }, { "type": "TableCell", "content": [ { "type": "MathFragment", "mathLanguage": "mathml", "text": "<", "meta": { "altText": "<" } }, "0.001" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "Amer. J. Math." ] } ] }, { "type": "TableCell", "content": [ "1.711" ] }, { "type": "TableCell", "content": [ "3.28" ] }, { "type": "TableCell", "content": [ "1.67" ] }, { "type": "TableCell", "content": [ "0.009" ] } ] }, { "type": "TableRow", "cells": [ { "type": "TableCell", "content": [ { "type": "Emphasis", "content": [ "Mathematics" ] } ] }, { "type": "TableCell", "content": [ "1.747" ] }, { "type": "TableCell", "content": [ "0.3" ] }, { "type": "TableCell", "content": [ "NA" ] }, { "type": "TableCell", "content": [ "NA" ] } ] } ] }, { "type": "Heading", "id": "Sx1", "depth": 1, "content": [ "Replacement for the impact factor" ] }, { "type": "Paragraph", "id": "Sx1.p1", "content": [ "The JIF has received serious criticism for various reasons, such as: lack of\nstatistical significance [", { "type": "Cite", "target": "bib-bib9", "content": [ "9" ] }, ", ", { "type": "Cite", "target": "bib-bib10", "content": [ "10" ] }, "], poor representativeness and\nrobustness [", { "type": "Cite", "target": "bib-bib5", "content": [ "5" ] }, "], insensitivity to field differences [", { "type": "Cite", "target": "bib-bib6", "content": [ "6" ] }, "],\ninsensitivity to the weight of the citing articles [", { "type": "Cite", "target": "bib-bib2", "content": [ "2" ] }, "] and\nmanipulability by editorial strategies [", { "type": "Cite", "target": "bib-bib8", "content": [ "8" ] }, "]. Here is a list of some of\nthe most significant limitations:" ] }, { "type": "Paragraph", "content": [] }, { "type": "List", "items": [ { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I1.i1.p1", "content": [ "it counts citations of articles that are not included in the denominator\nof the above formula;" ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I1.i2.p1", "content": [ "its analysis period is 2 years, which is not suitable for evaluation of\nmathematical research;" ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I1.i3.p1", "content": [ "it merely counts citations, without considering their quality. Therefore\nthe JIF may force some mathematicians to do research in topics on which a lot\nof people are working, who can potentially cite their papers. It is easy to\nfind evidence that such topics are mostly outside the mainstream of\nmathematics;" ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I1.i4.p1", "content": [ "it includes self-citations;" ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I1.i5.p1", "content": [ "it is relatively easy to manipulate JIFs and some other scientometrics.\nThere are “mutual citation groups” in which researchers in a certain circle\nheavily cite each other’s work in order to enhance the JIF of a certain\njournal and artificially inflate the impact of their own papers." ] } ] } ], "order": "Unordered" }, { "type": "Paragraph", "content": [ "The SJR aims to fix the above problems by providing a more effective\ncomputation formula, including a longer period of 3 years for counting\ncitations, attributing different weight to citations, and limiting\nself-citations. Some studies show that using the SJR can improve the situation\nto some extent. It is at any rate a first step towards avoiding some of the\nlimitations of JIF; see [", { "type": "Cite", "target": "bib-bib1", "content": [ "1" ] }, ", ", { "type": "Cite", "target": "bib-bib3", "content": [ "3" ] }, "]." ] }, { "type": "Paragraph", "id": "Sx1.p3", "content": [ "To illustrate the drawbacks and inadequacy of JIF in mathematics, let us take a\ncloser look at the JIF numbers. There are mathematical journals in the 2019\nlist of JCR-Q1 whose impact factors are “unexpectedly large”. For instance,\nthe ", { "type": "Emphasis", "content": [ "Iranian Journal of Fuzzy Systems" ] }, " is ranked 15 in the category of\nMathematics of the JCR list, while the very prestigious journal ", { "type": "Emphasis", "content": [ "Acta\nMathematica" ] }, ", launched in 1882, is ranked 13; also, ", { "type": "Emphasis", "content": [ "American Journal of\nMathematics" ] }, " and ", { "type": "Emphasis", "content": [ "Transactions of the American Mathematical Society" ] }, " are\nranked 32 and 60, respectively." ] }, { "type": "Paragraph", "id": "Sx1.p4", "content": [ "However, the SJR for ", { "type": "Emphasis", "content": [ "Iranian Journal of Fuzzy Systems" ] }, " is 0.51 but for\n", { "type": "Emphasis", "content": [ "Acta Mathematica" ] }, ", it is 5.77. Similarly, the Mathematical Citation\nQuotient (MCQ), a 5-year-impact factor computed by MathSciNet (an online\npublication of the American Mathematical Society), for ", { "type": "Emphasis", "content": [ "Iranian Journal of\nFuzzy Systems" ] }, " and ", { "type": "Emphasis", "content": [ "Acta Mathematica" ] }, " are 0.11 and 3.95, respectively." ] }, { "type": "Paragraph", "id": "Sx1.p5", "content": [ "This pattern can be seen in other journals. For example, ", { "type": "Emphasis", "content": [ "Journal of\nFunction Spaces" ] }, " is ranked 24, while the leading journal ", { "type": "Emphasis", "content": [ "Journal of\nFunctional Analysis" ] }, " is ranked 47! Again both the SJR and the MCQ of\n", { "type": "Emphasis", "content": [ "Journal of Functional Analysis" ] }, " are much greater than those of\n", { "type": "Emphasis", "content": [ "Journal of Function Spaces" ] }, "." ] }, { "type": "Paragraph", "id": "Sx1.p6", "content": [ "There is a similar situation regarding the ", { "type": "Emphasis", "content": [ "American Journal of\nMathematics" ] }, ", established in 1878, and a recently launched JCR journal named\n", { "type": "Emphasis", "content": [ "Mathematics" ] }, "." ] }, { "type": "Paragraph", "id": "Sx1.p7", "content": [ "Some important reasons for such unexpected JIFs are as follows:" ] }, { "type": "Paragraph", "content": [] }, { "type": "List", "items": [ { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I2.i1.p1", "content": [ "a high rate of publication on a topic. For instance, “fixed point\ntheory” is a popular topic that a lot of mathematicians work on;" ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I2.i2.p1", "content": [ "a considerable number of researchers working on a topic. For example, the\nnumber of mathematicians who are working on “fuzzy mathematics” is much\ngreater than those working on “", { "type": "MathFragment", "mathLanguage": "mathml", "text": "K", "meta": { "altText": "K" } }, "-theory”, and hence the general rate of\ncitations in such topics is high." ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I2.i3.p1", "content": [ "the open accessibility of a journal." ] } ] }, { "type": "ListItem", "content": [ { "type": "Paragraph", "id": "Sx1.I2.i4.p1", "content": [ "Non-ethical ways to increase JIF used by a few journals. While the term\n“predatory journal” is arguable, the mere appearance of this term shows that\nthe problem does exist.\n" ] } ] } ], "order": "Unordered" }, { "type": "Paragraph", "content": [ "The backlog between acceptance and publication in some mathematics journals may\nexceed two years. Journals with such large backlogs, which are usually good\njournals, may have unexpectedly low JIF. Nowadays, some journals have moved to\nthe continuous article publishing (CAP) model in which every article, after\nacceptance, is published immediately within the current issue." ] }, { "type": "Paragraph", "id": "Sx1.p9", "content": [ "We think that Clarivate Analytics should improve its formula for computing JIF.\nUntil then, we suggest that scientific committees should consider a package of\nindicators such as the JIF, SJR, Citescore, Eigenfactor together." ] }, { "type": "Paragraph", "id": "Sx1.p10", "content": [ "The scientometric indicators developed for journals, essentially based on\ncitations, should not be applied as a tool to assess the work of individual\nresearchers. In fact, as citation occurs after research, the direction of\nresearch should not be affected by any demand for citation. The scientometric\ndata reflect to some extent the quality of a journal, but not so much the\nactual quality of a single paper, since not all papers in a journal are cited\nequally." ] }, { "type": "Paragraph", "id": "Sx1.p11", "content": [ "As we explain in the next section, when a scientific committee uses only\nscientometric data to evaluate a mathematician’s achievement, without any human\nassessment, they are using a flawed approach that may result in an unfair\njudgement." ] }, { "type": "Heading", "id": "Sx2", "depth": 1, "content": [ "The role of human assessment" ] }, { "type": "Paragraph", "id": "Sx2.p1", "content": [ "A large number of universities around the world use scientometrics tools to\nevaluating the research of academic members, postdoctoral researchers, and\nPh.D. candidates for promotion, employment, or funding. It seems that such\nuniversities have no other reliable sources, and possibly suffer from lack of\nany peer-reviewed system in which the content of papers is expected to be\nevaluated by professional mathematicians. In addition, dealing with\nscientometric data is much easier than reading papers and assessing their\ncontent." ] }, { "type": "Paragraph", "id": "Sx2.p2", "content": [ "There are mathematicians who believe that scientometric data such as the SJR\nare reliable instruments for judgments, since they make assessments more\nobjective and free them from the crude or biased judgements of human beings.\nThey argue that quantitative indicators help funding organizations, publishers,\nand policy-makers to gain strategic intelligence that leads toward fairer\noutcomes and ensures that their budget is spent in the most effective way." ] }, { "type": "Paragraph", "id": "Sx2.p3", "content": [ "However, there are others who are against using scientometrics to measure\nscientific publications, due to the lack of transparency. Scientometrics may\ncause distortions that have detrimental effects on the development of\nscientific fields. For example, some supporters of the JIF subscribe to the\nidea that every paper published in a high-ranked journal must contain excellent\nmathematics, which is not entirely true in general; one can easily find some\ncounterexamples in the literature. Some mathematicians propose that citations\nare relevant only when dealing with large numbers. In small numbers, they can\nbe a misuse of statistics. These mathematicians continue to trust in evaluation\nby human beings, even though it may be subjective in the sense that it is\ninfluenced by the human dualities of love and hate, good and bad, as well as\ntrue and false. They believe that metrics put the worth and livelihood of our\nyoung mathematicians at risk and have undesirable impacts on the scientific\nlife of all mathematicians." ] }, { "type": "Paragraph", "id": "Sx2.p4", "content": [ "Although citations do not show all the good qualities of a paper, they (in\nparticular, non-self-citations by reputed researchers in prestigious journals)\nmay help experts in evaluating and documentating research work. Papers with no\ncitation over a ‘long period of time’ cannot be regarded as high-level papers.\nFor that matter, not all highly cited papers are necessarily high level papers.\nHowever, abuse of scientometric data such as the JIF and games with numbers can\nhappen, and may mislead people instead of being an indicator." ] }, { "type": "Heading", "id": "Sx3", "depth": 1, "content": [ "Conclusion" ] }, { "type": "Paragraph", "id": "Sx3.p1", "content": [ "Scientometrics tools can be used, provided that one keeps their disadvantages\nand distortions in mind, and they are considered together with the judgement of\nexperts based on depth and extent of papers. Such experts could be asked to\nlook at a candidate’s self-selected best papers, research programs, and\nstatements of major achievements. No assessment is complete without a peer\nreview. Furthermore, we need a modification of the policies of universities,\nfunding organizations, and so on to support human assessments.\n" ] }, { "type": "Paragraph", "id": "Sx3.p2", "content": [ "We hope that the various ideas discussed in this note may help not only\nmathematicians but the whole of the scientific community to improve their point\nof view and their assessment guidelines." ] }, { "type": "Paragraph", "id": "authorinfo", "content": [ "Mohammad Sal Moslehian is a Professor of Mathematics at Ferdowsi University of\nMashhad and a member of the Academy of Sciences of Iran. He was a member of the\nExecutive Committee of the Iranian Mathematical Society from 2004 to 2012 and a\nSenior Associate of ICTP in Italy. He is the editor-in-chief of the journals\n", { "type": "Emphasis", "content": [ "Banach J. Math. Anal." ] }, ", ", { "type": "Emphasis", "content": [ "Ann. Funct. Anal." ] }, ", and ", { "type": "Emphasis", "content": [ "Adv. Oper.\nTheory" ] }, " published by Birkhäuser/Springer.\n\n", { "type": "Link", "target": "mailto:moslehian@um.ac.ir", "content": [ "moslehian@um.ac.ir" ] } ] } ] }