Physicist Proposes New Way to Rank Scientific Output

November 8th, 2005

Publications in peer-reviewed journals are the yardstick by which academic scientists compare their work with their colleagues. But is the best measure of a scientist’s worth the total number of his or her published papers? Or the average quality of those papers, based on the number of times they are cited or the reputation of the journals in which they are published?

According to a physicist at the University of California, San Diego, neither of these methods—often used in academe or federal agencies to judge scientific publication records for hiring, promotion or grant awards—gives consistent and satisfactory comparisons. So Jorge E. Hirsch, a physics professor at UCSD, devised an alternative that appears to be a simpler and more reliable way to rank scientific output within a discipline than any now in use.

In a paper published in the November 15 issue of the Proceedings of the National Academy of Sciences, which appears this week in the journal’s early online edition, Hirsch explains that his “h-index” can give a reliable “estimate of the importance, significance and broad impact of a scientist’s cumulative research contributions.” What’s more, for each scientist, his method provides a single number, which takes only 30 seconds to compute, that can be used to compare a scientist’s relative rank within a discipline.

“For a person to have a high h-index is not an accident,” Hirsch says, after testing his method on scientists in a variety of disciplines and circulating his formula on physics bulletin boards for other scholars to test. “I myself was surprised to see how consistent an estimate you get with this method. It does seem to say something about a person’s overall academic achievement.”

The h-index is derived from the number of times a scientist’s publications are cited in other papers, but is calculated in a way to avoid some of the problems associated with counting large numbers of marginal papers or high-profile coauthors.

For example, Hirsch says that while the total number of publications gives some indication of a scientist’s productivity, it says little about the quality of those publications. And while the total number of times a scientist’s papers are cited in other publications says something about their quality, he says those measurements can be suspect if a scientist has high-performing coauthors, few publications or a lifetime of mediocre work skewed by one or two highly cited papers . Citation counts may also be skewed if a scientist publishes scientific review articles, which are not reports of original research, but summaries of other scientists’ work frequently referenced in subsequent journal articles.

Hirsch was motivated to develop the h-index because of his own problems publishing controversial papers on superconductivity in journals considered high-impact. Although these papers ended up in journals categorized as low-impact, they garnered many citations, evidence of their importance to the field.

His new method relies on the use of the Thomson ISI Web of Science database at http://isiknowledge.com To search for a scholar’s h-index, go to the Web of Science and enter the name in the “General Search” category. Clicking on “Search” brings up a list of papers over the entire lifetime by that author. To reorder the list from the most highly cited papers to least cited, click on “Sort by Times Cited” in the right hand column.

The h-index is obtained by moving down this list until the number of the paper—essentially the scholar’s h name—exceeds the number of citations from that paper. For example, a scholar will have an h value of 75 whose 76 th paper on the list has been cited 75 or fewer times, but whose 75 th paper has been cited 75 or more times. Put another way, this scholar has published 75 papers with at least 75 citations each.

Hirsch devotes a section in his paper to demonstrate mathematically why this method for “h”—which stands for “high citations”—seems to work. But the real proof of the pudding came when he applied the h-index to the scientific luminaries within various disciplines and found that they ended up where expected.

Edward Witten, a theoretical physicist at the Institute for Advanced Study in Princeton, N.J., who developed an extension of string theory and is widely regarded as one of the most brilliant physicists ever, has the highest h-index in physics, 110. By contrast, Nobel laureate Philip Anderson of Princeton University has an h-index of 91, while Nobel laureates Steven Weinberg of Harvard University has an h-index of 88, Frank Wilczek of the Massachusetts Institute of Technology (68) and David Gross of UC Santa Barbara (66).

Hirsch, whose own h-index is 49, notes that comparisons of h-index among scientists in different disciplines don’t work as well. High-impact biologists tend to have generally higher h-index values, he says, possibly because of their greater research resources, while social scientists tend to have lower h-index values, presumably because their other non-journal publications, such as books, are not factored into this calculation.

Nevertheless, Hirsch is able to make some generalizations. After 20-year career in science, he says in his paper, an h-index of 20 should generally indicate a “successful scientist,” while an h-index of 40 “characterizes outstanding scientists, likely to be found only at the top universities or major research laboratories.” An h-index of 60 after 20 years or 90 after a 30-year scientific career, meanwhile, he says, “characterizes truly unique individuals.”

Hirsch says he is concerned that his h-index, while useful to compare publication records, not be misused.

“It should only be used as one measure, not as the primary basis for evaluating people for awards or promotion,” he adds. “You surely wouldn’t want to say that in order to get tenure or to get into the National Academy of Sciences you need to have an h-index of such and such.”

Nonetheless, Hirsch’s h-index has generated intense interest among scientists who have found out about it and used it.

“The reaction I’ve gotten has been very favorable,” he says. “Scientists want to know how they compare to their colleagues. The h-index really says something about that person and their work.”

Source: UCSD (By Kim McDonald)