http://www.sciencemag.org/xxxx/content/full/317/5841/1026?etoc

EDUCATION:
Who Ranks the University Rankers?
Martin Enserink

....
Shanghai surprise
The Shanghai ranking avoids all of these problems by eschewing university-provided data and expert reviews. Instead, it uses only publicly available data, such as the number of publications in Nature and Science, the number of Nobel Prizes and Fields Medals won by alumni and staff, and the number of highly cited researchers. The result is a list based almost exclusively on research. Nian Cai Liu, who heads the Institute of Higher Education at Shanghai Jiao Tong University, started the ranking 5 years ago because he wanted to know how Chinese universities were placed in the global pecking order. When colleagues started asking for the data, Liu put them on a no-frills Web site, which now gets thousands of visits a day.

But as the Berlin quarrel shows, the ranking has its own problems. For example, Shanghai credits the institution where the Nobelist worked at the time of the award. And that can make a difference. Andrew Fire's 2006 Nobel in physiology or medicine helped his current institution, Stanford University in Palo Alto, California, move up from third to second place, even though Fire did his groundbreaking work on RNA interference while at the Carnegie Institution in Baltimore, Maryland.

Universities that focus on social sciences or humanities also tend to suffer under the Shanghai system. Recognizing that scientists in those disciplines gravitate to different journals, Liu doesn't count Nature and Science papers and redistributes that 20% share across other indicators. Still, the effect is noticeable: In 2006, the well-respected London School of Economics and Political Science ended up in the 201-300 tier (this far down the list, Liu no longer gives individual ranks), whereas the THES awarded the school 17th place.

Well aware of their influence, and the criticisms, the rankers themselves acknowledge that their charts aren't the last word. U.S. News & World Report, for instance, advises students to take many factors into account when choosing a college. A pop-up window on Liu's Web site warns that "there are still many methodological and technical problems" and urges "cautions" when using the results.

In response to the critics, some rankers are also continuously tinkering with their xxxxulas. But that opens them up to another criticism, namely, that a university can appear to become significantly better or worse in a single year. Many have accused U.S. News of changing its method precisely to shake up the tables and thus boost sales, a charge the magazine rejects.

In part to boost their credibility, the rankers have founded the International Rankings Expert Group (IREG), which in 2006 came up with a set of ranking guidelines. Called the Berlin Principles, they stress factors such as the importance of transparency, picking relevant indicators, and using verified data. Usher, an IREG member, concedes that the principles are quite general in nature because they are the "biggest common denominators" among groups of rankers with very different views. Many rankings aren't fully compliant with the rules yet, says CWTS researcher Henk Moed.

....