Untitled

On the increasing importance of authority proxies

(cit.) It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists' abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant's CV, they will have to rely at least in part on proxy indicators.

Correlation between IF and distributed measures. How does this apply to soft peer review metrics? Potential research direction.

(cit.) For a sample of 2,500 neuroscience papers listed on F1000 [Faculty 1000 - a website that seeks to identify and rank interesting papers based on the votes of handpicked expert faculty members], there was a strong correlation between the paper's F1000 factor and the impact factor of the journal in which it appeared.

Minimal criteria that any candidate system alternative to peer review should meet.

(cit.)

• It must be reliable – it must predict the significance of a paper with a level of accuracy comparable to or better than the current journal system.
• It must produce a recommendation that is easily digestible, allowing busy scientists to make quick decisions about what to read. [...]
• It must be economical, not only in terms of direct costs such as web operations, but also in terms of reviewer time invested.
• It must work fast. The peer review system produces clear-cut decisions relatively quickly (in part because editors pester reviewers to deliver their reports), whereas many forms of communal assessment – such as the emergence of a statistically significant pattern of citations or expert recommendations – are likely to be slow and gradual by comparison. Perhaps a popularity index (for example a ‘most emailed’ list) would provide a quick readout, but there is a danger of runaway amplification – the so-called ‘Matthew effect’, recognized by Robert Merton almost 40 years ago4 and likely to be exacerbated in the era of digital communication
• It must be resistant to ‘gaming’ by authors. Of course, savvy authors already know how to work the current system, but the separation of powers between editors and anonymous reviewers does – I believe – preserve some integrity to the process.