Marc is quite right – we can’t over-romanticize it, but we also can’t simply say “it’s corrupt; you can’t trust anything.” Yet the issues that complicate peer review are closely tied to the way we do science (and other scholarly work) these days, so it a good lens for looking at scholarship and its discontents.

In a course I teach, I have students read Michael Polanyi’s “Republic of Science” – an admittedly rosy view of how it works. Then I give them a piece by John Ziman, “Is Science Losing its Objectivity?” (from Nature 382 [1996]: 751-754). He sees the importance of “disinterestedness” – or rather, interest in truth regardless of personal gain – under threat as science focuses on growth, influence, and competition for resources. They help students grasp how knowledge is built and rebuilt by people who may share some ideals but often fall short.

I think the scare headline – “wow! most science is tainted!” – is one way to catch people’s eye and get them to to think about how science works, but you could just as easily say “wow! Most newspaper articles have errors!” or “You can’t trust books! They usually have mistakes!”

That said – the more we think about all scholarly communication in the broadest context: creators, conservators, publishers, consumers, funders – the whole political, economic, and cultural context in which new ideas are generated – the more interesting and rich the conversation will be. And we should have that conversation, often, because it matters.

Thanks for pointing out the Weller book, I’ll take a look.

The limitations of peer review have been well recognized for a very long time, as has the value. JAMA has been sponsoring an International Conference on Peer Review every 4 years since 1990 and there is now a considerable body of research as well as considerable debate within the publishing community about the value and limitations of peer review, the variety of forms that it can take, and various mechanisms that can be used to improve it.

Librarians have, indeed, been a part of the conversation. I can particularly recommend Ann C. Weller’s book “Editorial Peer Review: Its Strengths and Weaknesses” published in 2001. Ann is a health sciences librarian at the University of Illinois Chicago and her book is an excellent in-depth discussion of the facts and issues involved with peer review. (I should mention that she served as an editorial board member during part of my six-year tenure as editor of the Journal of the Medical Library Association (JMLA).)

No one, to my knowledge, has ever claimed that peer review, in any of its current manifestations, is sufficient to eliminate all cases of fraud or poor science. There is, however, wide agreement that it remains a very important tool, and, of course, the possibility of using the internet to experiment with new and, perhaps more open, mechanisms for improving peer review is one of the most promising developments of the current transformation in scholarly communication.

The PRISM rhetoric about open access being a threat to peer review is obviously overblown and easily argued against, but few publishers, commercial or otherwise, have made extreme or unsupported claims for the value that peer review brings. It is very important to separate serious debate and discussion of peer review from the rhetorical mudslinging (on both sides) that’s been generated by the PRISM rhetoric.