I have a friend who is in charge of managing the peer review process at some serious scientific journals. It’s a tough job requiring a set of skills that includes dealing with sometimes ornery people, managing multiple schedules, and expertise in the fields in which she works. She makes a good case for peer review, and for the journals that rely on it. Peer review has value and costs money, she says. So, journals have to charge fees to support the peer review process, and they have to hold onto the rights at least long enough to recover their costs.

I recognize the value of peer review. It not only directs our attention to worthwhile research, it is part of an editorial process that improves articles before they’re published. But peer review doesn’t scale. There’s so much research being done. A lot of it is good work but isn’t important enough to merit the investment in a traditional peer review process (including the failed hypotheses that we were taught in school were not failures at all). Peer review is valuable, but it’s a choke point required because traditional publishing’s neck is so thin. And it may — may! — turn out that the combination of crowds and quirky individuals can replace peer review’s value. Of course, we’d want the crowd to consist of people with some standing for evaluating the research. And we’d want to be sure that the quirky individuals who buck the crowd are not delusional psychotics. I of course don’t know what the world will look like (or what it does look like, when you come down to it), but I suspect that we’re going to have a mixed research ecology, with peer reviewed journals making recommendations we trust highly, and a wide variety of other ways of finding the research that matters to us. With PLoS and PLoS, and arXiv, and Nature’s version of arXiv, and all the rest of it, we’re already well on the way to filling the important niches in this new knowledge ecology.

In fact, peer review generally establishes two characteristics of a piece of work: It was performed properly and it is important enough to merit throwing some ink at it. Those are important criteria, but hardly the only ones. “This hastily performed work uses a flawed methodology but turns up an interesting fact worth considering” is the type of criterion researchers use when recommending articles to one another. There’s value there, and with research that has good data that it misanalyzes, research that is promising but incomplete, research that inadvertently demonstrates a flaw in some lab equipment, etc. etc. etc. And, as always, the value is in the long tail of et ceteras. [Tags: peer_review open_access science publishing everything_is_miscellaneous ]

Categories: Uncategorized dw