This past week was "Peer Review Week", when the institution of peer review has been much discussed. As I'm in the midst of preparing three grant applications, peer review is very much on my mind. Peer review has caused some researchers much grief and angst (e.g., here), but in my (albeit limited) experience, it isn't as bad as all that.
But one important question that I think even the worst detractors of peer review would ask is, "How do we improve peer review?"
I propose a sketch idea as follows:
First, allow authors to rate their peer reviewers on fairness, expertise in the research topic, thoughtfulness of comments, and so on. Let authors identify whose peer review comments were helpful and whose were not. Journals then compile data on the quality of peer reviewers and share these data with funding agencies. Funding agencies should then select the best possible peer reviewers to serve on study sections or review panels. Certainly, biased/uninformed/sloppy peer reviewers should be kept far, far away from making decisions on whose grants should be funded. Journals can also reward consistently good reviewers by inviting them to serve on editorial boards, raising good reviewers' profiles in their research communities.
This system would incentivize everyone to contribute to peer review and to do their best job at it. If we set the right incentives to reward fair, knowledgeable, thoughtful, helpful peer reviews, then the best science will be funded and published in a more transparent way. And the entire ecosystem of science will benefit tremendously.