Course Information:

Admin

• The course on Brightspace

Instructions

• Instructions for the reviewing a top data science paper assignment

Reviewing

Peer review is a central process in the conduct of science.

The resources below help understand this process, and provide background to writing excellent reviews.

Reviewing criteria

• UAI 2015;
• ICML 2015;
• SIGIR 2017
• SIGIR 2018

Why review

IP&M published an encouraging editorial Reviewer merits by Marchionini et al. (.pdf).

How to read a paper

• S. Keshav, How to read a paper (2007), published in ACM SIGCOMM Computer Communication Review, 37(3):83–84 .pdf; The paper is discussed briefly in a blog post by Charles Sutton.

How to review a paper

• Timothy Roscoe, Writing reviews for systems conferences (2007) .pdf;
• Alan Jay Smith, The task of the referee (1990);
• Nancy R. Gough, Training for Peer Review, Science Signaling 25 Aug 2009: Vol. 2, Issue 85, pp. tr2 DOI: 10.1126/scisignal.285tr2 (.pdf or .pdf);
• Science published A Peer Review How-To, a letter by Robert S. Zucker (.pdf).

Finally, Adrienne Shaw from Ideas on Fire, and academic publishing and consultancy agency, wrote a very clear blog post on how to write a review, worth the read even if it does not specifically focus on computer science research.

How not to review a paper

• Graham Cormode published How NOT to Review a Paper: The tools and techniques of the adversarial reviewer .pdf

See also

Finally, you might be interested in a few empirical studies into the reviewing process itself:

• the NIPS 2014 consistency experiment (and this blog post);
• The WSDM 2017 analysis of single vs. blind reviewing (and this blog post);
• The ESA 2018 analysis that compares decisions on papers between two teams of reviewers.