Reviewing Guidelines

Members of the Program Committee (PC) assist a submission’s Primary Area Chair (1AC) by writing a review for the paper, and participating actively in the discussion phase.

Every paper has been assigned a Primary Area Chair (1AC) and three first tier reviewers. As reviewer, your job is to provide critical assessments of the papers assigned to you based on the review form that we have prepared (see below). For each paper assigned to you, your review and the reviews by two fellow reviewers will form the basis for a discussion about the paper.

The Primary AC will lead the discussion, and if necessary flag the paper for additional review by a Secondary Area Chair (2AC). The 2AC will provide that review without seeing the other reviews, and then join the discussion. Based on the discussion, the 1AC will arrive at a recommended decision. This recommended decision will be passed on to the Program Chairs and used as input to the Program Committee Meeting.

The review form has nine questions to answer for every submission:

• Relevance to SIGIR
• Originality of work
• Technical soundness
• Quality of presentation
• Impact of ideas or results
• Adequacy of citations
• Reproducibility of methods
• Overall recommendation
• Reviewer’s confidence

Please assign a score for each of these questions. In the text area below the questions, you should supply a detailed rationale for the scores you have assigned across the first eight questions.

Your review is not just a vote for whether the paper will be accepted; it is essential input to a discussion amongst the reviewing team and to the Program Chairs. It is also the input to the authors to guide them with the changes suggested, and to help them understand the outcome of the review process. You are assisting your your fellow PC members, the Area Chairs, and the Program Chairs by providing arguments for or against acceptance.
In some cases, there will be divergence amongst reviewers’ numerical ratings of the paper; if you provide only a rating and terse summary, without an adequate rationale, it will not be helpful.

Start your review with an assessment of what you consider to be the main contribution of the paper. Please do not just repeat what the authors say they did. You should provide your own summary of what you gained by reading the paper.

Whether you like or dislike a paper, please say so in a manner that is helpful to the authors and informative to your area chairs. (You will be asked to rewrite reviews that do not meet this expectation.)

You will notice the “Reproducibility of methods” criterion. This question is not about the use of proprietary data. It is about whether you think the authors provide sufficient details to reproduce the work.
Do you think that other researchers would be able to reproduce the method and/or results presented in the paper if they had access to the same or similar resources? Are the descriptions of the methods used detailed and accurate? Given the resources used in the paper, or (if they are unavailable) similar resources, could researchers carry out similar experiments to verify the results? What further description could the authors provide?

In the text box labeled “Summary of your review”, please summarize your main points. It is important to point out weaknesses and validity issues, but it is equally important to identify the contribution of a submission. Ultimately, a submission’s acceptance depends on its novel contribution, not perfection. Note again that we are looking for an evaluation of the paper, not just a recommendation.

One issue that may arise is that authors miss some of the prior research that has been published in the area. This should be regarded as being a fatal flaw only if the missing work critically affects their conclusions; remember, authors will have opportunity to make small editorial changes to their papers (including adding missing references) if they are accepted

If you do regard a paper as being unacceptable because of lack of reference to prior work, you should supply sufficient detail about this prior research in the form a complete reference ideally including its DOI, so that the authors can understand why you believe their paper should not be accepted. We encourage you to provide this level of detail for all references that you consider missing in the submission.

Thank you for your contributions to creating an excellent program for SIGIR 2017!

Background information

Nancy R. Gough, Training for Peer Review, Science Signaling 25 Aug 2009: Vol. 2, Issue 85, pp. tr2 DOI: 10.1126/scisignal.285tr2 (.pdf).