Reviewing Guidelines

Dear PC members and external reviewers,

This information is to provide guidance for ISMAR 2019 reviewing, and is specifically directed towards those that are performing FULL REVIEWS of papers (e.g., secondary 2AC / committee member and external reviewers).

New this year, our review process will include a phase whereby the primary (1AC) reviewer / coordinator will examine all full reviews for quality.  If the primary deems that a review is insufficient in detail or quality, then reviewers will be asked to revisit their review and improve its quality.  In extreme cases, reviewers may be removed from the process, and other reviewers will be sought to replace the inadequate reviews.

So as is likely clear by now, we are very serious in ensuring the highest possible reviewing standards for ISMAR 2019.  As such, we ask that you carefully review this email, and if needed, seek additional resources to ensure you understand the level of reviewing quality we are committed to.

Before getting into the details of a quality review, we want to let you know that our ISMAR 2019 reviewing forms now require that you provide two separate scores; one for you to assess the submission’s quality with respect to a TVCG journal paper, and one to assess the submission’s quality with respect to an ISMAR conference paper.  

For guidance, we consider a paper of sufficient journal quality if it provides obvious and strong contribution to the field, either in theory, methods, engineering, design and/or evaluation approaches.  Journal papers should reflect an in-depth overview of related work on behalf of the authors (as compared to conference paper), and, in turn, the resulting paper is comprehensive and self-contained.

Alternatively, we consider a paper of sufficient conference quality if it presents a piece of research that may not be complete per se, but still affords a tangible contribution.  Conference papers are often earlier-term work (as compared to journal papers) and thus can allow authors to announce or mark an emerging idea.  In some cases, a conference paper may present preliminary findings, but, of course, more complete findings are always welcome!

Therefore, as you perform your reviews, we ask that you reflect on the overall contribution of the paper and how the contributions position the submission to be accepted into TVCG or ISMAR or neither.

In terms of the actual review, we strongly recommend that you read the entire (short) article by Ken Hinckley, which we found to give a lot of constructive advice:

Hinckley, K. (2016). So You're a Program Committee Member Now: On Excellence in Reviews and Meta-Reviews and Championing Submitted Work That Has Merit. https://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf

We’ve pulled some of the major elements from Ken’s paper, in some cases modified the text, added insights from further advice by Steve Mann and Mark Bernstein, and placed them in bullet form below:

From ISMAR’s perspective, we’d like to add that in your role as gatekeepers of high-quality papers, we ask that you not categorically find hidden flaws and assassinate papers wherever possible but instead accept papers for their merits.

When reviewing, keep in mind that almost every paper we review “could have done x, or y, or z”.  Don’t fall into this trap!  We cannot reject papers because authors did not perform their research the way we would have done it, or even how it is typically done.  Instead, we must judge each paper on its own merit, and whether or not the body of work presented can stand on its own, as presented.  Sure, every paper “could have done more”, but is the work that has been done of sufficient quality and impact?

Furthermore, we should not reject papers because “this experiment has been done before”.  This fallacy of novelty, ignores the long standing tradition in science of replication.  New work that performs similar research and finds consistent (or even conflicting) results are of value to the community and to science in general, and should be considered on their own merits.

In addition, we want to encourage you to make use of conditionally accepting papers. In contrast to previous years, we would like to encourage the conditional acceptance of more papers and give authors the chance to polish their work in the revision cycle. If these conditions are not fulfilled, then the paper can (and should) be rejected at the end.

And finally, to quote Ken Hinckley: "When in doubt, trust the literature to sort it out."

Thank you for work to ensure the highest quality ISMAR reviews,

ISMAR 2019 PC Chairs

Shimin Hu

Joseph L. Gabbard

Jens Grubert

Stefanie Zollmann


Thanks to our sponsors

We thank our sponsors for supporting the ISMAR 2019 conference.

DIAMOND Platinum GOLD      SILVER BRONZE

Support Units