SIG Governing Board Minutes: October 23, 2020

Attendees

*list is not all-inclusive

Srinivas Akru Jonathan Aldrich Sujata Banerjee Adam Bargteil Chris Brown Peter Brusilovsky Donna Cappo Sanmay Das Adrienne Decker Scott Delman Babak Falsafi Laurie Fox Juliana Freire Wayne Graves Marco Gruteser Vicki Hanson Sharon Hu

Natalie Enright Jerger Jeff Jortner Samir Khuller Shan Lu Barry Lunt Jeanna Matthews Renee McCauley Helena Mentis Fred Niederman Jens Palsberg Jian Pei Kalyan S. Perumalla Benjamin Pierce Franz Rothlauf Pat Ryan Shari Trewin

Duration: 3 Hours

Meeting starts at 7:06

Jonathan Aldrich (liaison to the ACM PUBS board) + Scott Delman

Jonathan speaks from 11:23 – 20:56

Publications

1. Double Blind Reviewing

Should ACM mandate some form of/partial Double Blind Reviewing?

Scott Delman: The pubs board is very clear that all authors including conferences and journal authors have the right to post their pre-prints at any stage in the process

Helena Mentis: Full transparency of reviewers so that bias and bad behavior cannot be hid behind anonymity. The movement right now proposes that a review should be signed to indicate that a reviewer stands by their review.

Shan Lu: Listing all reviewer names (tied to a paper) without linking the review to the reviewer’s name may help reduce ethical violations

Samir Khuller: Worries that reviews will become sugar coated if we require them to be signed, particularly in the case of junior employees reviewing the work of their superiors who may need their assistance in the future

Srinivas Akru: Reviewer anonymity should not be compromised. Compromised anonymity may lead to sugar coated reviews in the context of a junior position holder reviewing their senior’s work and in the context of people in power (those on funding panels) deliberating with a bias against people who rejected their work in the past.

Peter Brusilovsky: We should not have a mandate but instead a best practices in regards to double blind

Chris Brown: A double blind mandate policy would be problematic, and it may end the relationship between SIGSAM and one of our sponsored/in co-op conferences (ISSAC) which so happens to be an independent entity with their own bylaws

Jeanna Matthews: would like to see a Best Practices White Paper considering the differences in opinion being shared specifically in regard to marginalized groups. Proposed for more experimentation with DBR before adopting a new movement or mandate.

Adam Bargteil: Also a proponent for a detailed Best Practices White Paper that can be shared with Program Chairs

Shan Lu: There is a difference between discussion-oriented workshops and conferences. People are more interested in discussing a paper from a well-known company like Google as a workshop in comparison to a lesser known company, hence double blind is not practiced in this context by SIGOPS where discussion is prioritized over anonymity.

Sujata Banerjee: SIGCOMM’s operational systems track papers - it made no sense to anonymize the company name, so we still anonymized authors (edge case example similar to what Shan Lu discussed in regard to SIGOPS workshop)

Marco Gruteser: Supportive of sending a strong signal to reduce bias in review processes. An option to handling the issue of DBR would be to strongly suggest double blind but allowing Conferences the option of submitting an ultimate plan for reducing bias in their peer review processes.

Sanmay Das: scarcity of reasonable reviewers; are reviewers anonymous to each other? It would be great to have best practices to know what other folks are doing.

Kalyan S. Perumalla: wants guidance regarding transition strategies from non-double blind to double blind reviewing in conferences

Sharon Hu: What is the real reason behind a decline in review quality? Perhaps this is something the PUBS board should investigate. Sharon fully supports the notion of general best practices from ACM regarding DBR.

Juliana Freire: How do we reduce bias while simultaneously allowing people to do significant incremental work? Not being able to refer to prior work in the context of DBR can create issues for authors who have had their work rejected because they did not cite their own work.

Babak Falsafi: It would be good to discuss the journal first model in best practices

High Level Takeaways as per Jonathan Aldrich

• A lot of support for Double-Blind in general
• A lot of requests for a Best practices white paper & more information about how to implement it effectively
• A need for flexibility, we would not want to have a strict mandate because some of these conferences are coordinating, there is also a desire to experiment with some other models that may perhaps help marginalized communities in different ways

2. Peer Review Report

Jonathan speaks from 59:34 – 1:04:00 

Would creating an ACM Conference-wide standard timeline for completing reviews (3 weeks, for example) be welcomed?  Potential changes to the minimum number of reviews required to make publication decisions for journal and conference submissions (from 3 to 2).

Jens Palsberg: There is some experimentation with SIGPLAN conferences. For example, a paper was accepted with two strong reviews in the first round, whereas another was rejected that had two strong negative reviews. There was not a need for a third review in these circumstances.

Shan Lu: It would not be ideal to have a unified timeline across conferences considering the paper review workload varies across conferences. Perhaps eventually we will shift to a model in which papers are rejected with two reviews, but for now that seems like a difficult call to make.

Franz Rothlauf: There has been an increase in submissions with a simultaneous decrease in quality. SIGEVO is not pushing hard on decreasing the timeline regarding video submissions considering it ended up reducing the amount of video submissions received.

Sharon Hu: We should separate the discussion when it comes to journals and conferences. 3-week review timeline is not workable and should not be mandated. How can we create a mechanism to get more reviewers to sign up? Do we need so many conference and journal publications? Can we think about a model in which conferences are more about socializing?

Jian Pei: The conference publication acceptance process should be made easier, which is something we will implement in our KDD ’21 conference to provide a more inclusive community. Services like arxiv are a threat to the conference and journal publication business model, hence the need to be more accepting regarding publications.

Shan Lu: ASPLOS is doing an experiment this year in which they asked authors to submit a 2 page extended abstract and a 10 page full paper with the 2 page extended abstract being rejected with 5 reviews in the first round as opposed to the 10 page paper being rejected with 2 reviews in the first round. Advocating for the integration of TPMS with HotCRP

Adam Bargteil: The trend regarding publications seems to be moving away from conferences and more towards journals. SIGGRAPH is trying to come up with a procedure for reviewer continuity from conferences to journals.

Sharon Hu: Proponent for getting author’s feedback regarding their reviews and or rejections and giving them a booming voice in general. Would like to see ACM’s guidance around this.

 

3. Orcid (a way of identifying authors)

Jonathan speaks from 1:23:30 – 1:28:20 

Orcid Mandate

Jens Palsberg: Strongly in favor of open access because it helps eliminate ambiguity surrounding author identity

Juliana Freire: In favor of Orcid because it would help with conflict of interest detection

Helena Mentis: Can authors use pseudonyms to protect themselves against harm?

Scott Delman: Orcid does not prevent you from using a pseudonym to register ^. This mandate will not be burdensome to authors as we are not mandating them to take ownership of their bibliography, which is the time-consuming portion of this process. Scott along with a few other staff members will provide resources for anyone that needs assistance with Orcid.

Laurie Fox: We are missing networking on our f2f conferences, in support of Sharon Hu’s viewpoint. Worries that requirements such as DBR and Orcid will result in fewer conference submissions.

Peter Brusilovsky: A thoughtful implementation of Orcid will make a difference in terms of saving time as opposed to being burdensome

Fred Niederman: Orcid implementation should put the authors’ interests first, otherwise it may become an additional burden resulting in a lower amount of author participation. How can we make Orcid collection seamless by integrating it with pre-existing processes that authors are already participating in as part of their submission process?

High Level Takeaways as per Jonathan

• Making sure this stays very easy for Authors particularly Authors who do not publish very much

• This disproportionately helps groups like women and transgender people who are more likely to have multiple names

• There is also a very positive diversity related reason to go for this in addition to all the other benefits