ACL 2022 Chairs Blog Post about Paper Commitment and Anonymity Period

ACL 2022 PC Chairs Blog Post 3: Submissions for the (Last) November Deadline, Reviewing, Commitment to ACL 2022, and Recognition

ACL 2022 will be the 60th Meeting of the Association for Computational Linguistics. Along with the many changes in our community, this anniversary is also marked by the decision of the ACL 2022 and NAACL 2022 teams to help implement the vision of using the ACL Rolling Review (ARR) for our major *ACL conferences. As ACL 2022 is the first conference to fully adopt this review process, we have been working closely with ARR, and we have been coordinating our efforts with the NAACL 2022 PC chairs. As explained in our previous blog post, we joined ARR as Guest Editors-in-Chief (EiCs) to help with the September-November submissions, which primarily target ACL 2022. We thus had the opportunity to witness firsthand the enormous efforts made by the ARR team to enable this change in the review process for the *ACL conferences. We would also like to note that this is a community effort, and we are grateful for the support of the authors, the reviewers, the Action Editors (AEs), and the Senior Area Chairs (SACs), who have been constructively engaging and helping with ARR and ACL 2022. Below, we announce some initiatives we started, aiming to recognize their efforts. Yet, let us first give you some details about the submissions to ACL 2022, the reviewing, and the commitment process.

I. Submission to ACL 2022.

1. Submissions to ARR for the November 15 deadline. A total of 2,917 submissions for the November 15 deadline have been made to ARR, including 202 resubmissions. Since then, 18 submissions were withdrawn and 71 were desk-rejected (these might not be the final numbers, as more papers might be withdrawn, and there could be some additional desk-rejects). The primary reason for desk-rejects were formatting issues, e.g., using a non-ARR template (a number of submissions used the ACL-IJCNLP 2021 template, which uses a different page size and gives authors about half a page of extra space compared to the ARR template) or having more than eight pages of content (excluding references and appendices). In some very rare cases, a desk-reject was due to the paper being on a topic that is not relevant to *ACL. The number of desk-rejects is somewhat comparable to historical figures, e.g., ACL-IJCNLP 2021 desk-rejected 19 papers (out of 3,347 submissions), ACL 2020 desk-rejected 29 papers (out of 3,429 submissions), and ACL 2019 desk-rejected 88 papers (out of 2,820 submissions).

2. Number of submissions for ACL 2022. Note that the November 15 ARR deadline was the last one submissions to which could be considered for ACL 2022, if the authors chose to commit their paper to ACL 2022 (the December 15 submissions are not eligible for ACL 2022, but can be committed to NAACL 2022). For ACL 2022, we have a two-step submission process, where authors first submit their paper to ARR, and then once their paper has received reviews and a meta-review, the authors need to commit it to ACL 2022 in a second step (by January 15, 2022, as per the updated timeline). As we have explained in an earlier blog post, we plan to report the number of submissions and the acceptance rates in two different ways: (a) based on the declared intended venue at the time of submission to ARR (for all June–November submissions), and (b) based on actual commitment to ACL 2022. We will provide detailed statistics about both (a) and (b) in our next blog post.

II. Reviewing Process and Recognition for Reviewers and Action Editors

Next, let us have a look at the process of reviewing:

1. Reviewer and Action Editor pool. As we have explained in our earlier blog post, in anticipation of the higher number of submissions for the November 15 ARR deadline, together with ARR, we have put a lot of effort in extending the size of the pool of reviewers and Action Editors. Eventually, we managed to grow it from 1,273 for October to 3,433 for November. We also increased the number of Action Editors from 201 to 359 (in fact, we managed to recruit a total of 470 AEs, but we could not use some of them as they were SACs for ACL 2022 or were acting as reviewers, and we wanted to avoid people being involved in more than one role as much as possible). From the 3,433 recruited reviewers, a total of 2,530 who had complete OpenReview profiles are reviewing for November. Together with ARR, we are working to contact the remaining 984 reviewers to ensure that they fill in their profiles and can contribute in future months. Reviewers were asked to add to their Open Review profiles links to their Semantic Scholar and DBLP profiles, as well as to import some of their DBLP papers into their Open Review profile, but many did not perform this last step. In general, the algorithm for matching reviewer profiles to ARR papers can work with either/both Semantic Scholar and DBLP profiles, even with no DBLP papers imported, but for November, there was an algorithmic glitch and all reviewers with no DBLP papers imported were ignored. Due to time constraints, once this was discovered, ARR could not rerun the matching algorithm as it takes a long time to run with a large number of submissions; thus, we had to accept losing 984 reviewers for the initial assignments, but they can contribute as emergency reviewers. Eventually, many of the November reviewers were assigned 5 papers, as each paper gets an initial assignment of 4 reviewers. This algorithmic glitch will be fixed for the December reviewing cycle (however, December is not an ACL 2022 cycle). In the meantime, we ask reviewers to make sure they import some DBLP papers that reflect their research interest to their Open Review profiles; it is also fine to import all their papers. There are also 984 outstanding invitations in ARR to qualified reviewers for November, who have not replied to our invitations; we ask members of the *ACL community to accept to review for ARR if they are submitting papers to *ACL venues. Finally, about 500 of the reviewers in the November pool are authors of papers submitted to ACL 2022, who had not subscribed to be reviewers for ARR, but whom we asked to review anyway, as per the new policy that authors of papers submitted to ARR should review for the ARR cycles that they submit for, if asked. Note that we did not ask all authors of submitted papers, but only those who fulfill the ARR reviewer criteria.

2. Paper assignments and manual checking. As we have explained in our earlier blog post, for the November cycle, we added some of the ACL 2022 Senior Area Chairs (SACs) as SACs in ARR (a total of 39 SACs were added in this way). We then ran an automatic assignment of (i) papers to reviewers, (ii) reviewers to Action Editors (AEs), and (iii) AEs to SACs. We, the ACL 2022 PC chairs, then manually checked the assignment of AEs to the 39 SACs (about 10 AEs per SAC), as well as the assignment of AEs to papers for the Theme track. After that, we asked the SACs to check the assignment of AEs to papers (up to 5-6 papers per reviewer, with some reviewers getting a lower load as they had reviewed in previous cycles and ARR tries to balance their overall reviewing load over time) and to suggest AE reassignments if needed. Finally, we asked the AEs to check their papers for potential violations and to indicate candidates for desk-rejects as well as possible paper re-assignments.

3. Timing. The initial ARR review timeline was 35-days long: with submissions on the 15th day of the month, and notifications on the 20th day of the following month. Given the substantially higher number of submissions we anticipated for November and also given the additional steps of having SACs check the AE assignments, we initially worked with ARR to temporarily extend the timeline by 10 additional days, while keeping most reviewer and Action Editor duties to be before the holiday break, with reviewing starting on November 25 and reviews due by December 16, meta-reviews due by December 24, 2021, and author notification on December 31, 2021. However, the scale of the November submissions created some unanticipated technical difficulties along the process, which pushed the actual reviewing starting date to November 29, 2021. Keeping our original timeline would have meant that reviewers would have had to fulfill their duties in just about two weeks (from November 29 till December 16, 2021). Given that most reviewers were assigned five papers and given our desire to optimize for the quality of the reviews in order to reward the authors’ hard work, together with the ARR team, we decided to further extend the timeline of the reviewing process to allow for reviewers to finalize their reviews by December 27, 2021, have the meta-reviews in by January 7, 2022, the author notification by January 10, 2022, and finally push the ACL 2022 commitment deadline to January 15, 2022. We know that this timeline goes over the holiday and the break period, particularly for many of our Action Editors, and we are grateful for everyone’s dedication and hard work. Yet, we would like to note that most of the time interval covers a period before the holidays (for reviewers) or after the holidays (for Action Editors), which means that both reviewers and Action Editors should be able to do their duties without the need to work over the holidays. We understand that this is still far from ideal for people in the South hemisphere, as this time period coincides with a summer break, but we could not do any better given our time constraints. Unfortunately, we could not afford a longer extension in January given the tight timeline we have for the commitment period. We want to remind the community that ACL 2022 has been decided long in advance to be unusually early this year, in May 22-27, 2022, i.e., just 9.5 months after ACL 2021, which in turn was in August 2021, and thus we had little choice in choosing a more appropriate latest submission deadline for ACL 2022 with ARR than November 15, 2021. October 15, 2021 would have been too early and also too close to ACL 2021, and December 15, 2021 would still have overlapped with the holiday period and, critically, it would have been really too late for the full ACL 2022 reviewing and decision process to take place.

The revised timeline for November is as follows:

● November 20-23, 2021: SACs check AE assignments and notify the ACL 2022 PC chairs about any re-assignments needed;

● November 25, 2021: AE assignments finalized;

● November 27, 2021: AEs notified about the assignments;

● November 27-29, 2021: Desk-rejects and manual checking of the reviewer assignment by the AEs;

● November 29, 2021: Reviewers notified about the assignments;

● November 29 – December 27, 2021: Reviewing;

● December 28-29, 2021: AEs find emergency reviewers and lead the discussion;

● December 28-31, 2021: Emergency reviewing;

● December 28, 2021 – January 7, 2022: AEs prepare meta-reviews;

● January 10, 2022: ARR reviews and meta-reviews for the November 15 submissions sent to the authors;

● January 15, 2022: ACL 2022 commitment deadline.

4. Reviewing guides and tutorials. We wish the reviewers and the action editors a productive reviewing period. Anna Rogers and Isabelle Augenstein prepared a Reviewer Tutorial covering good practices for reviewing. For practical questions regarding how to perform common tasks in the Open Review platform, we collaborated with the ARR team to prepare materials such as a Reviewer’s Guide to Open Review and an Action Editor’s Guide to Open Review.

5. Recognition for reviewers and action editors. In coordination with ARR, we are announcing some initiatives to give recognition to the efforts of ARR reviewers and Action Editors, who have been effectively working primarily on potential ACL 2022 submissions.

● ARR Reviewers and Action Editors recognized as ACL 2022 Reviewers and Area Chairs. All ACL Rolling Review reviewers for papers submitted for the June-November cycles will be officially listed as Reviewers for ACL 2022. Similarly, the Action Editors for the June–November cycles will be officially listed as Area Chairs for ACL 2022.

● ACL 2022 Best Reviewer Awards. The Program Chairs of ACL 2022 jointly with the ACL Rolling Review Editors-in-Chief are delighted to announce the Best Reviewer Award, in recognition for outstanding contributions by reviewers. The award will acknowledge the quality of reviews as judged by Action Editors, fellow reviewers, and authors. The criteria include constructive and thorough reviews submitted on time, engagement in the reviewers’ discussion if initiated by the Action Editor, and availability for emergency reviewing. All reviewers who served in ARR for the June–November cycles will be considered for the Best Reviewer Award for ACL 2022.

● ACL 2022 Best Area Chair award. We further introduce an award for the best Action Editors for ACL 2022, which will be awarded to Action Editors for ARR from the June–November cycle, whom we also recognize as Area Chairs for ACL 2022.

● ARR and OpenReview team members recognized as part of the ACL 2022 team. We also recognize as part of the ACL 2022 team the following people from the ARR and OpenReview teams who have contributed tremendously to ACL 2022:

○ Amanda Stent and Goran Glavas as ARR Editors-in-Chief for ACL 2022;

○ Graham Neubig, Sebastin Santy, and Yoshitomo Matsubara as ARR Tech Team for ACL 2022.

○ Celeste Martinez Gomez, Melisa Bok, and Nadia L'Bahy as OpenReview Tech Team for ACL 2022.

III. ACL 2022 Commitment Process and Timeline.

Finally, remember that ACL 2022 has a two-step submission process, which requires authors to commit their paper to ACL 2022:

1. Timeline. The ARR reviewing for November is ongoing, and the authors will receive their reviews by January 10, 2022, which would be in time for the commitment deadline for ACL 2022, which we have now pushed to January 15, 2022. Here is the ACL 2022 timeline:

● Last deadline for submission to the ACL Rolling Review: November 15, 2021;

● Anonymity period for ACL 2022 starts: December 16, 2021;

● Commitment deadline for ACL 2022: January 15, 2022;

● Notification of acceptance: February 23, 2022;

● Deadline to withdraw papers: February 26, 2022;

● Camera-ready due: March 15, 2022.

2. Commitment form for ACL 2022. The Commitment Form for ACL 2022 will ask the authors to provide a link to their paper in ARR: this should be the latest version of the paper that has reviews and a meta-review. The authors will also need to select a track (including the special Theme track) they are submitting their paper for (this is needed as ACL 2022 has tracks, while ARR does not). Finally, the authors will be allowed to submit optional comments to the ACL 2022 Senior Area Chairs (SACs). Note that these comments will only be visible to the SACs, and they will NOT go to the reviewers or to the Action Editors (responding to reviewers and Action Editors should be handled in a response letter if the authors decided to do a resubmission in ARR, which is a completely different process than committing a paper to ACL 2022). These comments to the SACs are mainly to raise concerns about objective misunderstandings by the reviewers and/or by the Action Editor about the technical aspect of the paper that the authors believe will help the SACs in their decision making process.

3. Multiple Submission Policy. We follow the ARR Multiple Submission Policy. ACL 2022 will not consider any paper that is under review in a journal or another conference at the time of commitment, and committed papers must not be submitted/committed elsewhere during the ACL 2022 commitment period. This policy covers all refereed and archival conferences and workshops (e.g., IJCAI, SIGIR, NAACL, etc.), including all events registered in the ACL Rolling Review: at any given time, a paper could be committed to no more than one ARR event. We have made sure that the notification date for ACL 2022 (February 23, 2022) allows for a subsequent commitment to ACL 2022 workshops (which all have a submission/commitment deadline on February 28, 2022), and also to NAACL 2022 (whose commitment deadline is March 2, 2022). Moreover, we will not consider any paper that overlaps significantly in content or results with papers that will be (or have been) published elsewhere. Authors committing more than one paper to ACL 2022 must ensure that their submissions do not overlap significantly (>25%) with each other in content or results.

4. Anonymity Period for ARR and ACL 2022. Finally, a reminder that if you have received your reviews before December 15, 2021, and are planning to commit to ACL 2022, please remember NOT to post a deanonymized preprint (e.g., in arXiv) on or after December 16, 2021. ACL 2022 has a commitment anonymity period for papers committed to ACL 2022: it starts one month before the commitment deadline (i.e., on December 16, 2021 as the commitment deadline is January 15, 2022), and it ends on the date of notification (February 23, 2022). Note that this ACL 2022 commitment anonymity period is separate from the submission anonymity period for ARR reviewing (yet, both are mentioned in the ARR CFP guidelines): for ARR, the reviewing anonymity period starts one month before submission to ARR and ends either when the authors receive their reviews or when they withdraw their paper (see the ARR CFP). From the ARR CFP page:

“A paper can only be (re)submitted to ARR if no deanonymized preprint has been posted in the month prior to submission. Also, a paper can only be submitted to a venue if no deanonymized preprint has been posted in the month prior to submission. Submissions will be rejected if not properly anonymized.”

Be careful that the anonymity periods for ARR reviewing and for ACL 2022 commitment can overlap (again from the ARR CFP page):

“So, if you submit to the June deadline and get reviews in July, then on July 21st your paper is no longer under review. If you then post a deanonymized preprint on July 22nd, you can’t resubmit to ARR or to a conference in the August round but you can in September.”

From an ACL 2022 perspective, you need to comply (i) with the ARR reviewing anonymity, and (ii) with the ACL 2022 commitment anonymity. Both of these are as per the ARR policy.

The ACL 2022 Program Co-Chairs

Preslav Nakov, Aline Villavicencio, Smaranda Muresan