[D] Need advice on handling a difficult ACL ARR situation
A researcher's counter-narrative generation paper gets slammed for being both too open and not open enough.
An AI researcher working on counter-narrative generation—a technique to combat online hate speech—has sparked a viral discussion after detailing a frustrating peer-review experience with the ACL's ARR (ACL Rolling Review) system. The researcher first submitted their paper in October 2023, taking responsible AI steps like open-sourcing code and masking data. Reviewers provided constructive ethics feedback, but one criticized open-sourcing as having a "negative impact," and another argued the topic itself was unsuitable for the ACL conference, despite cited precedent.
For a January 2024 resubmission, the team made major revisions: reframing the paper, strengthening the ethics section, obtaining IRB approval, and adding human evaluation. The new reviews, however, presented fresh frustrations. One reviewer appeared to critique points from the older version rather than the updated manuscript and suggested a "hidden agenda" behind the research. Another reviewer incorrectly stated the code was not open source and argued that using five human evaluators was insufficient, a point the author contests by noting many highly-cited works in the field use three to five evaluators. The post has ignited a broader conversation about review quality, consistency, and the challenges of publishing responsible AI research, especially in sensitive areas like content moderation.
- Researcher's counter-narrative AI paper was criticized for both open-sourcing code (deemed a 'negative impact') and later for not being open-source.
- Major revisions including IRB approval and human evaluation were met with reviews critiquing old versions and alleging a 'hidden agenda'.
- The case highlights systemic peer-review inconsistencies in top AI venues like ACL ARR, particularly for ethics-heavy research.
Why It Matters
This case exposes critical flaws in AI peer review, potentially stifling important research on combating online harm and setting a worrying precedent for authors.