FirstPass AI predicts peer review outcomes with 80.5% accuracy

The paper tackles a critical gap in AI peer review: existing systems train only on CS/ML venues, ignore iterative reviewer-author dialogue, and optimize for stylistic mimicry rather than real editorial judgment. FirstPass addresses all three. The team curated 3,668 complete, multi-round peer-review dialogues from Nature Communications across biology, chemistry, neuroscience, physics, and earth science, using mandatory transparent peer review (since Nov 2022) and 100% content integrity audits. They fine-tuned Qwen2.5-7B-Instruct via LoRA on three tasks: review generation, reviewer updating, and revision-cycle prediction.

The key technical insight is that response-only loss masking is a prerequisite for meaningful learning—without it, accuracy drops to 62.0% (below majority baseline). With it, FirstPass achieves 80.5% accuracy and F1-macro 78.2% on predicting standard vs. extended revision cycles, outperforming Gemini-3.1-flash-lite-preview zero-shot by 10.4 percentage points (McNemar p<0.001). For generation, it produces 1,187-word reviews, much closer to human reference length (2,155 words) than baselines, with ROUGE-L 0.154. Deployed in a pre-submission loop, FirstPass acts as an anticipatory scientific co-author, giving researchers the judgment of a trusted colleague across five disciplines.