JupOtter beats LLMs at cell-level bug detection in Jupyter notebooks
New system outperforms static analyzers and GPT-4 on 2 of 3 datasets.
Jupyter Notebooks have become ubiquitous in data science and scientific computing, but their interactive, cell-based nature often leads to hard-to-find bugs. As notebooks grow from prototypes into complex programs, buggy code on platforms like GitHub has surged. To tackle this, Lukas Ottenhof and Thibaud Lutellier introduce JupOtter, a bug detection system purpose-built for the notebook format.
JupOtter makes three key contributions: a novel tokenization strategy that respects cell boundaries (unlike standard line-based tokenizers), a cell-level bug prediction technique that pinpoints which cell contains the error, and OtterDataset — a new labeled corpus of over 21,000 notebooks annotated for fine-grained cell-level bug detection. The system achieves higher F1 scores than both traditional static analyzers and state-of-the-art large language models (including GPT-4) on two out of three evaluation datasets. The paper has been accepted at the 42nd International Conference on Software Maintenance and Evolution (ICSME 2026).
- Three contributions: cell-aware tokenization, cell-level bug prediction, and OtterDataset (21,000+ annotated notebooks).
- Outperforms static analyzers and large language models (e.g., GPT-4) on 2 of 3 evaluation datasets.
- Accepted at ICSME 2026 (Research Papers Track) — a top venue for software maintenance and evolution.
Why It Matters
Cell-level debugging could save data scientists hours by pinpointing errors in complex notebooks, improving reproducibility and code quality.