V2N: First visual piano transcription system predicts velocity and offsets
KAIST and Georgia Tech's V2N nails piano notes from video, even with sustain pedal
Deep Dive
A new paper introduces V2N (Video to Notes), described as the first complete visual piano transcription system. It uses a shared temporal backbone with task-specific heads for onset, offset, key hold, and velocity, trained with per-frame supervision. According to the article, multi-task supervision enables offset and velocity prediction while also improving onset accuracy. V2N achieves state-of-the-art results on the PianoVAM and R3 benchmarks.
Key Points
- V2N is the first VPT system to predict note velocity and physical key-release offsets, not just onsets
- Multi-task learning with per-frame supervision improved onset accuracy while enabling offset and velocity prediction
- Achieved state-of-the-art results on both PianoVAM and R3 benchmarks; accepted to ISMIR 2026
Why It Matters
Video-based piano transcription with velocity could transform music education and automatic score generation without specialized sensors.