Pyannote 3.1 diarization gets 12x speedup on laptops with new clustering fix

Speaker diarization — identifying who spoke when — is critical for meeting transcription and voice agents, but running it on-device has been too expensive. In a new arXiv paper, Fumiaki Yamaguchi explores how far a Pyannote 3.1-based pipeline can be pushed on consumer GPUs (RTX 5070 Ti) and Apple Silicon (M4). The naive approach — coarser segmentation stride and per-chunk embedding — delivers multi-fold speedups and holds error rates steady on clean datasets like AMI. But on wild data (VoxConverse), accuracy collapses: diarization error rate (DER) jumps from 0.075 to 0.113.

The root cause is a fixed minimum cluster size that fails when fewer embeddings per speaker are generated. Yamaguchi's solution is elegantly simple: replace the fixed size with a relative minimum cluster size (mcs = round(0.01 * n)), where n is the number of embeddings. This single parameter recovers VoxConverse DER to 0.079 — about 89% of the lost accuracy — without harming AMI performance. The accelerated pipeline achieves up to 12.2x speedup on AMI using Apple's Metal Performance Shaders, making real-time on-device diarization practical for the first time.