Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR

A team of researchers has introduced Speaker-Reasoner, a new Speech Large Language Model (LLM) designed to solve the complex puzzle of multi-speaker conversation transcription. Unlike standard models that perform a single pass, Speaker-Reasoner operates like an AI agent, iteratively reasoning over the audio. It first analyzes the global structure, then autonomously predicts temporal boundaries for speech segments, and finally performs fine-grained analysis on each segment. This multi-turn process allows it to jointly model and output speaker identity, gender, precise timestamps, and the transcribed text all at once. A key innovation is a speaker-aware cache mechanism that enables the model to process audio sequences longer than its original training context window, a common limitation for LLMs.

The model was trained using a three-stage progressive strategy and evaluated on challenging real-world meeting datasets, AliMeeting and AISHELL-4. The results show consistent improvements over existing strong baselines, with notable gains in handling the most difficult aspects of conversation analysis: overlapping speech (crosstalk) and rapid, complex turn-taking with backchannels (like "uh-huh"). This represents a significant shift from treating speech recognition as a simple transcription task to framing it as a structured reasoning problem, where the model must actively disentangle and attribute the conversational flow. The work, detailed in a paper on arXiv, points toward more intelligent, context-aware AI systems for meeting analysis, customer service calls, and media production.