Developer builds real-time local voice chatbot with Qwen3.5-397B
Interruptible, context-aware AI voice chat runs fully on a single 24GB GPU.
A developer has created a fully local voice-to-voice chatbot that approaches real-time performance using a sophisticated stack. The core LLM is Qwen3.5-397B, running Unsloth's UD-Q3_K_XL quantization to fit the 24GB GPU limit while maintaining high quality. Speech recognition uses Whisper-small, and text-to-speech leverages Orpheus Q4_K_XL with a custom SNAC decoder deployed on ONNX for efficient inference. The system achieves near-real-time interaction through SSE streaming, and crucially allows interruption — the chatbot pauses, understands the new input, and continues without losing conversational context from earlier turns.
Memory management is a highlight: VRAM sits at 21.3GB, leaving headroom for compute graphs on a 24GB card. The MoE (Mixture of Experts) layers of Qwen3.5-397B are offloaded to system RAM, occupying about 150GB. The KV cache runs in bf16 (the developer notes Q8 caused instability) at a maximum context of 131,072 tokens — enough for multi-hour conversations. The developer plans to release the code on GitHub soon, making this a compelling blueprint for privacy-preserving, low-latency voice AI assistants that don't require cloud APIs.
- Runs fully local on a single 24GB GPU with 21.3GB VRAM used, leaving headroom for compute workloads.
- Consumes ~150GB system RAM for Qwen3.5-397B MoE experts; uses bf16 KV cache at 131,072 tokens for hours of context.
- Supports real-time SSE streaming and interruptible conversations while preserving prior context.
Why It Matters
Brings high-quality, private, real-time voice AI to consumer hardware, reducing reliance on cloud APIs.