Qwen3 gets a convolution boost with <0.01% added parameters
A tiny depthwise convolution layer boosts LLM accuracy by encoding locality
A team of researchers (Yuchuan Tian, Yingte Shu, et al.) shows that adding a tiny depthwise convolution layer to existing Transformer-based LLMs can improve accuracy without meaningful parameter overhead. Testing on Qwen3 models, they systematically ablated 17 possible insertion points and found the best results when convolution is applied to the projected queries, keys, and values right before the attention mechanism. The optimal configuration is a residual depthwise convolution with kernel size 3 and no additional normalization or activation.
Across multiple Qwen3 model sizes and pre-training data budgets, the design improves average accuracy on seven standard benchmarks (including commonsense reasoning and language understanding). The parameter increase is negligible—less than 0.01%. A representation-level case study suggests the convolution makes repeated token IDs more sensitive to their immediate context, helping the model handle local dependencies that pure self-attention might miss. This lightweight architectural tweak could be adopted broadly in Transformer-based LLMs without requiring retraining from scratch.
- Convolution is inserted into the Qwen3 Transformer block at the queries, keys, and values before attention, not after.
- The depthwise convolution uses kernel size 3, is residual, and adds no extra normalization or activation layers.
- Accuracy improves on 7 benchmarks (e.g., commonsense reasoning) with less than 0.01% parameter increase.
Why It Matters
Lightweight conv layers could enhance every Transformer-based LLM without significant compute cost.