Jet-Long boosts LLM context to 128K with dynamic RoPE scaling
Zero-shot context extension up to 1.39x faster on H100 with <4% overhead
Jet-Long addresses the fundamental trade-off in zero-shot context extension: a single fixed rescaling factor either sacrifices short-context fidelity or breaks at long inputs. The method uses two parallel RoPE windows — one local and faithful to pretraining, one long-range with a dynamically adjusted rescaling factor based on current sequence length. An inclusion-exclusion attention merge and on-the-fly RoPE correction make this bifocal construction nearly free at inference, with long-context prefill reaching up to 1.39× FA2 throughput on H100 and single-batch generation incurring ≤4% overhead at any length.
Evaluated on Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads the RULER benchmark by +4.79/+2.18/+2.03 pp over the strongest baseline, achieves best overall accuracy on HELMET-RAG (the most efficient predictor of downstream long-context performance), and attains the lowest PG-19 perplexity. It also generalizes to hybrid attention architectures such as Jet-Nemotron without retraining and remains hyperparameter-resilient, making it a practical drop-in for production deployments handling retrieval-augmented generation, repository-level coding, and agentic workflows.
- Uses dual RoPE windows: one local-faithful, one long-range with dynamic rescaling based on sequence length
- Achieves up to 1.39× FA2 throughput on H100 with single-batch generation overhead ≤4% at any length
- Outperforms baselines on RULER by up to +4.79pp and achieves best HELMET-RAG accuracy on Qwen3 models up to 128K
Why It Matters
Enables zero-shot long-context deployment for open-weight LLMs without retraining, boosting performance in RAG, coding, and agentic workflows.