New Rust proxy Turnstile captures token IDs for better agent RL training
Turnstile records exact token histories to fix retokenization drift in RL training.
Reinforcement learning (RL) for agentic tasks—like coding, web navigation, or research workflows—requires careful bookkeeping. During a rollout, the model's outputs (token IDs) must be recorded exactly as generated. However, agent harnesses often lose this fidelity by rerunning tokenizers on transcripts, causing retokenization drift or chat template drift. The resulting slight mismatches between the model's actual history and the training data degrade the RL signal, sometimes invisibly.
Turnstile solves this by acting as a lightweight Rust proxy between the harness and model backend. It captures the exact token IDs at generation time and exports a generic, framework-neutral rollout trajectory that feeds directly into any RL training pipeline. Open-sourced today, Turnstile has been validated on real training runs: a text-only coding agent and a multimodal computer-use agent both showed steady improvement using Turnstile's data. This small proxy eliminates a subtle but critical data integrity issue for RL-based agent improvement.
- Turnstile is a Rust proxy that captures token IDs at generation time to avoid retokenization drift during RL training.
- It is open source and framework-neutral, exporting trajectories that plug into any RL training stack.
- In validation runs, a text-only coding agent and a multimodal computer-use agent improved steadily using Turnstile's token-level data.
Why It Matters
Accurate token-level history is crucial for effective RL on agent tasks; Turnstile solves a subtle but critical data mismatch problem.