CRAFT slashes token costs 9x for enterprise coding agents
New post-training method boosts accuracy while cutting schema injection overhead
Enterprise coding agents need deep schema knowledge to translate natural-language requests into executable code over proprietary APIs. Current deployments inject exhaustive schema and tool documentation into every prompt — a costly approach that complicates schema evolution and undermines reliability in multi-turn analysis. CRAFT tackles this by acquiring stable schema knowledge and tool-use behavior through post-training, removing the need for prompt-time schema stuffing.
CRAFT operates in two stages. First, schema-stripped PLAN supervised fine-tuning teaches domain-structured plans and executable behaviors from validated trajectories. Second, execution-shaped reinforcement learning aligns tool selection, code quality, and recovery from failures. Training data is curated via a Tri-Gate filter combining execution validation, data-integrity checks, and LLM-judge reasoning audit. Evaluated in advertising analytics (campaign performance, metric drill-downs, entity-level analysis), CRAFT achieves a +9.6pp composite Agent Score improvement, +4.1pp consistency, and +4.2pp multi-turn coherence versus schema-stuffed baselines, while slashing input tokens by 9x and schema-discovery loops by 5x. The paper also discusses deployment tradeoffs and reward-shaping limitations.
- Two-stage post-training (PLAN SFT + execution-shaped RL) eliminates prompt-time schema injection
- Reduces input-token burden by ~9x and schema-discovery loops by up to 5x vs. schema-stuffed baselines
- Achieves +9.6pp Agent Score, +4.1pp consistency, +4.2pp multi-turn coherence in advertising analytics
Why It Matters
CRAFT dramatically cuts inference costs and complexity for production AI agents while boosting reliability and consistency.