HASTE: Hierarchical skill accumulation boosts ML agent medal rate to 77.3%

In a paper accepted at ICML 2026 Workshop on Deep Learning for Code (DL4C), researchers Yongbin Kim, Yashar Talebirad, and Osmar R. Zaiane from the University of Alberta propose HASTE (Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering). The system addresses a fundamental inefficiency in ML engineering agents: they waste compute rediscovering known techniques because every competition starts from scratch. HASTE organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction.

Key experimental results demonstrate the power of hierarchical skill management. Holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5% — the same as loading no skills — and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition. In warm-start runs where the system reloads skills from earlier competitions using only global and domain-level skills, refinement iterations drop by 52%, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.