New study reveals LLM function-calling gains are mostly format, not skill

A new paper from Wanyi Chen and colleagues at arXiv introduces a systematic method for attributing gains in structured-output benchmarks to either interface alignment (format compliance) or procedural transfer (actual skill). They propose a four-layer protocol: canonicalized rescoring, format-only controls, repaired/balanced induction, and portability checks. When applied to the Berkeley Function Calling Leaderboard (BFCL), the results are striking: format-only prompts (no skill injection) match or even exceed full skill prompts in multiple key BFCL cells. Additionally, after repairing and balancing induction examples, the largest sub-frontier gains vanish entirely. On API-Bank, target-native skill prompts are matched within 0.5 percentage points by length-matched generic procedural prompts, confirming that the performance boost comes from interface alignment rather than genuine procedural transfer.

These findings have significant implications for how the AI community interprets structured-output benchmarks. The paper argues that current evaluations conflate format compliance with transferable skill, leading to inflated claims about model capabilities. The authors release BFCL-CANONICAL, a canonicalized version of the dataset, and recommend that future work include format-only baselines and balanced induction to properly attribute gains. For practitioners, this means that many so-called improvements in function calling may simply reflect better prompt formatting rather than deeper reasoning or procedural knowledge. The work underscores the importance of careful experimental design when evaluating LLM skills, especially in production systems where format compliance is valuable but distinct from procedural competence.