CODEBLOCK trains code LLMs with 1.9% tokens, better pass@1
Selecting structure-complete code blocks, not tokens, boosts efficiency dramatically.
Supervised fine-tuning of code LLMs typically applies uniform cross-entropy loss to all tokens, assuming every token is equally useful. Recent token-level selection methods for natural language challenge this, but transferring them to code can break syntactically and semantically coherent units like function definitions or dependency chains.
To address this, Zhijie Deng and co-authors introduce CodeBlock. It first filters high-quality pairs, then partitions code responses into structure-complete coding items. Utility is estimated via generalized cross-entropy aggregated over core logic tokens, and items are reranked using data-flow reach and bridge signals that capture program dependencies. During training, loss is applied only to selected code items and informative NL tokens. Experiments on six benchmarks show CodeBlock outperforms full-token SFT and other baselines in pass@1 while supervising just 1.9% of response tokens.
- Uses only 1.9% of response tokens for supervision while improving performance.
- Preserves syntactic and semantic structure by operating on complete code blocks, not isolated tokens.
- Outperforms full-token SFT on 6 code generation benchmarks (pass@1 metric).
Why It Matters
Makes code LLM fine-tuning dramatically cheaper and more effective.