CATGen: Context-aware LLM workflow boosts unit test reliability 40%
Real-world projects see 90% fewer compilation failures with deterministic test skeletons...
Deep Dive
Researchers designed CATGen, a context-aware workflow for LLM-based unit test generation. Instead of relying on LLMs to infer project context, CATGen uses structured context retrieval, deterministic test skeleton construction, and lightweight static analysis. In evaluations on proprietary industrial projects and Defects4J, CATGen substantially improves compilation success and structural coverage while significantly reducing generation time and token consumption compared to existing LLM-based approaches.
Key Points
- CATGen reduces LLM unit test compilation failures from ~50% to >90% on real industrial codebases.
- Uses deterministic test skeletons and static analysis instead of iterative LLM repair, cutting token usage by 40-50%.
- Achieves 15-25% higher structural coverage on Defects4J benchmark compared to standard LLM-based generation.
Why It Matters
Proves that systematic engineering support, not just better prompts, is key to making LLM testing practical in enterprise.