Study finds LLMs overconfident in insecure code generation
GPT-4o-mini, Gemini, and Qwen3 often think their vulnerable code is safe.
Deep Dive
Researchers evaluated GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next on security calibration. They found models are overconfident, often assigning high confidence to vulnerable code. Functional calibration is worse than security calibration. Attempts to repair code based on calibration had limited success, and architectural gating deteriorated in realistic repository-level settings.
Key Points
- GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next show systematic overconfidence in insecure code.
- Functional calibration is consistently worse than security calibration across all models.
- Calibration-guided repair only marginally improved security while often breaking functionality; architectural gating failed in repository-level scenarios.
Why It Matters
LLMs used for coding may appear confident in vulnerable code, risking security breaches in production.