New 9D metric reveals LLM decompilation flaws in IoT vulnerability scans

As IoT devices increasingly handle sensitive data like audio and authentication tokens, firmware vulnerability detection is critical. Decompilation—turning binary firmware back into readable code—has become a key technique, especially with LLMs that produce high-quality, recompilable output. However, LLMs rely on probabilistic token prediction, prioritizing syntactic correctness over semantic equivalence. This means decompiled code can look valid but internally differ from the original binary, silently dropping critical details like error-handling flows and boundary checks that are exactly where vulnerabilities hide.

To address this, Minami Yoda and colleagues at the University of Electro-Communications developed a nine-dimensional quality evaluation metric organized into three categories: structural, behavioral, and semantic similarity. They tested it on 318 programs from OpenWrt, the open-source router platform behind many commercial devices, generating 19,625 decompilation results using five methods: one rule-based decompiler and four LLM-based approaches. Their statistical analysis revealed that the recompilation-success group scored significantly higher overall (Cohen's d=0.92), with behavioral similarity showing the strongest effect (d=0.96) and structural similarity at d=0.69. These metrics prove reliable predictors of decompilation quality, offering a generalizable framework for evaluating black-box generative models beyond IoT security.