AppEval: New benchmark reveals 68-point gap in LLM mobile app repair

A team of researchers has introduced AppEval, a benchmark and native-toolchain evaluation framework designed to test LLM agents that repair mobile applications across HarmonyOS/ArkTS, iOS/Swift, and Android/Kotlin. Unlike typical repository-level benchmarks that run tests on a build host, AppEval checks whether a repair survives the full mobile build-install-launch-test pipeline. Each task pairs a hidden behavior test with a reference production fix and is only accepted when the same installed app target fails on the defective revision and passes after the fix. Infrastructure issues—such as missing SDKs or offline devices—are explicitly tracked as distinct outcomes, preventing false program failures.

On the audited Android partition, AppEval includes 200 accepted instrumentation tasks from 24 independently buildable repositories. Evaluating five LLM agents, the researchers found Pass@1 scores from 22.00% to 90.50%—a dramatic 68.50-percentage-point spread under the same dynamic oracle. This wide variance shows mobile repair performance depends heavily on the chosen agent and highlights why runtime-aware acceptance is critical for meaningful comparison. The paper notes that quantitative findings are Android-specific; audited iOS and HarmonyOS results are still needed before cross-platform generalization. AppEval's common schema maps repair contracts to each platform's build system, runtime, and test runner, offering a needed standardized yardstick for agentic mobile development tools.