Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents

A research team has introduced RUSTFORGER, a novel LLM-based agent designed to automate repository-level issue resolution for the Rust programming language. The work addresses Rust's steep learning curve and the absence of large-scale benchmarks by creating Rust-SWE-bench, comprising 500 real-world software engineering tasks from 34 popular Rust repositories. Their comprehensive study of four representative agents and four state-of-the-art LLMs revealed that while ReAct-style agents could resolve up to 21.2% of issues, they struggled with repository-wide code comprehension and Rust's strict type/trait semantics. The research also identified issue reproduction as a critical bottleneck for task resolution.

The RUSTFORGER agent overcomes these limitations by integrating an automated test environment setup with a Rust metaprogramming-driven dynamic tracing strategy, enabling reliable issue reproduction and dynamic analysis. When evaluated with Claude-Sonnet-3.7, RUSTFORGER achieved a 28.6% task resolution rate on Rust-SWE-bench, representing a 34.9% improvement over the strongest baseline. Most notably, the agent uniquely solved 46 tasks that no other agent could handle across all tested advanced LLMs. This research, accepted to ICSE 2026, establishes a foundational understanding of LLM capabilities in the Rust ecosystem and provides a significant step toward making Rust more accessible through automated issue resolution.