ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

A team of researchers from Emory University and Georgia Tech has introduced ReCUBE, a novel benchmark designed to isolate and measure how effectively Large Language Models (LLMs) utilize repository-level context during code generation. Unlike existing benchmarks that test broad coding capabilities, ReCUBE specifically challenges models to reconstruct a masked file within a real-world software repository using only the remaining source files, dependencies, and documentation as context. It evaluates the reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting actual software usage patterns.

Experiments across eight leading models, including GPT-5 and Claude 3.5, revealed a significant performance gap. In a full-context setting, GPT-5 achieved a mere 37.57% strict pass rate, demonstrating that understanding and leveraging the interconnected nature of a full codebase remains a major challenge for even the most advanced AI. To address this, the researchers also developed the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide AI agents toward the most relevant files during repository exploration.

Agents augmented with the CCE toolkit consistently outperformed all baseline methods across all evaluated models, achieving improvements of up to 7.56% in strict pass rate. This finding highlights a promising direction for enhancing AI-powered coding assistants by giving them better navigational tools within complex codebases. The team has released the ReCUBE benchmark, code, and evaluation framework as open-source resources for the broader NLP and software engineering research communities.