New benchmark tests LLMs on Java code adaptation without explicit instructions

A new study from researchers at (likely) the University of Saskatchewan and other institutions introduces a rigorous evaluation framework for large language models on Java code snippet adaptation. The work, accepted at the 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026), targets a common developer practice: copying code fragments and adapting them to new contexts. Unlike existing benchmarks that rely on explicit step-by-step instructions or narrow change types (e.g., variable wiring), this framework uses a mutation-injection approach to systematically introduce controlled adaptations across a taxonomy derived from empirical developer behaviors.

The authors construct a dataset of Java code fragments from open-source repositories with strong test coverage. They apply adaptation operators (mutations) at the fragment level, ensuring precise knowledge of required changes. LLMs are then evaluated on instruction-free adaptation—meaning models receive no edit guidance, only the target context. The study measures performance across three dimensions: hardest adaptation types (RQ1), scaling with complexity (RQ2), and required surrounding context (RQ3). Correctness is assessed primarily via test-suite re-insertion, supplemented by manual mutation inspection. This controlled setup allows the field to move beyond function-level granularity and narrow edits, providing a more realistic benchmark for code reuse with LLMs.