New benchmark LibEvoBench reveals LLMs struggle with evolving APIs
Even state-of-the-art code models fail at version-aware API calls in Python.
Deep Dive
Researchers introduce LibEvoBench, a multi-task benchmark testing LLMs on multiple versions of Python APIs. Their new Software Evolution Understanding Score (SEUS) reveals models are version-oblivious: performance degrades on evolving APIs, while stable APIs show no change. Providing the target version doesn't help, but relevant documentation significantly boosts accuracy. This highlights a systematic limitation in current code generation training paradigms.
Key Points
- LibEvoBench tests LLMs across multiple versions of Python libraries like NumPy and pandas, detecting version-specific API errors.
- The SEUS metric quantifies model consistency when dealing with evolving APIs—current models score poorly.
- Providing the target library version doesn't help; only adding relevant documentation improves accuracy significantly.
Why It Matters
LLMs must master temporal knowledge to avoid critical API bugs in real-world codebases.