Persistent Memory Boosts Autonomous Data Science Agents with Small Language Models
A new CBR-augmented agent uses Gemma 4 locally, achieving higher accuracy with lower variance
Most autonomous data science agents today depend on massive frontier cloud models and have no persistent memory across sessions. In a new paper, Felix Stocker tackles both gaps by augmenting Microsoft’s R&D-Agent framework with a Case-Based Reasoning (CBR) layer that uses a locally deployable small language model — Google’s Gemma 4 31B Dense. The CBR layer overrides three phases of the R&D loop via a single environment variable toggle, storing structured case records that include executable code snapshots and quality metadata. A five-gate quality filter and heuristic reuse-detection mechanism combine embedding similarity, code-fingerprint overlap, and injection provenance to assess knowledge transfer.
Evaluated on two Kaggle competitions (NOMAD 2018 and Spaceship Titanic) across four seeds and eight improvement loops, the CBR-augmented agent achieved directionally higher accuracy on Spaceship Titanic (0.8147 vs. 0.8098, Cohen’s d = -1.41) with substantially lower variance. Heuristic reuse detection across 108 retrieval events revealed high semantic relevance (mean embedding similarity 0.882) but variable structural proximity (mean code-fingerprint similarity 0.305), confirming the agent benefits from conceptual guidance rather than verbatim code copying. This work marks the first published end-to-end evaluation of Gemma 4 as an autonomous data science agent backbone, proving that small, locally deployable models can compete with cloud giants — especially when paired with persistent, quality-controlled memory.
- CBR-augmented R&D-Agent uses Gemma 4 31B Dense, a locally deployable small language model, as its agent backbone
- On Kaggle's Spaceship Titanic, achieved 0.8147 accuracy vs 0.8098 baseline, with lower variance and a Cohen's d of -1.41
- Heuristic reuse detection over 108 events showed 0.882 mean embedding similarity but only 0.305 code-fingerprint overlap
Why It Matters
Enables autonomous data science agents to run locally with persistent memory, reducing cloud dependency and costs.