New Haystack Update Lets AI Work Longer Without Forgetting or Overspending
Long AI tasks get cheaper and stop forgetting — here's why your AI bill stays predictable.
Haystack is free, open-source software (anyone can download and use it) that companies use to build AI assistants — especially ones that look things up in your documents, a technique called RAG, or "letting AI look things up." On September 24, the team shipped version 3.2.0. It's a tool for developers, not a product you buy, but the features inside will shape the AI helpers you use at work.
The first change tackles a well-known weakness. AI models can only hold so much text in mind at once — that limit is called the context window, think of it like a desk that only fits so many papers. On long tasks, assistants forget earlier instructions or crash into that limit. The new SummarizationCompactor uses a small, inexpensive AI model to summarize old conversation turns, keeping a short record of earlier goals and decisions. The newest steps are never touched, so the assistant stays sharp on what it's doing right now.
The second change is about money. Tokens are the word-chunks AI companies charge by, like minutes on a phone plan. The new TokenBudgetHook lets a developer set a hard ceiling, say 100,000 tokens. When the AI hits it, the assistant stops before its next paid step, explains why it stopped, and hands back whatever work it finished. No surprise invoices, no half-finished mess.
There's also a convenience upgrade: connecting the pieces of an AI system now takes fewer lines of code, and developers can chain those steps together. The release does include breaking changes, meaning some older setups will need small fixes — the kind of plumbing work users never see. The bigger picture: AI assistants are getting longer memories and spending limits, which is what it takes to trust them with real, everyday jobs.
- A new memory feature lets AI assistants summarize old conversations instead of forgetting them, using a small cheap model to do the summarizing.
- A spending cap of, for example, 100,000 tokens (the word-chunks AI bills by) stops a runaway assistant before it costs you money.
- Developers can now connect an AI system's parts with fewer lines of code, making these tools faster to build — and cheaper to run.
Why It Matters
Longer AI jobs become affordable and predictable — fewer surprise bills and assistants that don't forget your earlier instructions.