b8133
The latest commit eliminates storage of output IDs and logits, reducing memory overhead for recurrent models.
The llama.cpp project, maintained by ggml-org, has pushed a pivotal update with commit b8133. This technical release fundamentally changes how the inference engine handles session state persistence. The core modification removes the storage of output IDs, logits, and embeddings within the `llama_context` state that gets saved to session files. Previously, this data was serialized, leading to larger file sizes and unnecessary memory allocation for data that can be recalculated.
The change necessitates updates to the session handling logic. The `completion` tool now implements a 'replay' mechanism: after loading a session state, it reprocesses the last token to regenerate the logits needed for the sampling step to continue text generation. This is a more efficient on-demand computation versus storing the results. Furthermore, the `save-load-state` example was updated to use the new `llama_state_load_file` function and a related fix sets `n_seq_max = 2` for a specific test context (`ctx3`), resolving a crash that occurred with recurrent/hybrid models when attempting to use a second sequence with a parallel value of 1.
This update reflects ongoing optimization for production deployment scenarios. By streamlining the session state, llama.cpp reduces its memory footprint and I/O overhead for saving and loading long-running conversations or document processing jobs. It particularly benefits the operation of stateful, recurrent models (like Mamba or Griffin architectures) which are designed for efficient long-context processing. The commit is part of a broader effort to enhance the library's robustness and performance across its wide range of supported platforms, including Apple Silicon, CUDA, Vulkan, and ROCm.
- Commit b8133 removes serialization of output IDs/logits/embeddings from session state, reducing file size and memory use.
- Introduces a token replay mechanism in the completion tool to regenerate logits on-demand after loading a session.
- Includes a fix for recurrent/hybrid models by setting n_seq_max=2 in an example, preventing a sequence allocation error.
Why It Matters
Enables more efficient long-running AI sessions with lower memory overhead, crucial for deploying models on edge devices and in cost-sensitive environments.