v0.22.1
Batch sampling across sequences boosts local AI model efficiency...
Ollama, the popular open-source tool for running large language models locally, has released v0.22.1 as a pre-release. This update introduces several technical improvements aimed at enhancing performance and compatibility. A standout feature is batch sampling across multiple sequences, contributed by jessegross in PR #15736, which allows the MLX runner to process multiple inference sequences simultaneously, potentially speeding up tasks like chat completions and batch generation.
Other notable changes include a fix for multi-regex BPE offset handling in the tokenizer by dhiltgen, addressing potential tokenization errors. The update also adds support for importing NVIDIA TensorRT Model Optimizer models, expanding hardware compatibility for users with NVIDIA GPUs. A critical bug fix resolves an issue where the desktop app startup would kill active Ollama launch sessions, improving user experience. New models were added by dhiltgen in PR #15861. This release builds on v0.21.3 and includes contributions from dhiltgen, jessegross, and hoyyeva.
- Batch sampling across multiple sequences improves inference speed in MLX runner
- Added support for NVIDIA TensorRT Model Optimizer imports for better GPU compatibility
- Fixed desktop app startup bug that killed active Ollama sessions
Why It Matters
Faster local AI inference and broader GPU support make running models more efficient and accessible.