llama.cpp b10003 revamps tokenize tool with unified CLI and HF support

llama.cpp b10003, released July 14 by ggml-org, focuses on standardizing the tokenize command-line tool. The commit by Adrien Gallouët (HuggingFace) migrates tokenize from hand-rolled argv parsing to common_params_parse, the shared argument infrastructure used by other llama.cpp tools. This brings automatic Windows UTF-8 handling, consistent file reading, and unified error logging via LOG_ERR instead of fprintf. The update exposes model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to tokenize, allowing users to download and tokenize models directly from HuggingFace. Additionally, new tokenize-specific options include --ids (output token IDs), --stdin (read from standard input), --no-bos (omit beginning-of-sequence token), --no-parse-special (disable special token parsing), and --show-count (display token count). The --parse-special flag defaults to true to maintain backward compatibility.

For developers building local AI pipelines, this release reduces boilerplate by aligning tokenize with the rest of llama.cpp's CLI. The ability to fetch models via HuggingFace identifiers simplifies scripting, while stdin support enables piping text from other tools. The shared argument system also ensures fewer edge cases across platforms, especially on Windows. Although a minor version bump, b10003 improves the developer experience for the 120K-star open-source project that powers countless on-device LLM deployments. Build artifacts are available for macOS (Apple Silicon, Intel), Linux (x64, arm64, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, OpenCL), Android (arm64), and iOS.