Developer's JSON repair library fixes LLM output failures across 288 model calls
From markdown fences to truncated objects, here's how local models break JSON.
A developer running 288 structured output calls across models like Llama 3, Mistral, Command R, DeepSeek, and Qwen on OpenRouter catalogued failure modes for JSON generation. The most common breakages: markdown fences wrapping the JSON, trailing commas (from JavaScript training data), Python True/False/None instead of JSON true/false/null, truncated objects from token limits, unescaped quotes inside strings, // or # comments, and literal '...' placeholders. Failure patterns were nearly identical across both open and closed-source models; only frequency varied.
In response, they built outputguard, a Python library that validates against JSON Schema and runs 15 repair strategies in a specific order (encoding before structure, re-parsing between fixes to avoid undoing previous work). It also handles YAML, TOML, and Python literals—formats models output when no JSON mode is available. The library is MIT licensed, has zero LLM provider dependencies, and can be installed via pip. The developer noted that "just use JSON mode" advice doesn't cover locally-run models, and grammar-based generation has speed/compatibility tradeoffs.
- 288 calls across Llama 3, Mistral, Command R, DeepSeek, Qwen showed nearly identical JSON failure modes across models
- Top breakages: markdown fences, trailing commas, Python literals (True/False/None), truncation, unescaped quotes, comments, and ellipsis
- outputguard library runs 15 ordered repair strategies, supports YAML/TOML/Python literals, MIT licensed, pip installable
Why It Matters
Reliable JSON output from local LLMs is critical for automation; this library fills a gap where JSON mode or grammar constraints aren't available.