CHaystack Benchmark Reveals Qwen's 5x Lead in Chinese Document Retrieval

Researchers led by Hanxi Li have released CHaystack, a new benchmark for Chinese document retrieval and visual question answering (DocumentVQA). Unlike existing benchmarks focused on English or single-document tasks, CHaystack covers four diverse document categories: academic papers, advertisements, web pages, and real-world photographed documents. This design enables comprehensive evaluation of large-scale retrieval-and-generation pipelines. Alongside the benchmark, the team introduces CDocRAG, a Chinese DocumentVQA system that uses a vision-language model (VLM) based relevance filter to verify retrieved document images before answer generation, aiming to reduce noise in retrieval results.

The evaluation of representative open-source embedding and generation models on CHaystack reveals stark performance differences. Qwen-family models, particularly Qwen3-VL, dominate text-rich categories such as web pages and papers, achieving 71.91 Recall@1. In contrast, the best non-Qwen model (likely from other families) reaches only 14.40 Recall@1—a nearly five-fold gap. Other models perform competitively only on visually rich categories like advertisements but degrade sharply on text-dense documents. These results indicate that the primary bottleneck in Chinese large-scale DocumentVQA is textual encoding, not visual understanding. The findings underscore substantial room for improvement in Chinese document retrieval systems and highlight the need for better multilingual text encoders.