Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents

A research team including Kaifeng Wu, Junyan Wu, Qiang Liu, Jiarui Zhang, and Wen Xu has published a breakthrough paper titled 'Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents' on arXiv. The work addresses critical limitations in long-document topic segmentation, where traditional discriminative models struggle with document-level semantics and generative LLMs face prohibitive computational costs. Their novel approach combines a Qwen3-0.6B backbone with specialized architectural innovations to create a system that efficiently identifies paragraph boundaries in documents far exceeding typical context windows.

The technical framework introduces a cross-window context fusion layer and a boundary classification head, deployed with an overlapping sliding-window strategy that enables processing of up to 13k tokens in a single pass. For downstream efficiency, the team also developed a vector fusion method with scalar correction to compress ultra-long segment representations without semantic loss. On the WIKI-727K benchmark, their model outperformed three generative baselines (based on Qwen2-0.5B from Jina) in macro-averaged F1 score while achieving a staggering two orders of magnitude (100x) faster inference. This dramatic performance leap substantially improves the practicality and scalability of processing legal documents, research papers, and other lengthy texts for retrieval-augmented generation (RAG) systems and document understanding pipelines.