Co-Scraper achieves 94.78% F1 with AI-powered DOM pruning
Fine-tuned Qwen3-8B model turns messy HTML into reusable scrapers at 90% success.
Co-Scraper, introduced by Shoupeng Wang and colleagues, tackles the challenge of extracting structured data from diverse web pages. The framework operates in two stages: first, a query-aware DOM pruning mechanism intelligently filters out irrelevant parts of the HTML tree, reducing noise and focusing on content relevant to the extraction goal. Second, it uses a fine-tuned Qwen3-8B language model to induce stable, reusable extraction strategies that are compiled into lightweight programmatic wrappers. This approach allows the scraper to generalize across similar pages without manual reconfiguration.
On the widely-used SWDE dataset, Co-Scraper outperforms previous methods, achieving an F1 score of 94.78% and a reuse success rate of 90.39%. The combination of targeted DOM pruning with LLM-based strategy synthesis enables robust handling of HTML hierarchy and variation. For professionals dealing with large-scale web data collection, this means more reliable automation with less custom scripting, reducing both development time and maintenance overhead. The framework is especially valuable for e-commerce, news aggregation, and competitive intelligence where page structures frequently change.
- Uses query-aware DOM pruning to handle long, complex HTML documents efficiently.
- Fine-tunes a Qwen3-8B model to induce stable, reusable extraction strategies.
- Achieves state-of-the-art 94.78% F1 and 90.39% reuse success rate on SWDE benchmark.
Why It Matters
Automates web data extraction with reusable scrapers, saving developers hours of manual parsing and maintenance.