The Self-Hosted Secret to Web Access for Local Agents — No Paid APIs Required
Self-hosted metasearch and smart page extraction replace Tavily and Firecrawl for free.
The first piece is SearXNG, a self-hosted metasearch engine run in Docker. The agent queries its JSON endpoint with a simple GET request, returning normalized results (title, url, description). Configuration requires adding JSON to search.formats in settings.yml. Public instances are discouraged; the user controls their own instance to avoid vendor APIs like Tavily or Serper.
For page extraction, Scrapling provides two paths: a fast path using Fetcher.get() with Chrome impersonation (no browser) for normal pages, and a stealth path with a headless browser for blocked or challenge-filled sites. Once HTML is retrieved, Trafilatura converts it to clean Markdown (with links/tables) — far better for LLMs than raw HTML. Additional components handle PDFs via pypdf, detect CAPTCHA/security pages, guard against SSRF, and optionally summarize large pages with a configurable model. The tradeoff: stealth path is slow, and SearXNG quality depends on upstream engines.
- SearXNG provides self-hosted metasearch via a simple JSON GET endpoint, avoiding paid search APIs.
- Scrapling uses a fast HTTP fetcher for normal pages and a stealth headless browser fallback for challenged sites.
- Trafilatura converts HTML to Markdown; PDFs handled by pypdf, with built-in SSRF and challenge detection.
Why It Matters
Enables cost-free, privacy-respecting web access for local AI agents without reliance on proprietary APIs.