Local Embeddings and Rerankers Beat Local LLMs for Paid AI Users
For ChatGPT Pro subscribers, running local memory models is more productive than running local LLMs.
A Reddit user who already subscribes to ChatGPT Pro found that running local LLMs like Qwen 3.6 27B or Gemma 4 31B offered diminishing returns for productivity. Since they have near-unlimited access to GPT models via Codex OAuth, the real gap was in embedding and reranker models—services that typically require paid API usage. This realization led them to shift focus from running local LLMs to deploying local embedding and reranker models for a memory system they call GBrain.
The system uses Qwen3 Embedding 4B and Qwen3 Reranker 4B running on llama.cpp, with PostgreSQL and pgvector for vector storage, Ceph for object storage, and GitLab for Markdown-based memories. When the user interacts with Codex or ChatGPT Web, important information is saved as Markdown files via an MCP interface. GBrain then ingests these files, generates embeddings, and extracts concise facts using an LLM. On retrieval requests, it first pulls relevant memories via embeddings, then reranks them to surface only the most pertinent results. This approach avoids forcing the LLM to digest full documents and enables seamless context sharing between different chat clients—making local AI a practical productivity tool rather than just a hobby.
- Uses Qwen3 Embedding 4B and Qwen3 Reranker 4B locally for memory retrieval instead of running a local LLM.
- Stack includes llama.cpp, PostgreSQL with pgvector, Ceph for S3 storage, and GitLab for Markdown file management.
- GBrain indexes memories, extracts facts with an LLM, and combines embeddings with reranking for high-accuracy retrieval.
Why It Matters
Makes local AI a productivity asset by enhancing LLM memory without expensive API calls for embeddings and rerankers.