Closed models like Claude may not beat open models fairly — hidden tricks skew benchmarks

A recent Reddit analysis challenges the assumption that closed-source AI models like Anthropic’s Claude consistently outperform open alternatives such as GLM-5.2. The author points out that benchmarks typically compare the final output of a commercial product (e.g., Claude) against raw model inference from open models. However, closed providers can layer on undisclosed enhancements: RAG for real-time knowledge injection, prompt preprocessing, context-dependent system prompts, hidden internal tool calls, and even a “clown-car MoE” approach that routes queries to specialized expert models. These tricks can dramatically improve perceived performance without reflecting the base model’s intrinsic reasoning ability.

If true, this undermines the narrative that closed models have superior architectures or training pipelines. The author suggests that Anthropic, for example, could be serving a composite system under the “Claude” brand while open models stand alone. This makes benchmark comparisons effectively “apples to oranges.” The implication is that open models may be much closer to closed ones than commonly believed—and that the real advantage lies in proprietary orchestration layers, not core intelligence. For the AI community, this emphasizes the need for transparent, standardized evaluation that accounts for all backend components.