OpenAI API outputs for code benchmarks: fine-tuning vs. evaluation debate

A developer on Reddit is navigating the gray area of OpenAI’s API terms of service while building a domain-specific code generation benchmark. They propose creating programming tasks, reference solutions, and verification tests for a specialised Python library (used in a technical/scientific field) using OpenAI API outputs. After human review and validation, they want to use this silver dataset either to fine-tune an open-source code model (Scenario 1) or purely as an evaluation benchmark (Scenario 2). The core question: does using API outputs to train another model violate OpenAI’s policy, even if the scope is narrow and the output is heavily curated?

OpenAI’s terms generally prohibit using API outputs to develop models that compete with OpenAI, but the definition of “compete” is ambiguous. In Scenario 1, the fine-tuned open-source model would likely be a direct competitor in code generation, potentially violating the usage policy even with human filtering. Scenario 2 (benchmark only) is safer because no model training occurs, but OpenAI may still restrict publishing datasets derived from its API if they resemble evaluation benchmarks for competing models. The post underscores the need for explicit legal guidance from OpenAI or counsel before proceeding, as the boundary between permissible AI-assisted work and prohibited model training remains unclear for researchers and developers.