Consistency as a measure of LLM philosophical competence: a preliminary experiment

The researchers address a key challenge in training language models for conceptual reasoning: lack of reliable reward signals when questions have no clear right answer (e.g., consciousness, decision theory). They hypothesize that requiring models to give consistent answers across paraphrased or logically equivalent prompts could serve as a proxy for philosophical competence. In a small pilot study using the LMCA dataset (which tests negative examples in argumentation), they created 398 rewrites of critiques by slightly changing wording, hand‑approving each. The experiment then measured how well a model’s loss on the original LMCA task correlated with its consistency on these rewrites.

Preliminary results indicate a non‑trivial correlation between consistency and performance, suggesting that consistency constraints could be used as a training target to improve reasoning without needing explicit labels. The authors caution that further work is needed to confirm scalability and to explore other forms of consistency (e.g., transitive probability judgments). They also note that automatic generation of consistency checks (using LLMs themselves) could make the approach fully scalable. If validated, this method could unlock progress in training models for philosophy, ethics, and other domains where objective feedback is scarce.