LLMography: New AI audit framework scores human contribution at 86.8/100
A framework that turns chat logs into KPIs for AI transparency.
A new academic framework called LLMography, developed by Mohammed Bousmah, aims to solve the problem of assessing not just final AI outputs but the entire human-AI interaction process. By analogy with bibliography, LLMography documents the dynamic trajectory of a conversation as a structured trace of co-production. Its prototype analyzes conversation logs and generates a suite of Key Performance Indicators (KPIs), such as Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, and Privacy Risk Level.
In a preliminary study using 19 anonymized student audit reports, the framework classified most interactions as Human-AI co-produced, with an average Human Direction Score of 86.8/100, Prompt Quality of 81.9/100, Auditability of 72.8/100, and Final Output Traceability of 77.1/100. Bousmah even applied LLMography retroactively to the paper's own writing, classifying it as “human-originated, human-directed, AI-assisted co-production.” The work suggests AI transparency must move beyond simple detection toward documenting the history of interaction.
- LLMography generates 7 KPIs per conversation, including Prompt Quality Score (81.9 avg) and AI Dependency Level.
- Pilot evaluation on 19 student audit reports found most interactions are Human-AI co-produced, not purely AI-generated.
- Framework is self-referential: applied to its own writing process to demonstrate transparency labeling.
Why It Matters
Moves AI transparency from output guessing to verifiable process auditing, critical for education and compliance.