AI Root Cause Analysis Just Got a Much-Needed Accountability Upgrade — But There's a Catch

JustDiag! tackles a critical gap in AI-assisted root cause analysis: LLMs can generate convincing final diagnoses, but in high-stakes operations like software incident response, engineers need to see the reasoning behind those conclusions. The system maintains an explicit process state documenting what evidence was considered, which competing hypotheses were evaluated, where contradictions remained, and whether the case was resolved or uncertainty preserved. This approach transforms RCA from a black-box answer to an accountable audit trail.

Evaluated on 66 real-world incidents using a two-layer protocol that separately scores final-answer quality and process quality, JustDiag! outperformed a matched control without such justification. It achieved stronger scores on both outcome and process metrics, while accepting slightly lower terminal completion due to more calibrated non-closure—meaning it correctly preserves uncertainty when evidence is insufficient. The work suggests that accountable RCA requires explicit diagnostic justification artifacts and process-aware evaluation, not just fluent answers.