PyTorch's CI Advisor now logs full AI reasoning to debug bad verdicts
AI verdicts can now be traced from input to reasoning to catch confabulations.
PyTorch's CI Advisor, the automated system that decides whether to revert suspicious commits, has gained a critical observability upgrade. Previously, when the AI rendered a disputed verdict, only the final JSON was recoverable—engineers couldn't see what input the model actually received or how it reached its conclusion. That made it impossible to distinguish a genuine model confabulation from a wrong or truncated input feed after the fact.
To fix this, PR #191545 adds two debug captures to the advisor workflow. First, per-job input payloads—including suspect_commit, pr_number, and the verbatim signal_pattern—are saved as an 'advisor-input' artifact and uploaded to S3. Second, the full claude-code-action execution_file output (a turn-by-turn log covering thinking, every tool call and result, and final output) is persisted as an 'advisor-reasoning-trace' artifact. Both steps run on always() with continue-on-error: true, ensuring pathological runs (empty or failed verdicts) are exactly the ones that can be inspected, while debug logging never gates or fails the advisor itself. Keyed by run_id/run_attempt and reusing existing OIDC credentials, the change adds no modification to the verdict pipeline—just pure transparency.
- Saves per-job input payload (suspect_commit, pr_number, signal_pattern) as 'advisor-input' artifacts to S3
- Persists full claude-code-action reasoning trace (thinking, tool calls, results) as 'advisor-reasoning-trace' JSON
- Runs on always() with continue-on-error:true so debug logging can never fail the CI advisor
- Enables distinguishing confabulation from truncated input after runs, keyed by run_id and run_attempt
Why It Matters
For AI agents in CI, observability of inputs and reasoning—not just outputs—is now a required practice.