Developer Tools

PyTorch's CI Advisor now logs full AI reasoning to debug bad verdicts

⚡AI verdicts can now be traced from input to reasoning to catch confabulations.

Deep Dive

PyTorch's CI Advisor, the automated system that decides whether to revert suspicious commits, has gained a critical observability upgrade. Previously, when the AI rendered a disputed verdict, only the final JSON was recoverable—engineers couldn't see what input the model actually received or how it reached its conclusion. That made it impossible to distinguish a genuine model confabulation from a wrong or truncated input feed after the fact.

To fix this, PR #191545 adds two debug captures to the advisor workflow. First, per-job input payloads—including suspect_commit, pr_number, and the verbatim signal_pattern—are saved as an 'advisor-input' artifact and uploaded to S3. Second, the full claude-code-action execution_file output (a turn-by-turn log covering thinking, every tool call and result, and final output) is persisted as an 'advisor-reasoning-trace' artifact. Both steps run on always() with continue-on-error: true, ensuring pathological runs (empty or failed verdicts) are exactly the ones that can be inspected, while debug logging never gates or fails the advisor itself. Keyed by run_id/run_attempt and reusing existing OIDC credentials, the change adds no modification to the verdict pipeline—just pure transparency.

Key Points
  • Saves per-job input payload (suspect_commit, pr_number, signal_pattern) as 'advisor-input' artifacts to S3
  • Persists full claude-code-action reasoning trace (thinking, tool calls, results) as 'advisor-reasoning-trace' JSON
  • Runs on always() with continue-on-error:true so debug logging can never fail the CI advisor
  • Enables distinguishing confabulation from truncated input after runs, keyed by run_id and run_attempt

Why It Matters

For AI agents in CI, observability of inputs and reasoning—not just outputs—is now a required practice.

📬 Get the top 10 AI stories daily