CAPRA multi-agent LLM system automates feedback on software architecture at 88.8% accuracy

CAPRA (Configurable Architecture Proficiency Report Assessment) is a multi-agent LLM system designed to automate feedback generation for software architecture deliverables — a task that has resisted full automation due to its reliance on structural completeness and requirements traceability. The system coordinates multiple specialized agents and uses a Python-based microservice for multi-modal document extraction, leveraging PyMuPDF and vision-enabled gpt-4o to parse both text and UML diagrams. To ensure educational reliability and mitigate hallucinations, CAPRA introduces a deterministic Evidence Anchoring step using fuzzy matching via normalized Levenshtein distance, along with a ConsistencyManager agent that cross-verifies, deduplicates, and merges findings. The system outputs personalized, template-compliant LaTeX feedback.

In a preliminary empirical evaluation on 10 student reports using a structured eight-criterion binary taxonomy, CAPRA satisfied 88.8% of the evaluated criteria under a strict two-rater aggregation rule and achieved moderate inter-rater agreement with human evaluators (Cohen's kappa = 0.582). Each report was processed in slightly over 4 minutes. While these results demonstrate the viability of LLM-supported architectural feedback, the authors emphasize that human oversight remains essential for subjective assessment dimensions. The paper has been accepted for publication at the 38th International Conference on Software Engineering Education and Training (CSEE&T 2026).