LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance
Study reveals full fine-tuning creates more focused AI reasoning than efficient methods like LoRA.
A team of researchers has published a groundbreaking study analyzing how different fine-tuning methods affect the internal reasoning of Large Language Models (LLMs) used for automated building code compliance. The paper, accepted at the International Conference on Computing in Civil and Building Engineering (ICCCBE 2026), employs a perturbation-based attribution analysis to compare how models interpret regulatory text. This moves beyond treating LLMs as black boxes and examines how training decisions shape their interpretive behavior.
The study specifically compares full fine-tuning (FFT) against parameter-efficient methods like Low-Rank Adaptation (LoRA) and quantized LoRA. The key finding is that FFT produces attribution patterns that are statistically different and more focused than those from efficient fine-tuning methods. This suggests that the more computationally intensive FFT leads to models that pay attention to more specific, relevant parts of the input text when generating compliance rules.
Furthermore, the research explores the impact of model scale, analyzing LLMs of varying parameter sizes. The results show that as models grow larger, they develop more sophisticated interpretive strategies, such as prioritizing numerical constraints and rule identifiers within building codes. However, a crucial insight is that performance gains—measured by the semantic similarity of generated rules to reference rules—plateau for models larger than 7 billion parameters. This indicates a point of diminishing returns for this specific task.
- Full fine-tuning (FFT) creates more focused model reasoning than efficient methods like LoRA, according to attribution analysis.
- Larger LLMs (over 7B parameters) develop specific strategies like prioritizing numbers, but performance gains plateau.
- The research provides crucial explainability insights for using transparent LLMs in critical AEC industry regulation tasks.
Why It Matters
This work is a major step toward building transparent, trustworthy AI systems for high-stakes regulatory and compliance applications in engineering and construction.