Hierarchical Control Framework Integrating LLMs with RL for Decarbonized HVAC Operation

A research team from Tsinghua University and collaborating institutions has published a novel AI framework that marries large language models (LLMs) with reinforcement learning (RL) to tackle one of the biggest energy drains in buildings: HVAC systems. The core innovation is a hierarchical structure where a fine-tuned LLM, trained on historical operational data, acts as a high-level guide. It analyzes the current state of the building (temperature, occupancy, etc.) and generates "action masks"—essentially a filtered list of plausible control actions for each zone. This prunes the otherwise exponentially large and inefficient action space that a vanilla RL agent would have to explore blindly.

This LLM-guided masking allows a downstream value-based RL agent to perform constrained optimization within a much smaller, more sensible set of actions. The result is dramatically improved training stability and exploration efficiency. The framework was rigorously evaluated using a high-fidelity simulator calibrated with real sensor and occupancy data from a 7-zone office building. It achieved a mean Predicted Percentage of Dissatisfied (PPD) comfort metric of just 7.30%, representing a 39.1% improvement over the best standard DQN RL baseline. Crucially, it did this while reducing daily HVAC energy consumption to 140.90 kWh, outperforming all vanilla RL baselines and beating the best LLM-only approach by 53.1% in energy reduction.

The study, available on arXiv, demonstrates a practical pathway for using the semantic reasoning and knowledge-encoding capabilities of LLMs to ground and accelerate numerical optimization processes like RL. This hybrid approach overcomes the individual weaknesses of each method: RL's struggle with massive action spaces and LLMs' lack of reliable, closed-loop optimization. The success in the complex, multi-zone environment suggests significant potential for scaling this method to larger commercial buildings, directly contributing to decarbonization goals through intelligent, adaptive control.