A Multi-Level Reinforcement Learning Framework for Smart Building Management

The growing complexity of urban environments renders the conventional approach of optimizing individual smart city subsystems, such as transportation and building energy management, ineffective, leading to fragmented decision-making and systemic inefficiencies. This paper proposes a unified, multilevel, and multi-objective reinforcement learning framework for coordinated, distributed optimization across various smart city sectors to address this issue. In this paper, we examine the smart-building domain across three buildings to clearly illustrate the approach’s core principles. The upper-level agents make strategic policy decisions. The lower-level agents, on the other hand, continuously manage low-level operations. The framework enables dynamic interaction among decision layers: policies at the upper level affect the environment perceived by lower-level agents. In contrast, lower-level actions provide feedback to the upper-level to modify their policy. The lower-level policies stabilize within the 10−2-10−3 range, as indicated by an experimental evaluation of a multi-building simulation environment. At the same time, the high-level agent continues fine-grained adaptation, as expected from a two-time-scale learning design. This closed-loop interaction facilitates multi-objective optimization, enabling system-wide coordination and synchronized policy learning across domains. Compared to the rule-based strategy, the bi-level PPO achieves smoother temperature regulation, more efficient battery dispatch, reduced discomfort induced by demand response (DR), and improved utilization of renewable energy. We plan to deploy our framework on the OpenCyberCity testbed to evaluate its real-world applicability.