Xiao Du 「杜潇」

Deep reinforcement learning (DRL) has emerged as a promising paradigm for home energy management systems (HEMSs) due to its model-free nature and ability to handle complex dynamics. However, existing DRL-based approaches typically employ a unified reward function that aggregates multiple objectives into a single scalar, failing to account for the heterogeneous roles of controllable energy devices (CEDs) and their distinct causal relationships with control objectives. This leads to reward misattribution, where CEDs with simpler constraints dominate the optimization process while critical components such as battery storage remain underutilized. To address this challenge, we propose a causally aligned multiagent reinforcement learning (MARL) framework that explicitly models CED-objective causal pathways using a structural causal model (SCM). A causal surgery procedure decomposes shared objectives into CED-specific variants, enabling individualized reward signals aligned with each CED's causal responsibility. The proposed heterogeneous reward-aware multiagent soft actor–critic (HR-MASAC) algorithm features a multihead centralized critic for learning vectorized Q-values and agent-specific entropy coefficients for heterogeneous exploration. Experiments across diverse home scenarios demonstrate that our method achieves 40.1% cost reduction and 57.1% comfort improvement over unified-reward baselines, with robust performance under sensor noise and household heterogeneity.

Home energy management systems (HEMS) must balance competing objectives—electricity cost, thermal comfort, and carbon emissions—according to user preferences that are personalized, evolving, and often expressed in natural language. Conventional deep reinforcement learning (DRL) methods cannot directly interpret such preferences, while existing reinforcement learning (RL)+LLM integrations either invoke large language models (LLMs) at every control timestep or rely on one-shot preference parsing without feedback. This paper proposes LA-UPAHEM, an iterative LLM-augmented framework in which three specialized agents collaborate in a closed loop: a Code Generation Agent translates preferences into reward and state modification functions, a Result Analysis Agent diagnoses policy–preference misalignment from evaluation metrics, and an Optimal Performance Search Agent identifies the best-performing policy as the warm-start for the next iteration. LLMs are invoked only during this offline refinement phase; the deployed policy operates independently without LLM inference. Experiments on real residential energy datasets show that LA-UPAHEM outperforms classical DRL and existing RL+LLM baselines in both macro-level key performance indicator (KPI) alignment (cost, comfort, emissions) and micro-level rule compliance, achieving a Weighted Improvement Ratio (WIR) of 0.13 and a Rule Compliance Rate (RCR) of 0.84, while reducing the failure rate from 34.3% (static parsing) to 5.4%. The framework is robust to environmental noise, evaluation weight perturbation, and preference paraphrasing, and generalizes across multiple DRL algorithms, LLM backbones, and preference languages.

Deep reinforcement learning (DRL) is widely used in home energy management for its ability to handle nonlinearity and uncertainty. However, its reliance on trial-and-error interaction makes early unsafe and inefficient behaviors impractical in real households. To address this, we propose RB-ZeroHEM, a zero-shot knowledge transfer framework based on representation learning and behavioral cloning. RB-ZeroHEM employs contrastive learning to extract stable, physics-aligned representations of household energy dynamics from historical control trajectories, enabling clustering-based similarity measurement without hand-crafted features. For a new household, it identifies the most similar source cases and clones a deployable policy requiring zero target-environment interactions. Experiments on 12 simulated households driven by real-world energy data (San Diego smart meter and weather records spanning four seasons) demonstrate that when transferring to physically similar environments, RB-ZeroHEM reduces discomfort ratio by 33% (from 12.34% to 8.24%) and electricity costs by 15% (from $228 to $193) compared to rule-based control, while achieving 81% of the grid energy savings of the best online DRL method, soft actor-critic (SAC), with zero interactions versus SAC's 24,192-step exploration budget. The learned representations exhibit strong physics alignment and remain robust across seasonal shifts and deployment noise. With minimal fine-tuning (576 interactions, 2.4% of SAC's budget), performance matches or surpasses online DRL baselines. When source–target physical dynamics are mismatched, zero-shot performance degrades gracefully, indicating that similarity-based retrieval reliability is critical. These results establish RB-ZeroHEM as a practical, low-risk solution for deploying intelligent energy management in real households, enabling rapid scaling without the safety and economic penalties of online exploration.

Personalizing home energy management systems (HEMSs) to align with dynamic user preferences remains challenging for intelligent energy control. Existing deep reinforcement learning (DRL) approaches typically rely on static reward functions and fixed constraints, limiting their ability to adapt to diverse user behaviors. We propose a framework that leverages user override feedback—instances where users manually correct system decisions—as implicit preference signals. Our method combines contrastive preference learning from pairwise comparisons, context-aware preference embeddings from interaction history, and preference-conditioned DRL policies. This enables the controller to continuously align with personalized preferences without explicit reward engineering. Validation on real-world residential energy data shows that the framework achieves an average override rate of 0.033 across four diverse preference profiles—a 37% reduction over the strongest baseline—while maintaining the lowest user interaction cost (0.394) among all feedback-driven methods. The framework also demonstrates efficient transfer across prototype preferences and generalization to novel behavioral rules.