A Hybrid LSTM–XGBoost Framework Optimized by Particle Swarm Optimization for Short-Term Energy Demand Forecasting Using Large-Scale Smart Meter Data

Abstract

Abstract

Rapid proliferation of cloud storage services has enlarged the issue of protecting user data from

The short term electricity demand forecast is an essential for the operation of smart grid, demand response programs, peak load management and sustainable energy planning. Given the nonlinear, dynamic and time-dependent nature of the energy demand patterns, however, predicting electricity consumption remains difficult. To tackle these challenges, the present study proposes to optimize the Hybrid LSTM–XGBoost forecasting framework by exploiting the advantage of temporal sequence learning of LSTM networks and the nonlinear regression power of Extreme Gradient Boosting (XGBoost). Moreover, Particle Swarm Optimization (PSO) algorithm is used to automatically optimize the hyperparameters of XGBoost component to improve the forecasting accuracy and eliminate the need for extensive manual tuning. A Walk-Forward Validation (WFV) strategy is used during the experimental process to achieve realistic and leakage-free model assessment. The proposed framework is evaluated based on a large-scale real world data set of smart meters gathered by the Pecan Street project which produces high-resolution measurements of residential electricity consumption. A scalable pre-processing pipeline and extensive feature engineering approach are designed to capture temporal, lag-based and statistical features that represent the electricity consumption behavior well. The experimental results validate that the proposed Hybrid LSTM–XGBoost + PSO model always has better performance than the baseline XGBoost model and the non-optimized Hybrid LSTM–XGBoost model. The proposed approach gives the minimum forecasting error, maximum coefficient of determination, and consistent results in all the walk forward period validation folds. The results suggest that the deep temporal feature extraction, ensemble regression, intelligent hyperparameter tuning, and leakage-free time-aware assessment methods are an accurate, robust, and practical approach to short-term electricity demand forecasting in smart grid settings.