Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning

Document Type : Research paper

Authors

1 Department of Engineering of Electrical Machines and Drives, Tashkent State Technical University, University Street No2, Tashkent, Uzbekistan.

2 Tashkent Institute of Irrigation and Agricultural Mechanization Engineers Institute" National Research University, Kari Niyazov Street 39, 100000, Tashkent, Uzbekistan.

3 Kimyo International University in Tashkent, Shota Rustaveli Street 156, 100121, Tashkent, Uzbekistan.

4 PhD, Assistant Professor, Alfraganus University, Uzbekistan.

5 Tashkent Institute of Irrigation and Agricultural Mechanization Engineers National Research University, Uzbekistan.

6 Urgench State University named after Abu Rayhan Biruni, Urgench, Uzbekistan.

7 Research Institute of Environmental and Nature Protection Technologies, 100000, Tashkent, Uzbekistan.

8 Department of General Professional Subjects, Mamun University, Khiva, Uzbekistan.

Abstract

This paper proposes a fully decentralized reinforcement learning–based energy management framework for hybrid electrical–thermal microgrids with distributed energy resources. Uncertainties in renewable energy generation, variations in load demand, and the nonlinear nature of battery systems make it difficult to achieve optimal energy management in microgrids. Additionally, using centralized controller techniques in large-scale systems increases computational complexity and makes controller procedure implementation more challenging. This study proposes a fully decentralized multi-agent architecture in which the stochastic performance of agents in the microgrid is modeled using Markov decision processes. This model treats consumers, batteries, and distributed thermal and electrical resources as intelligent, self-governing agents that learn from their surroundings and converge to their best policies through decentralized exploitation. The proposed model-free learning-based approach is designed to not only maximize the profits of producers but also minimize the costs for consumers and reduce the microgrid's reliance on the main grid. Finally, using real-world data from renewable power plants and electricity market data, the performance of the proposed method is evaluated through simulation and accuracy assessment.

Keywords

Main Subjects

References

  1. S. Makaba, U. Mardianto, K. Jumintono, and N. Nugrohowati, “Genetic algorithm optimization with machine learning to check the primary health of hospital visitors,” Procedia Environ. Sci. Eng. Manag., vol. 12, no. 1, pp. 7–15, 2025.
  2. M. Rezaee and V. A. Maleki, “On the complex mode shapes and natural frequencies of clamped–clamped fluid-conveying pipe,” Appl. Ocean Res., vol. 150, p. 104113, 2024.
  3. S. Gulandom, G. Shuhratovich, R. Ramatjanovna, K. Yaqipbay, B. Baltabayevna, J. Olimjon, and G. Aleksandrovna, “Development of optimization process for improving the educational classes scheduling,” Procedia Environ. Sci. Eng. Manag., vol. 12, no. 1, pp. 71–80, 2024.
  4. A. Anuchin, N. Kuraev, L. Rassudov, D. Savkin, and G. Demidova, “Multi-functional test benches for electric drive instructional laboratories,” Int. J. Ind. Eng. Manag., vol. 16, no. 2, pp. 161–175, 2025.
  5. N. Sharafkhani, “An ultra-thin multi-layered metamaterial for power transformer noise absorption,” Build. Acoust., vol. 29, no. 1, pp. 53–62, 2022.
  6. N. Sharafkhani, J. O. Orwa, S. D. Adams, J. M. Long, G. Lissorgues, L. Rousseau, and A. Z. Kouzani, “An intracortical polyimide microprobe with piezoelectric-based stiffness control,” J. Appl. Mech., vol. 89, no. 9, p. 091008, 2022.
  7. N. Sharafkhani, A. Z. Kouzani, S. D. Adams, J. M. Long, and J. O. Orwa, “A pneumatic-based mechanism for inserting a flexible microprobe into the brain,” J. Appl. Mech., vol. 89, no. 3, p. 031010, 2022.
  8. H. Dasari and E. Eisenbraun, “Predicting the effect of silicon electrode design parameters on thermal performance of a lithium-ion battery,” J. Electrochem. Sci. Eng., vol. 13, no. 4, pp. 659–672, 2023.
  9. L. Kruitwagen, J. E. Hinkel, M. C. Lioris, M. Stephan, J. L. Hacke, M. D. Islam, and S. A. Kurtz, “A global inventory of photovoltaic solar energy generating units,” Nature, vol. 598, no. 7882, pp. 604–610, 2021.
  10. M. W. Akram, G. Li, Y. Jin, and X. Chen, “Failures of photovoltaic modules and their detection: A review,” Appl. Energy, vol. 313, p. 118822, 2022.
  11. H. M. Hussein, A. Aghmadi, M. S. Abdelrahman, S. M. S. H. Rafin, and O. Mohammed, “A review of battery state of charge estimation and management systems: Models and future prospective,” WIREs Energy Environ., vol. 13, no. 1, p. e507, 2024.
  12. S. Wang, Q. Tan, X. Ding, and J. Li, “Efficient microgrid energy management with neural-fuzzy optimization,” Int. J. Hydrogen Energy, vol. 64, pp. 269–281, 2024.
  13. C. Álvarez Arroyo, S. Vergine, A. Sánchez de la Nieta, L. Alvarado-Barrios, and G. D’Amico, “Optimising microgrid energy management: Leveraging flexible storage systems and full integration of renewable energy sources,” Renewable Energy, vol. 229, p. 120701, 2024.
  14. M. R. Khan, Z. M. Haider, F. H. Malik, F. M. Almasoudi, K. S. S. Alatawi, and M. S. Bhutta, “A comprehensive review of microgrid energy management strategies considering electric vehicles, energy storage systems, and AI techniques,” Processes, vol. 12, no. 2, p. 270, 2024.
  15. G. Liu, M. F. Ferrari, T. B. Ollis, and K. Tomsovic, “An MILP-based distributed energy management for coordination of networked microgrids,” Energies, vol. 15, no. 19, p. 6971, 2022.
  16. P. Buchibabu and J. Somlal, “Sustainable energy management in microgrids: A multi-objective approach for stochastic load and intermittent renewable energy resources,” Electr. Eng., pp. 1–15, 2024.
  17. J. S. Giraldo, J. A. Castrillon, J. C. López, M. J. Rider, and C. A. Castro, “Microgrids energy management using robust convex programming,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4520–4530, 2019.
  18. Z. Shen, C. Wu, L. Wang, and G.-L. Zhang, “Real-time energy management for microgrid with EV station and CHP generation,” IEEE Trans. Netw. Sci. Eng., vol. 8, no. 2, pp. 1492–1501, 2021.
  19. Q. Duan, W. Sheng, H. Wang, C. Zhao, C. Ma, and S. Cheng, “A two-stage robust optimization method based on the expected scenario for islanded microgrid energy management,” Discrete Dyn. Nat. Soc., vol. 2021, p. 7079296, 2021.
  20. S. Haddadipour, V. Amir, and S. J. Arani, “Simultaneous purchase and sale of electricity in a multi-agent microgrid energy market,” Comput. Intell. Electr. Eng., vol. 11, no. 4, pp. 93–110, 2020.
  21. S. Umetani, Y. Fukushima, and H. Morita, “A linear programming-based heuristic algorithm for charge and discharge scheduling of electric vehicles in a building energy management system,” Omega, vol. 67, pp. 115–122, 2019.
  22. A. Mohammad, M. Zuhaib, and I. Ashraf, “An optimal home energy management system with integration of renewable energy and energy storage with home-to-grid capability,” Int. J. Energy Res., vol. 46, no. 6, pp. 8352–8366, 2022.
  23. A. Seifi, M. H. Moradi, M. Abedini, and A. Jahangiri, “Assessing the impact of load response on microgrids considering uncertainty in renewable generation,” Comput. Intell. Electr. Eng., vol. 12, no. 1, pp. 87–98, 2021.
  24. M. Yadipour, F. Hashemzadeh, and M. Baradarannia, “Controller design to enlarge the domain of attraction for a class of nonlinear systems,” in Proc. Int. Conf. Research and Education in Mechatronics (REM), pp. 1–6, IEEE, 2019.
  25. S. Sourani Yancheshmeh, A. Ebrahimpour, and T. Deemyad, “Optimizing chassis design for autonomous vehicles in challenging environments based on finite element analysis and genetic algorithm,” in Proc. ASME Int. Mech. Eng. Congr. Expo. (IMECE), ASME, 2024.
  26. K. C. Bingham, S. Sourani Yancheshmeh, G. Vaidya, A. Ebrahimpour, and T. Deemyad, “Advanced material selection and design strategies for optimized robotic systems,” in Proc. ASME Int. Mech. Eng. Congr. Expo. (IMECE), ASME, 2024.
  27. N. M. Manousakis, P. S. Karagiannopoulos, G. J. Tsekouras, and F. D. Kanellos, “Integration of renewable energy and electric vehicles in power systems: A review,” Processes, vol. 11, no. 5, p. 1544, 2023.
  28. B. Javanmard, M. Tabrizian, M. Ansarian, and A. Ahmarinejad, “Energy management of multi-microgrids based on a game theory approach,” J. Energy Storage, vol. 42, p. 102971, 2021.
  29. A. R. Jordehi, “Two-stage stochastic programming for risk-aware scheduling of energy hubs participating in electricity markets,” Sustainable Cities Soc., vol. 81, p. 103823, 2022.
  30. S. Mahjoubi and Y. Bao, “Game theory-based metaheuristics for structural design optimization,” Comput.-Aided Civ. Infrastruct. Eng., vol. 36, no. 10, pp. 1337–1353, 2021.
  31. B. Zhang, W. Hu, A. M. Ghias, X. Xu, and Z. Chen, “Multiagent deep reinforcement learning-based distributed control architecture for interconnected multi-energy microgrids,” Energy Convers. Manag., vol. 277, p. 116647, 2023.
  32. A. Churkin, J. Bialek, D. Pozo, E. Sauma, and N. Korgin, “Review of cooperative game theory applications in power system expansion planning,” Renewable Sustain. Energy Rev., vol. 145, p. 111056, 2021.
  33. X. Xu, Y. Jia, Y. Xu, Z. Xu, S. Chai, and C. S. Lai, “A multi-agent reinforcement learning-based data-driven method for home energy management,” IEEE Trans. Smart Grid, vol. 11, no. 4, pp. 3201–3211, 2020.
  34. G. K. Venayagamoorthy, R. K. Sharma, P. K. Gautam, and A. Ahmadi, “Dynamic energy management system for a smart microgrid,” IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 8, pp. 1643–1656, 2019.
  35. B. Lami, M. Alsolami, A. Alferidi, and S. B. Slama, “A smart microgrid platform integrating AI and deep reinforcement learning for sustainable energy management,” Energies, vol. 18, no. 5, p. 1157, 2025.
  36. F. D. Li, M. Wu, Y. He, and X. Chen, “Optimal control in microgrid using multi-agent reinforcement learning,” ISA Trans., vol. 51, no. 6, pp. 743–751, 2022.
  37. W. Liu, P. Zhuang, H. Liang, J. Peng, and Z. Huang, “Distributed economic dispatch in microgrids based on cooperative reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 6, pp. 2192–2203, 2019.
  38. R. B. Diddigi, C. Kamanchi, and S. Bhatnagar, “A generalized minimax Q-learning algorithm for two-player zero-sum stochastic games,” IEEE Trans. Autom. Control, vol. 67, no. 9, pp. 4816–4823, 2022.
  39. P. A. Tsividis, J. Loula, J. Burga, N. Foss, A. Campero, T. Pouncy, S. J. Gershman, and J. B. Tenenbaum,
    “Human-level reinforcement learning through theory-based modeling, exploration, and planning,” arXiv preprint, vol. arXiv:2107.12544, 2021.
  40. C. Guo, X. Wang, Y. Zheng, and F. Zhang, “Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning,” Energy, vol. 238, p. 121873, 2022.
  41. K. Deshpande, P. Möhl, A. Hämmerle, G. Weichhart, H. Zörrer, and A. Pichler, “Energy management simulation with multi-agent reinforcement learning: Reliability and resilience,” Energies, vol. 15, no. 19, p. 7381, 2022.
  42. M. Andreasson, D. V. Dimarogonas, H. Sandberg, and K. H. Johansson, “Distributed PI-control with applications to power systems frequency control,” in Proc. American Control Conf., pp. 3184–3189, IEEE, 2024.
  43. A. Al-Shetwi, M. Hannan, H. Al-Masri, and M. Sujod, “Latest advancements in smart grid technologies and their transformative role in shaping the power systems of tomorrow: An overview,” Progress in Energy, vol. 7, no. 1, p. 012004, 2024.
  44. R. Darshi, M. A. Bahreini, and S. A. Ebrahim, “Prediction of short-term electricity consumption using artificial neural networks,” in Proc. 5th Iranian Conf. Signal Process. Intell. Syst. (ICSPIS), IEEE, 2019.
  45. T. Chen, Z. Wang, and M. Zhou, “Diffusion policies creating a trust region for offline reinforcement learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 50098–50125, 2024.
  46. Y. Du and F. Li, “Intelligent multi-microgrid energy management based on deep neural networks and model-free reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 2, pp. 1066–1076, 2019.
  47. E. Foruzan, L. K. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Trans. Power Syst., vol. 33, no. 5, pp. 5749–5758, 2020.
Journal of Operation and Automation in Power Engineering
Volume 13, Special Issue
Intelligent and Sustainable Power Systems (ISPS): AI-Driven Innovations for Renewable Integration and Smart Grid Resilience
2025
Pages 45-61

Files

History

  • Receive Date: 27 November 2025
  • Revise Date: 23 December 2025
  • Accept Date: 25 December 2025
  • First Publish Date: 25 December 2025

Share

How to cite

Statistics