Shayeghi, H., Younesi, A. (2019). Mini/Micro-Grid Adaptive Voltage and Frequency Stability Enhancement Using Q-learning Mechanism. Journal of Operation and Automation in Power Engineering, 7(1), 107-118. doi: 10.22098/joape.2019.5542.1417

H. Shayeghi; A. Younesi. "Mini/Micro-Grid Adaptive Voltage and Frequency Stability Enhancement Using Q-learning Mechanism". Journal of Operation and Automation in Power Engineering, 7, 1, 2019, 107-118. doi: 10.22098/joape.2019.5542.1417

Shayeghi, H., Younesi, A. (2019). 'Mini/Micro-Grid Adaptive Voltage and Frequency Stability Enhancement Using Q-learning Mechanism', Journal of Operation and Automation in Power Engineering, 7(1), pp. 107-118. doi: 10.22098/joape.2019.5542.1417

Shayeghi, H., Younesi, A. Mini/Micro-Grid Adaptive Voltage and Frequency Stability Enhancement Using Q-learning Mechanism. Journal of Operation and Automation in Power Engineering, 2019; 7(1): 107-118. doi: 10.22098/joape.2019.5542.1417

Mini/Micro-Grid Adaptive Voltage and Frequency Stability Enhancement Using Q-learning Mechanism

^{}Department of Electrical Engineering, University of Mohaghegh Ardabili, Ardabil, Iran

Abstract

This paper develops an adaptive control method for controlling frequency and voltage of an islanded mini/micro grid (M/µG) using reinforcement learning method. Reinforcement learning (RL) is one of the branches of the machine learning, which is the main solution method of Markov decision process (MDPs). Among the several solution methods of RL, the Q-learning method is used for solving RL in this paper because it is a model-free strategy and has a simple structure. The proposed control mechanism is consisting of two main parts. The first part is a classical PID controller which is fixed tuned using Salp swarm algorithm (SSA). The second part is a Q-learning based control strategy which is consistent and updates it's characteristics according to the changes in the system continuously. Eventually, the dynamic performance of the proposed control method is evaluated in a real M/µG compared to fuzzy PID and classical PID controllers. The considered M/µG is a part of Denmark distribution system which is consist of three combined heat and power (CHP) and three WTGs. Simulation results indicate that the proposed control strategy has an excellent dynamic response compared to both intelligent and traditional controllers for damping the voltage and frequency oscillations.

[1] "IEEE recommended practice for excitation system models for power system stability studies," 1992.

[2] M. J. Morshed and A. Fekih, "A fault-tolerant control paradigm for microgrid-connected wind energy systems," IEEE Syst. J., vol. 12, pp. 360-372, 2018.

[3] R. Ghanizadeh, M. Ebadian, and G. B. Gharehpetian, "Control of inverter-interfaced distributed generation units for voltage and current harmonics compensation in grid-connected microgrids," J. Oper. Autom. Power Eng., vol. 4, pp. 66-82, 2016.

[4] D. O. Amoateng, M. A. Hosani, M. S. Elmoursi, K. Turitsyn, and J. L. Kirtley, "Adaptive voltage and frequency control of islanded multi-microgrids," IEEE Trans. Power Syst., vol. 33, pp. 4454-4465, 2018.

[5] X. Wu, C. Shen, and R. Iravani, "A distributed, cooperative frequency and voltage control for microgrids," IEEE Trans. Smart Grid, vol. 9, pp. 2764-2776, 2018.

[6] Y. Hirase, K. Abe, K. Sugimoto, K. Sakimoto, H. Bevrani, and T. Ise, "A novel control approach for virtual synchronous generators to suppress frequency and voltage fluctuations in microgrids," Appl. Energy, vol. 210, pp. 699-710, 2018/01/15/ 2018.

[7] J. W. Simpson-Porco, F. Dörfler, and F. Bullo, "Voltage stabilization in microgrids via quadratic droop control," IEEE Trans. Autom. Cont., vol. 62, pp. 1239-1253, 2017.

[8] F. Gao, S. Bozhko, A. Costabeber, C. Patel, P. Wheeler, C. I. Hill, et al., "Comparative stability analysis of droop control approaches in voltage-source-converter-based DC microgrids," IEEE Trans. Power Electron., vol. 32, pp. 2395-2415, 2017.

[9] F. Asghar, M. Talha, and S. Kim, "Robust frequency and voltage stability control strategy for standalone AC/DC hybrid microgrid," Energies, vol. 10, p. 760, 2017.

[10] H. Zhao, M. Hong, W. Lin, and K. A. Loparo, "Voltage and frequency regulation of microgrid with battery energy storage systems," IEEE Trans. Smart Grid, vol. PP, pp. 1-1, 2017.

[11] A. Ahmarinejad, B. Falahjoo, and M. Babaei, "The stability control of micro-grid after islanding caused by error," Energy Procedia, vol. 141, pp. 587-593, 12// 2017.

[12] W. Issa, S. M. Sharkh, R. Albadwawi, M. Abusara, and T. K. Mallick, "DC link voltage control during sudden load changes in AC microgrids," Proc. IEEE 26^{th} Int. Symp. Ind. Electron., 2017, pp. 76-81.

[13] D. Ernst, M. Glavic, and L. Wehenkel, "Power systems stability control: reinforcement learning framework," IEEE Trans. Power Syst., vol. 19, pp. 427-435, 2004.

[14] T. Yu and W. G. Zhen, "A reinforcement learning approach to power system stabilizer," Proc. IEEE Power & Energy Soc. Gen. Meet., Calgary, AB, 2009, pp. 1-5.

[15] J. G. Vlachogiannis and N. D. Hatziargyriou, "Reinforcement learning for reactive power control," IEEE Trans. Power Syst., vol. 19, pp. 1317-1325, 2004.

[16] V. Nanduri and T. K. Das, "A reinforcement learning model to assess market power under auction-based energy pricing," IEEE Trans. Power Syst., vol. 22, pp. 85-95, 2007.

[17] A. Younesi, H. Shayeghi, and M. Moradzadeh, "Application of reinforcement learning for generating optimal control signal to the IPFC for damping of low‐frequency oscillations," Int. Trans. Electr. Energy Syst., vol. 28, p. e2488, 2018.

[18] H. Shayeghi and A. Younesi, "An online q-learning based multi-agent LFC for a multi-area multi-source power system including distributed energy resources," Iran. J. Electr. Electron. Eng., vol. 13, pp. 385-398, 2017.

[19] C. Weber, M. Elshaw, and N. M. Mayer, Reinforcement learning, theory and applications: I-TECH Education and Publishing, 2008.

[20] L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: a survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, 1996.

[21] R. S. Sutton and A. G. Barto, Reinforcement learning: an introduction: MIT Press, 2005.

[22] C. J. C. H. Watkins and P. Dayan, "Technical note: q-learning," Mach. Learn., vol. 8, pp. 279-292.

[23] S. Mirjalili, A. H. Gandomi, S. Z. Mirjalili, S. Saremi, H. Faris, and S. M. Mirjalili, "Salp Swarm Algorithm: A bio-inspired optimizer for engineering design problems," Adv. Eng. Software, vol. 114, pp. 163-191, 2017/12/01/ 2017.

[24] P. Mahat, Z. Chen, and B. Bak-Jensen, "Control and operation of distributed generation in distribution systems," Electric Power Syst. Res., vol. 81, pp. 495-502, 2011/02/01/ 2011.

[25] H. Shayeghi, A. Younesi, and Y. Hashemi, "Optimal design of a robust discrete parallel FP + FI + FD controller for the Automatic Voltage Regulator system," Int. J. Electr. Power Energy Syst., vol. 67, pp. 66-75, 2015.

[26] R. Hadidi and B. Jeyasurya, "Reinforcement learning based real-time wide-area stabilizing control agents to enhance power system stability," IEEE Trans. Smart Grid, vol. 4, pp. 489-497, 2013.