On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach

WANG Li; WU Xuewei; WANG Yanhui; XIAO Zhe; LI Liang; FEI Aiguo

doi:10.23919/cje.2021.00.326

Volume 32 Issue 4

Jul. 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2023 > 32(4): 760-772

WANG Li, WU Xuewei, WANG Yanhui, et al., “On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach,” Chinese Journal of Electronics, vol. 32, no. 4, pp. 760-772, 2023, doi: 10.23919/cje.2021.00.326

Citation:

WANG Li, WU Xuewei, WANG Yanhui, et al., “On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach,” Chinese Journal of Electronics, vol. 32, no. 4, pp. 760-772, 2023, doi: 10.23919/cje.2021.00.326

Citation:

PDF( 4115 KB)

On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach

doi: 10.23919/cje.2021.00.326

WANG Li^1
,,
WU Xuewei^2
,,
WANG Yanhui^2
,,
XIAO Zhe^1
,,
LI Liang^1
,,
FEI Aiguo^1
,

1.
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100867, China
2.
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100867, China

Funds: This work was supported by the Beijing Municipal Natural Science Foundation (L192030), the National Key Research and Development Program of China (2020YFC1511801), and the National Natural Science Foundation of China (62171054, U2066201, 61871416)

More Information

Author Bio:
Li WANG received the Ph.D. degree from the Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2009. She is currently a Full Professor with the School of Computer Science (National Pilot Software Engineering School), BUPT, where she is also an Associate Dean and heads the High Performance Computing and Networking Laboratory. She is also a member of the Key Laboratory of the Universal Wireless Communications, Ministry of Education, China. She also held Visiting Positions with the School of Electrical and Computer Engineering, Georgia Tech, Atlanta, GA, USA, from December 2013 to January 2015, and with the Department of Signals and Systems, Chalmers University of Technology, Gothenburg, Sweden, from August to November 2015 and July to August 2018. She has authored/coauthored almost 50 journal papers and two books. Her current research interests include wireless communications, distributed networking and storage, vehicular communications, social networks, and edge AI. She currently serves on the Editorial Boards for IEEE Transactions on Vehicular Technology, IEEE Transactions on Green Communications and Networking, Computer Networks, IEEE Access, and China Communications. She was the Symposium Chair of IEEE ICC 2019 on Cognitive Radio and Networks Symposium and a Tutorial Chair of IEEE VTC 2019-fall. She was the vice chair of Meetings and Conference Committee (MCC) for IEEE Communication Society (ComSoc) Asia Pacific Board (APB) for the term of 2021, and chairs the special interest group (SIG) on Sensing, Communications, Caching, and Computing (C3) in Cognitive Networks for IEEE Technical Committee on Cognitive Networks. She was the recipient of the 2013 Beijing Young Elite Faculty for Higher Education Award, Best Paper Awards from several IEEE conferences, e.g., IEEE ICCC 2017, IEEE GLOBECOM 2018, IEEE WCSP 2019, and so forth. She was also the recipient of the Beijing Technology Rising Star Award in 2018. She has served on TPC of multiple IEEE conferences, including IEEE Infocom, Globecom, International Conference on Communications, IEEE Wireless Communications and Networking Conference, and IEEE Vehicular Technology Conference in recent years. (Email: liwang@bupt.edu.cn)

Xuewei WU received the bachelor’s degree from Beijing University of Posts and Telecommunications, Beijing, China, in 2018, where she is currently pursuing the M.S. degree in electronics and communication engineering. Her research interests include UAV communications, machine learning-based wireless communications, and cooperative cache. (Email: xuewei.wu@bupt.edu.cn)

Yanhui WANG received the B.S. degree from Beijing University of Posts and Telecommunications Beijing, China, in 2019, where he is currently pursuing the M.S. degree in electronics and communication engineering. His research interests include distributed caching, deep reinforcement learning, and incentive mechanism design in wireless communications. (Email: 1911128152@qq.com)

Zhe XIAO received the M.S. degree from the Beijing University of Posts and Telecommunications, Beijing, China, in 2019, where he is currently pursuing the Ph.D. degree in computer science and technology. His research interests include UAV-deployment, deep reinforcement learning, and wireless communications. (Email: xiaozhe@bupt.edu.cn)

Liang LI received the Ph.D. degree in the School of Telecommunications Engineering at Xidian University, China, in 2021. She is currently a postdoctoral faculty member with the School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications. She was also a visiting Ph.D. student with the Department of Electrical and Computer Engineering, University of Houston, Houston, TX, USA, from 2018 to 2020. Her research interests include edge computing, federated learning, data-driven robust optimization, and differential privacy. (Email: liliang1127@bupt.edu.cn)

Aiguo FEI received the M.S. degree from the Beijing University of Posts and Telecommunications, Beijing, China, in 1981, and the Ph.D. degree from the University of Science and Technology Beijing, Beijing, in 2004. He is a Professor with the School of Computer Science (National Pilot Software Engineering School), BUPT. He is also a Member of the State Key Laboratory of Networking and Switching Technology and the Academician of the Chinese Academy of Engineering, Beijing. His current research interests include Internet of things, intelligent emergency communication systems, intelligent information systems, big data, cloud computing, and intelligent software development and testing. (Email: aiguofei@bupt.edu.cn)
Received Date: 2021-09-01
Accepted Date: 2022-01-10

Available Online: 2022-07-23

Publish Date: 2023-07-05

Abstract

Abstract

Unmanned aerial vehicles (UAVs) can be effectively used as serving stations in emergency communications because of their free movements, strong flexibility, and dynamic coverage. In this paper, we propose a coordinated multiple points based UAV deployment framework to improve system average ergodic rate, by using the fuzzy C-means algorithm to cluster the ground users and considering exclusive forest channel models for the two cases, i.e., associated with a broken base station or an available base station. In addition, we derive the upper bound of the average ergodic rate to reduce computational complexity. Since deep reinforcement learning (DRL) can deal with the complex forest environment while the large action and state space of UAVs leads to slow convergence, we use a ratio cut method to divide UAVs into groups and propose a hierarchical clustering DRL (HC-DRL) approach with quick convergence to optimize the UAV deployment. Simulation results show that the proposed framework can effectively reduce the complexity, and outperforms the counterparts in accelerating the convergence speed.
- UAV deployment,
- Forest channel model,
- Deep reinforcement learning

FullText(HTML)

References(21)

References

[1]	K. Gomez, S. Kandeepan, M. M. Vidal, et al., “Aerial base stations with opportunistic links for next generation emergency communications,” IEEE Communications Magazine, vol.54, no.4, pp.31–39, 2016. doi: 10.1109/MCOM.2016.7452263
[2]	C. C. Lai, C. T. Chen, and L. C. Wang, “On-demand density-aware UAV base station 3D placement for arbitrarily distributed users with guaranteed data rates,” IEEE Wireless Communications Letters, vol.8, no.3, pp.913–916, 2019. doi: 10.1109/LWC.2019.2899599
[3]	A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum coverage,” IEEE Wireless Communications Letters, vol.3, no.6, pp.569–572, 2014. doi: 10.1109/LWC.2014.2342736
[4]	R. I. Bor-Yaliniz, A. El-Keyi, and H. Yanikomeroglu, “Efficient 3-D placement of an aerial base station in next generation cellular networks,” in Proceedings of the 2016 IEEE International Conference on Communications (ICC), Kuala Lumpur, Malaysia, pp.1–5, 2016.
[5]	M. Alzenad, A. El-Keyi, F. Lagum, et al., “3-D placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage,” IEEE Wireless Communications Letters, vol.6, no.4, pp.434–437, 2017. doi: 10.1109/LWC.2017.2700840
[6]	M. Z. Chen, M. Mozaffari, W. Saad, et al., “Caching in the sky: Proactive deployment of cache-enabled unmanned aerial vehicles for optimized quality-of-experience,” IEEE Journal on Selected Areas in Communications, vol.35, no.5, pp.1046–1061, 2017. doi: 10.1109/JSAC.2017.2680898
[7]	L. Liu, S. W. Zhang, and R. Zhang, “CoMP in the sky: UAV placement and movement optimization for multi-user communications,” IEEE Transactions on Communications, vol.67, no.8, pp.5645–5658, 2019. doi: 10.1109/TCOMM.2019.2907944
[8]	L. P. Zhu, J. Zhang, Z. Y. Xiao, et al., “Millimeter-wave full-duplex UAV relay: Joint positioning, beamforming, and power control,” IEEE Journal on Selected Areas in Communications, vol.38, no.9, pp.2057–2073, 2020. doi: 10.1109/JSAC.2020.3000879
[9]	Z. Y. Xiao, L. P. Zhu, and X. G. Xia, “UAV communications with millimeter-wave beamforming: Potentials, scenarios, and challenges,” China Communications, vol.17, no.9, pp.147–166, 2020. doi: 10.23919/JCC.2020.09.012
[10]	Q. Wei, L. Wang, L. M. Xu, et al., “Hierarchical coded caching for multiscale content sharing in heterogeneous vehicular networks,” IEEE Transactions on Vehicular Technology, vol.71, no.6, pp.5770–5786, 2022. doi: 10.1109/TVT.2022.3158363
[11]	J. Hejselbæk, J. Ø. Nielsen, W. Fan, et al., “Empirical study of near ground propagation in forest terrain for internet-of-things type device-to-device communication,” IEEE Access, vol.6, pp.54052–54063, 2018. doi: 10.1109/ACCESS.2018.2871368
[12]	M. A. Weissberger, “An initial critical summary of models for predicting the attenuation of radio waves by trees,” Technical Report, ESD-TR-81-101, Electromagnetic Compatibility Analysis Center, Annapolis, MD, USA, 1982.
[13]	G. A. J. Van Dooren, H. J. F. G. Govaerts, and M. H. A. J. Herben, COST 235: Radiowave Propagation Effects on Next-Generation Fixed-Services Terrestrial Telecommunications Systems, Technische Universiteit Eindhoven, 1997.
[14]	ITU-R P.833-9 (09/2016): 2016, Attenuation in Vegetation, P Series: Radio propagation, Available at: https://www.itu.int/rec/R-REC-P.833-9-201609-S/en.
[15]	F. Teschl, F. P. Fontan, M. Schonhuber, et al., “Attenuation of spruce, pine, and deciduous woodland at C-band,” IEEE Antennas and Wireless Propagation Letters, vol.11, pp.109–112, 2012. doi: 10.1109/LAWP.2012.2184253
[16]	X. Z. Lu, L. Xiao, and C. H. Dai, UAV-aided 5G communications with deep reinforcement learning against jamming, arXiv preprint, arXiv: 1805.06628, 2018.
[17]	Y. W. Liu, Z. J. Qin, Y. L. Cai, et al., “UAV communications based on non-orthogonal multiple access,” IEEE Wireless Communications, vol.26, no.1, pp.52–57, 2019. doi: 10.1109/MWC.2018.1800196
[18]	C. H. Liu, Z. Y. Chen, J. Tang, et al., “Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach,” IEEE Journal on Selected Areas in Communications, vol.36, no.9, pp.2059–2070, 2018. doi: 10.1109/JSAC.2018.2864373
[19]	J. B. Seo, S. Y. Kim, and V. C. M. Leung, “Outage probability characterization of CoMP-joint transmission with path-loss and Rayleigh fading,” IEEE Communications Letters, vol.19, no.1, pp.78–81, 2015. doi: 10.1109/LCOMM.2014.2377039
[20]	Y. Z. Jiang, F. L. Chung, S. T. Wang, et al., “Collaborative fuzzy clustering from multiple weighted views,” IEEE Transactions on Cybernetics, vol.45, no.4, pp.688–701, 2015. doi: 10.1109/TCYB.2014.2334595
[21]	P. Bayan, S. Ghazi-Maghrebi, and A. Rahmanian, “A new radio wave propagation model optimization approach for Tehran city based on standard model tuning,” in Proceedings of the 2018 International Symposium on Networks, Computers and Communications (ISNCC), Rome, Italy, pp.1–5, 2018.