MADRL-Based 3D Deployment and User Association of Cooperative mmWave Aerial Base Stations for Capacity Enhancement

ZHAO Yikun; ZHOU Fanqin; FENG Lei; LI Wenjing; YU Peng

doi:10.23919/cje.2021.00.327

Volume 32 Issue 2

Mar. 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2023 > 32(2): 283-294

ZHAO Yikun, ZHOU Fanqin, FENG Lei, et al., “MADRL-Based 3D Deployment and User Association of Cooperative mmWave Aerial Base Stations for Capacity Enhancement,” Chinese Journal of Electronics, vol. 32, no. 2, pp. 283-294, 2023, doi: 10.23919/cje.2021.00.327

Citation:

ZHAO Yikun, ZHOU Fanqin, FENG Lei, et al., “MADRL-Based 3D Deployment and User Association of Cooperative mmWave Aerial Base Stations for Capacity Enhancement,” Chinese Journal of Electronics, vol. 32, no. 2, pp. 283-294, 2023, doi: 10.23919/cje.2021.00.327

Citation:

PDF( 4461 KB)

MADRL-Based 3D Deployment and User Association of Cooperative mmWave Aerial Base Stations for Capacity Enhancement

doi: 10.23919/cje.2021.00.327

1.
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China

Funds: This work was supported by the National Natural Science Foundation of China (61971053)

More Information

Author Bio:
Yikun ZHAO received the B.E. degree in communication engineering from Inner Mongolia University in 2019. She is currently pursuing the Ph.D. degree in computer science and technology in Beijing University of Posts and Telecommunications (BUPT). Her research interest is in aerial base station networks. (Email: yizhao@bupt.edu.cn)

Fanqin ZHOU received the Ph.D. degree in automation from BUPT, China, in 2019. He is currently a Lecturer with the State Key Laboratory of Networking and Switching Technology in BUPT. His current research interests include intelligent network slicing and resource management of mobile edge networks

Lei FENG (corresponding author) received the B.E. and Ph.D. degrees in communication and information systems from BUPT in 2009 and 2015. He is an Associated Professor at present in State Key Laboratory of Networking and Switching Technology, BUPT. His research interests are resources management in wireless network and smart grid. (Email: fenglei@bupt.edu.cn)

Wenjing LI is a Professor at BUPT and serves as a Director in the Key Laboratory of Network Management Research Center. Meanwhile, she is the Leader of TC7/WG1 in the China Communications Standards Association (CCSA). Her research interests are wireless network management and automatic healing in SONs

Peng YU received the B.E. and Ph.D. degrees from BUPT in 2008 and 2013 respectively. He is an Associate Professor at present in State Key Laboratory of Networking and Switching Technology, BUPT. His research interests are intelligent and green network management for 5G/6G networks and smart grid communication networks
Received Date: 2021-08-31
Accepted Date: 2022-03-01

Available Online: 2022-03-12

Publish Date: 2023-03-05

Abstract

Abstract

Although millimeter-wave aerial base station (mAeBS) gains rich wireless capacity, it is technically difficult for deploying several mAeBSs to solve the surge of data traffic in hotspots when considering the amount of interference from neighboring mAeBS. This paper introduces coordinated multiple points transmission (CoMP) into the mAeBS-assisted network for capacity enhancement and designs a two-timescale approach for three-dimensional (3D) deployment and user association of cooperative mAeBSs. Specially, an affinity propagation clustering based mAeBS-user cooperative association scheme is conducted on a large timescale followed by modeling the capacity evaluation, and a deployment algorithm based on multi-agent (MA) deep deterministic policy gradient (MADDPG) is designed on the small timescale to obtain the 3D position of mAeBS in a distributed manner. Simulation results show that the proposed approach has significant throughput gains over conventional schemes without CoMP, and the MADDPG is more efficient than centralized deep reinforcement learning (DRL) algorithms in deriving the solution.
- Aerial base station,
- mmWave,
- Capacity enhancement,
- Cooperative communication,
- Multi-agent deep reinforcement learning (MADRL)

FullText(HTML)

References(38)

References

[1]	L. Zhu, Z. Xiao, X. -G. Xia, et al., “Millimeter-wave communications with non-orthogonal multiple access for B5G/6G,” IEEE Access, vol.7, pp.116123–116132, 2019.
[2]	Z. Xiao, L. Zhu, Y. Liu, et al., “A survey on millimeter-wave beamforming enabled UAV communications and networking,” IEEE Communications Surveys & Tutorials, vol.24, no.1, pp.557–610, 2022.
[3]	Z. Xiao, L. Zhu, and X. -G. Xia, “UAV communications with millimeter-wave beamforming: Potentials, scenarios, and challenges,” China Communications, vol.17, no.9, pp.147–166, 2020. doi: 10.23919/JCC.2020.09.012
[4]	L. Zhang, H. Zhao, S. Hou, et al., “A survey on 5G millimeter wave communications for UAV-assisted wireless networks,” IEEE Access, vol.7, pp.117460–117504, 2019.
[5]	X. Zhao, Y. Zhang, P. Qin, et al., “Key technologies and development trends for a space-air-ground integrated wireless optical communication network,” Acta Electronica Sinica, vol.50, no.1, pp.1–17, 2022. (in Chinese)
[6]	L. Zhu, J. Zhang, Z. Xiao, et al., “Optimization of multi-UAV-BS aided millimeter-wave massive MIMO networks,” in Proceedings of 2020 IEEE Global Communications Conference, Taipei, China, pp.1–6, 2020.
[7]	S. Kumar, S. Suman, and S. De, “Dynamic resource allocation in UAV-enabled mmWave communication networks,” IEEE Internet of Things Journal, vol.8, no.12, pp.9920–9933, 2021. doi: 10.1109/JIOT.2020.3027476
[8]	D. Lee, H. Seo, B. Clerckx, et al., “Coordinated multipoint transmission and reception in LTE-advanced: Deployment scenarios and operational challenges,” IEEE Communications Magazine, vol.50, no.2, pp.148–155, 2012. doi: 10.1109/MCOM.2012.6146494
[9]	Q. Cui, H. Song, H. Wang, et al., “Capacity analysis of joint transmission CoMP with adaptive modulation,” IEEE Transactions on Vehicular Technology, vol.66, no.2, pp.1876–1881, 2017. doi: 10.1109/TVT.2016.2564106
[10]	P. Peng, F. Zhu, Q. Liu, et al., “Achieving safe deep reinforcement learning via environment comprehension mechanism,” Chinese Journal of Electronics, vol.30, no.6, pp.1049–1058, 2021. doi: 10.1049/cje.2021.07.025
[11]	H. Bayerlein, R. Gangula, and D. Gesbert, “Learning to rest: A Q-learning approach to flying base station trajectory design with landing spots,” in Proceedings of 2018 52nd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, pp.724–728, 2018.
[12]	P. Yu, J. Guo, Y. Huo, et al., “Three-dimensional aerial base station location for sudden traffic with deep reinforcement learning in 5G mmWave networks,” International Journal of Distributed Sensor Networks, vol.16, no.5, DOI: 10.1177/1550147720926374, 2020.
[13]	Q. Wang, W. Zhang, Y. Liu, et al., “Multi-UAV dynamic wireless networking with deep reinforcement learning,” IEEE Communications Letters, vol.23, no.12, pp.2243–2246, 2019. doi: 10.1109/LCOMM.2019.2940191
[14]	X. Liu, Y. Liu, Y. Chen, et al., “Machine learning aided trajectory design and power control of multi-UAV,” in Proceedings of 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA, pp.1–6, 2019.
[15]	Z. Xiao, P. Xia, and X. -G. Xia, “Enabling UAV cellular with millimeter-wave communication: potentials and approaches,” IEEE Communications Magazine, vol.54, no.5, pp.66–73, 2016. doi: 10.1109/MCOM.2016.7470937
[16]	S. G. Sanchez, S. Mohanti, D. Jaisinghani, et al., “Millimeter-wave base stations in the sky: An experimental study of UAV-to-ground communications,” IEEE Transactions on Mobile Computing, vol.21, no.2, pp.644–662, 2022. doi: 10.1109/TMC.2020.3013575
[17]	M. Gapeyenko, D. Moltchanov, S. Andreev, et al., “Line-of-sight probability for mmWave-based UAV communications in 3D urban grid deployments,” IEEE Transactions on Wireless Communications, vol.20, no.10, pp.6566–6579, 2021. doi: 10.1109/TWC.2021.3075099
[18]	L. Zhu, J. Zhang, Z. Xiao, et al., “3D beamforming for flexible coverage in millimeter-wave UAV communications,” IEEE Wireless Communications Letters, vol.8, no.3, pp.837–840, 2019. doi: 10.1109/LWC.2019.2895597
[19]	H. -L. Chiang, K. -C. Chen, W. Rave, et al., “Machine-learning beam tracking and weight optimization for mmWave multi-UAV links,” IEEE Transactions on Wireless Communications, vol.20, no.8, pp.5481–5494, 2021. doi: 10.1109/TWC.2021.3068206
[20]	F. Zhou, W. Li, L. Meng, et al., “Capacity enhancement for hotspot area in 5G cellular networks using mmWave aerial base station,” IEEE Wireless Communications Letters, vol.8, no.3, pp.677–680, 2019. doi: 10.1109/LWC.2018.2882445
[21]	X. Guo, Y. Chen, and Y. Wang, “Learning-based robust and secure transmission for reconfigurable intelligent surface aided millimeter wave UAV communications,” IEEE Wireless Communications Letters, vol.10, no.8, pp.1795–1799, 2021. doi: 10.1109/LWC.2021.3081464
[22]	L. Zhu, J. Zhang, Z. Xiao, et al., “Millimeter-wave NOMA with user grouping, power allocation and hybrid beamforming,” IEEE Transactions on Wireless Communications, vol.18, no.11, pp.5065–5079, 2019. doi: 10.1109/TWC.2019.2932070
[23]	Y. Liu, K. Liu, J. Han, et al., “Resource allocation and 3-D placement for UAV-enabled energy-efficient IoT communications,” IEEE Internet of Things Journal, vol.8, no.3, pp.1322–1333, 2021. doi: 10.1109/JIOT.2020.3003717
[24]	X. Liu, Y. Liu, and Y. Chen, “Reinforcement learning in multiple-UAV networks: Deployment and movement design,” IEEE Transactions on Vehicular Technology, vol.68, no.8, pp.8036–8049, 2019. doi: 10.1109/TVT.2019.2922849
[25]	I. Valiulahi and C. Masouros, “Multi-UAV deployment for throughput maximization in the presence of co-channel interference,” IEEE Internet of Things Journal, vol.8, no.5, pp.3605–3618, 2021. doi: 10.1109/JIOT.2020.3023010
[26]	C. Shen, T. Chang, J. Gong, et al., “Multi-UAV interference coordination via joint trajectory and power control,” IEEE Transactions on Signal Processing, vol.68, pp.843–858, 2020. doi: 10.1109/TSP.2020.2967146
[27]	B J. Frey and D. Dueck, “Clustering by passing messages between data points,” Science, vol.315, no.5814, pp.972–976, 2007. doi: 10.1126/science.1136800
[28]	H. Zhang, C. Jiang, J. Cheng, et al., “Cooperative interference mitigation and handover management for heterogeneous cloud small cell networks,” IEEE Wireless Communications, vol.22, no.3, pp.92–99, 2015. doi: 10.1109/MWC.2015.7143331
[29]	H. Zhang, H. Liu, C. Jiang, et al., “A practical semidynamic clustering scheme using affinity propagation in cooperative picocells,” IEEE Transactions on Vehicular Technology, vol.64, no.9, pp.4372–4377, 2015. doi: 10.1109/TVT.2014.2361931
[30]	J. Zhang, G. Chuai, and W. Gao, “Power control and clustering-based interference management for UAV-assisted networks,” Sensors, vol.20, no.14, article no.3864, 2020. doi: 10.3390/s20143864
[31]	C. Qiu, Z. Wei, X. Yuan, et al., “Multiple UAV-mounted base station placement and user association with joint fronthaul and backhaul optimization,” IEEE Transactions on Communications, vol.68, no.9, pp.5864–5877, 2020. doi: 10.1109/TCOMM.2020.3001136
[32]	A. Alzidaneen, A. Alsharoa, and M. Alouini, “Resource and placement optimization for multiple UAVs using backhaul tethered balloons,” IEEE Wireless Communications Letters, vol.9, no.4, pp.543–547, 2020. doi: 10.1109/LWC.2019.2961906
[33]	R. Lowe, Y. Wu, A. Tamar, et al., “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Long Beach, CA, USA, pp.6382–6393, 2017.
[34]	Y. Zhang, Z. Mou, F. Gao, et al., “UAV-enabled secure communications by multi-agent deep reinforcement learning,” IEEE Transactions on Vehicular Technology, vol.69, no.10, pp.11599–11611, 2020. doi: 10.1109/TVT.2020.3014788
[35]	L. Wang, K. Wang, C. Pan, et al., “Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing,” IEEE Transactions on Cognitive Communications and Networking, vol.7, no.1, pp.73–84, 2021. doi: 10.1109/TCCN.2020.3027695
[36]	A. Gao, Q. Wang, W. Liang, et al., “Game combined multi-agent reinforcement learning approach for UAV assisted offloading,” IEEE Transactions on Vehicular Technology, vol.70, no.12, pp.12888–12901, 2021. doi: 10.1109/TVT.2021.3121281
[37]	J. Yao and J. Xu, “Joint 3D maneuver and power adaptation for secure UAV communication with CoMP reception,” IEEE Transactions on Wireless Communications, vol.19, no.10, pp.6992–7006, 2020. doi: 10.1109/TWC.2020.3007648
[38]	A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum coverage,” IEEE Wireless Communications Letters, vol.3, no.6, pp.569–572, 2014. doi: 10.1109/LWC.2014.2342736