Trajectory Optimization and Power Allocation Scheme Based on DRL in Energy Efficient UAV-Aided Communication Networks

WANG Chaowei; CUI Yuling; DENG Danhao; WANG Weidong; JIANG Fan

doi:10.1049/cje.2021.00.314

Volume 31 Issue 3

May 2022

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2022 > 31(3): 397-407

WANG Chaowei, CUI Yuling, DENG Danhao, et al., “Trajectory Optimization and Power Allocation Scheme Based on DRL in Energy Efficient UAV-Aided Communication Networks,” Chinese Journal of Electronics, vol. 31, no. 3, pp. 397-407, 2022, doi: 10.1049/cje.2021.00.314

Citation:

WANG Chaowei, CUI Yuling, DENG Danhao, et al., “Trajectory Optimization and Power Allocation Scheme Based on DRL in Energy Efficient UAV-Aided Communication Networks,” Chinese Journal of Electronics, vol. 31, no. 3, pp. 397-407, 2022, doi: 10.1049/cje.2021.00.314

Citation:

PDF( 2905 KB)

Trajectory Optimization and Power Allocation Scheme Based on DRL in Energy Efficient UAV-Aided Communication Networks

doi: 10.1049/cje.2021.00.314

WANG Chaowei^{1, 2
,},
CUI Yuling^1
,,
DENG Danhao^1
,,
WANG Weidong^{1, 2
,},
JIANG Fan^3
,

1.
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China
2.
Key Laboratory of Universal Wireless Communications, Ministry of Education, Beijing 100876, China
3.
School of Telecommunications and Information Engineering, Xi’an University of Posts and Telecommunications, Xi’an 710121, China

Funds: This work was supported by the National Key R&D Program of China (2020YFB1807204)

More Information

Author Bio:
was born in Sichuan Province in 1982. He received the Ph.D. degree from Beijing University of Posts and Telecommunications in 2010. He is currently an Associate Professor at the School of Electronic Engineering, Beijing University of Posts and Telecommunications. His research interests include wireless communications and IoT applications. (Email: wangchaowei@bupt.edu.cn)

was born in Hebei Province in 1996. She is pursuing the M.S. degree in electronics and communication engineering at Beijing University of Posts and Telecommunications. Her research interests include UAV-assisted communications and resource management. (Email: cuiyuling@bupt.edu.cn)

was born in Hebei Province in 1996. She is currently pursuing the Ph.D. degree with Beijing University of Posts and Telecommunications. Her research interests include unmanned aerial vehicle communication networks, space-earth convergence networks and resource management. (Email: dengdanhao@bupt.edu.cn)

(corresponding author) was born in Inner Mongolia in 1967. He received the Ph.D. degree from Beijing University of Posts and Telecommunications in 2002. He is currently a Full Professor of School of Electronic Engineering at Beijing University of Posts and Telecommunications. His research interests include communication system, radio resource management, Internet of things and signal processing. (Email: wangweidong@bupt.edu.cn)

was born in Shaan-xi Province in 1982. She received the Ph.D. degree from Beijing University of Posts and Telecommunications in 2010. She is currently a Full Professor at Xi’an University of Posts and Telecommunications. Her research interests include wireless and mobile communications. (Email: fjiangwbc@gmail.com)
Received Date: 2021-08-31
Accepted Date: 2021-09-24

Available Online: 2022-03-08

Publish Date: 2022-05-05

Abstract

Abstract

With flexibility, convenience and mobility, unmanned aerial vehicles (UAVS) can provide wireless communication networks with lower costs, easier deployment, higher network scalability and larger coverage. This paper proposes the deep deterministic policy gradient algorithm to jointly optimize the power allocation and flight trajectory of UAV with constrained effective energy to maximize the downlink throughput to ground users. To validate the proposed algorithm, we compare with the random algorithm, Q-learning algorithm and deep Q network algorithm. The simulation results show that the proposed algorithm can effectively improve the communication quality and significantly extend the service time of UAV. In addition, the downlink throughput increases with the number of ground users.
- Unmanned aerial vehicles,
- Deep deterministic policy gradient,
- Trajectory optimization,
- Power allocation,
- Energy efficient

FullText(HTML)

References(51)

References

[1]	Y. Du, H. Zhu, H. Yang, et al., “Review of 5G mobile communication technology standard,” Telecommunications Science, vol.34, no.8, pp.2–9, 2018.
[2]	S. Jin, X. Zhang, and Q. Zhou, “A statistical model for the UAV communication channel,” Acta Aeronautica et Astronautica Sinica, vol.25, no.1, pp.62–65, 2004.
[3]	Zeng Y, Lyu J, Zhang R, “Cellular-connected UAV: Potential, challenges, and promising technologies,” IEEE Wireless Communications, vol.26, no.1, pp.120–127, 2018. doi: 10.1109/MWC.2018.1800023
[4]	Y. Zeng, R. Zhang, and T. J. Lim, “Wireless communications with unmanned aerial vehicles: Opportunities and challenges,” IEEE Commun. Mag., vol.54, no.5, pp.36–42, 2016.
[5]	J. Zhao, F. Gao, G. Ding, et al., “Integrating communications and control for UAV systems: Opportunities and challenges,” IEEE Access, vol.6, pp.67519–67527, 2018. doi: 10.1109/ACCESS.2018.2879637
[6]	M. Mozaffari, W. Saad, M. Bennis, et al., “Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage,” IEEE Commun. Lett., vol.20, no.8, pp.1647–1650, 2016. doi: 10.1109/LCOMM.2016.2578312
[7]	M. Alzenad, A. El-Keyi, F. Lagum, et al., “3-D placement of an unmanned aerial vehicle base station (UAV-BS) for energyefficient maximal coverage,” IEEE Wireless Communications Letters, vol.6, no.4, pp.434–437, 2017. doi: 10.1109/LWC.2017.2700840
[8]	R. I. Bor-Yaliniz, A. El-Keyi, and H. Yanikomeroglu, “Efficient 3-D placement of an aerial base station in next generation cellular networks,” in Proc. of 2016 IEEE International Conference on Communications (ICC), Kuala Lumpur, Malaysia, pp.1–5, 2016.
[9]	J. Lyu, Y. Zeng, R. Zhang, et al., “Placement optimization of UAV-mounted mobile base stations,” IEEE Commun. Lett., vol.21, no.3, pp.604–607, 2017. doi: 10.1109/LCOMM.2016.2633248
[10]	J. Zhao, F. Gao, Q. Wu, S. Jin, et al., “Beam tracking for UAV mounted satcom on-the-move with massive antenna array,” IEEE Journal on Selected Areas in Communications, vol.36, no.2, pp.363–375, 2018.
[11]	L. Xiao, X. Lu, D. Xu, Y. Tang, et al., “UAV relay in vanets against smart jamming with reinforcement learning,” IEEE Transactions on Vehicular Technology, vol.67, no.5, pp.4087–4097, 2018.
[12]	D. Orfanus, E. P. de Freitas, and F. Eliassen, “Self-organization as a supporting paradigm for military uav relay networks,” IEEE Communications Letters, vol.20, no.4, pp.804–807, 2016.
[13]	N. Cheng, F. Lyu, W. Quan, et al., “Space/aerial-assisted computing offloading for IoT applications: A learning-based approach,” IEEE Journal on Selected Areas in Communications, vol.37, no.5, pp.1117–1129, 2019. doi: 10.1109/JSAC.2019.2906789
[14]	F. Cheng, G. Gui, N. Zhao, et al., “UAV-relayingassisted secure transmission with caching,” IEEE Transactions on Communications, vol.67, no.5, pp.3140–3153, 2019. doi: 10.1109/TCOMM.2019.2895088
[15]	H. Wang, J. Wang, G. Ding, et al., “Spectrum sharing planning for full-duplex uav relaying systems with underlaid D2D communications,” IEEE Journal on Selected Areas in Communications, vol.36, no.9, pp.1986–1999, 2018. doi: 10.1109/JSAC.2018.2864375
[16]	Y. Yu, X. Bu, K. Yang, et al., “UAV-aided low latency mobile edge computing with mmwave backhaul,” in Proc. of 2019 IEEE International Conference on Communications (ICC), Shanghai, China, pp.1–7, 2019.
[17]	M. Erdelj, E. Natalizio, K. R. Chowdhury, and I. F. Akyildiz, “Help from the sky: Leveraging UAVs for disaster management,” IEEE Pervasive Computing, vol.16, no.1, pp.24–32, 2017. doi: 10.1109/MPRV.2017.11
[18]	C. She, C. Liu, T. Q. S. Quek, et al., “UAV-assisted uplink transmission for ultra-reliable and low-latency communications,” in Proc. of 2018 IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, MO, USA, pp.1–6, 2018.
[19]	M. Mozaffari, W. Saad, M. Bennis, et al., “Wireless communication using unmanned aerial vehicles (UAVs): Optimal transport theory for hover time optimization,” IEEE Transactions on Wireless Communications, vol.16, no.12, pp.8052–8066, 2017.
[20]	W. Shi, J. Li, N. Cheng, et al., “Multi-drone 3-D trajectory planning and scheduling in drone-assisted radio access networks,” IEEE Transactions on Vehicular Technology, vol.68, no.8, pp.8145–8158, 2019.
[21]	J. Wang, C. Jiang, Z. Wei, et al., “Joint UAV hovering altitude and power control for space-air-ground IoT networks,” IEEE Internet of Things Journal, vol.6, no.2, pp.1741–1753, 2019. doi: 10.1109/JIOT.2018.2875493
[22]	W. Wang, J. Tang, N. Zhao, X. Liu, et al., “Joint precoding optimization for secure swipt in uav-aided noma networks,” IEEE Trans. Commun., vol.68, no.8, pp.5028–5040, 2020.
[23]	X. Mao, Y. Gu, and W. Yin, “Walk proximal gradient: An energyefficient algorithm for consensus optimization,” IEEE Internet Things J., vol.6, no.2, pp.2048–2060, 2019.
[24]	J. Zhao, X. Guan, X. Li, et al., “Power allocation based on genetic simulated annealing algorithm in cognitive radio networks,” Chin. J. Electron., vol.22, no.1, pp.177–180, 2013.
[25]	F. Cheng, S. Zhang, Z. Li, et al., “UAV trajectory optimization for data offloading at the edge of multiple cells,” IEEE Transactions on Vehicular Technology, vol.67, no.7, pp.6732–6736, 2018.
[26]	J. Zhao, Q. Li, Y. Gong, et al., “Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks,” IEEE Trans. Veh. Tech., vol.68, no.8, pp.7944–7956, 2019.
[27]	X. Xi, X. Cao, P. Yang, J. Chen, et al., “Joint user association and UAV location optimization for UAV-aided communications,” IEEE Wireless Communications Letters, vol.8, no.6, pp.1688–1691, 2019.
[28]	P. Yang, X. Xi, K. Guo, et al., “Proactive UAV network slicing for URLLC and mobile broadband service multiplexing,” IEEE Journal on Selected Areas in Communications, vol.39, no.10, pp.3225–3244, 2021. doi: 10.1109/JSAC.2021.3088624
[29]	L. Zhu, J. Zhang, Z. Xiao, et al., “Millimeter-wave full-duplex UAV relay: Joint positioning, beamforming, and power control,” IEEE Journal on Selected Areas in Communications, vol.38, no.9, pp.2057–2073, 2020.
[30]	Y. Wang, Z. Gao, J. Zhang, et al., “Trajectory design for UAV-based Internet-of-things data collection: A deep reinforcement learning approach,” IEEE Internet of Things Journal, vol.9, no.5, pp.3899–3912, 2022. doi: 10.1109/JIOT.2021.3102185
[31]	X. Yuan, Z. Feng, W. Xu, et al., “Capacity analysis of UAV communications: Cases of random trajectories,” IEEE Transactions on Vehicular Technology, vol.67, no.8, pp.7564–7576, 2018.
[32]	Q. Zhang, M. Jiang, Z. Feng, et al., “IoT enabled UAV: Network architecture and routing algorithm,” IEEE Internet of Things Journal, vol.6, no.2, pp.3727–3742, 2019.
[33]	X. Yuan, Z. Feng, W. Ni, et al., “Connectivity of UAV swarms in 3D spherical spaces under (un)intentional ground interference,” IEEE Transactions on Vehicular Technology, vol.69, no.8, pp.8792–8804, 2020.
[34]	C. Qiu, Z. Wei, X. Yuan, et al., “Multiple UAV-mounted base station placement and user association with joint fronthaul and backhaul optimization,” IEEE Transactions on Communications, vol.68, no.9, pp.5864–5877, 2020.
[35]	J. Chen, P. Chen, Q. Wu, et al., “A game-theoretic perspective on resource management for large-scale UAV communication networks,” China Communications, vol.18, no.1, pp.70–87, 2021.
[36]	Z. Wang, F. Zhou, Y. Wang, et al., “Joint 3D trajectory and resource optimization for a UAV relay-assisted cognitive radio network,” China Communications, vol.18, no.6, pp.184–200, 2021.
[37]	B. Liu, Y. Wan, F. Zhou, et al., “Robust trajectory and beamforming design for cognitive MISO UAV networks,” IEEE Wireless Communications Letters, vol.10, no.2, pp.396–400, 2021.
[38]	Z. Xiao, H. Dong, L. Bai, et al., “Unmanned aerial vehicle base station (UAV-BS) deployment with millimeter-wave beamforming,” IEEE Internet of Things Journal, vol.7, no.2, pp.1336–1349, 2020.
[39]	L. Zhu, J. Zhang, Z. Xiao, et al., “3-D Beamforming for flexible coverage in millimeter-wave UAV communications,” IEEE Wireless Communications Letters, vol.8, no.3, pp.837–840, 2019.
[40]	Z. Xiao, L. Zhu, and X. -G. Xia, “UAV communications with millimeter-wave beamforming: Potentials, scenarios, and challenges,” China Communications, vol.17, no.9, pp.147–166, 2020. doi: 10.23919/JCC.2020.09.012
[41]	L. Zhu, J. Zhang, Z. Xiao, et al., “Millimeter-wave NOMA with user grouping, power allocation and hybrid beamforming,” IEEE Transactions on Wireless Communications, vol.18, no.11, pp.5065–5079, 2019. doi: 10.1109/TWC.2019.2932070
[42]	J. Wang, C. Jiang, Z. Han, et al., “Taking drones to the next level: Cooperative distributed unmanned-aerial-vehicular networks for small and mini drones,” IEEE Vehicular Technology Magazine, vol.12, no.3, pp.73–82, 2017. doi: 10.1109/MVT.2016.2645481
[43]	Y. Zeng and R. Zhang, “Energy-efficient UAV communication with trajectory optimization,” IEEE Trans. Wireless Commun., vol.16, no.6, pp.3747–3760, 2017.
[44]	R. Ding, F. Gao, and X. S. Shen, “3D UAV trajectory design and frequency band allocation for energy-efficient and fair communication: A deep reinforcement learning approach,” IEEE Transactions on Wireless Communications, vol.19, no.12, pp.7796–7809, 2020. doi: 10.1109/TWC.2020.3016024
[45]	X. Li, H. Yao, J. Wang, et al., “A near-optimal UAV-aided radio coverage strategy for dense urban areas,” IEEE Transactions on Vehicular Technology, vol.68, no.9, pp.9098–9109, 2019. doi: 10.1109/TVT.2019.2927425
[46]	V. Mnih, K. Kavukcuoglu, D. Silver, et al., “Human-level control through deep reinforcement learning,” Nature, vol.518, pp.529–533, 2015.
[47]	Z. Mou, Y. Zhang, F. Gao, et al., “Deep reinforcement learning based three-dimensional area coverage with UAV swarm,” IEEE Journal on Selected Areas in Communications, vol.39, no.10, pp.3160–3176, 2021. doi: 10.1109/JSAC.2021.3088718
[48]	C. H. Liu, Z. Chen, J. Tang, et al., “Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach,” IEEE J. Selected Areas in Commun., vol.36, no.9, pp.2059–2070, 2018. doi: 10.1109/JSAC.2018.2864373
[49]	R. Ding, Y. Xu, F. Gao, et al., “Trajectory design and access control for air-ground coordinated communications system with multi-agent deep reinforcement learning,” IEEE Internet of Things, vol.9, no.8, pp.5785–5798, 2022. doi: 10.1109/JIOT.2021.3062091
[50]	Y. Zeng, R. Zhang, and T. J. Lim, “Throughput maximization for UAV-enabled mobile relaying systems,” IEEE Trans. Commun., vol.64, no.12, pp.4983–4996, 2016. doi: 10.1109/TCOMM.2016.2611512
[51]	Y. Zeng, J. Xu, and R. Zhang, “Energy minimization for wireless communication with rotary-wing UAV,” IEEE Transactions on Wireless Communications, vol.18, no.4, pp.2329–2345, 2019. doi: 10.1109/TWC.2019.2902559