Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method

WU Qiong; SHI Shuai; WAN Ziyang; FAN Qiang; FAN Pingyi; ZHANG Cui

doi:10.23919/cje.2022.00.093

Volume 32 Issue 6

Nov. 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2023 > 32(6): 1230-1244

WU Qiong, SHI Shuai, WAN Ziyang, et al., “Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method,” Chinese Journal of Electronics, vol. 32, no. 6, pp. 1230-1244, 2023, doi: 10.23919/cje.2022.00.093

Citation:

WU Qiong, SHI Shuai, WAN Ziyang, et al., “Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method,” Chinese Journal of Electronics, vol. 32, no. 6, pp. 1230-1244, 2023, doi: 10.23919/cje.2022.00.093

Citation:

PDF( 4020 KB)

Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method

doi: 10.23919/cje.2022.00.093

1.
School of Internet of Things Engineering, Jiangnan University, Wuxi 214122, China
2.
State Key Laboratory of Integrated Services Networks, Xidian University, Xi’an 710071, China
3.
Qualcomm, San Jose CA 95110, USA
4.
Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
5.
Banma Network Technology Co., Ltd., Shanghai 200000, China

Funds: This work was supported by the National Natural Science Foundation of China (61701197), the Open Research Fund of State Key Laboratory of Integrated Services Networks (ISN23-11), the National Key Research and Development Program of China (2021YFA1000500(4)), and the 111 Project (B12018)

More Information

Author Bio:
Qiong WU received the Ph.D. degree in information and communication engineering from the National Mobile Communications Research Laboratory, Southeast University, Nanjing, China, in 2016. From 2018 to 2020, he was a Postdoctoral Researcher with the Department of Electronic Engineering, Tsinghua University. He is currently an Associate Professor with the School of Internet of Things Engineering, Jiangnan University, Wuxi, China. His current research interest focuses autonomous driving communication technology. (Email: qiongwu@jiangnan.edu.cn)

Shuai SHI was born in Shandong Province, China. He received the B.S. degree from Dezhou University, Dezhou, China, in 2019. He is currently working toward the M.S. degree at Jiangnan University. His current research interests include data freshness and fairness in autonomous driving communications and federated learning for edge network. (Email: shuaishi@stu.jiangnan.edu.cn)

Ziyang WAN was born in Shandong Province, China. He received the B.S. degree from Ludong University, China, in 2019. He is currently working toward the M.S. degree at Jiangnan University. His current research interests include data freshness and fairness in autonomous driving communications. (Email: ziyangwan@stu.jiangnan.edu.cn)

Qiang FAN received the Ph.D. degree in electrical and computer engineering from New Jersey Institute of Technology in 2019, and the M.S. degree in electrical engineering from Yunnan University of Nationalities, China, in 2013. He was a Postdoctor Researcher in the Department of Electrical and Computer Engineering, Virginia Tech. Currently, he is a Staff Engineer in Qualcomm, USA. His current research interests include wireless communications and networking, mobile edge computing, machine learning and drone assisted networking. (Email: qiangfan29@gmail.com)

Pingyi FAN (corresponding author) received the B.S. and M.S. degrees from the Department of Mathematics of Hebei University in 1985 and Nankai University in 1990, respectively, received the Ph.D. degree from the Department of Electronic Engineering, Tsinghua University, Beijing, China, in 1994. He is a Professor of Department of Electronic Engineering of Tsinghua University currently. Dr. Fan is a Senior Member of IEEE and an Oversea Member of IEICE. He has attended to organize many international conferences including as TPC Co-chair of Chinacom 2020, IEEE WCNIS 2010 and TPC Member of IEEE ICC, Globecom, WCNC, VTC, Infocom, etc. He has served as an Editor of several journals for IEEE, Inderscience, Wiley, MDPI, etc. He is also a reviewer of more than 30 international journals including 24 IEEE journals and 8 EURASIP journals. He has received some academic awards, including the IEEE ICC20, TAOS20, Globecom14, WCNC08 Best Paper Awards, ACM IWCMC10 Best Paper Award and IEEE ComSoc Excellent Editor Award for IEEE Transactions on Wireless Communications in 2009. His main research interests include B5G technology in wireless communications, machine learning and AI in wireless communications, information theory in big data analysis, network coding and network information theory, etc. (Email: fpy@tsinghua.edu.cn)

Cui ZHANG received the Ph.D. degree from Institute of Information Engineering, CAS, in 2017. From 2017 to 2021, she worked in Huawei 2012 Laboratory for Trustworthy System and Formal Verification Research. Currently, she is an Expert at security technique of real-time operating system (RTOS) for intelligent driving in Banma Network Technology Co., Ltd. Her main research interests include mobile computing, RTOS, security and formal verification. (Email: faircas85@163.com)
Received Date: 2022-04-20
Accepted Date: 2022-06-21

Available Online: 2022-11-04

Publish Date: 2023-11-05

Abstract

Abstract

Vehicles on the road exchange data with base station frequently through vehicle to infrastructure (V2I) communications to ensure the normal use of vehicular applications, where the IEEE 802.11 distributed coordination function is employed to allocate a minimum contention window (MCW) for channel access. Each vehicle may change its MCW to achieve more access opportunities at the expense of others, which results in unfair communication performance. Moreover, the key access parameter MCW is privacy information and each vehicle is not willing to share it with other vehicles. In this uncertain setting, age of information (AoI), which measures the freshness of data and is closely related with fairness, has become an important communication metric. On this basis, we design an intelligent vehicular node to learn the dynamic environment and predict the optimal MCW, which can make the intelligent node achieve age fairness. In order to allocate the optimal MCW for the vehicular node, we employ a learning algorithm to make a desirable decision by learning from replay history data. In particular, the algorithm is proposed by extending the traditional deep-Q-learning (DQN) training and testing method. Finally, by comparing with other methods, it is proved that the proposed DQN method can significantly improve the age fairness of the intelligent node.
- Vehicles,
- Fair,
- Chennel access,
- Age of information (AoI),
- Deep-Q-learning

1Simulation codes are provided to reproduce the results in this paper: https://github.com/qiongwu86/Age-Fairness

FullText(HTML)

References(37)

References

[1]	Q. Wu, H. X. Liu, C. Zhang, et al., “Trajectory protection schemes based on a gravity mobility model in IoT,” Electronics, vol.8, no.2, article no.148, 2019. doi: 10.3390/electronics8020148
[2]	H. B. Zhu, Q. Wu, X. J. Wu, et al., “Decentralized power allocation for MIMO-NOMA vehicular edge computing based on deep reinforcement learning,” IEEE Internet of Things Journal, vol.9, no.14, pp.12770–12782, 2022. doi: 10.1109/JIOT.2021.3138434
[3]	J. H. Zhao, Q. P. Li, Y. Gong, et al., “Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks,” IEEE Transactions on Vehicular Technology, vol.68, no.8, pp.7944–7956, 2019. doi: 10.1109/TVT.2019.2917890
[4]	Q. Wu and J. Zheng, “Performance modeling and analysis of the ADHOC MAC protocol for VANETs,” in Proceedings of IEEE International Conference on Communication (ICC), London, UK, pp.3646–3652, 2015.
[5]	Q. Wu, Y. Zhao, and Q. Fan, “Time-dependent performance modeling for platooning communications at intersection,” IEEE Internet of Things Journal, vol.9, no.19, pp.18500–18513, 2022. doi: 10.1109/JIOT.2022.3161028
[6]	Defense Advanced Research Projects Agency, “Spectrum collaboration challenge (SC2),” Available at: https://www.darpa.mil/program/spectrum-collaboration-challenge, 2023.
[7]	J. H. Zhao, L. H. Yang, M. H. Xia, et al., “Unified analysis of coordinated multipoint transmissions in mmWave cellular networks,” IEEE Internet of Things Journal, vol.9, no.14, pp.12166–12180, 2022. doi: 10.1109/JIOT.2021.3134017
[8]	Q. Wu and J. Zheng, “Performance modeling and analysis of the ADHOC MAC protocol for vehicular networks,” Wireless Networks, vol.22, no.3, pp.799–812, 2016. doi: 10.1007/s11276-015-1000-6
[9]	J. H. Zhao, X. K. Sun, Q. P. Li, et al., “Edge caching and computation management for real-time Internet of vehicles: An online and distributed approach,” IEEE Transactions on Intelligent Transportation Systems, vol.22, no.4, pp.2183–2197, 2021. doi: 10.1109/TITS.2020.3012966
[10]	Q. Wu, H. M. Ge, P. Y. Fan, et al., “Time-dependent performance analysis of the 802.11p-based platooning communications under disturbance,” IEEE Transactions on Vehicular Technology, vol.69, no.12, pp.15760–15773, 2020. doi: 10.1109/TVT.2020.3034622
[11]	W. W. Lu, S. L. Gong, Y. H. Zhu, “Timely data delivery for energy-harvesting IoT devices,” Chinese Journal of Electronics, vol.31, no.2, pp.322–336, 2022. doi: 10.1049/cje.2021.00.005
[12]	R. D. Yates, Y. Sun, D. R. Brown, et al., “Age of information: An introduction and survey,” IEEE Journal on Selected Areas in Communications, vol.39, no.5, pp.1183–1210, 2021. doi: 10.1109/JSAC.2021.3065072
[13]	X. F. Chen, C. Wu, T. Chen, et al., “Age of information aware radio resource management in vehicular networks: A proactive deep reinforcement learning perspective,” IEEE Transactions on Wireless Communications, vol.19, no.4, pp.2268–2281, 2020. doi: 10.1109/TWC.2019.2963667
[14]	M. K. Abdel-Aziz, S. Samarakoon, C. F. Liu, et al., “Optimized age of information tail for ultra-reliable low-latency communications in vehicular networks,” IEEE Transactions on Communications, vol.68, no.3, pp.1911–1924, 2020. doi: 10.1109/TCOMM.2019.2961083
[15]	Q. Wu, Z. Y. Wan, Q. Fan, et al., “Velocity-adaptive access scheme for MEC-assisted platooning networks: Access fairness via data freshness,” IEEE Internet of Things Journal, vol.9, no.6, pp.4229–4244, 2022. doi: 10.1109/JIOT.2021.3103325
[16]	J. Lv, X. M. Zhang, X. J. Han, et al., “A novel adaptively dynamic tuning of the contention window (CW) for distributed coordination function in IEEE 802.11 Ad hoc networks,” in Proceedings of 2007 International Conference on Convergence Information Technology (ICCIT 2007), Gwangju, Korea (South), pp.290–294, 2007.
[17]	C. Wu, S. Ohzahata, Y. S. Ji, et al., “A MAC protocol for delay-sensitive VANET applications with self-learning contention scheme,” in Proceedings of 2014 IEEE 11th Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA, pp.438–443, 2014.
[18]	A. Jamali, S. M. S. Hemami, M. Berenjkoub, et al., “An adaptive MAC protocol for wireless LANs,” Journal of Communications and Networks, vol.16, no.3, pp.311–321, 2014. doi: 10.1109/JCN.2014.000052
[19]	X. Zhou, C. W. Zheng, and X. X. He, “Adaptive contention window tuning for IEEE 802.11,” in Proceedings of 2015 22nd International Conference on Telecommunications (ICT), Sydney, NSW, Australia, pp.74–79, 2015.
[20]	A. Pressas, Z. G. Sheng, F. Ali, et al., “Contention-based learning MAC protocol for broadcast vehicle-to-vehicle communication,” in Proceedings of 2017 IEEE Vehicular Networking Conference (VNC), Torino, Italy, pp.263–270, 2017.
[21]	C. Wu, Z. Liu, F. Q. Liu, et al., “Collaborative learning of communication routes in edge-enabled multi-access vehicular environment,” IEEE Transactions on Cognitive Communications and Networking, vol.6, no.4, pp.1155–1165, 2020. doi: 10.1109/TCCN.2020.3002253
[22]	X. W. Wu, X. H. Li, J. Li, et al., “Deep reinforcement learning for IoT networks: Age of information and energy cost tradeoff,” in Proceedings of GLOBECOM 2020–2020 IEEE Global Communications Conference, Taipei, China, pp.1–6, 2020.
[23]	F. Y. Wu, H. L. Zhang, J. J. Wu, et al., “UAV-to-device underlay communications: Age of information minimization by multi-agent deep reinforcement learning,” IEEE Transactions on Communications, vol.69, no.7, pp.4461–4475, 2021. doi: 10.1109/TCOMM.2021.3065135
[24]	S. H. Wang, M. Z. Chen, W. Saad, et al., “Reinforcement learning for minimizing age of information under realistic physical dynamics,” in Proceedings of GLOBECOM 2020–2020 IEEE Global Communications Conference, Taipei, China, pp.1–6, 2020.
[25]	B. Han, Y. Zhu, Z. Y. Jiang, et al., “Fairness for freshness: Optimal age of information based OFDMA scheduling with minimal knowledge,” IEEE Transactions on Wireless Communications, vol.20, no.12, pp.7903–7919, 2021. doi: 10.1109/TWC.2021.3088719
[26]	S. Y. Leng and A. Yener, “An actor-critic reinforcement learning approach to minimum age of information scheduling in energy harvesting networks,” in Proceedings of ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, ON, Canada, pp.8128–8132, 2021.
[27]	I. Kadota and E. Modiano, “Minimizing the age of information in wireless networks with stochastic arrivals,” IEEE Transactions on Mobile Computing, vol.20, no.3, pp.1173–1185, 2021. doi: 10.1109/TMC.2019.2959774
[28]	G. Bianchi, “Performance analysis of the IEEE 802.11 distributed coordination function,” IEEE Journal on Selected Areas in Communications, vol.18, no.3, pp.535–547, 2000. doi: 10.1109/49.840210
[29]	P. Peng, F. Zhu, Q. Liu, et al., “Achieving safe deep reinforcement learning via environment comprehension mechanism,” Chinese Journal of Electronics, vol.30, no.6, pp.1049–1058, 2021. doi: 10.1049/cje.2021.07.025
[30]	R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1998.
[31]	H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA, pp.2094–2100, 2016.
[32]	Y. Zhang, P. Sun, Y. H. Yin, et al., “Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning,” in Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, pp.1251–1256, 2018.
[33]	Z. Y. Wang, T. Schaul, M. Hessel, et al., “Dueling network architectures for deep reinforcement learning,” in Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, pp.1995–2003, 2016.
[34]	M. Hessel, J. Modayil, H. van Hasselt, et al., “Rainbow: Combining improvements in deep reinforcement learning,” in Proceedings of the 32nd AAAI conference on Artificial Intelligence, New Orleans, LA, USA, pp.3215–3222, 2018.
[35]	B. Li, S. Y. Liang, D. Q. Chen, et al., “A decision-making method for air combat maneuver based on hybrid deep learning network,” Chinese Journal of Electronics, vol.31, no.1, pp.107–115, 2022. doi: 10.1049/cje.2020.00.075
[36]	M. Fortunato, M. G. Azar, B. Piot, et al., Noisy networks for exploration, arXiv preprint, arXiv: 1706.10295, 2017.
[37]	H. van Hasselt, A. Guez, M. Hessel, et al., “Learning values across many orders of magnitude,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp.4294–4302, 2016.