Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning

WU Yuqin; SHEN Congqi; CHEN Shuhan; WU Chunming; LI Shunbin; Wei Ruan

doi:10.1049/cje.2020.00.417

Article Contents

Article Navigation > Chinese Journal of Electronics > 2022 > Corrected proof

WU Yuqin, SHEN Congqi, CHEN Shuhan, WU Chunming, LI Shunbin, Wei Ruan. Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning[J]. Chinese Journal of Electronics. doi: 10.1049/cje.2020.00.417

Citation:

WU Yuqin, SHEN Congqi, CHEN Shuhan, WU Chunming, LI Shunbin, Wei Ruan. Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning[J]. Chinese Journal of Electronics. doi: 10.1049/cje.2020.00.417

Citation:

PDF( 4824 KB)

Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning

doi: 10.1049/cje.2020.00.417

WU Yuqin^{1, 2
,},
SHEN Congqi^{3, 4},
CHEN Shuhan^{1, 4},
WU Chunming^{1, 4
,},
LI Shunbin³,
Wei Ruan^5
,

1.
College of Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang 310027, China
2.
College of Information and Electrical Engineering, Ningde Normal University, Ningde, Fujian 352100, China
3.
Zhejiang Lab, Hangzhou, Zhejiang 310012, China
4.
Zhejiang University NGICS Platform
5.
College of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang 310027, China

Funds: The paper is supported by National Key Research and Development Project(2018YFB2100404), Fujian Natural Science Foundation (2020J01431), Ningde Normal University Innovation Team Program(2018T04), the Fundamental Research Funds for the Central Universities (Zhejiang University NGICS Platform: K20200002), and the Major Scientific Project of Zhejiang Lab (2018FD0ZX01)

More Information

Author Bio:
received the BS degree in computer science and technology from the Department of Computer Science and Technology, Xi’an University, Xi’an, China, in 2004. Her research interests include computer network communication and network security, machine learning theory, and IoT. (Email: wuyu-qin@163.com)

received the PhD degree and is a Professor in the Department of Computer Science and Technology, Zhejiang University, China. His research interests include the nextgeneration network, network security, and network virtualization. (Email: wuchunming@cs.zju.edu.cn)
Corresponding author: Wei Ruan is the corresponding author
Received Date: 2020-12-12
Accepted Date: 2021-01-07

Available Online: 2021-11-05

Abstract

Abstract

With the recent increase in the number of Internet of Things (IoT) services, an intelligent scheduling strategy is needed to manage these services. In this paper, the problem of automatic choreography of microservices in IoT is explored. A type of reinforcement learning (RL) algorithm called TD3 is used to generate the optimal choreography policy under the framework of a softwaredefined network. The optimal policy is gradually reached during the learning procedure to achieve the goal, despite the dynamic characteristics of the network environment. The simulation results show that compared with other methods, the TD3 algorithm converges faster after a certain number of iterations, and it performs better than other non-RL algorithms by obtaining the highest reward. The TD3 algorithm can effciently adjust the traffc transmission path and provide qualified IoT services.
- Internet of Things,
- Microservice,
- Traffc Scheduling,
- Reinforcement Learning

FullText(HTML)

References(27)

References

[1]	Ueda T., Nakaike T., Ohara M., "Workload characterization for microservices," IEEE International Symposium on Workload Characterization. Providence, RI, USA, pp. 1-10, 2016.
[2]	Jr A. R. S., Kadiyalax H., Hux B., et al., "Supporting microservice evolution," IEEE International Conference on Software Maintenance & Evolution, Shanghai, China, pp. 529-543, 2017.
[3]	Niu Y., Liu F., Li Z., "Load balancing across microservices," IEEE INFOCOM, Honolulu, HI, USA, pp. 198-206, 2018.
[4]	Drutskoy D., Keller E., Rexford J., “Scalable network virtualization in software-defined networks,” Internet Computing, IEEE, vol.17, no.2, pp.20–27, 2013. doi: 10.1109/MIC.2012.144
[5]	Sarkar C., Nambi S. N. A. U., Prasad R. V., et al. "A scalable distributed architecture for unifying IoT applications," IEEE World Forum on Internet of Things (WF-IoT) , Seoul, Korea (South), pp. 508-513, 2014.
[6]	Madden S., Franklin M. J., Hellerstein J. M., et al., “TAG: A tiny aggregation service for ad-hoc sensor networks,” Acm Sigops Operating Systems Review, vol.36, no.SI, pp.131–146, 2002. doi: 10.1145/844128.844142
[7]	Madden S., Franklin M. J., Hellerstein J. M., et al., “Design of an acquisitional query processor for sensor networks,” Acm Sigmod, New York, NY, USA , pp.491–502, 2003.
[8]	Gummadi R., Gnawali O., Govindan R. "Macro-programming wireless sensor networks using Kairos," Proc of the IEEE International Conference on Distributed Computing in Sensor Systems. DCOSS 2005. Lecture Notes in Computer Science, vol.3560. Springer, Berlin, Heidelberg.
[9]	Koroniotis N., Moustafa N., Sitnikova E, “Forensics and deep learning mechanisms for botnets in Internet of Things: A survey of challenges and solutions,” IEEE Access, vol.7, pp.61764–61785, 2019. doi: 10.1109/ACCESS.2019.2916717
[10]	Patel P, Cassou D, “Enabling high-level application development for the Internet of Things,” Journal of Systems and Software, vol.103, pp.62–84, 2015. doi: 10.1016/j.jss.2015.01.027
[11]	Cassou D, Bertran B, Loriant N, et al. "A generative programming approach to developing pervasive computing systems,"Proceedings of GPCE ’09. ACM, pp.137-146, 2009.
[12]	Katasonov A. "Enabling non-programmers to develop smart environment applications," The IEEE symposium on Computers and Communications,pp. 1059-1064, 2010.
[13]	Thoma M, Meyer S, Sperner K, et al. "On IoT-services: Survey, classification and enterprise integration,"2012 IEEE International Conference on Green Computing and Communications (GreenCom), 2012.
[14]	Guinard D, Trifa V, Karnouskos S, et al., “Interacting with the SOA-based Internet of Things: Discovery, Query, Selection, and On-Demand Provisioning of Web Services,” IEEE Transactions on Services Computing, vol.3, no.3, pp.223–235, 2010. doi: 10.1109/TSC.2010.3
[15]	Resnick M, Maloney J, Andrés Monroy-Hernández, et al., “Scratch: Programming for all,” Communications of the Acm, vol.52, no.11, pp.60–67, 2009. doi: 10.1145/1592761.1592779
[16]	Gans P, “The benefits of using scratch to introduce basic programming concepts in the elementary classroom: Poster session,” Journal of Computing Sciences in Colleges, vol.25, no.6, pp.235–236, 2010. doi: 10.5555/1791129.1791176
[17]	Nordmann A, Hochgeschwender N, Wrede S. "A survey on domain-specific languages in robotics," Simulation, Modeling, and Programming for Autonomous Robots. SIMPAR 2014,Lecture Notes in Computer Science, vol 8810. Springer, Cham.
[18]	H.Y. Li, "Application Service and Resource Management System for Smart Community based on Micro-service Architecture," M.S. Thesis, Shanghai Jiao Tong University, 2016.(in Chinese)
[19]	S. Li, Y. H. Yan, J. Ren, et al. "A sample-efficient actor-critic algorithm for recommendation diversification," Chinese Journal of Electronics, vol. 29, no. 1, pp. 89-96, 2020.
[20]	Z. Xu, J. Tang, J. Meng, et al. "Experience-driven networking: A deep reinforcement learning based approach," IEEE Conference on Computer Communications,Honolulu, HI, USA, pp. 1871-1879, 2018.
[21]	S. C. Lin, I. F. Akyildiz, P. Wang, et al. "QoS-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach," IEEE International Conference on Services Computing, San Francisco, CA, USA, pp. 25-33, 2016.
[22]	Mckeown N, Anderson T, Balakrishnan H, et al., “OpenFlow: Enabling innovation in campus networks,” Acm Sigcomm Computer Communication Review, vol.38, no.2, pp.69–74, 2008. doi: 10.1145/1355734.1355746
[23]	Chun S, Jung S, Yi S, et al. "Method and apparatus for transmitting and receiving signal to and from network at user equipment in a wireless communication system,"Patent, 20220117003 ,USA, 2022.
[24]	Li L E, Mao Z M, Rexford J. "Toward software-defined cellular networks," European Workshop on Software Defined Networking. IEEE Computer Society, pp. 99-106, 2012.
[25]	Donald Gross, John F. Shortle, James M. Thompson, et al., Fundamentals of Queueing Theory, Fourth Edition, Wiley, 2008.
[26]	Balalaie A, Heydarnoori A, Jamshidi P, “Microservices architecture enables DevOps: An experience report on migration to a cloud-native architecture,” IEEE Software,, vol.33, no.3, pp.42–52, 2016.
[27]	Peters J, Schaal S, “Natural actor-critic,” Neurocomputing, vol.71, no.7-9, pp.1180–1190, 2008. doi: 10.1016/j.neucom.2007.11.026