Turn off MathJax
Article Contents
WU Yuqin, SHEN Congqi, CHEN Shuhan, WU Chunming, LI Shunbin, Wei Ruan. Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning[J]. Chinese Journal of Electronics. doi: 10.1049/cje.2020.00.417
Citation: WU Yuqin, SHEN Congqi, CHEN Shuhan, WU Chunming, LI Shunbin, Wei Ruan. Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning[J]. Chinese Journal of Electronics. doi: 10.1049/cje.2020.00.417

Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning

doi: 10.1049/cje.2020.00.417
Funds:  The paper is supported by National Key Research and Development Project(2018YFB2100404), Fujian Natural Science Foundation (2020J01431), Ningde Normal University Innovation Team Program(2018T04), the Fundamental Research Funds for the Central Universities (Zhejiang University NGICS Platform: K20200002), and the Major Scientific Project of Zhejiang Lab (2018FD0ZX01)
More Information
  • Author Bio:

    received the BS degree in computer science and technology from the Department of Computer Science and Technology, Xi’an University, Xi’an, China, in 2004. Her research interests include computer network communication and network security, machine learning theory, and IoT. (Email: wuyu-qin@163.com)

    received the PhD degree and is a Professor in the Department of Computer Science and Technology, Zhejiang University, China. His research interests include the nextgeneration network, network security, and network virtualization. (Email: wuchunming@cs.zju.edu.cn)

  • Corresponding author: Wei Ruan is the corresponding author
  • Received Date: 2020-12-12
  • Accepted Date: 2021-01-07
  • Available Online: 2021-11-05
  • With the recent increase in the number of Internet of Things (IoT) services, an intelligent scheduling strategy is needed to manage these services. In this paper, the problem of automatic choreography of microservices in IoT is explored. A type of reinforcement learning (RL) algorithm called TD3 is used to generate the optimal choreography policy under the framework of a softwaredefined network. The optimal policy is gradually reached during the learning procedure to achieve the goal, despite the dynamic characteristics of the network environment. The simulation results show that compared with other methods, the TD3 algorithm converges faster after a certain number of iterations, and it performs better than other non-RL algorithms by obtaining the highest reward. The TD3 algorithm can effciently adjust the traffc transmission path and provide qualified IoT services.
  • loading
  • [1]
    Ueda T., Nakaike T., Ohara M., "Workload characterization for microservices," IEEE International Symposium on Workload Characterization. Providence, RI, USA, pp. 1-10, 2016.
    Jr A. R. S., Kadiyalax H., Hux B., et al., "Supporting microservice evolution," IEEE International Conference on Software Maintenance & Evolution, Shanghai, China, pp. 529-543, 2017.
    Niu Y., Liu F., Li Z., "Load balancing across microservices," IEEE INFOCOM, Honolulu, HI, USA, pp. 198-206, 2018.
    Drutskoy D., Keller E., Rexford J., “Scalable network virtualization in software-defined networks,” Internet Computing, IEEE, vol.17, no.2, pp.20–27, 2013. doi: 10.1109/MIC.2012.144
    Sarkar C., Nambi S. N. A. U., Prasad R. V., et al. "A scalable distributed architecture for unifying IoT applications," IEEE World Forum on Internet of Things (WF-IoT) , Seoul, Korea (South), pp. 508-513, 2014.
    Madden S., Franklin M. J., Hellerstein J. M., et al., “TAG: A tiny aggregation service for ad-hoc sensor networks,” Acm Sigops Operating Systems Review, vol.36, no.SI, pp.131–146, 2002. doi: 10.1145/844128.844142
    Madden S., Franklin M. J., Hellerstein J. M., et al., “Design of an acquisitional query processor for sensor networks,” Acm Sigmod, New York, NY, USA , pp.491–502, 2003.
    Gummadi R., Gnawali O., Govindan R. "Macro-programming wireless sensor networks using Kairos," Proc of the IEEE International Conference on Distributed Computing in Sensor Systems. DCOSS 2005. Lecture Notes in Computer Science, vol.3560. Springer, Berlin, Heidelberg.
    Koroniotis N., Moustafa N., Sitnikova E, “Forensics and deep learning mechanisms for botnets in Internet of Things: A survey of challenges and solutions,” IEEE Access, vol.7, pp.61764–61785, 2019. doi: 10.1109/ACCESS.2019.2916717
    Patel P, Cassou D, “Enabling high-level application development for the Internet of Things,” Journal of Systems and Software, vol.103, pp.62–84, 2015. doi: 10.1016/j.jss.2015.01.027
    Cassou D, Bertran B, Loriant N, et al. "A generative programming approach to developing pervasive computing systems,"Proceedings of GPCE ’09. ACM, pp.137-146, 2009.
    Katasonov A. "Enabling non-programmers to develop smart environment applications," The IEEE symposium on Computers and Communications,pp. 1059-1064, 2010.
    Thoma M, Meyer S, Sperner K, et al. "On IoT-services: Survey, classification and enterprise integration,"2012 IEEE International Conference on Green Computing and Communications (GreenCom), 2012.
    Guinard D, Trifa V, Karnouskos S, et al., “Interacting with the SOA-based Internet of Things: Discovery, Query, Selection, and On-Demand Provisioning of Web Services,” IEEE Transactions on Services Computing, vol.3, no.3, pp.223–235, 2010. doi: 10.1109/TSC.2010.3
    Resnick M, Maloney J, Andrés Monroy-Hernández, et al., “Scratch: Programming for all,” Communications of the Acm, vol.52, no.11, pp.60–67, 2009. doi: 10.1145/1592761.1592779
    Gans P, “The benefits of using scratch to introduce basic programming concepts in the elementary classroom: Poster session,” Journal of Computing Sciences in Colleges, vol.25, no.6, pp.235–236, 2010. doi: 10.5555/1791129.1791176
    Nordmann A, Hochgeschwender N, Wrede S. "A survey on domain-specific languages in robotics," Simulation, Modeling, and Programming for Autonomous Robots. SIMPAR 2014,Lecture Notes in Computer Science, vol 8810. Springer, Cham.
    H.Y. Li, "Application Service and Resource Management System for Smart Community based on Micro-service Architecture," M.S. Thesis, Shanghai Jiao Tong University, 2016.(in Chinese)
    S. Li, Y. H. Yan, J. Ren, et al. "A sample-efficient actor-critic algorithm for recommendation diversification," Chinese Journal of Electronics, vol. 29, no. 1, pp. 89-96, 2020.
    Z. Xu, J. Tang, J. Meng, et al. "Experience-driven networking: A deep reinforcement learning based approach," IEEE Conference on Computer Communications,Honolulu, HI, USA, pp. 1871-1879, 2018.
    S. C. Lin, I. F. Akyildiz, P. Wang, et al. "QoS-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach," IEEE International Conference on Services Computing, San Francisco, CA, USA, pp. 25-33, 2016.
    Mckeown N, Anderson T, Balakrishnan H, et al., “OpenFlow: Enabling innovation in campus networks,” Acm Sigcomm Computer Communication Review, vol.38, no.2, pp.69–74, 2008. doi: 10.1145/1355734.1355746
    Chun S, Jung S, Yi S, et al. "Method and apparatus for transmitting and receiving signal to and from network at user equipment in a wireless communication system,"Patent, 20220117003 ,USA, 2022.
    Li L E, Mao Z M, Rexford J. "Toward software-defined cellular networks," European Workshop on Software Defined Networking. IEEE Computer Society, pp. 99-106, 2012.
    Donald Gross, John F. Shortle, James M. Thompson, et al., Fundamentals of Queueing Theory, Fourth Edition, Wiley, 2008.
    Balalaie A, Heydarnoori A, Jamshidi P, “Microservices architecture enables DevOps: An experience report on migration to a cloud-native architecture,” IEEE Software,, vol.33, no.3, pp.42–52, 2016.
    Peters J, Schaal S, “Natural actor-critic,” Neurocomputing, vol.71, no.7-9, pp.1180–1190, 2008. doi: 10.1016/j.neucom.2007.11.026
  • 加载中


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索


    Article Metrics

    Article views (125) PDF downloads(16) Cited by()
    Proportional views


    DownLoad:  Full-Size Img  PowerPoint