Volume 31 Issue 5
Sep.  2022
Turn off MathJax
Article Contents
WU Yuqin, SHEN Congqi, CHEN Shuhan, et al., “Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning,” Chinese Journal of Electronics, vol. 31, no. 5, pp. 930-937, 2022, doi: 10.1049/cje.2020.00.417
Citation: WU Yuqin, SHEN Congqi, CHEN Shuhan, et al., “Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning,” Chinese Journal of Electronics, vol. 31, no. 5, pp. 930-937, 2022, doi: 10.1049/cje.2020.00.417

Intelligent Orchestrating of IoT Microservices Based on Reinforcement Learning

doi: 10.1049/cje.2020.00.417
Funds:  This paper was supported by National Key Research and Development Project (2018YFB2100404), the key R&D Program of Zhejiang Province (2020C01077), Fujian Natural Science Foundation (2020J01431), Ningde Normal University Innovation Team Program (2018T04), the Fundamental Research Funds for the Central Universities (Zhejiang University NGICS Platform), and the Major Scientific Project of Zhejiang Lab (2018FD0ZX01)
More Information
  • Author Bio:

    (corresponding author) received the B.S. degree in computer science and technology from the Department of Computer Science and Technology, Xi’an University, Xi’an, China, in 2004. Her research interests include computer network communication and network security, machine learning theory, and IoT. (Email: wuyu-qin@163.com)

    received the Ph.D. degree and is a Professor in the Department of Computer Science and Technology, Zhejiang University, China. His research interests include the next generation network, network security, and network virtualization.(Email: wuchunming@cs.zju.edu.cn)

  • Received Date: 2020-12-12
  • Accepted Date: 2021-01-07
  • Available Online: 2021-11-05
  • Publish Date: 2022-09-05
  • With the recent increase in the number of Internet of things (IoT) services, an intelligent scheduling strategy is needed to manage these services. In this paper, the problem of automatic choreography of microservices in IoT is explored. A type of reinforcement learning (RL) algorithm called TD3 is used to generate the optimal choreography policy under the framework of a softwaredefined network. The optimal policy is gradually reached during the learning procedure to achieve the goal, despite the dynamic characteristics of the network environment. The simulation results show that compared with other methods, the TD3 algorithm converges faster after a certain number of iterations, and it performs better than other non-RL algorithms by obtaining the highest reward. The TD3 algorithm can effciently adjust the traffic transmission path and provide qualified IoT services.
  • loading
  • [1]
    Ueda T., Nakaike T., and Ohara M., “Workload characterization for microservices,” IEEE International Symposium on Workload Characterization, Providence, RI, USA, pp.1–10, 2016.
    [2]
    Jr A. R. S., Kadiyalax H., Hux B., et al., “Supporting microservice evolution,” IEEE International Conference on Software Maintenance & Evolution, Shanghai, China, pp.529–543, 2017.
    [3]
    Niu Y., Liu F., and Li Z., “Load balancing across microservices,” IEEE INFOCOM, Honolulu, HI, USA, pp.198–206, 2018.
    [4]
    Drutskoy D., Keller E., and Rexford J., “Scalable network virtualization in software-defined networks,” IEEE Internet Computing, vol.17, no.2, pp.20–27, 2013. doi: 10.1109/MIC.2012.144
    [5]
    Sarkar C., Nambi S. N. A. U., Prasad R. V., et al., “A scalable distributed architecture for unifying IoT applications,” IEEE World Forum on Internet of Things (WF-IoT), Seoul, Korea (South), pp.508–513, 2014.
    [6]
    Madden S., Franklin M. J., Hellerstein J. M., et al., “TAG: A tiny aggregation service for ad-hoc sensor networks,” ACM Sigops Operating Systems Review, vol.36, no.SI, pp.131–146, 2002. doi: 10.1145/844128.844142
    [7]
    Madden S., Franklin M. J., Hellerstein J. M., et al., “Design of an acquisitional query processor for sensor networks,” in Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, San Diego, CA, USA, pp.491–502, 2003.
    [8]
    Gummadi R., Gnawali O., and Govindan R. “Macro-programming wireless sensor networks using Kairos,” in Proc of the IEEE International Conference on Distributed Computing in Sensor Systems (DCOSS 2005), Marina del Rey, CA, USA, pp.126–140, 2005
    [9]
    Koroniotis N., Moustafa N., and Sitnikova E., “Forensics and deep learning mechanisms for botnets in Internet of things: A survey of challenges and solutions,” IEEE Access, vol.7, pp.61764–61785, 2019. doi: 10.1109/ACCESS.2019.2916717
    [10]
    Patel P and Cassou D, “Enabling high-level application development for the Internet of things,” Journal of Systems and Software, vol.103, pp.62–84, 2015. doi: 10.1016/j.jss.2015.01.027
    [11]
    Cassou D, Bertran B, Loriant N, et al., “A generative programming approach to developing pervasive computing systems,” ACM SIGPLAN Notices, vol.45, no.2, pp.137–146, 2010. doi: 10.1145/1621607.1621629
    [12]
    Katasonov A, “Enabling non-programmers to develop smart environment applications,” The IEEE Symposium on Computers and Communications, Riccione, Italy, pp.1059–1064, 2010.
    [13]
    Thoma M, Meyer S, Sperner K, et al., “On IoT-services: Survey, classification and enterprise integration,” 2012 IEEE International Conference on Green Computing and Communications (GreenCom), Besancon, France, pp.257–260, 2012.
    [14]
    Guinard D, Trifa V, Karnouskos S, et al., “Interacting with the SOA-based Internet of things: Discovery, query, selection, and on-demand provisioning of Web services,” IEEE Transactions on Services Computing, vol.3, no.3, pp.223–235, 2010. doi: 10.1109/TSC.2010.3
    [15]
    Resnick M, Maloney J, Andrés Monroy-Hernández, et al., “Scratch: Programming for all,” Communications of the ACM, vol.52, no.11, pp.60–67, 2009. doi: 10.1145/1592761.1592779
    [16]
    Gans P., “The benefits of using scratch to introduce basic programming concepts in the elementary classroom: Poster session,” Journal of Computing Sciences in Colleges, vol.25, no.6, pp.235–236, 2010. doi: 10.5555/1791129.1791176
    [17]
    Nordmann A, Hochgeschwender N, and Wrede S., “A surve–y on domain-specific languages in robotics,” International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR 2014), Bergamo, Italy, pp.195–206, 2014.
    [18]
    H. Y. Li, “Application service and resource management system for smart community based on micro-service architecture,” M.S. Thesis, Shanghai Jiao Tong University, China, 2016. (in Chinese)
    [19]
    S. Li, Y. H. Yan, J. Ren, et al. “A sample-efficient actor-critic algorithm for recommendation diversification,” Chinese Journal of Electronics, vol. 29, no.1, pp.89–96, 2020.
    [20]
    Z. Xu, J. Tang, J. Meng, et al., “Experience-driven networking: A deep reinforcement learning based approach,” IEEE Conference on Computer Communications, Honolulu, HI, USA, pp.1871–1879, 2018.
    [21]
    S. C. Lin, I. F. Akyildiz, P. Wang, et al., “QoS-aware adaptive routing in multi-layer hierarchical software defined networks: A reinforcement learning approach,” IEEE International Conference on Services Computing, San Francisco, CA, USA, pp.25–33, 2016.
    [22]
    Mckeown N, Anderson T, Balakrishnan H, et al., “OpenFlow: Enabling innovation in campus networks,” ACM SIGCOMM Computer Communication Review, vol.38, no.2, pp.69–74, 2008. doi: 10.1145/1355734.1355746
    [23]
    Chun S., Jung S., Yi S., et al., “Method and apparatus for transmitting and receiving signal to and from network at user equipment in a wireless communication system,” Patent, 20220117003, USA, 2022.
    [24]
    Li L. E., Mao Z. M., Rexford J., “Toward software-defined cellular networks,” 2012 European Workshop on Software Defined Networking, IEEE Computer Society, Darmstadt, Germany, pp.99–106, 2012.
    [25]
    Donald Gross, John F. Shortle, James M. Thompson, et al., Fundamentals of Queueing Theory, 4th ed., John Wiley & Sons, Inc., 2008.
    [26]
    Balalaie A., Heydarnoori A., Jamshidi P., “Microservices architecture enables DevOps: An experience report on migration to a cloud-native architecture,” IEEE Software, vol.33, no.3, pp.42–52, 2016.
    [27]
    Peters J. and Schaal S., “Natural actor-critic,” Neurocomputing, vol.71, no.7-9, pp.1180–1190, 2008. doi: 10.1016/j.neucom.2007.11.026
    [28]
    F. Ortin and D. O’Shea, “Towards an easily programmable IoT framework based on micro services,” JSW, vol.13, no.2, pp.90–102, 2018. doi: 10.17706/jsw.13.2.90-120
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(6)

    Article Metrics

    Article views (479) PDF downloads(43) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return