A Sample-Efficient Actor-Critic Algorithm for Recommendation Diversification

LI Shuang; YAN Yanghui; REN Ju; ZHOU Yuezhi; ZHANG Yaoxue

doi:10.1049/cje.2019.10.004

Volume 29 Issue 1

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2020 > 29(1): 89-96

LI Shuang, YAN Yanghui, REN Ju, et al., “A Sample-Efficient Actor-Critic Algorithm for Recommendation Diversification,” Chinese Journal of Electronics, vol. 29, no. 1, pp. 89-96, 2020, doi: 10.1049/cje.2019.10.004

Citation:

LI Shuang, YAN Yanghui, REN Ju, et al., “A Sample-Efficient Actor-Critic Algorithm for Recommendation Diversification,” Chinese Journal of Electronics, vol. 29, no. 1, pp. 89-96, 2020, doi: 10.1049/cje.2019.10.004

Citation:

PDF( 646 KB)

A Sample-Efficient Actor-Critic Algorithm for Recommendation Diversification

doi: 10.1049/cje.2019.10.004

1.
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China;
2.
Kwai Inc., Beijing 100083, China;
3.
School of Information Science and Engineering, Central South University, Changsha 410083, China

Funds: This work is supported by Tsinghua University Initiative Scientific Research Program (No.20161080066).

Received Date: 2018-11-12
Rev Recd Date: 2018-11-26
Publish Date: 2020-01-10

Abstract

Abstract

Diversifying recommendation results gains benefits from satisfying user's existing interests as well as exploring novel information needs. Recently proposed Monte-Carlo based reinforcement learning method suffers from sample inefficiency, large variance, and even failing to perform well in large action space. We propose a novel actor-critic reinforcement learning algorithm for recommendation diversification in order to solve the above mentioned problems. The actor acts as the ranking policy, while the introduced critic predicts the expected future rewards of each candidate action. The critic target is updated by full Bellman equation and the actor network is optimized using expected gradient in the whole action space. To further stabilize and improve the performance, we also add policy-filtered critic supervision loss. Experiments on MovieLens dataset well demonstrate the effectiveness of our approach over multiple competitive methods.
- Recommender system,
- Diversity,
- Reinforcement learning,
- Actor-critic

FullText(HTML)

References(20)

References

Y. Zhang, J. Ren, J. Liu, et al., "A survey on emerging computing paradigms for big data", Chinese Journal of Electronics, Vol.26, No.1, pp.1-12, 2017.

J. Ren, H. Guo, C. Xu, and Y. Zhang, "Serving at the edge:A scalable iot architecture based on transparent computing", IEEE Network, Vol.31, No.5, pp.96-105, 2017.

X. Peng, J. Ren, L. She, D. Zhang, J. Li, and Y. Zhang, "Boat:A block-streaming app execution scheme for lightweight iot devices", IEEE Internet of Things Journal, Vol.5, No.3, pp.1816-1829, 2018.

Yue Shi, Martha Larson and Alan Hanjalic, "List-wise learning to rank with matrix factorization for collaborative filtering", In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys'10, pp.269-272, 2010.

Paolo Cremonesi, Yehuda Koren and Roberto Turrin, "Performance of recommender algorithms on top-n recommendation tasks", Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys'10, pp.39-46, 2010.

Francesco Ricci, Lior Rokach and Bracha Shapira, Recommender Systems Handbook, Springer Publishing Company, Incorporated, 2nd edition, 2015.

Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen, "Improving recommendation lists through topic diversification", Proceedings of the 14th International Conference on World Wide Web, WWW'05, pp.22-32, 2005.

Long Xia, Jun Xu, Yanyan Lan, et al., "Adapting Markov decision process for search result diversification", Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'17, pp.535-544, 2017.

Jaime Carbonell and Jade Goldstein, "The use of MMR, diversity-based reranking for reordering documents and producing summaries", Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'98, pp.335-336, 1998.

Azin Ashkan, Branislav Kveton, Shlomo Berkovsky, and Zheng Wen, "Optimal greedy diversity for recommendation", Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pp.1742-1748, 2015.

Chaofeng Sha, Xiaowei Wu, and Junyu Niu, "A framework for recommending relevant and diverse items", Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16, pp.3868-3874, 2016.

S. Li, Y. Zhou, D. Zhang, Y. Zhang, and X. Lan, "Learning to diversify recommendations based on matrix factorization", 2017 IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech), pp.68-74, 2017.

Yadong Zhu, Yanyan Lan, Jiafeng Guo, et al., "Learning for search result diversification", Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'14, pp.293-302, 2014.

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning:An Introduction, the MIT Press, 2nd edition, 2017.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, et al., "Continuous control with deep reinforcement learning", Proceedings of the Conference on Learning Representations (ICLR), 2016.

Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, et al., "An actor-critic algorithm for sequence prediction", Proceedings of the Conference on Learning Representations (ICLR), 2017.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, et al., "Asynchronous methods for deep reinforcement learning", Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48, ICML'16, pp.1928-1937, 2016.

John Schulman, Philipp Moritz, Sergey Levine, et al., "Highdimensional continuous control using generalized advantage estimation", Proceedings of the Conference on Learning Representations (ICLR), 2016.

Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, et al., "Novelty and diversity in information retrieval evaluation", Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'08, pp.659-666, 2008.

Olivier Chapelle, Shihao Ji, Ciya Liao, et al., "Intent-based diversification of web search results:Metrics and algorithms", Inf. Retr., Vol.14, No.6, pp.572-592, 2011.

Relative Articles

Supplements(0)

Cited By

Proportional views