Volume 29 Issue 6
Dec.  2020
Turn off MathJax
Article Contents
ZHANG Junwei and LI Zhao, “Quantum Contextuality for Training Neural Networks,” Chinese Journal of Electronics, vol. 29, no. 6, pp. 1178-1184, 2020, doi: 10.1049/cje.2020.10.003
Citation: ZHANG Junwei and LI Zhao, “Quantum Contextuality for Training Neural Networks,” Chinese Journal of Electronics, vol. 29, no. 6, pp. 1178-1184, 2020, doi: 10.1049/cje.2020.10.003

Quantum Contextuality for Training Neural Networks

doi: 10.1049/cje.2020.10.003
Funds:  This work is supported by the National Key R&D Program of China (No.2017YEF0111900), the National Natural Science Foundation of China (No.61876129), the National Natural Science Foundation of China (No.U1636203), the Alibaba Innovation Research Foundation 2017 and the European Unions Horizon 2020 research and innovation programme under the Marie Skodowska-Curie grant agreement (No.721321), Part of the work was performed when Junwei Zhang visited the Alibaba Inc. in 2019.
  • Received Date: 2019-08-06
  • Publish Date: 2020-12-25
  • In the training process of Neural networks (NNs), the selection of hyper-parameters is crucial, which determines the final training effect of the model. Among them, the Learning rate decay (LRD) can improve the learning speed and accuracy; the Weight decay (WD) improves the over-fitting to varying degrees. However, the decay methods still have problems such as hysteresis and stiffness of parameter adjustment, so that the final model will be inferior. Based on the Quantum contextuality (QC) theory, we propose a Quantum contextuality constraint (QCC) to constrain the weights of nodes in NNs to further improve the training effect. In the simplest classification model, we combine this constraint with different methods of LRD and WD to verify that QCC can further improve the training effect on the decay method. The performance of the experiments shows that QCC can significantly improve the convergence and accuracy of the model.
  • loading
  • I. Goodfellow, Y. Bengio, A. Courville, et al., "Deep learning", Cambridge:MIT press, 2016.
    Y. Bengio, "Learning deep architectures for AI", Now Publishers Inc, 2009.
    W.U. Yujia, L.I. Jing, S. Chengfang, et al., "Words in Pairs Neural Networks for Text Classification", Chinese Journal of Electronics, Vol.29, No.3, pp.491-500, 2020.
    Y. Bengio, "Practical recommendations for gradient-based training of deep architectures", Neural networks:Tricks of the trade, Berlin, Heidelberg, pp.437-478, 2012.
    L.N. Smith, "Cyclical learning rates for training neural networks", IEEE Winter Conference on Applications of Computer Vision, pp.464-472, 2017.
    Y. Dauphin, H.D. Vries and Y. Bengio, "Equilibrated adaptive learning rates for non-convex optimization", Advances in neural information processing systems, pp.1504-1512, 2015.
    C. Gulcehre, M. Moczulski and Y. Bengio, "Adasecant:robust adaptive secant method for stochastic gradient", arXiv preprint arXiv:1412.7419, 2014.
    A. Krogh and J.A. Hertz, "A simple weight decay can improve generalization", Advances in neural information processing systems. pp.950-957, 1992.
    P. Kurzyński, R. Ramanathan and D. Kaszlikowski, "Entropic test of quantum contextuality", Physical review letters, Vol.109, No.2, pp.020-404, 2012.
    A. Cabello, S. Severini and A. Winter, "Graph-theoretic approach to quantum correlations", Physical review letters, Vol.112, No.4, pp.040-401, 2014.
    J.B. Vega, N. Delfosse, D.E. Browne, et al., "Contextuality as a resource for models of quantum computation with qubits", Physical review letters, Vol.119, No.12, pp.120-505, 2017.
    A.W. Simmons, "How (Maximally) Contextual Is Quantum Mechanics?", Quantum, Probability, Logic. Springer, Cham, pp.505-519, 2020.
    A. Grudka, K. Horodecki, M. Horodecki, et al., "Quantifying contextuality", Physical review letters, Vol.112, No.12, pp.120-401, 2014.
    S. Mansfield, "The Mathematical Structure of Non-locality and Contextuality", 2013.
    A. Cabello, S. Severini and A. Winter, "(Non-) contextuality of physical theories as an axiom", arXiv preprint arXiv:1010.2163, 2010.
    S. Kochen, and E.P. Specker, "The problem of hidden variables in quantum mechanics", The logico-algebraic approach to quantum mechanics. Springer, Dordrecht, pp.293-328, 1975.
    S. Popescu and D. Rohrlich, "Quantum nonlocality as an axiom", Foundations of Physics, Vol.24, No.3, pp.379-385, 1994.
    H.W. Lin, M. Tegmark and D. Rolnick, "Why does deep and cheap learning work so well?", Journal of Statistical Physics, Vol.168, No.6, pp.1223-1247, 2017.
    K. Horodecki, M. Horodecki, P. Horodecki, et al., "Contextuality offers device-independent security", arXiv preprint arXiv:1006.0468, 2010.
    L.I. Panchi and Z. Ya, "Model and algorithm of sequencebased quantum-inspired neural networks", Chinese Journal of Electronics, Vol.27, No.1, pp.9-18, 2018.
    B.R. La Cour, "Quantum contextuality in the Mermin-Peres square:A hidden-variable perspective", Physical Review A, Vol.79, No.1, pp.012-102, 2009.
    A.Y. Khrennikov, "Contextual approach to quantum formalism", Springer Science Business Media, 2009.
    M. Kleinmann, O. Gühne, J.R. Portillo, et al., "Memory cost of quantum contextuality", New Journal of Physics, Vol.13, No.11, pp.113-011, 2011.
    K. Svozil, "How much contextuality?", Natural Computing, Vol.11, No.2, pp.261-265, 2012.
    A. Grudka, K. Horodecki, M. Horodecki, et al., "Quantifying contextuality", Physical review letters, Vol.1121, No.2, pp.120-401, 2014.
    A.A. Klyachko, M.A. Can, S. Binicioğlu, et al., "Simple test for hidden variables in spin-1 systems", Physical review letters, Vol.101, No.2, pp.020-403, 2008.
    S. Kochen and E.P. Specker, "The problem of hidden variables in quantum mechanics", The logico-algebraic approach to quantum mechanics. Springer, Dordrecht, pp.293-328, 1975.
    S. Yu and C.H. Oh, "State-independent proof of KochenSpecker theorem with 13 rays", Physical review letters, Vol.108, No.3, pp.030-402, 2012.
    L. Lovász, "On the Shannon capacity of a graph", IEEE Transactions on Information theory, Vol.25, No.1, pp.1-7, 1979.
    M. Grötschel, L. Lovász and A. Schrijver, "The ellipsoid method and its consequences in combinatorial optimization", Combinatorica, Vol.1, No.2, pp.169-197, 1981.
    Y. LeCun, C. Cortes and C.J. Burges, "MNIST handwritten digit database", http://yann.lecun.com/exdb/mnist, 2010-7-23.
    H. Xiao, K. Rasul and R. Vollgraf, "Fashion-mnist:a novel image dataset for benchmarking machine learning algorithms", arXiv preprint arXiv:1708.07747, 2017.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Article Metrics

    Article views (456) PDF downloads(64) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return