Volume 29 Issue 6
Dec.  2020
Turn off MathJax
Article Contents
ZHANG Naimin and ZHANG Ting, “Recurrent Neural Networks for Computing the Moore-Penrose Inverse with Momentum Learning,” Chinese Journal of Electronics, vol. 29, no. 6, pp. 1039-1045, 2020, doi: 10.1049/cje.2020.02.005
Citation: ZHANG Naimin and ZHANG Ting, “Recurrent Neural Networks for Computing the Moore-Penrose Inverse with Momentum Learning,” Chinese Journal of Electronics, vol. 29, no. 6, pp. 1039-1045, 2020, doi: 10.1049/cje.2020.02.005

Recurrent Neural Networks for Computing the Moore-Penrose Inverse with Momentum Learning

doi: 10.1049/cje.2020.02.005
Funds:  This work is supported by the National Natural Science Foundation of China (No.61572018).
  • Received Date: 2019-07-15
  • Publish Date: 2020-12-25
  • We are concerned with a kind of iterative method for computing the Moore-Penrose inverse, which can be considered as a discrete-time form of recurrent neural networks. We study the momentum learning scheme of the method and discuss its semi-convergence when computing the Moore-Penrose inverse of a rankdeficient matrix. We prove the semi-convergence for our new acceleration algorithm and obtain the optimal momentum factor which makes the fastest semi-convergence. Numerical tests demonstrate the effectiveness of our new acceleration algorithm.
  • loading
  • A. Ben-Israel and T.N.E. Greville, Generalized Inverses:Theory and Applications (2nd ed.), Springer-Verlag, New York, USA, pp.7-38, 2003.
    Y. Xia, S.C. Zhang and P.S. Stanimirović, "Neural network for computing pseudoinverses and outer inverses of complexvalued matrices", Applied Mathematics and Computation, Vol.273, pp.1107-1121, 2016.
    P.S. Stanimirović, I. Živković and Y.M. Wei, "Recurrent neural network for computing the Drazin inverse", IEEE Transactions on Neural Networks and Learning Systems, Vol.26, No.11, pp.2830-2843, 2015.
    P.S. Stanimirović, I. Živković and Y.M. Wei, "Recurrent neural network approach based on the integral representation of the Drazin inverse", Neural Computation, Vol.27, No.10, pp.2107-2131, 2015.
    B. Liao and Y. Zhang, "Different complex ZFs leading to different complex ZNN models for time-varying complex generalized inverse matrices", IEEE Transactions on Neural Networks and Learning Systems, Vol.25, No.9, pp.1621-1631, 2014.
    Y. Zhang, Y. Yang, N. Tan, et al., "Zhang neural network solving for time-varying ful-rank matrix Moore-Penrose inverse", Computing, Vol.92, No.2, pp.97-121, 2011.
    M.D. Petkovic and P.S. Stanimirović, "Iterative method for computing the Moore-Penrose inverse based on penrose equations", Journal of Computational and Applied Mathematics, Vol.235, No.6, pp.1604-1613, 2011.
    G.B. Huang, Q.Y. Zhu and C.K. Siew, "Extreme learning machine:Theory and applications", Neurocomputing, Vol.70, No.1-3, pp.489-501, 2006.
    W.B. Guo and T.Z. Huang, "Method of elementary transformation to compute Moore-Penrose inverse", Applied Mathematics and Computation, Vol.216, No.5, pp.1614-1617, 2010.
    F. Huang and X. Zhang, "An improved Newton iteration for the weighted Moore-Penrose inverse", Applied Mathematics and Computation, Vol.174, No.2, pp.1460-1486, 2006.
    F. Toutounian and A. Ataei, "A new method for computing Moore-Penrose inverse matrices", Journal of Computational and Applied Mathematics, Vol.228, No.1, pp.412-417, 2009.
    V.N. Katsikis and D. Pappas, "Fast computing of the MoorePenrose inverse matrix", Electronic Journal of Linear Algebra, Vol.17, No.1, pp.637-650, 2008.
    V.N. Katsikis, D. Pappas and A. Petralias, "An improved method for the computation of the Moore-Penrose inverse matrix", Applied Mathematics and Computation, Vol.217, No.23, pp.9828-9834, 2011.
    H. Chen and Y. Wang, "A family of higher-order convergent iterative methods for computing the Moore-Penrose inverse", Applied Mathematics and Computation, Vol.218, No.8, pp.4012-4016, 2011.
    W.G. Li and Z. Li, "A family of iterative methods for computing the approximate inverse of a square matrix and inner inverse of a non-square matrix", Applied Mathematics and Computation, Vol.215, No.9, pp.3433-3442, 2010.
    J. Wang, "Recurrent neural networks for computing pseudoinverses of rank-deficient matrices", SIAM Journal on Scientific Computing, Vol.18, No.5, pp.1479-1493, 1997.
    Y.M. Wei, "Recurrent neural networks for computing weighted Moore-Penrose inverse", Applied Mathematics and Computation, Vol.116, No.3, pp.279-287, 2000.
    S. Boyd and L. Vandenberghe, "Convex optimization", IEEE Transactions on Automatic Control, Vol.51, No.11, pp.1859-1859, 2006.
    Y. Xia, T. Chen and J. Shan, "A novel iterative method for computing generalized inverse", Neural Computation, Vol.26, No.2, pp.449-465, 2014.
    D.R. Jensen, "Minimal properties of Moore-Penrose inverses", Linear Algebra and Its Applications, Vol.196, pp.175-182, 1994.
    Y. Si, T. Li, J. Pan, et al., "A prefix tree based n-best list rescoring strategy for recurrent neural network language mode", Chinese Journal of Electronics, Vol.23, No.1, pp.70-74, 2014.
    L. Xiao and R. Lu, "A finite-time recurrent neural network for computing quadratic minimization with time-varying coefficients", Chinese Journal of Electronics, Vol.28, No.2, pp.253-258, 2019.
    Y. Zhang, Y. Jiang and Y. Tong, "Study of sentiment classification for Chinese microblog based on recurrent neural network", Chinese Journal of Electronics, Vol.25, No.4, pp.601-607, 2016.
    D.E. Rumelhart, G.E. Hinton and R.J. Williams, "Learning representations by back-propagating errors", Nature, Vol.323, No.6088, pp.533-536, 1986.
    N. Qian, "On the momentum term in gradient descent learning algorithms", Neural Networks, Vol.12, No.1, pp.145-151, 1999.
    M. Torii and M.T. Hagan, " Stability of steepest descent with momentum for quadratic functions", IEEE Transactions on Neural Networks, Vol.13, No.3, pp.752-756, 2002.
    A. Bhaya and E. Kaszkurewicz, "Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method", Neural Networks, Vol.17, No.1, pp.65-71, 2004.
    H. Zweiri, L.D. Seneviratne and K. Althoefer, "Stability analysis of a three-term backpropagation algorithm", Neural Networks, Vol.18, No.10, pp.1341-1347, 2005.
    N.M. Zhang, W. Wu and G.F. Zheng, "Convergence of gradient method with momentum for two-layer feedforward neural networks", IEEE Transactions on Neural Networks, Vol.17, No.2, pp.522-525, 2006.
    W. Wu, N.M. Zhang, Z.X. Li, et al., "Convergence of gradient method with momentum for back-propagation neural networks", Journal of Computational Mathematics, Vol.26, No.4, pp.613-623, 2008.
    H.M. Shao and G.F. Zheng, "Convergence analysis of a back-propagation algorithm with adaptive momentum", Neurocomputing, Vol.74, No.5, pp.749-752, 2011.
    H.M. Shao, D.P. Xu, G.F. Zheng, et al., "Convergence of an online gradient method with inner-product and adaptive momentum", Neurocomputing, Vol.77, No.1, pp.243-252, 2012.
    N.M. Zhang, "A study on the optimal double parameters for steepest descent with momentum", Neural Computation, Vol.27, No.4, pp.982-1004, 2015.
    N.M. Zhang, "On parameter acceleration methods for saddle point problems", J. Comput. Appl. Math., Vol.288, pp.169-179, 2015.
    A. Berman and R.J. Plemmons, Nonnegative Matrices in the Mathematical Sciences, Philadelphia:SIAM, New York, USA, pp.195-209, 1994.
    J. J. H. Miller, "On the location of zeros of certain classes of polynomials with applications to numerical analysis", J. Inst. Math. Appl., Vol.8, No.3, pp.397-406, 1971.
    R. Boisvert, R. Pozo, K. Remington, et al., "The Matrix Market", https://math.nist.gov/Matrix Market, 2018-4-15.
    N.J. Higham, "The matrix computation toolbox", http://www.ma.man.ac.uk/higham/mctoolbox/, 2018-4-15.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Article Metrics

    Article views (712) PDF downloads(231) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return