Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm

GAO Xiaoguang; LI Fei; WAN Kaifang

doi:10.1049/cje.2018.03.013

Volume 27 Issue 3

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2018 > 27(3): 483-487

GAO Xiaoguang, LI Fei, WAN Kaifang. “Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm”. Chinese Journal of Electronics, vol. 27 no. 3. doi: 10.1049/cje.2018.03.013

Citation:

GAO Xiaoguang, LI Fei, WAN Kaifang. “Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm”. Chinese Journal of Electronics, vol. 27 no. 3. doi: 10.1049/cje.2018.03.013

Citation:

PDF( 256 KB)

Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm

doi: 10.1049/cje.2018.03.013

Northwestern Polytechnical University, Xi'an 710129, China

Funds: This work is supported by the National Natural Science Foundation of China (No.61573285, No.61305133) and the Fundamental Research Funds for the Central Universities (No.3102015BJ(Ⅱ)GH01, No.3102016CG002).

Received Date: 2016-11-25
Rev Recd Date: 2017-12-06
Publish Date: 2018-05-10

Abstract

Abstract

We investigated two commonly used momentum algorithms, Classical momentum (CM) and Nesterov momentum (NM). We found that, when used in Restricted Boltzmann machine (RBM), they have two main problems:The first one is their performances are not obvious and not as good as expected. The second one is they may lose accelerating ability in the later stage of training process. Aiming at these two problems, we proposed the Weight momentum algorithm and evaluated our approach on four datasets. It has been demonstrated that our methods can achieve better performance under both reconstruction error and classification rate criterions.
- Deep learning,
- Restricted Boltzmann machine (RBM),
- Momentum algorithm,
- Weight momentum (WM)

FullText(HTML)

References(34)

References

G.E. Hinton and R.R. Salakhutdinov, "Reducing the dimensionality of data with neural networks", Science, Vol.313, No.5786, pp.504-507, 2006.

G.E. Hinton, S. Osindero and Y.W. Teh, "A fast learning algorithm for deep belief nets", Neural Computation, Vol.18, No.7, pp.1527-1554, 2006.

A. Krizhevsky, I. Sutskever and G.E. Hinton, "Imagenet classification with deep convolutional neural networks", Proc. of Advances in Neural Information Processing Systems, Harrahs and Harveys, Lake Tahoe, USA, pp.1097-1105, 2012.

Y. Bengio, "Learning deep architectures for AI", Foundations and Trends in Machine Learning, Vol.21, No.6, pp.1601-1621, 2009.

L. Deng, O. Abdel-Hamid and D. Yu, "A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion", Proc. of the International Conference on Acoustics Speech and Signal Processing, Vancouver, Canada, pp.6669-6673, 2013.

L. Deng, "Design and learning of output representations for speech recognition", Proc. of Advances in neural information processing systems Workshop on Learning Output Representations, Lake Tahoe, USA, 2013.

C. Chet and C. Eswaran, "Reconstruction and recognition of face and digit images using auto-encoders", Neural Computing and Application, Vol.19, pp.1069-1079, 2010.

L. Deng, G. Hinton and B. Kingsbury, "New types of deep neural network learning for speech recognition and related applications:An overview", Proc. of International Conference on Acoustics Speech and Signal Processing, 2013.

L.Y. Gu, et al., "Deception detection study based on PCANet and support vector machine", Acta Electronica Sinica, Vol.44, No.8, pp.1969-1973, 2016. (in Chinese)

D. Li and Y. Dong, "Deep learning:methods and applications", Foundations and Trends^R in Signal Processing, Vol.7, No.3, pp.197-387, 2014.

Z.F. Lian, et al., "Drop connect regularization method with sparsity constraint for neural networks", Chinese Journal of Electronics, Vol.25, No.1, pp.152-158, 2016.

R.C. Li, et al., "Rejecting character recognition errors using CNN based confidence estimation", Chinese Journal of Electronics, Vol.25, No.3, pp.520-526, 2016.

X.G. Deng and Y.D. Ma, "PCNN model analysis and its automatic parameters determination in image segmentation and edge detection", Chinese Journal of Electronics, Vol.23, No.1, pp.97-103, 2014.

K. Swersky, B. Chen, B. Marlin and N. de Freitas, "A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets", Proc. of Information Theory and Applications Workshop, La Jolla, California, USA, pp.1-10, 2010.

G.E. Hinton and R.R. Salakhutdinov, "Reducing the dimensionality of data with neural networks", Science, Vol.313, No.5786, pp.504-507, 2006.

D. Erhan, A. Courville, Y. Bengio and P. Vincent, "Why does unsupervised pre-training help deep learning?", Proc. of the 13th International Conference on Artificial Intelligence and Statistics, Lauderdale, USA, Vol.9, pp.201-208, 2010.

B.T. Polyak, "Some methods of speeding up the convergence of iteration methods", USSR Computational Mathematics and Mathematical Physics, Vol.4, No.5, pp.1-17, 1964.

I. Sutskever, J. Martens, G.E. Dahl and G.E. Hinton, "On the importance of initialization and momentum in deep learning', Proc. of The 30th International Conference on Machine Learning, Atlanta, USA, Vol.28 pp.1139-1147, 2013.

A. Fischer and C. Igel, "Bounding the bias of contrastive divergence learning", Neural Computation, Vol.23, pp.664-673, 2011a.

G.E. Hinton, "Training products of experts by minimizing contrastive divergence", Neural Computation, Vol.14, No.8, pp.1771-1800, 2002.

Y. Bengio and O. Delalleau, "Justifying and generalizing contrastive divergence", Neural Computation, Vol.21, No.6, pp.1601-1621, 2009.

T. Tieleman, "Training restricted Boltzmann machines using approximations to the likelihood gradient", Proc. the 25th International Conference on Machine Learning, Helsinki, Finland, pp.1064-1071, ACM, 2008.

T. Tieleman and G.E. Hinton, "Using fast weights to improve persistent contrastive divergence", Proc. the 26th International Conference on Machine Learning (ICML), Montreal, Canada, pp.1033-1040, ACM, 2009.

A. Fischer and C. Igel, "Parallel tempering, importance sampling, and restricted Boltzmannmachines", Proc. of The 5th Workshop on Theory of Randomized Search Heuristics, Copenhagen, Denmark, 2011b.

G. Desjardins, A. Courville and Y. Bengio, "Adaptive parallel tempering for stochastic maximum likelihood learning of RBMs", Proc. of Advances in Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning, Hyatt Regency, Vancouver, Canada, 2010.

K. Cho, T. Raiko and A. Ilin, "Parallel tempering is efficient for learning restricted Boltzmann machines", Proc. the International Joint Conference on Neural Networks, Barcelona, Spain, pp.3246-3253, 2010.

P. Brakel, S. Dieleman and B. Schrauwen, "Training restricted Boltzmann machines with multi-tempering:Harnessing parallelization", Proc. of European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, pp.287-292, 2012.

G.E. Hinton, "A practical guide to training restricted Boltzmann machines", Neural Networks:Tricks of the Trade, pp.599-619, 2012.

S. Zarea, A. Gonczarek, J.M. Tomczak and J. Swiatek, "Accelerated learning for restricted Boltzmann machine with momentum term", Progress in Systems Engineering, pp.187-192, 2015

M.A. Carreira Perpi and G.E. Hinton, "On contrastive divergence learning", Proc. of 10th International Workshop on Artifial Intelligence and Statistics, Barbados, pp.59-66, 2005.

Y. LeCun L. Bottou Y. Bengio and P. Haffner, "Gradient-based learning applied to document recognition", Proc. Proceedings of the IEEE, Vol.86, No.11, pp.2278-2324, 1998a.

Y. LeCun, F.J. Huang and L. Bottou, "Learning methods for generic object recognition with invariance to pose and lighting", Proc. of Computer Vision and Pattern Recognition, Washington, DC, USA, Vol.2, pp.Ⅱ-104, 2004.

A. Krizhevsky, et al., "Learning multiple layers of features from tiny images", available at https://www.cs.toronto.du/~kriz/learning-features-2009-TR.pdf, 2018-3-15.

A. Torralba, R. Fergus and W.T. Freeman, "80 million tiny images:A large dataset for non-parametric object and scene recognition", IEEE transactions on pattern analysis and machine intelligence, Vol.30, No.11, pp.1958-1970, 2008.

Relative Articles

Supplements(0)

Cited By

Proportional views