Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm

We investigated two commonly used momentum algorithms, Classical momentum (CM) and Nesterov momentum (NM). We found that, when used in Restricted Boltzmann machine (RBM), they have two main problems:The first one is their performances are not obvious and not as good as expected. The second one is they may lose accelerating ability in the later stage of training process. Aiming at these two problems, we proposed the Weight momentum algorithm and evaluated our approach on four datasets. It has been demonstrated that our methods can achieve better performance under both reconstruction error and classification rate criterions.