Research of Neural Network Structural Optimization Based on Information Entropy

WANG Danyang; SHAO Fangming

doi:10.1049/cje.2020.05.006

Volume 29 Issue 4

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2020 > 29(4): 632-638

WANG Danyang and SHAO Fangming, “Research of Neural Network Structural Optimization Based on Information Entropy,” Chinese Journal of Electronics, vol. 29, no. 4, pp. 632-638, 2020, doi: 10.1049/cje.2020.05.006

Citation:

WANG Danyang and SHAO Fangming, “Research of Neural Network Structural Optimization Based on Information Entropy,” Chinese Journal of Electronics, vol. 29, no. 4, pp. 632-638, 2020, doi: 10.1049/cje.2020.05.006

Citation:

PDF( 267 KB)

Research of Neural Network Structural Optimization Based on Information Entropy

doi: 10.1049/cje.2020.05.006

School of Science, East China University of Science and Technology, Shanghai 200237, China

Funds: This work is supported by the National Natural Science Foundation of China(No.61040040).

Received Date: 2019-05-09
Rev Recd Date: 2019-09-04
Publish Date: 2020-07-10

Abstract

Abstract

In the application of deep learning, the depth and width of the neural network structure have a great influence on the learning performance of the neural network. This paper focuses on structural optimization of depth and width, leveraging the information entropy model and decision tree strategy as feature selection and structural adjustment to optimize neural network candidates. Therefore, a decision tree-based heuristic optimization algorithm for neural network structural adjustment is proposed. Furthermore, the proposed approach is applied to fully-connected neural networks trained on the Iris dataset, and the proposed approach is verified effective via experimental simulation.
- Information entropy,
- Decision tree,
- Fully connected neural network,
- Heuristic algorithm

FullText(HTML)

References(28)

References

Y. Lecun, Y. Bengio and G. Hinton, “Deep learning”, Nature, Vol.521, No.7553, Page 436, 2015.

L. Deng, J. Li, J.T. Huang, et al., “Recent advances in deep learning for speech research at Microsoft”, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, pp.8604-8608, 2013.

A. Krizhevsky, I. Sutskever, G. Hinton, et al., “Imagenet classification with deep convolutional neural networks”, Neural Information Processing Systems, pp.1097-1105, 2012.

C. Chen, A. Seff, A. Kornhauser, et al., “Deepdriving: Learning affordance for direct perception in autonomous driving”, International Conference on Computer Vision, pp.2722-2730, 2015.

A. Esteva, B. Kuprel, R. Novoa, et al., “Dermatologistlevel classification of skin cancer with deep neural networks”, Nature, Vol.542, No.7639, pp.115-118, 2017.

D. Silver, A. Huang, C. Maddison, et al., “Mastering the game of Go with deep neural networks and tree search”, Nature, Vol.529, No.7587, pp.484-489, 2016.

F.Y. Wang, J.J. Zhang, X. Zheng, et al., “Where does AlphaGo go: From church-turing thesis to AlphaGo thesis and beyond”, IEEE/CAA Journal of Automatica Sinica, Vol.3, No.2, pp.113-120, 2016.

J.R. Long, H.T. Mayfield, M.V. Henley, et al., “Pattern recognition of jet fuel chromatographic data by artificial neural networks with back-propagation of error”, Analytical Chemistry, Vol.63, No.13, pp.1256-1261, 1991.

M.V. Butz, D.E. Goldberg, P.L. Lanzi, et al., “Gradient descent methods in learning classifier systems: Improving XCS performance in multistep problems”, IEEE Transactions on Evolutionary Computation, Vol.9, No.5, pp.452-473, 2005.

B. Kosko and J.C. Burgess, “Neural Networks for Signal Processing”, Journal of the Acoustical Society of America, Vol.103, No.6, pp.25-38, 1992.

I. Guyon and P.S.P. Wang, “Advances in pattern recognition systems using neural network technologies”, World Scientific Publishing, 1994.

V. Sze, Y.H. Chen, T.J. Yang, et al., “Efficient processing of deep neural networks: A tutorial and survey”, Proceedings of the IEEE, Vol.105, No.12, pp.2295-2329, 2017.

S.R. Safavian and D. Landgrebe, “A survey of decision tree classifier methodology”, IEEE Transactions on Systems, Man and Cybernetics, Vol.21, No.3, pp.660-674, 1991.

C. Lee and G. Lee, “Information gain and divergencebased feature selection for machine learning-based text categorization”, Information Processing Management, Vol.42, No.1, pp.155-165, 2006.

A. Jain and D. Zongker, “Feature selection: Evaluation, application, and small sample performance”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.19, No.2, pp.153-158, 1997.

R. Vidal, J. Bruna, R. Giryes, et al., “Mathematics of deep learning”, arXiv preprint arXiv: 1712.04741, 2017.

K. Hornik, “Multilayer feedforward networks are universal approximators”, Neural Networks, Vol.2, No.5, pp.359-366, 1989.

K. Hornik, “Approximation capabilities of multilayer feedforward networks”, Neural Networks, Vol.4, No.2, pp.251-257, 1991.

Y. Bengio, N.L. Roux, P. Vincent, et al., “Convex neural networks”, Neural Information Processing Systems, pp.123-130, 2005.

J.H. Friedman, “Greedy function approximation: A gradient boosting machine”, The Annals of Statistics, Vol.29, No.5, pp.1189-1232, 2001.

J. Cheng, J. Chen, Y.N. Guo, et al., “Adaptive CCRELM with variable-length brain storm optimization algorithm for class-imbalance learning”, Natural Computing, DOI： 10.1007/s11047-019-09735-9, 2019.

L.K. Yang, Y.N Guo, J. Cheng, et al., “Manifold distancebased over-sampling technique for class imbalance learning”, Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, Vol.33, pp.10071-10072, 2019.

S.E. Fahlman and C. Lebiere, “The cascade-correlation learning architecture”, Advances in Neural Information Processing Systems, 1990.

R. Reed, “Pruning algorithms-A survey”, IEEE Transactions on Neural Networks, Vol.4, No.5, pp.740-747, 1993.

J.H. Holland, Adaptation in Natural and Artificial Systems, MIT Press, 1992.

R.F. Nalewajski, “Entropy descriptors of the chemical bond in information theory. I. Basic concepts and relations”, Molecular Physics, Vol.102, No.6, pp.531-546, 2004.

I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016.

H.R.A. Ghayab, Y. Li, S. Siuly, et al., “Epileptic seizures detection in EEGs blending frequency domain with information gain technique”, Soft Computing, Vol.23, No.1, pp.227-239, 2019.

Relative Articles

Supplements(0)

Cited By

Proportional views