Video Stabilization via Prediction with Time-Series Network and Image Inpainting with Pyramid Fusion

CHENG Keyang; LI Shichao; RONG Lan; WANG Wenshan; SHI Wenxi; ZHAN Yongzhao

doi:10.1049/cje.2021.08.006

Volume 30 Issue 6

Nov. 2021

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2021 > 30(6): 1103-1110

CHENG Keyang, LI Shichao, RONG Lan, et al., “Video Stabilization via Prediction with Time-Series Network and Image Inpainting with Pyramid Fusion,” Chinese Journal of Electronics, vol. 30, no. 6, pp. 1103-1110, 2021, doi: 10.1049/cje.2021.08.006

Citation:

CHENG Keyang, LI Shichao, RONG Lan, et al., “Video Stabilization via Prediction with Time-Series Network and Image Inpainting with Pyramid Fusion,” Chinese Journal of Electronics, vol. 30, no. 6, pp. 1103-1110, 2021, doi: 10.1049/cje.2021.08.006

Citation:

PDF( 3761 KB)

Video Stabilization via Prediction with Time-Series Network and Image Inpainting with Pyramid Fusion

doi: 10.1049/cje.2021.08.006

1. School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China;
2. National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data, Beijing 100846, China;
3. Xinjiang Lianhaichuangzhi Information Technology Co., Ltd., Urumqi 830011, China;
4. Jiangsu Province Big Data Ubiquitous Perception and Intelligent Agricultural Application Engineering Research Center, Zhenjiang 212013, China;
5. Cyber Space Security Academy of Jiangsu University, Zhenjiang 212013, China

Funds:

This work is supported by the National Natural Science Foundation of China (No.61972183) and the Director Foundation Project of National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data (PSRPC).

Received Date: 2020-06-15
Rev Recd Date: 2020-11-30

Available Online: 2021-09-23

Publish Date: 2021-11-05

Abstract

Abstract

Due to the poor filling effect of the video image defect commonly used in the video stabilization field, the video is seemed still unstable after the image stabilization process, which seriously affects the visual effect. To solve this problem, we improve a video stabilization method based on time-series network prediction and pyramid fusion restoration is proposed to optimize the visual effect after image stabilization. The flow of the proposed method is as follows:First, it is adaptive to determine whether the defect of the corresponding frame at the current time needs padding inpainting. Then, for the frame that needs to be inpainting, the frames generated before the current moment are sent to the model combining the convolutional neural networks and the gate recurrent unit to predict the part to be filled. Next the current defect image and the complete image to be filled are brought into the Laplacian pyramid reconstruction, and the improved weighted optimal suture is introduced for splicing during the fusion. Finally, the video frame is cut after reconstruction. The method is tested on a data set composed of videos commonly used in the field of video stabilization. The experimental results show that the average peak signal to noise ratio of the method is 2 to 5dB higher than that of the comparison algorithm, and the average structural similarity index is improved by about 2% to 7% compared with the contrast algorithm.
- Video stabilization,
- Video inpainting,
- Time series network,
- Multi-resolution fusion,
- Optimal seam

FullText(HTML)

References(32)

References

Y. Matsushita, E. Ofek, W. Ge, et al., "Full-frame video stabilization with motion inpainting", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.28, No.7, pp.1150-1163, 2006.

Y. G. Ryu and M. J. Chung, "Robust online digital image stabilization based on point-feature trajectory withoutaccumulative global motion estimation ", IEEE Signal Processing Letters, Vol.19, No.4, pp.223-226, 2012.

S. Yoo, A. K. Katsaggelos, G. Jo, et al., "Video completion using block matching for video stabilization", Proc. of The 18th IEEE International Symposium on Consumer Electronics, JeJu Island, South Korea, pp.1-2, 2014.

Fan Yeping, Guo Zheng and Zhang Rui, "Research on electronic image stabilization algorithm based on subsample gray-scale projection", Industry and Mine Automation, No.4, pp.22-27, 2017. (in Chinese)

K. A. Patwardhan, G. Sapiro and M. Bertalmío, "Video inpainting under constrained camera motion", IEEE Transactions on Image Processing, Vol.16, No.2, pp.545-553, 2007.

H. A. Hsu, C. K. Chiang and S. H. Lai, "Spatio-temporally consistent view synthesis from video-plus-depth data with global optimization", IEEE Transactions on Circuits and Systems for Video Technology, Vol.24, No.1, pp.74-84, 2014.

Qiao Xiaotian, "3D Reconstruction and display of multi-view video", Ph. D. Thesis, University of Zhejiang, China, 2016. (in Chinese)

A. Newson, A. Almansa, M. Fradet, et al., "Video inpainting of complex scenes", SIAM Journal on Imaging Sciences, Vol.7, No.4, pp.1993-2019, 2014.

G. B. Luo, Y. S. Zhu, Z. T. Li, et al., "A hole filling approach based on background reconstruction for view synthesis in 3D video", Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp.1781-1789, 2016.

Yu Haibao, Shen Qi and Feng Guocan, "Introduce numerical solution to visualize convolutional neuron networks based on numerical solution", Computer Science, No.S1, pp.146-150, 2017. (in Chinese)

Liu Cun, Li Yuanxiang, Zhou Yongjun, et al., "Video image super-resolution reconstruction method based on convolutional neural network", Application Research of Computers, Vol.36, No.4, pp.1256-1260, 2019. (in Chinese)

L. He, J. Tan, C. Xie, et al., "A novel two-step approach for the super-resolution reconstruction of video sequences", Proc. of International Conference on Digital Home, Guangzhou, China, pp.85-90, 2014.

Li Sumei, Lei Guoqing and Fan Ru, "Depth map superresolution reconstruction based on convolutional neural networks", Acta Optica Sinica, Vol.37, No.12, pp.124-132, 2017. (in Chinese)

K. Cho, van B. Merrienboer, D. Bahdanau, et al., "On the properties of neural machine translation:Encoderdecoder approaches", Proc. of the 8th Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, pp.103-111, 2014.

P. J. Burt and E. H. Adelson, "A multiresolution spline with application to image mosaics", ACM Transactions on Graphics, Vol.2, No.4, pp.217-236, 1983.

R. Mao, X. Fu, P. Niu, et al., "Multi-directional Laplacian pyramid image fusion algorithm", Proc. of International Conference on Mechanical, Control and Computer Engineering, Huhhot Inner Mongolia, China, pp.568-572, 2018.

S. K. Verma, M. Kaur and R. Kumar, "Hybrid image fusion algorithm using Laplacian pyramid and PCA method", Proc. of the Second International Conference on Information and Communication Technology for Competitive Strategies, The Papandayan Hotel, Bandung, Indonesia, pp.68-72, 2016.

M. J. Li, Y. B. Dong and X. L. Wang, "Image fusion algorithm based on wavelet transform and Laplacian pyramid", Advanced Materials Research, Vol.860, No.3, pp.2846-2849, 2014.

M. L. Duplaquet, "Building large image mosaics with invisible seam lines", Proc. of Visual Information Processing VII, Tokyo, Japan, pp.369-377, 1998.

Gu Yu, Zhou Yang, Ren Gang, et al., "Image stitching by combining optimal seam and multi-resolution fusion", Journal of Image and Graphics, Vol.22, No.6, pp.0842-0851, 2017. (in Chinese)

Z. Qu, T. Wang, S. An, et al., "Image seamless stitching and straightening based on the image block", IET Image Processing, Vol.12, No.8, pp.1361-1369, 2018.

D. Lee and S. Lee, "Seamless image stitching by homography refinement and structure deformation using optimal seam pair detection", Journal of Electronic Imaging, Vol.26, No.6, pp.1-6, 2017.

W. X. Yan, C. C. Liu and J. Hu, "Optimal seam line detection in laplacian pyramid domain for image stitching", Journal of Computers, Vol.29, No.1, pp.209-219, 2018.

J. Xue, S. Chen, X. Cheng, et al., "A new optimal seam method for seamless image stitching", Proc. of Ninth International Conference on Digital Image Processing, Hong Kong, China, pp.1-6, 2017.

F. Liu, M. Gleicher, H. Jin, et al., "Content-preserving warps for 3D video stabilization", ACM Transactions on Graphics, San Francisco, USA, pp.1-9, 2009.

W. C. Hu, C. H. Chen, Y. J. Su, et al., "Featurebased real-time video stabilization for vehicle video recorder system", Multimedia Tools and Applications, Vol.77, No.5, pp.5107-5127, 2018.

Liu Guanglong, "Research on electronic image stabilization based on feature optical flow", Ph.D. Thesis, University of Harbin Institute of Technology, China, 2015. (in Chinese)

M. Wang, G. Yang, J. Lin, et al., "Deep Online Video Stabilization With Multi-Grid Warping Transformation Learning", IEEE Transactions on Image Processing, Vol.28, No.5, pp.2283-2292, 2019.

BAI Dongdong, WANG Chaoqun, ZHANG Bo, et al., "CNN feature boosted SeqSLAM for real-time loop closure detection", Chinese Journal of Electronics, Vol.27, No.3, pp.488-499, 2018.

QU Hua, ZHANG Yanpeng, LIU Wei, et al., "A robust fuzzy time series forecasting method based on multi-partition and outlier detection", Chinese Journal of Electronics, Vol.28, No.5, pp.899-905, 2019.

WANG Haitao, HE Jie, ZHANG Xiaohong, et al., "A short text classification method based on N-gram and CNN", Chinese Journal of Electronics, Vol.29, No.2, pp.248-254, 2020.

G. Fei, W. Meng, W. Jun, et al., "A novel separability objective function in CNN for feature extraction of SAR images", Chinese Journal of Electronics, Vol.28, No.2, pp.634-640, 2019.

Relative Articles

Supplements(0)

Cited By

Proportional views