Deep Guided Attention Network for Joint Denoising and Demosaicing in Real Image

ZHANG Tao; FU Ying; ZHANG Jun

doi:10.23919/cje.2022.00.414

Volume 33 Issue 1

Jan. 2024

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2024 > 33(1): 303-312

Tao ZHANG, Ying FU, Jun ZHANG, “Deep Guided Attention Network for Joint Denoising and Demosaicing in Real Image,” Chinese Journal of Electronics, vol. 33, no. 1, pp. 303–312, 2024 doi: 10.23919/cje.2022.00.414

Citation:

Tao ZHANG, Ying FU, Jun ZHANG, “Deep Guided Attention Network for Joint Denoising and Demosaicing in Real Image,” Chinese Journal of Electronics, vol. 33, no. 1, pp. 303–312, 2024 doi: 10.23919/cje.2022.00.414

Citation:

PDF( 4957 KB)

Deep Guided Attention Network for Joint Denoising and Demosaicing in Real Image

doi: 10.23919/cje.2022.00.414

ZHANG Tao^1
,,
FU Ying^{1, 2
,
,},
ZHANG Jun^2
,

1.
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
2.
Advanced Reasearch Institute of Multidisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

More Information

Author Bio:
Tao ZHANG received the B.S. degree from the School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China, in 2017. He is currently working toward the Ph.D. degree in the School of Computer Science and Technology, Beijing Institute of Technology, Beijing. His research interests include deep learning, image processing, and computational photography. (Email: tzhang@bit.edu.cn)

Ying FU received the B.S. degree in electronic engineering from Xidian University in 2009, the M.S. degree in automation from Tsinghua University in 2012, and the Ph.D. degree in information science and technology from the University of Tokyo in 2015. She is a Professor at the School of Computer Science and Technology, Beijing Institute of Technology. Her research interests include physics-based vision, image processing, and computational photography. (Email: fuying@bit.edu.cn)

Jun ZHANG received the B.S., M.S., and Ph.D. degrees in communications and electronic systems from Beihang University, Beijing, China, in 1987, 1991, and 2001, respectively. He was a Professor with Beihang University. He has served as the Dean for the School of Electronic and Information Engineering, and the Vice President and the Secretary for the Party Committee, Beihang University. He is currently a Professor with Beijing Institute of Technology, where he is also the Secretary. His research interests are networked and collaborative air traffic management systems, covering signal processing, integrated and heterogeneous networks, and wireless communications. He is a member of the Chinese Academy of Engineering. He has won the awards for science and technology in China many times. (Email: buaazhangjun@vip.sina.com)
Corresponding author: Email: fuying@bit.edu.cn
Received Date: 2022-12-05
Accepted Date: 2023-03-16

Available Online: 2023-07-12

Publish Date: 2024-01-05

Abstract

Abstract

Denoising (DN) and demosaicing (DM) are the first crucial stages in the image signal processing pipeline. Recently, researches pay more attention to solve DN and DM in a joint manner, which is an extremely undetermined inverse problem. Existing deep learning methods learn the desired prior on synthetic dataset, which limits the generalization of learned network to the real world data. Moreover, existing methods mainly focus on the raw data property of high green information sampling rate for DM, but occasionally exploit the high intensity and signal-to-noise (SNR) of green channel. In this work, a deep guided attention network (DGAN) is presented for real image joint DN and DM (JDD), which considers both high SNR and high sampling rate of green information for DN and DM, respectively. To ease the training and fully exploit the data property of green channel, we first train DN and DM sub-networks sequentially and then learn them jointly, which can alleviate the error accumulation. Besides, in order to support the real image JDD, we collect paired raw clean RGB and noisy mosaic images to conduct a realistic dataset. The experimental results on real JDD dataset show the presented approach performs better than the state-of-the-art methods, in terms of both quantitative metrics and qualitative visualization.
- Image denoising,
- Image demosaicing,
- Joint processing,
- Guided attention,
- Paired real dataset

FullText(HTML)

References(47)

References

[1]	M. Gharbi, G. Chaurasia, S. Paris, et al., “Deep joint demosaicking and denoising,” ACM Transactions on Graphics, vol. 35, no. 6, article no. 191, 2016. doi: 10.1145/2980179.2982399
[2]	K. Hirakawa and T. W. Parks, “Joint demosaicing and denoising,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2146–2157, 2006. doi: 10.1109/TIP.2006.875241
[3]	L. Condat and S. Mosaddegh, “Joint demosaicking and denoising by total variation minimization,” in Proceedings of the 19th IEEE International Conference on Image Processing, Orlando, FL, USA, pp. 2781–2784, 2012.
[4]	F. Heide, M. Steinberger, Y. T. Tsai, et al., “FlexISP: A flexible camera image processing framework,” ACM Transactions on Graphics, vol. 33, no. 6, article no. 231, 2014. doi: 10.1145/2661229.2661260
[5]	H. L. Tan, X. R. Zeng, S. M. Lai, et al., “Joint demosaicing and denoising of noisy Bayer images with ADMM,” in Proceedings of 2017 International Conference on Image Processing, Beijing, China, pp. 2951–2955, 2017.
[6]	D. S. Tan, W. Y. Chen, and K. L. Hua, “DeepDemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks,” IEEE Transactions on Image Processing, vol. 27, no. 5, pp. 2408–2419, 2018. doi: 10.1109/TIP.2018.2803341
[7]	F. Kokkinos and S. Lefkimmiatis, “Deep image demosaicking using a cascade of convolutional residual denoising networks,” in Proceedings of the 15th European Conference on Computer Vision, Munich, Germany, pp. 317–333, 2018.
[8]	L. Liu, X. Jia, J. Z. Liu, et al., “Joint demosaicing and denoising with self guidance,” in Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 2240–2249, 2020.
[9]	R. Timofte, S. H. Gu, J. Q. Wu, et al., “NTIRE 2018 challenge on single image super-resolution: Methods and results,” in Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, pp. 852–863, 2018.
[10]	R. Timofte, E. Agustsson, L. Van Gool, et al., “NTIRE 2017 challenge on single image super-resolution: Methods and results,” in Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, pp. 114–125, 2017.
[11]	D. Khashabi, S. Nowozin, J. Jancsary, et al., “Joint demosaicing and denoising via learned nonparametric random fields,” IEEE Transactions on Image Processing, vol. 23, no. 12, pp. 4968–4981, 2014. doi: 10.1109/TIP.2014.2359774
[12]	G. C. Qian, Y. H. Wang, J. J. Gu, et al., “Rethinking learning-based demosaicing, denoising, and super-resolution pipeline,” in Proceedings of 2022 IEEE International Conference on Computational Photography, Pasadena, CA, USA, pp. 1–12, 2022.
[13]	J. L. Schnapf, T. W. Kraft, and D. A. Baylor, “Spectral sensitivity of human cone photoreceptors,” Nature, vol. 325, no. 6103, pp. 439–441, 1987. doi: 10.1038/325439a0
[14]	O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, pp. 234–241, 2015.
[15]	A. Buades, B. Coll, J. M. Morel, et al., “Self-similarity driven color demosaicking,” IEEE Transactions on Image Processing, vol. 18, no. 6, pp. 1192–1202, 2009. doi: 10.1109/TIP.2009.2017171
[16]	J. Mairal, M. Elad, and G. Sapiro, “Sparse representation for color image restoration,” IEEE Transactions on Image Processing, vol. 17, no. 1, pp. 53–69, 2008. doi: 10.1109/TIP.2007.911828
[17]	I. Pekkucuksen and Y. Altunbasak, “Gradient based threshold free color filter array interpolation,” in Proceedings of 2010 IEEE International Conference on Image Processing, Hong Kong, China, pp. 137–140, 2010.
[18]	G. S. Yu, G. Sapiro, and S. Mallat, “Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity,” IEEE Transactions on Image Processing, vol. 21, no. 5, pp. 2481–2499, 2012. doi: 10.1109/TIP.2011.2176743
[19]	Y. Monno, D. Kiku, M. Tanaka, et al., “Adaptive residual interpolation for color and multispectral image demosaicking,” Sensors, vol. 17, no. 12, article no. 2787, 2017. doi: 10.3390/s17122787
[20]	L. M. Tang, Z. Fang, C. C. Xiang, et al., “A variational model for staircase reduction in image denoising,” Chinese Journal of Electronics, vol. 26, no. 2, pp. 358–366, 2017. doi: 10.1049/cje.2017.01.015
[21]	Y. N. Xie, J. J. Huang, and Y. J. He, “One dictionary vs. two dictionaries in sparse coding based denoising,” Chinese Journal of Electronics, vol. 26, no. 2, pp. 367–371, 2017. doi: 10.1049/cje.2017.01.014
[22]	G. Y. Chen, G. C. Luo, L. Tian, et al., “Noise reduction for images with non-uniform noise using adaptive block matching 3D filtering,” Chinese Journal of Electronics, vol. 26, no. 6, pp. 1227–1232, 2017. doi: 10.1049/cje.2017.09.031
[23]	R. Sadiq, M. B. Qureshi, and M. M. Khan, “De-convolution and De-noising of SAR based GPS images using hybrid particle swarm optimization,” Chinese Journal of Electronics, vol. 32, no. 1, pp. 166–176, 2023. doi: 10.23919/cje.2021.00.138
[24]	Z. K. Ni, K. K. Ma, H. Q. Zeng, et al., “Color image demosaicing using progressive collaborative representation,” IEEE Transactions on Image Processing, vol. 29, pp. 4952–4964, 2020. doi: 10.1109/TIP.2020.2975978
[25]	B. Fu, Y. H. Dong, S. L. Fu, et al., “Multistage supervised contrastive learning for hybrid-degraded image restoration,” Signal, Image and Video Processing, vol. 17, no. 2, pp. 573–581, 2023. doi: 10.1007/s11760-022-02262-8
[26]	Q. Zhang, Q. Q. Yuan, M. P. Song, et al., “Cooperated spectral low-rankness prior and deep spatial prior for HSI unsupervised denoising,” IEEE Transactions on Image Processing, vol. 31, pp. 6356–6368, 2022. doi: 10.1109/TIP.2022.3211471
[27]	W. Z. Xing and K. Egiazarian, “End-to-end learning for joint image demosaicing, denoising and super-resolution,” in Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, pp. 3507–3516, 2021.
[28]	K. M. He, J. Sun, and X. O. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013. doi: 10.1109/TPAMI.2012.213
[29]	L. Zeng and Y. Z. Dai, “Single image dehazing based on combining dark channel prior and scene radiance constraint,” Chinese Journal of Electronics, vol. 25, no. 6, pp. 1114–1120, 2016. doi: 10.1049/cje.2016.08.006
[30]	Y. Monno, D. Kiku, M. Tanaka, et al., “Adaptive residual interpolation for color image demosaicking,” in Proceedings of 2015 IEEE International Conference on Image Processing, Quebec City, Canada, pp. 3861–3865, 2015.
[31]	T. W. Hui, C. C. Loy, and X. O. Tang, “Depth map super-resolution by deep multi-scale guidance,” in Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, pp. 353–369, 2016.
[32]	Y. J. Li, J. B. Huang, N. Ahuja, et al., “Deep joint image filtering,” in Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, pp. 154–169, 2016.
[33]	H. T. Zheng, M. Q. Ji, H. Q. Wang, et al., “CrossNet: An end-to-end reference-based super resolution network using cross-scale warping,” in Proceedings of the 15th European Conference on Computer Vision, Munich, Germany, pp. 88–104, 2018.
[34]	Y. M. Zhou, G. C. Wu, Y. Fu, et al., “Cross-MPI: Cross-scale stereo for image super-resolution using multiplane images,” in Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, pp. 14842–14851, 2021.
[35]	Y. Fu, T. Zhang, Y. Q. Zheng, et al., “Hyperspectral image super-resolution with optimized RGB guidance,” in Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, pp. 11661–11670, 2019.
[36]	X. T. Wang, K. Yu, C. Dong, et al., “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 606–615, 2018.
[37]	S. Guo, Z. T. Liang, and L. Zhang, “Joint denoising and demosaicking with green channel prior for real-world burst images,” IEEE Transactions on Image Processing, vol. 30, pp. 6930–6942, 2021. doi: 10.1109/TIP.2021.3100312
[38]	T. Zhang, Y. Fu, and C. Li, “Deep spatial adaptive network for real image demosaicing,” in Proceedings of the 36th AAAI Conference on Artificial Intelligence, Palo Alto, CA, USA, pp. 3326–3334, 2022.
[39]	T. Zhang, Y. Fu, and J. Zhang, “Guided hyperspectral image denoising with realistic data,” International Journal of Computer Vision, vol. 130, no. 11, pp. 2885–2901, 2022. doi: 10.1007/s11263-022-01660-2
[40]	Y. Fu, T. Zhang, L. Z. Wang, et al., “Coded hyperspectral image reconstruction using deep external and internal learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3404–3420, 2022. doi: 10.1109/TPAMI.2021.3059911
[41]	S. H. Gu, Y. W. Li, L. Van Gool, et al., “Self-guided network for fast image denoising,” in Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South), pp. 2511–2520, 2019.
[42]	C. L. Tsai, W. C. Tu, and S. Y. Chien, “Efficient natural color image denoising based on guided filter,” in Proceedings of 2015 IEEE International Conference on Image Processing, Quebec City, Canada, pp. 43–47, 2015.
[43]	H. S. Zhao, J. Y. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 10076–10085, 2020.
[44]	C. Chen, Q. F. Chen, J. Xu, et al., “Learning to see in the dark,” in Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 3291–3300, 2018.
[45]	D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint, arXiv: 1412.6980, 2014.
[46]	K. X. Wei, Y. Fu, J. L. Yang, et al., “A physics-based noise formation model for extreme low-light raw denoising,” in Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 2758–2767, 2020.
[47]	Y. L. Zhang, K. P. Li, K. Li, et al., “Image super-resolution using very deep residual channel attention networks,” in Proceedings of the 15th European Conference on Computer Vision, Munich, Germany, pp. 294–310, 2018.