Hyperspectral Image Super-Resolution Based on Spatial-Spectral Feature Extraction Network

LI Yanshan; CHEN Shifu; LUO Wenhan; ZHOU Li; XIE Weixin

doi:10.23919/cje.2021.00.081

Volume 32 Issue 3

May 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2023 > 32(3): 415-428

LI Yanshan, CHEN Shifu, LUO Wenhan, et al. “Hyperspectral Image Super-Resolution Based on Spatial-Spectral Feature Extraction Network”. Chinese Journal of Electronics, vol. 32 no. 3. doi: 10.23919/cje.2021.00.081

Citation:

LI Yanshan, CHEN Shifu, LUO Wenhan, et al. “Hyperspectral Image Super-Resolution Based on Spatial-Spectral Feature Extraction Network”. Chinese Journal of Electronics, vol. 32 no. 3. doi: 10.23919/cje.2021.00.081

Citation:

PDF( 4613 KB)

Hyperspectral Image Super-Resolution Based on Spatial-Spectral Feature Extraction Network

doi: 10.23919/cje.2021.00.081

1.
ATR National Key Laboratory of Defense Technology, Shenzhen University, Shenzhen 518060, China
2.
Tencent, Shenzhen 518057, China

Funds: This work was partially supported by the National Natural Science Foundation of China (61771319, 61871154), the Natural Science Foundation of Guangdong Province (2017A030313343, 2019A1515011307), and the Shenzhen Science and Technology Project (JCYJ20180507182259896)

More Information

Author Bio:
Yanshan LI received the M.S. degree from the Zhejiang University of Technology in 2005 and the Ph.D. degree from the South China University of Technology, China, in 2015. He is currently an Associate Professor with the ATR National Key Laboratory of Defense Technology, Shenzhen University, China. His research interests include computer vision, machine learning, and image analysis. (Email: lys@szu.edu.cn)

Shifu CHEN received the B.E. degree in School of Computer, Electronics and Information from Guangxi University, Nanning, China, in 2018. He is currently pursuing the M.S. degree in information and engineering from Shenzhen University, Shenzhen, China. He is a member of the ATR National Key Laboratory of Defense Technology, Shenzhen University. His research interests include intelligent information processing, video processing, and pattern recognition

Wenhan LUO received the B.E. degree from Huazhong University of Science and Technology, China, in 2009, M.E. degree from Institute of Automation, Chinese Academy of Sciences, China, in 2012, and Ph.D. degree from Imperial College London, UK, in 2016. His research interests include several topics in computer vision and machine learning, such as motion analysis (especially object tracking), image/video quality restoration, object detection and recognition, and reinforcement learning

Li ZHOU received the B.E. degree in automation from South-Central University for Nationalities, Wuhan, China, in 2018. He is currently pursuing the M.S. degree in information and engineering from Shenzhen University, Shenzhen, China. He is a member of the ATR National Key Laboratory of Defense Technology, Shenzhen University. His research interests include intelligent information processing, video processing, and pattern recognition

Weixin XIE received the degree from Xidian University, Xi’an. He was a Faculty Member with Xidian University in 1965. From 1981 to 1983, he was a Visiting Scholar at the University of Pennsylvania, USA. In 1989, he was a Visiting Professor with the University of Pennsylvania. He is currently with the School of Information Engineering, Shenzhen University, China. His research interests include intelligent information processing, fuzzy information processing, image processing, and pattern recognition
Received Date: 2021-03-01
Accepted Date: 2021-12-07

Available Online: 2022-06-08

Publish Date: 2023-05-05

Abstract

Abstract

Constrained by the physics of hyperspectral sensors, the spatial resolution of hyperspectral images (HSI) is low. Hyperspectral image super-resolution (HSI SR) is a task to obtain high-resolution hyperspectral images from low-resolution hyperspectral images. Existing algorithms have the problem of losing important spectral information while improving spatial resolution. To handle this problem, a spatial-spectral feature extraction network (SSFEN) for HSI SR is proposed in this paper. It enhances the spatial resolution of the HSI while preserving the spectral information. The SSFEN is composed of three parts: spatial-spectral mapping network, spatial reconstruction network, and spatial-spectral fusing network. And a joint loss function with spatial and spectral constraints is designed to guide the training of the SSFEN. Experiment results show that the proposed method improves the spatial resolution of the HSI and effectively preserves the spectral information simultaneously.
- Hyperspectral image,
- Super-resolution,
- Spectral difference,
- Spatial-spectral feature extraction

FullText(HTML)

References(43)

References

[1]	J. J. Li, Y. S. Li, R. Song, et al., “Local spectral similarity preserving regularized robust sparse hyperspectral unmixing,” IEEE Transactions on Geoscience and Remote Sensing, vol.57, no.10, pp.7756–7769, 2019. doi: 10.1109/TGRS.2019.2916296
[2]	M. M. Zhang, W. Li, and Q. Du, “Diverse region-based CNN for hyperspectral image classification,” IEEE Transactions on Image Processing, vol.27, no.6, pp.2623–2634, 2018. doi: 10.1109/TIP.2018.2809606
[3]	W. Li, G. D. Wu, and Q. Du, “Transferred deep learning for anomaly detection in hyperspectral imagery,” IEEE Geoscience and Remote Sensing Letters, vol.14, no.5, pp.597–601, 2017. doi: 10.1109/LGRS.2017.2657818
[4]	J. J. Li, Q. Du, Y. S. Li, et al., “Hyperspectral image classification with imbalanced data based on orthogonal complement subspace projection,” IEEE Transactions on Geoscience and Remote Sensing, vol.56, no.7, pp.3838–3851, 2018. doi: 10.1109/TGRS.2018.2813366
[5]	W. Li, G. D. Wu, F. Zhang, et al., “Hyperspectral image classification using deep pixel-pair features,” IEEE Transactions on Geoscience and Remote Sensing, vol.55, no.2, pp.844–853, 2017. doi: 10.1109/TGRS.2016.2616355
[6]	H. Y. Yu, L. R. Gao, J. Li, et al., “Spectral-spatial hyperspectral image classification using subspace-based support vector machines and adaptive Markov random fields,” Remote Sensing, vol.8, no.4, article no.355, 2016. doi: 10.3390/rs8040355
[7]	W. Li, Q. Du, F. Zhang, et al., “Hyperspectral image classification by fusing collaborative and sparse representations,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol.9, no.9, pp.4178–4187, 2016. doi: 10.1109/JSTARS.2016.2542113
[8]	B. Du, M. F. Zhang, L. F. Zhang, et al., “PLTD: Patch-based low-rank tensor decomposition for hyperspectral images,” IEEE Transactions on Multimedia, vol.19, no.1, pp.67–79, 2017. doi: 10.1109/TMM.2016.2608780
[9]	B. Du, Y. X. Zhang, L. P. Zhang, et al., “Beyond the sparsity-based target detector: A hybrid sparsity and statistics-based detector for hyperspectral images,” IEEE Transactions on Image Processing, vol.25, no.11, pp.5345–5357, 2016. doi: 10.1109/TIP.2016.2601268
[10]	H. Y. Zhang, L. P. Zhang, and H. F. Shen, “A super-resolution reconstruction algorithm for hyperspectral images,” Signal Processing, vol.92, no.9, pp.2082–2096, 2012. doi: 10.1016/j.sigpro.2012.01.020
[11]	K. F. Fan, J. Y. Liang, F. Li, et al., “CNN based no-reference HDR image quality assessment,” Chinese Journal of Electronics, vol.30, no.2, pp.282–288, 2021. doi: 10.1049/cje.2021.01.008
[12]	C. Dong, C. C. Loy, K. M. He, et al., “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.38, no.2, pp.295–307, 2016. doi: 10.1109/TPAMI.2015.2439281
[13]	E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, pp.1122–1131, 2017.
[14]	N. Liu, P. Zhou, W. J. Liu, et al., “Sparse representation based image super-resolution using large patches,” Chinese Journal of Electronics, vol.27, no.4, pp.813–820, 2018. doi: 10.1049/cje.2018.05.011
[15]	H. Irmak, G. B. Akar, and S. E. Yuksel, “A MAP-based approach for hyperspectral imagery super-resolution,” IEEE Transactions on Image Processing, vol.27, no.6, pp.2942–2951, 2018. doi: 10.1109/TIP.2018.2814210
[16]	Y. F. Zhang, A. Duijster, and P. Scheunders, “A Bayesian restoration approach for hyperspectral images,” IEEE Transactions on Geoscience and Remote Sensing, vol.50, no.9, pp.3453–3462, 2012. doi: 10.1109/TGRS.2012.2184122
[17]	S. Ayas, E. T. Gormus, and M. Ekinci, “An efficient pan sharpening via texture based dictionary learning and sparse representation,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol.11, no.7, pp.2448–2460, 2018. doi: 10.1109/JSTARS.2018.2835573
[18]	Y. Q. Zhao, J. X. Yang, and J. C. W. Chan, “Hyperspectral imagery super-resolution by spatial–spectral joint nonlocal similarity,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol.7, no.6, pp.2671–2679, 2014. doi: 10.1109/JSTARS.2013.2292824
[19]	W. S. Dong, F. Z. Fu, G. M. Shi, et al., “Hyperspectral image super-resolution via non-negative structured sparse representation,” IEEE Transactions on Image Processing, vol.25, no.5, pp.2337–2352, 2016. doi: 10.1109/TIP.2016.2542360
[20]	R. Kawakami, Y. Matsushita, J. Wright, et al., “High-resolution hyperspectral imaging via matrix factorization,” in Proceedings of CVPR 2011, Colorado Springs, CO, USA, pp.2329–2336, 2011.
[21]	N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol.50, no.2, pp.528–537, 2012. doi: 10.1109/TGRS.2011.2161320
[22]	C. Lanaras, E. Baltsavias, and K. Schindler, “Hyperspectral super-resolution by coupled spectral unmixing,” in Proceedings of 2015 IEEE International Conference on Computer Vision, Santiago, Chile, pp.3586–3594, 2015.
[23]	W. Huang, L. Xiao, H. Y. Liu, et al., “Hyperspectral Imagery super-resolution by compressive sensing inspired dictionary learning and spatial-spectral regularization,” Sensors, vol.15, no.1, pp.2041–2058, 2015. doi: 10.3390/s150102041
[24]	R. Fernandez-Beltran, P. Latorre-Carmona, and F. Pla, “Single-frame super-resolution in remote sensing: A practical overview,” International Journal of Remote Sensing, vol.38, no.1, pp.314–354, 2017. doi: 10.1080/01431161.2016.1264027
[25]	J. Hu, Y. S. Li, and W. Y. Xie, “Hyperspectral image super-resolution by spectral difference learning and spatial error correction,” IEEE Geoscience and Remote Sensing Letters, vol.14, no.10, pp.1825–1829, 2017. doi: 10.1109/LGRS.2017.2737637
[26]	S. H. Mei, X. Yuan, J. Y. Ji, et al., “Hyperspectral image spatial super-resolution via 3D full convolutional neural network,” Remote Sensing, vol.9, no.11, article no.1139, 2017. doi: 10.3390/rs9111139
[27]	S. Lei, Z. W. Shi, and Z. X. Zou, “Super-resolution for remote sensing images via local–global combined network,” IEEE Geoscience and Remote Sensing Letters, vol.14, no.8, pp.1243–1247, 2017. doi: 10.1109/LGRS.2017.2704122
[28]	C. Ledig, L. Theis, F. Huszar, et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, pp.105–114, 2016.
[29]	Y. L. Zhang, Y. P. Tian, Y. Kong, et al., “Residual dense network for image super-resolution,” in Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp.2472–2481, 2018.
[30]	B. Lim, S. Son, H. Kim, et al., “Enhanced deep residual networks for single image super-resolution,” in Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, pp.1132–1140, 2017.
[31]	Q. Li, Q. Wang, and X. L. Li, “Mixed 2D/3D convolutional network for hyperspectral image super-resolution,” Remote Sensing, vol.12, no.10, article no.1660, 2020. doi: 10.3390/rs12101660
[32]	L. Loncan, L. B. de Almeida, J. M. Bioucas-Dias, X. et al., “Hyperspectral pansharpening: A review,” IEEE Geoscience and Remote Sensing Magazine, vol.3, no.3, pp.27–46, 2015. doi: 10.1109/MGRS.2015.2440094
[33]	N. Yokoya, C. Grohnfeldt, and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine, vol.5, no.2, pp.29–56, 2017. doi: 10.1109/MGRS.2016.2637824
[34]	S. Ozkan, B. Kaya, and G. B. Akar, “EndNet: Sparse AutoEncoder network for endmember extraction and hyperspectral unmixing,” IEEE Transactions on Geoscience and Remote Sensing, vol.57, no.1, pp.482–496, 2019. doi: 10.1109/TGRS.2018.2856929
[35]	F. Yasuma, T. Mitsunaga, D. Iso, et al., “Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum,” IEEE Transactions on Image Processing, vol.19, no.9, pp.2241–2253, 2010. doi: 10.1109/TIP.2010.2046811
[36]	A. Chakrabarti and T. Zickler, “Statistics of real-world hyperspectral images,” in Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, pp.193–200, 2011.
[37]	D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA, pp.1–15, 2015.
[38]	J. X. Yang, Y. Q. Zhao, C. Yi, et al., “No-reference hyperspectral image quality assessment via quality-sensitive features learning,” Remote Sensing, vol.9, no.4, article no.305, 2017. doi: 10.3390/rs9040305
[39]	N. Yokoya, “Texture-guided multisensor superresolution for remotely sensed images,” Remote Sensing, vol.9, no.4, article no.316, 2017. doi: 10.3390/rs9040316
[40]	R. Yuhas, A. Goetz, and J. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm,” in Summaries of the Third Annu. JPL Airborne Geoscience Workshop, JPL, Volume 1: AVIRIS Workshop, pp.147–149, 1992.
[41]	Y. Qu, H. R. Qi, and C. Kwan, “Unsupervised sparse dirichlet-net for hyperspectral image super-resolution,” in Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp.2511–2520, 2018.
[42]	C. Z. Rong, G. H. Liu, Z. L. Ping, et al., “Fusion of infrared and visible images based on infrared object extraction,” Chinese Journal of Electronics, vol.30, no.2, pp.339–348, 2021. doi: 10.1049/cje.2020.11.013
[43]	D. W. Zhou, R. Duan, L. J. Zhao, et al., “Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network,” Signal Processing, vol.166, article no.107251, 2020. doi: 10.1016/j.sigpro.2019.107251