Transformer-Based Under-sampled Single-Pixel Imaging

TIAN Ye; FU Ying; ZHANG Jun

doi:10.23919/cje.2022.00.284

Volume 32 Issue 5

Sep. 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Electronics > 2023 > 32(5): 1151-1159

TIAN Ye, FU Ying, ZHANG Jun, “Transformer-Based Under-sampled Single-Pixel Imaging,” Chinese Journal of Electronics, vol. 32, no. 5, pp. 1151-1159, 2023, doi: 10.23919/cje.2022.00.284

Citation:

TIAN Ye, FU Ying, ZHANG Jun, “Transformer-Based Under-sampled Single-Pixel Imaging,” Chinese Journal of Electronics, vol. 32, no. 5, pp. 1151-1159, 2023, doi: 10.23919/cje.2022.00.284

TIAN Ye, FU Ying, ZHANG Jun, “Transformer-Based Under-sampled Single-Pixel Imaging,” Chinese Journal of Electronics, vol. 32, no. 5, pp. 1151-1159, 2023, doi: 10.23919/cje.2022.00.284

Citation:

TIAN Ye, FU Ying, ZHANG Jun, “Transformer-Based Under-sampled Single-Pixel Imaging,” Chinese Journal of Electronics, vol. 32, no. 5, pp. 1151-1159, 2023, doi: 10.23919/cje.2022.00.284

PDF( 4598 KB)

Transformer-Based Under-sampled Single-Pixel Imaging

doi: 10.23919/cje.2022.00.284

TIAN Ye^{1, 4
,},
FU Ying^{2, 3
,},
ZHANG Jun^{1, 2, 4
,}

1.
School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China
2.
Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing 314019, China
3.
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
4.
Advanced Research Institute of Multidisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

Funds: This work was supported by the National Key R&D Program of China (2022YFC3300704) and the National Natural Science Foundation of China (62171038, 61827901, 62088101).

More Information

Author Bio:
Ye TIAN received the B.S. degree from School of Information Science and Engineering, Lanzhou University, Lanzhou, China, in 2017, the M.S. degree from Peking University, Beijing, China, in 2020. She is currently pursuing the Ph.D. degree with the School of Information and Electronics, Beijing Institute of Technology, Beijing, China. Her current research interests include deep learning, image processing, and computational imaging. (Email: 3220205110@bit.edu.cn)

Ying FU (corresponding author) received the B.S. degree in electronic engineering from Xidian University in 2009, the M.S. degree in automation from Tsinghua University in 2012, and the Ph.D. degree in information science and technology from the University of Tokyo in 2015. She is currently a Professor at the School of Computer Science and Technology, Beijing Institute of Technology. Her research interests include physics-based vision, image processing, and computational photography. (Email: fuying@bit.edu.cn)

Jun ZHANG received the B.S., M.S., and Ph.D. degrees in communications and electronic systems from Beihang University, Beijing, China, in 1987, 1991, and 2001, respectively. He was a Professor with Beihang University. He has served as the Dean for the School of Electronic and Information Engineering, and the Vice President and the Secretary for the Party Committee, Beihang University. He is currently a Professor with Beijing Institute of Technology, where he is also the President. His research interests are networked and collaborative air traffic management systems, covering signal processing, integrated and heterogeneous networks, and wireless communications. He is a Member of the Chinese Academy of Engineering. He has won the awards for science and technology in China many times. (Email: buaazhangjun@vip.sina.com)
Received Date: 2022-08-22
Accepted Date: 2023-01-18

Available Online: 2023-02-23

Publish Date: 2023-09-05

Abstract

Abstract

Single-pixel imaging, as an innovative imaging technique, has attracted much attention during the last decades. However, it is still a challenging task for single-pixel imaging to reconstruct high-quality images with fewer measurements. Recently, deep learning techniques have shown great potential in single-pixel imaging especially for under-sampling cases. Despite outperforming traditional model-based methods, the existing deep learning-based methods usually utilize fully convolutional networks to model the imaging process which have limitations in long-range dependencies capturing, leading to limited reconstruction performance. In this paper, we present a transformer-based single-pixel imaging method to realize high-quality image reconstruction in under-sampled situation. By taking advantage of self-attention mechanism, the proposed method is good at modeling the imaging process and directly reconstructs high-quality images from the measured one-dimensional light intensity sequence. Numerical simulations and real optical experiments demonstrate that the proposed method outperforms the state-of-the-art single-pixel imaging methods in terms of reconstruction performance and noise robustness.
- Computational imaging,
- Single-pixel imaging,
- Vision transformer,
- Under-sampled ratio

FullText(HTML)

References(45)

References

[1]	M. P. Edgar, G. M. Gibson, and M. J. Padgett, “Principles and prospects for single-pixel imaging,” Nature Photonics, vol.13, no.1, pp.13–20, 2019. doi: 10.1038/s41566-018-0300-7
[2]	G. M. Gibson, S. D. Johnson, and M. J. Padgett, “Single-pixel imaging 12 years on: A review,” Optics Express, vol.28, no.19, pp.28190–28208, 2020. doi: 10.1364/OE.403195
[3]	Q. H. Dai, J. M. Wu, J. T. Fan, et al., “Recent advances in computational photography,” Chinese Journal of Electronics, vol.28, no.1, pp.1–5, 2019. doi: 10.1049/cje.2018.10.006
[4]	Y. Tian, Y. Fu, and J. Zhang, “Plug-and-play algorithm for under-sampling Fourier single-pixel imaging,” Science China Information Sciences, vol.65, no.10, article no.209303, 2022. doi: 10.1007/s11432-022-3553-1
[5]	M. J. Sun, M. P. Edgar, G. M. Gibson, et al., “Single-pixel three-dimensional imaging with time-based depth resolution,” Nature Communications, vol.7, article no.12010, 2016. doi: 10.1038/ncomms12010
[6]	G. M. Gibson, B. Q. Sun, M. P. Edgar, et al., “Real-time imaging of methane gas leaks using a single-pixel camera,” Optics Express, vol.25, no.4, pp.2998–3005, 2017. doi: 10.1364/OE.25.002998
[7]	L. Zanotto, R. Piccoli, J. L. Dong, et al., “Single-pixel terahertz imaging: A review,” Opto-Electronic Advances, vol.3, no.9, article no.200012, 2020. doi: 10.29026/oea.2020.200012
[8]	J. W. Ma, “Single-pixel remote sensing,” IEEE Geoscience and Remote Sensing Letters, vol.6, no.2, pp.199–203, 2009. doi: 10.1109/LGRS.2008.2010959
[9]	Z. B. Zhang, X. Y. Wang, G. A. Zheng, et al., “Fast Fourier single-pixel imaging via binary illumination,” Scientific Reports, vol.7, no.1, article no.12029, 2017. doi: 10.1038/s41598-017-12228-3
[10]	M. F. Duarte, M. A. Davenport, D. Takhar, et al., “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, vol.25, no.2, pp.83–91, 2008. doi: 10.1109/MSP.2007.914730
[11]	C. B. Li, “An efficient algorithm for total variation regularization with applications to the single pixel camera and compressive sensing,” Master Thesis, Rice University, Houston, TX, USA, 2010.
[12]	M. Lyu, W. Wang, H. Wang, et al., “Deep-learning-based ghost imaging,” Scientific Reports, vol.7, no.1, article no.17865, 2017. doi: 10.1038/s41598-017-18171-7
[13]	Y. C. He, G. Wang, G. X. Dong, et al., “Ghost imaging based on deep learning,” Scientific Reports, vol.8, no.1, article no.6469, 2018. doi: 10.1038/s41598-018-24731-2
[14]	L. S. Sui, L. W. Zhang, Y. Cheng, et al., “Computational ghost imaging based on the conditional adversarial network,” Optics Communications, vol.492, article no.126982, 2021. doi: 10.1016/j.optcom.2021.126982
[15]	T. Wolf, L. Debut, V. Sanh, et al., “Transformers: state-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp.38–45, 2020.
[16]	K. Han, Y. H. Wang, H. T. Chen, et al., “A survey on vision transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.45, no.1, pp.87–110, 2023. doi: 10.1109/TPAMI.2022.3152247
[17]	M. H. Guo, T. X. Xu, J. J. Liu, et al., “Attention mechanisms in computer vision: A survey,” Computational Visual Media, vol.8, no.3, pp.331–368, 2022. doi: 10.1007/s41095-022-0271-y
[18]	W. L. Gong and S. S. Han, “A method to improve the visibility of ghost images obtained by thermal light,” Physics Letters A, vol.374, no.8, pp.1005–1008, 2010. doi: 10.1016/j.physleta.2009.12.030
[19]	F. Ferri, D. Magatti, L. A. Lugiato, et al., “Differential ghost imaging,” Physical Review Letters, vol.104, no.25, article no.253603, 2010. doi: 10.1103/PhysRevLett.104.253603
[20]	L. H. Bian, J. L. Suo, Q. H. Dai, et al., “Experimental comparison of single-pixel imaging algorithms,” Journal of the Optical Society of America A, vol.35, no.1, pp.78–87, 2018. doi: 10.1364/JOSAA.35.000078
[21]	K. K. Guo, S. W. Jiang, and G. A. Zheng, “Multilayer fluorescence imaging on a single-pixel detector,” Biomedical Optics Express, vol.7, no.7, pp.2425–2431, 2016. doi: 10.1364/BOE.7.002425
[22]	C. F. Higham, R. Murray-Smith, M. J. Padgett, et al., “Deep learning for real-time single-pixel video,” Scientific Reports, vol.8, no.1, article no.2369, 2018. doi: 10.1038/s41598-018-20521-y
[23]	F. Wang, H. Wang, H. C. Wang, et al., “Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging,” Optics Express, vol.27, no.18, pp.25560–25572, 2019. doi: 10.1364/OE.27.025560
[24]	H. Wu, R. Z. Wang, G. P. Zhao, et al., “Deep-learning denoising computational ghost imaging,” Optics and Lasers in Engineering, vol.134, article no.106183, 2020. doi: 10.1016/j.optlaseng.2020.106183
[25]	I. Hoshi, T. Shimobaba, T. Kakue, et al., “Single-pixel imaging using a recurrent neural network combined with convolutional layers,” Optics Express, vol.28, no.23, pp.34069–34078, 2020. doi: 10.1364/OE.410191
[26]	X. G. Wang, A. G. Zhu, S. S. Lin, et al., “Learning-based high-quality image recovery from 1D signals obtained by single-pixel imaging,” Optics Communications, vol.521, article no.128571, 2022. doi: 10.1016/j.optcom.2022.128571
[27]	T. Bian, Y. X. Yi, J. L. Hu, et al., “A residual-based deep learning approach for ghost imaging,” Scientific Reports, vol.10, no.1, article no.12149, 2020. doi: 10.1038/s41598-020-69187-5
[28]	T. Bian, Y. M. Dai, J. L. Hu, et al., “Ghost imaging based on asymmetric learning,” Applied Optics, vol.59, no.30, pp.9548–9552, 2020. doi: 10.1364/AO.405120
[29]	W. Feng, X. Y. Sun, X. H. Li, et al., “High-speed computational ghost imaging based on an auto-encoder network under low sampling rate,” Applied Optics, vol.60, no.16, pp.4591–4598, 2021. doi: 10.1364/AO.422641
[30]	Y. Fu, T. Zhang, L. Z. Wang, et al., “Coded hyperspectral image reconstruction using deep external and internal learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, no.7, pp.3404–3420, 2021. doi: 10.1109/TPAMI.2021.3059911
[31]	F. Wang, C. L. Wang, C. J. Deng, et al., “Single-pixel imaging using physics enhanced deep learning,” Photonics Research, vol.10, no.1, pp.104–110, 2022. doi: 10.1364/PRJ.440123
[32]	A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, pp.6000–6010, 2017.
[33]	A. Gillioz, J. Casas, E. Mugellini, et al., “Overview of the transformer-based models for NLP tasks,” in Proceedings of the 2020 15th Conference on Computer Science and Information Systems, Sofia, Bulgaria, pp.179–183, 2020.
[34]	C. F. R. Chen, Q. F. Fan, and R. Panda, “CrossViT: Cross-attention multi-scale vision transformer for image classification,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, pp.347–356, 2021.
[35]	B. W. Cheng, A. G. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” in Proceedings of the 35th Advances in Neural Information Processing Systems, Virtual Conference, pp.17864–17875, 2021.
[36]	N. Carion, F. Massa, G. Synnaeve, et al., “End-to-end object detection with transformers,” in Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, pp.213–229, 2020.
[37]	Y. H. Cai, Z. C. Wang, Z. X. Luo, et al., “Learning delicate local representations for multi-person pose estimation,” in Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, pp.455–472, 2020.
[38]	A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” arXiv preprint, arXiv: 2010.11929, 2020.
[39]	W. H. Wang, E. Z. Xie, X. Li, et al., “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, pp.548–558, 2021.
[40]	Z. Liu, Y. T. Lin, Y. Cao, et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, pp.9992–10002, 2021.
[41]	S. Wu, J. Tian, and W. Cui, “A novel parameter estimation algorithm for DSSS signals based on compressed sensing,” Chinese Journal of Electronics, vol.24, no.2, pp.434–438, 2015. doi: 10.1049/cje.2015.04.035
[42]	X. S. Wang, Y. H. Cheng, and J. Ji, “Semi-supervised regression algorithm based on optimal combined graph,” Chinese Journal of Electronics, vol.22, no.4, pp.724–728, 2013.
[43]	D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA, Available at: https://arxiv.org/pdf/1412.6980v9.pdf, 2015.
[44]	S. S. Channappayya, A. C. Bovik, and R. W. Heath, “Rate bounds on SSIM index of quantized images,” IEEE Transactions on Image Processing, vol.17, no.9, pp.1624–1639, 2008. doi: 10.1109/TIP.2008.2001400
[45]	K. Zhang, Y. W. Li, W. M. Zuo, et al., “Plug-and-play image restoration with deep denoiser prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, no.10, pp.6360–6376, 2022. doi: 10.1109/TPAMI.2021.3088914