Colour Variation Minimization Retinex Decomposition and Enhancement with a Multi-Branch Decomposition Network

I.   Introduction

Currently, information technology is developing rapidly. As the carrier of visual information, image has the advantages of intuitive information and rich content, and has become an important means to obtain, process and transmit information. In a light-rich environment, we obtain images with clear content and detailed structure. However, in low-light/uneven-light scenes such as night, cloudy, and backlit, image suffers from low contrast, noise interference, and colour distortion. One or more problems have been solved with the advancement of image capture equipment and image acquisition technology, but the expensive equipment and complex auxiliary methods have greatly increased the hardware cost. In daily life, image can be collected under very low-light conditions such as mobile devices pictures [1] and urban road monitoring. They show varying degrees of degradation, which seriously affects their further use. Dark image enhancement [2] is an important part of image restoration technology. The technique restores detail in low-light areas by changing the brightness of the source images. On the one hand, this technology can help consumers optimize image details and highlight the more attractive parts of the image during photography; on the other hand, this technique plays an important role in the image preprocessing stage. It can provide better image to advanced visual tasks such as video surveillance [3] and intelligent driving [4]. Therefore, it is important to develop practical technology for large-scale image data enhancement.

To improve low-quality images, low-light image enhancement technology must address four major problems: low contrast, high noise, colour distortion and uneven brightness. Among the traditional methods, histogram equalization (HE) [5] is the most representative. The grayscale histogram of the dark images is distributed over a small range. HE extends the original grayscale images to an approximately uniform distribution over the entire area, improving image contrast [6]. This algorithm also causes overexposure when it improves the image contrast. Retinex theory states that colour texture and light are constant feature and variable feature, respectively. The theory suggests that adjusting variable features to the retina can distinguish constant features and minimize the loss of constant features [7]. Homomorphic filtering establishes a reflection-illumination model in the frequency domain. The model treats the illumination components in the low-frequency domain with approximately uniform treatment, and improves the reflectivity contrast in the high-frequency domain. However, noise problems remain in the enhanced images [8]. These traditional methods focus on brightness enhancement and contrast improvement to restore the details of shadowed image blocks. The enhancement ability is substantial for some specific images, but these approaches cannot meet the needs of large-scale image processing.

Data-driven deep learning model is a hot area for dark image enhancement technique. It builds a black-box model that illuminates low-light images on the basis of large-scale data and restores shadow detail. According to the different algorithm requirements, deep learning methods can be divided into high-reference image enhancement, low-reference image enhancement, and non-reference image enhancement. High-reference image enhancement is a mainstream low-light image enhancement method based on deep learning. High-reference image enhancement is the earliest studied deep learning method. The method inputs the whole process of bright image matching, constraint, dark image feature extraction and enhancement [9]. The enhancement lighting is close to the reference lighting, but the training data relies on strictly matched image pairs. Low-reference image enhancement reduces the requirement of image pairs to a certain extent. The input images can help unpaired training to learn information through a global-local discriminator structure [10]. However, we need to rigorously select unpaired reference images and investigate how to correlate common features of the reference images. Non-reference image enhancement is the most popular research direction in the field of low-light image enhancement in the past year. Some scholars have attempted to completely abandon the reference images. For example, maximum entropy theory [11] proposes that the maximum channel of the enhanced images is consistent with that of the original images. The theory is applied to the Retinex model to decompose the constant reflectivity. The estimated reflectance is the enhanced images obtained by this method, which does not consider the influence of real illumination. ZeroDCE [12] merges the dark images and the bright images to enhance the image brightness by drawing a nonlinear depth estimation curve. However, the method still needs to extract detailed lighting information from carefully selected bright images to draw a brightness curve. Therefore, decomposition and enhancement models face difficulties in predicting detailed reflection and natural brightness without reference images.

To address the above problem and improve the practical value of the non-reference dark images, this paper proposes a colour variation minimization retinex decomposition and enhancement with a multi-branch decomposition network (CvmD-net) based on retinex decomposition theory. First, an input constant feature prior mechanism (ICFP) is proposed to compensate for the lack of paired images. We use ICFP to extract constant features of the original images and optimize the decomposition training process. Second, to estimate the reflectance and illumination in the training process, the interaction between the two maps causes instability in the reflectance estimation. Therefore, we design two subnetworks with different structures and generate better image components. The reflectance subnet is used to learn the colour and detail characteristics of the original images. The illumination subnet is used to learn the initial exposure and structure of dark images. Finally, we propose colour variation minimization on the basis of structure consistency and unify the constant attribute between the input image and the estimated reflectivity. We refine the following innovative work:

1) To solve the dependence of the deep learning-based retinex algorithm on bright reference images, we first propose an input constant feature prior mechanism. This mechanism optimizes model decomposition and enhancement by extracting colour features and structural features from input low-quality images, and achieves an optimization effect that is not worse than that of the paired real reference images. The mechanism successfully replaces the reference images, greatly reducing the difficulty of constructing low-light image datasets.

2) To address reflectance estimation overfitting in the current retinex model during the process of training image decomposition, this paper designs a multi-branch decomposition network. The model is divided into a reflectance subnet and an illumination subnet. The reflectance subnet uses a U-net structure to estimate the reflectivity, and the illumination subnet uses a shallow convolutional neural network to estimate the illumination. The network structure reduces the negative effects caused by the interaction between reflectance and illumination, and estimates the reflectance with better quality.

3) To overcome noise interference in the enhanced images, this paper proposes a colour variation minimization loss function that uses the pixel intensity difference between the input-output images R, G, and B channels to adjust the predicted reflectance. This function effectively suppresses the amplified noise in the reflectance estimation and reduces colour distortion.

II.   Related Work

Scholars at home and abroad have long paid attention to dark image enhancement technology and have been applied to many scenes. The vigorous development of artificial intelligence disciplines has led to the development of traditional image restoration and data-driven image restoration.

1   Traditional image restoration

Image contrast enhancement　HE design an inverse equilibrium function for gray histogram. This function extends the original grayscale interval to the full range $[0,\;255]$. For example, Kim et al. [5] propose an average brightness retention algorithm (BBHE). The work sets an average from the dark images to parse two separate gray histograms to maintain the degree of equalization. BBHE generates two subimages based on the mean: a sample set lower than the input mean value and a sample set higher than the input mean value. Then the independent histogram is proposed to balance the two subimages and constrain the resulting balanced subimages to be bounded around each other near the input mean. Naik et al. [13] propose a colour preserving function for the colour gamut problem caused by image space transformation. This function does not depend on the image intensity and normalizes the three vectors of the image at the same scale. After the histogram is equalized, the greyscale images are constrained by the function to fix the transformed values within RGB space. Celik et al. [14] propose a pixel-level contrast enhancement method (CVC). This method studies the nonlinear mapping of 2D histogram and extracts pixel neighborhood information that is beneficial to colour image. In addition, image equalization calculates the Frobinius norm for gray intervals and assigns a higher probability to a high grey-level difference. Lee et al. [15] propose a pixel difference enhancement (LDR) algorithm based on 2D histogram equalization. This work observes that differences between neighboring pixels can change image contrast. It counts the hierarchical anomaly of pixels and derives transformation functions to limit such anomaly. Instead of filtering the input data, these methods make noise more disturbed.

Illumination estimation　Retinex model is the primary method of brightness enhancement. Inspired by the human retina, it believes that brightness changes with the environment, and reflectivity is inherent property of object. The model reconstructs the image light level by adjusting the illumination. For example, Fu et al. [16] propose a new logarithmic transformation prior model to restore better original reflectivity. Given the weighted gradient term of the target stimulus signal, the model adopted an alternate minimization scheme to estimate the reflectance and illumination simultaneously. This method can suppress part of the noise, but the brightness is very poor in dark areas. Based on retinex theory, Guo et al. [17] propose a lighting estimation method based on reflection optimization (LIME). It designs different optimization strategies to iteratively predict illuminance. The structure prior uses the augmented Lagrange method to transform the optimized illumination into an optimization problem. This method considers only the brightness factor and can enhance the shadow details, but it requires cumbersome adjustment of the parameters to find the optimal value for a single image. Li et al. [18] add noise cancellation to the retinex model. This is the first attempt to estimate the measurement noise. Instead of logarithmic transformation, ADM based on the augmented Lagrange multiple was proposed to optimize the model. This method combines the image enhancement with denoising to over-smooth the texture of the image structure.

Homomorphic filtering　This method enhances the high-frequency image information based on the frequency domain transformation of the acquired images, and transforms the enhanced result to preserve the spatial domain. For example, Tiwari et al. [19] apply homomorphic filtering to the medical image domain to sharpen local structure. It uses the average difference between the enhanced brightness and the original brightness to optimize the final illumination. Luan [20] study subimage segmentation and homomorphic filter cut-off frequency selection. They divide variance corresponding to different sensitivity regions into two subimages. The two subimages are subjected to independent homomorphic filtering with the cut-off frequency as the boundary. This method has strong applicability for restoring strong light areas of an image, but it also enlarges the useless dark area structures.

2   Data-driven image restoration

High-reference image enhancement　Data-driven image recovery learns how to remove darkness from the trained images. On the basis of deep learning, researchers create different directions. For example, Gharbi et al. [9] propose that the two-sided network architecture seeks a balance between performance improvement and execution speed. This method improves the accuracy of prediction by increasing the color dimension, and provides two paths to extract global and local features. Wei et al. [21] are the first to implement retinex theory using deep learning network. This method uses a two-stage network to simulate image decomposition and brightness enhancement. The primary network biases to predict reflectance, and selects the predicted reflectance-weighted illumination. The two-stage network focuses on enhanced illumination with reflectivity guidance. Wang et al. [22] develop an encoder-decoder network to estimate the global illumination. The network splices the input images and global illumination into a large-core convolutional neural network to reconstruct image details. Chen et al. [23] choose two standard U-net networks to process full-resolution images and compare different denoising enhancement schemes through the controlled variable method. The input images collect the original sensor data to make a new dataset that contains short-exposure dark images and corresponding long-exposure matching images. To reduce colour distortion, Wang et al. [24] introduce adversarial optimization in the Retinex decomposition model. In this method, they fuse the feature of the reflectance map, the CRM correction image and the input image predicted by the decomposition network. The decomposition reflectance of the reference image enhance the result of the adversarial constraints. These methods can extract many reference features from paired data, but reference images are difficult to obtain in practice. Some models fuse different exposure images to produce enhanced images with more detail [1]–[3]. The output image can enrich the texture of dark area. The source image is preprocessed to produce a set of matching images. The optimization of different exposure image has obvious influence on the output image.

Low-reference image enhancement　These methods consider the connection between dark images and arbitrary light images. For example, Jiang et al. [10] propose a dual-discriminator adversarial network to balance global and local features. This method lowers the mapping guidelines for matching image. A self-regularized perception loss for network training is used to normalize unpaired data extracted from the input images. Ignatov et al. [25] transform image enhancement into an image style transformation problem. A cyclic consistency confrontation network that maps the enhanced images back to the source image spaces are proposed to relax the need for paired images. Chen et al. [26] improve the bidirectional generative adversarial network with a more advanced global feature enhancement architecture. The model generator selects global features of the enhanced U-net to increase the advanced information and uses the adaptive weighting strategy A-WGAN to accelerate network training convergence. A generative adversarial network is introduced to generate enhanced images, which reduce the threshold of training data appropriately. However, compared with the paired image enhancement methods, the model volume is increased, and more computing resources are consumed.

Non-reference image enhancement　Scholars have just begun to study non-reference images based on deep learning, while traditional method has matured to focus on single-frame image enhancement. For example, Zhang et al. [11] propose the maximum entropy constraint of the fusion of HE and retinex. This constraint also trains the depth model and improves brightness when there is only a single image. The constrained reflectance is used as the enhancement result. However, the model works only on the specific scale and format pictures. Guo et al. [12] convert image enhancement into the image depth curve estimation problem. The special image curve is trained by a convolutional neural network. The method fits a high-level pixel intelligence curve within the input dynamic range for pixel-level adjustments. Although this method does not require a reference image, the brightness must be increased by capturing bright images in the dataset.

The above high-reference image enhancement and low-reference image enhancement methods have achieved good results on low-resolution images, but they are difficult to generalize for practical applications. At present, it is relatively lacking on non-reference low-light image enhancement research. Moreover, we need to research further improvements in the qualitative and quantitative aspects.

III.   The Proposed Method

Classical retinex theory is inspired by the human visual perception system. It suggests that the reflection of natural light should be taken into account in object imaging, as shown in Fig.1. The retinex theory can be expressed as

Figure  1.  Reflectance and illumination decomposed by retinex theory.

DownLoad: Full-Size Img PowerPoint

where $R$ and $I$ are reflectance and illumination respectively. $S$ is the observed images and “·” represents element multiplication. The $R$ is constant under any lighting conditions. The $I$ is not evenly distributed in low-light images.

Inspired by MUDE [27], this paper proposes a colour variation minimization retinex decomposition and enhancement with a multi-branch decomposition network to minimize colour variation. As shown in Fig.2, The CvmD-net includes four major modules: 1) Input constant feature prior mechanism (ICFP). To replace the reference images, we extract the average structural features of the input images based on structural consistency and map them to the decomposition model for optimization. 2) Multi-branch decomposition network (MD-net). We separate the image decomposition and propose a multi-branch decomposition model. The model predicts reflectance and illumination from the difference network. 3) Colour variation minimization. We propose a colour variation minimization function to constrain the minimum difference between the R, G, and B channels of reflectance. 4) Initial illumination equalization adjustment. We extract the average brightness from the equalized images and adjust the initial illumination to match the average brightness.

Figure  2.  MD-net and Ba-net constitute the framework of CvmD-net, and the reflectance uses ICFP to parse the input image.

DownLoad: Full-Size Img PowerPoint

The proposed model is divided into two stages: original image decomposition and brightness enhancement. The decomposition expression $S_{D}$ consists of reconstruction loss $L_{Drecon}$, decomposition adjustment loss $L_{ic}$, illumination smoothness loss $L_{ci}$, and colour variation minimization loss $L_{r}$.

The reconstruction loss $L_{Rrecon}$, colour variation minimization loss $L_{bc}$, and illumination smoothness loss $L_{bi}$ are combined into brightness expression $S_{R}$ as follows:

1   Input constant feature prior

The deep learning algorithm based on retinex decomposition theory [21] trains the decomposition network with paired low-/normal-light images during the image decomposition phase. The Low-decomposition and high-decomposition do not have feature intersections, but are executed alternately in sequence. We predict the colour feature and structural feature of the reflectance by the low-decomposition. The feature information is few when hidden in the dark. The high-decomposition guides the low-decomposition to restore the correct constant feature. When there is not normal-light image guidance, we attempt to obtain prior feature from the input image and constrain the predicted reflectivity. Since the constancy characteristic is the only property that determines the transaction, we disperse only a part of the darkness to restore the overall image content. According to (1), we assume initial illumination $I_{il}$ to estimate initial reflection $R_{ir}$.

We propose ICFP based on initial illumination $I_{il}$ and initial reflection $R_{ir}$ [28]. In decomposition, the initial reflection guides to highlight the colour and texture of the decomposed image. The initial light is generated more naturally during the brightness enhancement phase. Prior illumination $I_{il}$ is based on the maximum value of RGB space, which can calculate prior reflection $R_{ir}$ as

where $I_{il}$ is the maximum value of R, G, and B, i.e., ${\rm{max}}\{R,G,B\}$. 0.0001 is to prevent the denominator from being zero.

Inspred by Zhu et al. [29], we average the three RGB channel histograms of dark image to highlight the colour and structure. According to the equalized R, G and B channels and the maximum value of the three input channels, the decomposition network simulates the high decomposition process, and extracts the constant features of $S_{c\in R,G,B}$ to accelerate the dark area recovery of the low-light images.

where $S_{ij}$ signifies the input R, G and B feature space before and after equalization, $R_{i}$ represents the predicted reflectance, and $I_{j}$ denotes the predicted illumination. Equaiton (2) represents the reconstruction reciprocity loss that enhances the details of the reflection by entering the high contrast of the constant feature prior.

2   Multi-branch decomposition network

The deep learning decomposition [21] model uses a single network to predict reflectance and illumination simultaneously. The two components share the extracted image features, and their interaction hinders the best decomposition results. A large amount of spatial noises are enhanced and retained in the reflectance component. Therefore, this paper proposes a multi-branch decomposition network (MD-net) to predict the reflectance and illumination separately. The network is divided into the reflectance subnetwork (Re-net) based on the U-net [30] and the illumination subnetwork (Il-net) based on the simple network [31].

We choose the reconstruction loss [32] as the basis function, which is based on retinex decomposition theory. The $R$ and $I$ generated by the one-stage network can be reversed into the acquired image $S$, as shown below:

where $R_{i}$ and $I_{i}$ are reflectance and illumination from the initial image, $S_{i}$ represents the input RGB image, and $sy$ signifies the constant characteristic of the input RGB image after equalization.

Excessive structural information from the illumination map makes the enhanced image worse. The illumination must ensure the integrity of the global structure while smoothing the detailed information as much as possible. Here, we choose the illumination smoothness loss [21]:

where $\lambda_{ci}$ and $\lambda$ represent the illumination smoothness coefficient and the enhanced structure gradient weighting coefficient and $G$ denotes the weighting factor of the illumination gradient.

The entire MD-net consists of two branch networks with different structures [33]. The low-light image first input a 3 × 3 convolutional layer to extract low-level image features; then, the standard U-net uses the features to map the reflectance and the convolutional neural network with 4 layers uses the features to map the illumination. The illumination structure smoothly selects the weighted illumination gradient of the reflectance gradient to keep the overall structure of the illumination consistent with the reflectance structure. Finally, the single-layer convolutional layer maps the eigenspace of U-net and estimates the reflectivity. The last layer of the shallow convolutional neural network is set up with 1 × 1 convolution kernel to map initial illumination.

3   Colour variation minimization

The greyscale world theory [34] states that the average intensity of the three channels of the RGB images tends to the same greyscale. Inspired by this theory, we hypothesize that the difference in intensity between RGB primary colour channels is independent of luminance intensity. Based on differences in primary color channels, difference mapping can constrain the reflectivity of the observed image. The reflectance guided by this hypothesis can suppress the spatial noise to approximate the original level [35].

To preserve the colour fidelity and minimize the noise while predicting the reflectance, we propose a colour variation minimization loss. Based on the input constant feature prior, the function shows that the reflected RGB channel area tends to the average intensity. The expression is:

where $\Delta C_{R}$ describes the difference between reflectivity and the mapping of the observed image in the primary color channel. $S^{2}_{o}$ and $S^{2}_{i}$ are the total colour channel deviation from the reflectance and the input image, respectively.

Based on the colour change minimization function, this paper selects the colour change of the input image $S_{{\rm{in}}}$ to constrain the product of illuminance $I_{o}$ and reflectivity $R_{{\rm{in}}}$, so as to prevent overexposure caused by excessive initial illumination.

where $\lambda_{r}$ denotes the chromatic aberration minimization coefficient of reflectance prediction. $R_{ir}$ Estimated initial reflection.

In the brightness adjustment stage, this paper uses the colour variation minimization function by selecting the colour variation of the input image $S_{{\rm{in}}}$ to constrain the product of the enhanced illumination $I_{o}$ and the reflectance $R_{{\rm{in}}}$ to prevent overexposure caused by excessive initial illumination.

where $\lambda_{bc}$ is the colour variation minimization coefficient of brightness adjustment.

4   Initial illumination equalization optimization

The initial illumination brightness predicted by the multi-branch decomposition network is low and has many dark areas. To light up these areas, the common deep learning methods refer to the bright real image. This type of method overcomes the concurrency problem caused by dim light and is better than traditional method. The latest non-reference Retinex model [11] ignores initial lighting optimization. For this problem, the observed mean luminance value is selected as the reference brightness according to the ICFP mechanism. We constrain The light optimized by the brightness enhancement network (Ba-net) to approximate the natural light.

Considering that the images are full resolution in practical applications, the brightness enhancement network adopts the U-net structure. We reduce the scaling operation and perform only 2 downsamplings to simplify the network. The reconstruction function of the illumination from the Ba-net and the reflectance from the initial image is as follows:

where $R_{{\rm{in}}}$ and $I_{o}$ signify the input reflectance and the enhanced brightness. $f(S_{{\rm{in}}})$ represents the reference feature after equalization.

The redundant structure information in $I_{o}$ can contaminate the enhanced image. Thus, we use the illumination smoothness function [21] to reduce the negative impact of the illumination structure. The expression refers to (7) and (8), shown as

where $\lambda_{bi}$ is the coefficient of illumination smoothness, and $f_{s}(\cdot)$ represents the light smoothing operation.

As shown in Fig.3, (b) and (d) are the reflection and illumination predicted by MD-net respectively, and (c) is the final illumination lit by Ba-net. The brightness of $I_{high}$ is significantly higher than the input image and $I_{low}$. $R_{low}$ highlights many detail texture in the dark. The enhanced image after $I_{high}$ adjustment is closer to the real-world scenarios.

Figure  3.  The effect of our network’s decomposition module and brightness enhancement module.

DownLoad: Full-Size Img PowerPoint

IV.   Experiment

We collate the research of similar algorithms in recent years, and select representative results to compare with the work in this paper. This article uses multiple evaluation criteria to assess performance.

1   Experimental settings

1) Datasets

To verify the effect of the proposed method, we use LOL [21] for paired image enhancement experiments and ZeroDCE [12] for non-reference image enhancement experiments respectively. The evaluation images come from the widely used datasets DICM [36], LIME [17], LOL [21], MEF [37], SRIE [16], and VIP. We collect Skynet images at night to test the advantages of our method in the field of video surveillance.

LOL dataset　The first dataset that contains the real images was organized by Zhang et al. [21] and made public on the Internet. All the images are 600 × 400. They change the exposure of the captured image by adjusting the parameters of the camera and combine the results into pairs. We select the fifteen of the 500 real images to evaluate the network. To expand the amount of data, 1000 synthetic image pairs are selected from RAISE [38].

ZeroDCE dataset　The latest non-reference low-light image enhancement algorithm ZeroDCE [12] constructed a dataset according to its own characteristics. All 512 × 512 images are derived from part 1 of the public dataset SICE [39] and 360 multi-exposure sequences. The 2002 total images include the dark images, the normal images and the overexposure images.

Skynet image dataset　To verify the superiority of our work in real-image application, we collect the 1512 night surveillance images from the Skynet system to establish a skynet image dataset. The sample images of the skynet image dataset are shown in Fig.4. For convenience during training, all images are uniformly adjusted to 600 × 400.

Figure  4.  Sample images of the skynet image dataset.

DownLoad: Full-Size Img PowerPoint

2) Evaluation metrics

To comprehensively validate our method from a quantitative perspective, We consider three different evaluation benchmarks to measure the comparative experiments, namely, LOE [40], PSNR [41], SSIM [42].

We use luminance order error (LOE) to measure the difference between enhanced and real light intensity. LOE is expressed as

where ${\rm{RD}}(i,j)$ represents the relative luminance order difference between the real image and the predictive image.

In the evaluation method without subjective judgment, we choose the peak signal-to-noise ratio (PSNR) to calculate the pixel difference after the original image is enhanced. The PSNR can highlight the effect of the image noise.

where ${\rm{MSE}}$ signifies the mean square error between the input image and the enhanced image. ${\rm{MAX}}^{2}$ represents the maximum value of the image colour.

Structural similarity (SSIM) extends the pixel difference to the surrounding area and calculates the structural characteristic and the brightness change in image blocks.

where $L^{2}_{xy}$, $C^{2}_{xy}$ , and $S^{2}_{xy}$ describe the brightness measurement value comparison of the observed image and its estimated result, the contrast measurement value comparison and the structure measurement value comparison. $x$ is the dark image, and $y$ is the enhanced contrast image.

3) Parameter settings

Our experiment is performed on an NVIDIA GTX 1080 GPU using the TensorFlow framework [43]. The batch size is 16, the patch size is 48, $\lambda_{ic}$, $\lambda_{ci}$, $\lambda_{r}$, $\lambda_{bi}$, $\lambda_{bc}$, and $\lambda$ respectively are 0.001, 0.01, 0.05, 5, 0.01, and 0.00009, and the initial learning rate is 0.001.

2   Ablation study

To be clear about the true impact of our focused work on the imagery. This article discusses network loss and two-level network parameters.

1) Colour variation minimization loss

We show the results of training with the three loss of network architectures in Fig.5. $L_{{\rm{SSIM}}}={\rm{SSIM}}()$ and $L_{ir}=\|R_{low}-R_{high}\|_{1}$ are the most widely used reflection estimation loss, which obtain details by formulating strict constraints on the spatial structure. $L_{{\rm{SSIM}}}$ and $L_{ir}$ can amplify the noise considerably, and the dense noise interferes with the detail presentation. $L_{ir}$ ignores the correlation between adjacent regions, which inevitably produces artifacts. Colour variation minimization loss $f_{r}$ takes into account the correlation between the three channels of the image and the consistency of spatial structure, and recovers more texture from noise.

Figure  5.  The contribution of each loss in the network structure.

DownLoad: Full-Size Img PowerPoint

2) The effect of parameter setting changes

We evaluate the reflection estimation of parameters $\lambda_{ic}$, $\lambda_{ci}$ and $\lambda_{r}$ at different combined values. After the same number of iterations, as shown in Fig.6. Fig.6(b) is the worst, almost only useless features visible from the figure. Fig.6(c) lowers the exposure, ignoring textured edges in low-light areas. Fig.6(d) adds structural constraints but creates color pollution at the edges of the structure. Fig.6(e) reduces the smoothness of the light and suppresses noise of large particles. Building on Fig.6(e), we improve the setting of colour variation minimization loss. Fig.6(f) restores more dark detail, and suppresses noise to a low level.

Figure  6.  The conparison of parameter setting changes.

DownLoad: Full-Size Img PowerPoint

3   Performance comparison

1) Assessments on baseline datasets

This paper studies representative retinex algorithms in recent years and analyzes their excellent performance on LoL and ZeroDCE datasets. In the comparison, SRIE, LIME and Robust are traditional image decomposition-enhancement methods. Retinex-net and GLAD-net are deep learning methods without the ability to suppress noise. They train enhanced model by means of the high-reference normal-light image. This paper studies enhancement technology without reference image and need only low-light image. We select several different scene images from DICM [36] and LIME [17] and subjectively compare the enhanced image described above. In Fig.7 and Fig.8, (c) and (f) have a large deviation for the colours, and (c) and (e) is excessive for the brightness and noise. The local information is blocked with the white light. (d) is better than (c) and (4) for the brightness, but it has a few areas with overexposure remain. HE and GLAD-net cause serious colour variation. SRIE achieves insufficient enhancement of the original image brightness. LIME and Retinex-net are relatively complete to restore the image structure and texture, but overexposure occurs. RetinexDIP [44] focuses on enhancing the texture of the structure. Due to parameter limitations, the light changes of the enhanced image are the most deliberate. This paper adopts the non-reference image to enhance the image colour recovery the best. Compared with (e) and (g), (h) is closer to the natural image. The edges of the image texture obtained by our method are smoother, which may be the inevitable reduction of edge differences while suppressing noise.

Figure  7.  Based on the LOL dataset, we select pictures from DICM and LIME to test all comparison methods.

DownLoad: Full-Size Img PowerPoint

Figure  8.  Based on the ZeroDCE dataset, we select images from DICM and SIRE to test a single image approach based on deep learning.

DownLoad: Full-Size Img PowerPoint

2) Evaluation on surveillance images

We use the low-resolution images to conduct the above experiments, while many high-resolution images exist in real scenes. We use a real dataset based on the Skynet images to train our algorithm, and several low-light images with a scale of 3392 × 2072 are collected from video surveillance as the test data. As shown in Fig.9, our work shows good generalization ability in practical applications.

Figure  9.  We use Skynet images to compare the performance of representative methods.

DownLoad: Full-Size Img PowerPoint

3) Numerical analysis

We perform many experiments on baseline datasets, and use mainstream metrics to perform quantitative comparisons on the multiple public test sets. Table 1 lists the values of the three baseline evaluation test comparison methods. The data in the table quantifies the luminance fidelity, pixel-level distortion, and structural similarity of the enhanced observation images. Among them, we change the text colour to highlight the best value on each benchmark. The bold font in the table indicates the best result. Our method is the highest for The SSIM and LOE. The result indicates that our method focuses on restoring the structure and texture characteristics and the enhanced illumination is more in accordance with the changing law of the natural light.

Table  1.  For the comparison of the three benchmark evaluation images, the maximum value of PSNR and SSIM is the best, and the minimum value of LOE is the best

$\text{Method}$	${\rm{SRIE}}$	${\rm{LIME}}$	${\rm{Robust }}$	${\rm{Retinex}} \text{-} {\rm{net}}$	${\rm{GLAD}}\text{-}{\rm{net}}$	$\text{Zhang}\; et\; al.$	${\rm{ ZeroDCE}}$	${\rm{RetinexDIP }}$	${\rm{Ours}}$
${\rm{PSNR}}↑$	$14.41$	$16.07$	$15.06$	$12.05$	$14.63$	$\bf{17.27}$	$16.22$	$17.21$	$17.25$
${\rm{SSIM}}↑$	$0.54$	$0.59$	$0.62$	$0.51$	$0.66$	$0.75$	$0.61$	$0.72$	$\bf{0.76}$
${\rm{ LOE}}↓$	$268.33$	$936.34$	$355.22$	$570.06$	$237.62$	$878.18$	$494.78$	$550.38$	$\bf{230.4}$

 | Show Table

DownLoad: CSV

Fig.10 shows the change in loss as iteration increases, Retinex-Net and ours are both decomposition and enhancement models. We record the change in decomposition loss and enhanced loss. It is obvious that the loss of each method drops significantly at the beginning, and our decomposition loss is less volatile than other methods at a gentle time, with stable iterations.

Figure  10.  Comparison of training effects.

DownLoad: Full-Size Img PowerPoint

V.   Conclusions

In this paper, we propose a two-stage framework CvmD-net which is independent of reference images, and optimize MD-net to estimate reflectance and illuminance by extracting invariant features of original images. The two sub-decomposition networks share the image preprocessing layer and use different frames to extract high-level features. Additionally, we propose a colour variation minimization regularization condition that uses low-light images to constrain the reflectance estimation. The paper also proposes an input constant feature prior mechanism that takes synthesized high-contrast/brightness information to guide the network decomposition and brightness adjustment. Extensive experiments have proven the superiority of our method in the application of uneven lighting.

In the future work, we may begin to recover image details under extremely dark conditions and extend our algorithm to low-light video enhancement.