View Synthesis in Tidal Flat Environments with Spherical Harmonics and Neighboring Views Integration

I.   Introduction

By basing the new view synthesis on a set of sequential images of a scene, the goal is to generate realistic images of the same scene at the new viewpoint. Early research in new viewpoint generation focused on geometric and image rendering techniques to generate new viewpoint images by modeling the geometric structure of the scene and rendering the images. The 3D structure of the scene is reconstructed from multiple images with different viewpoints, and then the reconstructed 3D model is used to generate new viewpoint images [1]–[4].

With further technological developments, light field rendering [5], [6], view-dependent texturing [7], and more modern learning-based approaches have been proposed, where image rendering methods typically operate by deforming, resampling, and/or blending the source view to the target viewpoint. Such approaches can achieve high-resolution rendering but usually require very dense input views or explicit proxy geometries, which are difficult to estimate with high quality, leading to artifacts in the rendering.

Recently, NeRF has achieved incredible results in new viewpoint synthesis, obtaining extremely realistic new viewpoint images in complex real-world situations by using neural network weights to record scene features through 5D inputs, combined with high-dimensional position-encoded multi-layer perceptron (MLP) representations of the scene. Although NeRF has achieved impressive results in new view fusion, when using the NeRF technique for view fusion of tidal flats, an overly blurred rendering occurs. NeRF operates by sampling a set of points along the ray and approximating the segmented successive integrals as an accumulation of estimated volumetric features at the sampling intervals. However, the MLP can only be used at a fixed set of discrete positions where queries are made, and the same fuzzy point sampling features are used to represent opacity at all points with intervals between samples, leading to ambiguity in the neural rendering. The features of the samples can vary significantly on different rays projected from different viewing directions. Thus, simultaneous supervision of these rays to produce isotropic volumetric features may result in artifacts like blurred surfaces and blurred textures.

To overcome this limitation of NeRF and to improve the rendering quality, some works [8]–[10] have introduced shape-based sampling into scene representation. By embedding new spatial paradigms such as Gaussian ellipsoids, truncated bodies, and spheres, these models can reduce representation ambiguity, improve rendering quality, and reduce blurring and aliasing artifacts.

In this paper, we present a new radial field representation of the density and appearance of tidal flats in neural rendering, better reconstructing the geometry and appearance of the scene and generating more realistic images from new viewpoints. The viewpoint orientation information is queried using a spherical harmonic function and transformed into a set of coefficients that compactly and efficiently represent the complex orientation distribution. While rendering light, image features from adjacent viewpoints are used for fusion, performing density, occlusion, visibility inference, and color blending. This allows our system to operate without any scene-specific optimizations or precomputed proxy geometry. The core of our approach is to go beyond considering only the information in the current viewpoint during rendering, aggregating information from adjacent views along a given ray to compute their final colors. For 3D positions sampled along the ray, potential 2D features from nearby source views are acquired and then aggregated at each sampled position via a transformer to produce density features. The final density value is then computed via the ray converter module by considering these density features along the entire ray, enabling visibility inference across larger spatial scales.

II.   Related Work

1   Based on neural scene representation

Constructing radial fields from image features of the input view to render new viewpoints has become a hot research topic, using neural networks to represent the shape and appearance of the scene. Earlier research [11]–[15] has shown that MLPs can be used as an implicit shape representation to approximate highly complex shapes, utilizing MLPs’ ability to model nonlinear functions, ensuring a smooth transition of the SDF (Signed Distance Function) values throughout the space by mapping any given 3D point coordinates to the corresponding signed distance values.

With the proliferation of microrenderable methods [16]–[19], eliminating the need for hard-to-obtain 3D ground truth data and using two-dimensional images for supervision, the process of microrendering allows for end-to-end training of the network, making it possible to learn 3D representations directly from 2D image data. Some scholars focus on extracting geometric features of a scene from simple geometry and multiple views of diffuse materials [20]–[23]. NeRF [24] achieved impressive results in novel view synthesis by optimizing the 5D neural radiation field of a scene, an achievement that has attracted the attention of many researchers. However, traditional NeRF techniques are difficult to apply directly to large-scale, high-resolution UAV imagery.

To address this, Jonathan T. Barron et al. proposed mip-NeRF [8], which uses a conical table instead of traditional light sampling to solve aliasing issues in the reconstruction process. Matthew Tancik et al. presented Block-NeRF, which divides a large-scale, borderless scene into chunks, rendering an entire neighborhood of San Francisco [25]. Turki et al. proposed Mega-NeRF, applying a weighted averaging strategy to filter out differences in overlapping regions, successfully training NeRF on UAV images [26]. Many researchers are now validating the potential of NeRF in urban environments [27]–[29], as well as in remote sensing and satellite imagery [30]–[32].

However, there are fewer studies using similar techniques applied to tidal flat conservation, where significant differences exist relative to urban environments and satellite imagery due to the complexity and dynamics of tidal flats’ natural ecology.

2   Tidal flats monitoring and protection

Environmental conservation through image acquisition by UAVs has been a noteworthy research focus and currently plays an important role in forests [33], artificial wetlands [34], and species conservation [34]–[36]. In this paper, UAVs collect images of tidal flats for perspective fusion to monitor the health of these areas.

Current research largely focuses on monitoring the health of tidal flats from a globally vast macroscopic perspective using time series images [37]–[39]. However, relatively few studies integrate multi-scale perspectives on tidal flats at smaller scales. Small-scale 3D reconstruction can capture subtle changes on the surface of tidal flats, such as erosion, sedimentation, and vegetation changes, which are often not accurately reflected in large-scale monitoring. Monitoring key areas of the tidal flat ecosystem at a smaller scale, such as salt marshes, mudflats, and intertidal zones, can more accurately capture these subtle changes in critical areas.

3   Image rendering technology

Early work in image rendering was based on weighted blending of reference pixels to synthesize a set of reference images into a new perspective picture [6], [40], where blending weights are computed by calculating ray-space proximity or by proxy geometries [5], [40], [41]. There has been increasing research interest in proxy geometries [42], [43], optical flow corrections [44]–[46], and soft blending [47], [48], which improve the quality of generated images. For example, two types of multi-view stereo [49], [50] are used to generate a mesh surface associated with a view, and then a CNN computes the blending weights. Alternatively, a radiation field is generated based on a mesh surface [51] or a point cloud [52], [53].

While these methods achieve good results in some cases and can handle sparser views than other methods, they are fundamentally limited by the performance of 3D reconstruction algorithms [49], [50]. 3D reconstruction tends to fail in low-textured or reflective regions and is not well-suited for handling partially translucent surfaces, whereas tidal flats have many low-textured or reflective regions. In our method, continuous volume density learning is used to optimize the quality of synthetic new view images for better performance in challenging scenes like tidal flats.

III.   Methods

1   Neural radiance field

Neural Radiance Fields (NeRF) were introduced as a groundbreaking method for synthesizing 3D scenes through the application of deep learning, signifying a significant advancement in computer graphics and 3D modeling. NeRF utilizes a fully connected deep neural network to map 5D coordinates (spatial $x, y, z$ and 2D viewing directions) to color and volume density.

The radiation field can be conceptualized as a function where the input is a ray in space $r(t) = o + t \cdot d$ (where $r \in {\mathbb{R}}$). This function allows us to query the density $\sigma$ of the ray $r(t)$ at each point $(x, y, z)$ in space, as well as the color $C(r)$ rendered in the direction $d$ of the ray. The density $\sigma$ also signifies the probability that a ray will terminate at this point in space and controls the amount of radiation absorbed by other rays as they pass through the point.

When rendering an image for a given position $o$ and direction $d$, the radiation from all points along a given ray $r(t)$ is accumulated to compute the color value $C(r)$ of the corresponding point in the image. Formally, this is represented as:

where $T(r(t))$ represents the cumulative transmittance along the ray up to $r(t)$ and is defined by

In this formulation, $t$ represents time, with $t_n$ and $t_f$ as the start and end points. The rendering outcome results from the interaction of three critical factors: cumulative transmittance $T(r(t))$, density $\sigma(r(t))$, and color $c(r(t), d)$. The interaction of $T(r(t))$ and $\sigma(r(t))$ serves as a “color weight” parameter, quantifying the remaining light intensity at a specific point and its corresponding density. As depicted in Equation (2), this relationship follows an inverse exponential pattern, meaning that a higher density at a given point results in a lesser amount of light penetrating beyond that point.

In the actual rendering process, the discrete forms of Equation (1) and Equation (2) are represented as follows:

In Equation (3), $\delta_i = t_{i+1} - t_i$ represents the distance between consecutive sampling points along the ray. The relation between $\sigma(r(t))$ and $1 - \exp(-\sigma(r(i)) \cdot \delta_i)$ has been demonstrated in previous work [54].

The optimization of NeRF involves minimizing the mean square error (MSE) loss between the predicted image $\hat{C}(r)$ and the ground truth image $C(r)$, specifically:

2   feature extraction network

As shown in Figure 1, at a given location of the generated image, we collect images from adjacent viewpoints and use these adjacent viewpoint images in the rendering process of the new viewpoint. This approach leverages the colors and densities in the images from neighboring viewpoints to generate a more realistic rendering at a specified viewpoint. Additionally, a spherical harmonic function is introduced to process the viewpoint information, ultimately providing the density $\sigma$ and color of the new viewpoint. The loss function is then calculated to complete the rendering of the new viewpoint image.

Figure  1.  System overview.

DownLoad: Full-Size Img PowerPoint

To render the target view, we first select the view whose direction is closest to that of the target viewpoint from the acquired neighboring views. The number of views $N$ is selected based on GPU capacity and forms the set of neighboring views used for rendering the new viewpoint. To prepare for rendering, features are extracted from these neighboring viewpoint images using a shared-weight network.

The image features are extracted using the network architecture outlined in Table 1. We implement the feature extraction network using the architecture based on ResNet34 [55], implemented in PyTorch [56]. We replace all batch normalization [57] with instance normalization [58], as shown in [59], and remove the maximum pooling layer, substituting it with a cross-row convolution. Our network is fully convolutional and can accept variable-sized input images. For instance, we use a single image of size $640 \times 480 \times 3$ as an example input.

Table  1.  Generate image feature network structure

Input (id: dimension)	Layer	Output (id: dimension)
0: 640 × 480 × 3	7 × 7 Conv, 64, stride 2	1: 320 × 240 × 64
1: 320 × 240 × 64	Residual Block 1	2: 160 × 120 × 64
2: 160 × 120 × 64	Residual Block 2	3: 80 × 60 × 128
3: 80 × 60 × 128	Residual Block 3	4: 40 × 30 × 256
5: 40 × 30 × 256	3 × 3 Upconv, 128, factor 2	6: 80 × 60 × 128
[3,6]: 80 × 60 × 256	3 × 3 Conv, 128	7: 80 × 60 × 128
7: 80 × 60 × 128	3 × 3 Upconv, 64, factor 2	8: 160 × 120 × 64
[2,8]: 160 × 120 × 128	3 × 3 Conv, 64	9: 160 × 120 × 64
9: 160 × 120 × 64	1 × 1 Conv, 64	Out: 160 × 120 × 64

 | Show Table

DownLoad: CSV

Use $I_i \in [0,1]^{H_i \times W_i \times 3}$ to denote the $i$-th neighbouring view of the target perspective, and $P_i \in {\mathbb{R}}^{3 \times 4}$ to denote the $i$-th camera projection matrix of the target perspective. The image features $F_i$ are obtained by the above shared weight network structure, and for each new viewpoint rendering, the input tuple $\{(I_i, P_i, F_i)\}_{i = 1}^N$ is constructed as the input to the network.

3   Volume density prediction

Our proposed method renders the final 2D image by accumulating the colours and densities on the rays, using the classical neural radiation field as a baseline for generating the new view image. The difference is that our method aggregates the body densities and colours of adjacent views used to generate the new viewpoint image.

The image features are extracted using a network of common parameters to aggregate the features of the views under the target viewpoint $d$. The image is then rendered as a 2D image. Also, in order to make it more likely that 3D points on a surface will have a consistent low-local appearance across multiple views, an effective way to maintain the consistency of $F_i$, we use a point network architecture [60], as shown in Figure 2. Multi-view features are employed and variance is used as the global pool operator. Firstly, the mean $\mu$ and variance $v$ of each image extracted feature vector $F_i$ are calculated to consider the image extracted features from a global perspective and construct the global link between image features. Subsequently, the feature vectors $F_i$, mean $\mu$ and variance $v$ are concatenated to obtain the new feature vector $F_N'$ and the weight parameter $w_n$ through MLP, and then these two parameters are mapped to the density feature $F_\sigma$ using MLP.

Figure  2.  Rendering a new perspective picture flowchart using adjacent views.

DownLoad: Full-Size Img PowerPoint

Our network inputs consider neighbouring view information, but do not expect all neighbouring views to carry the same weight in the final density and colour generation process, and expect neighbouring views that are close to the target viewpoint to carry more weight in the rendering process. We use the pooling technique proposed by Sun et al [61]. Our weighting function is defined as follows:

Similarly, we set $s$ as a learnable parameter to further control the weight occupied by neighbouring views with different viewpoint gaps in the rendering treatments through the neural network. Obtaining the density feature $F_\sigma$ can be done directly using a neural network to turn it into the $\sigma$ needed for rendering, but direct conversion produces images with blurring and artefacts in the new view. We believe that further use of global information is needed, and we employ a ray transformation module that is able to make the most of the information on the ray. We use the classical Transformer consisting of two core components: positional encoding and self-attention.

The ray converter consists of two key components from the classical transformer [62]: positional encoding and self-attention. Given $M$ samples along a ray, our ray transformer processes the samples as a sequence from near to far, applying positional encoding and multi-head self-attention to the dense sequence of features $(F_\sigma(X_1), \dots, F_\sigma(X_M))$. It then predicts the final density value $\sigma$ based on the features provided by each sample.

4   Fusion colour prediction

For the set consisting of the selected neighboring views, we consider that a smaller distance from the object view $d$ to the neighborhood view $d_i$ implies a greater likelihood that the color of the target view is similar to the respective color of view $i$ and vice versa. To predict the weights of neighboring views for each target view, as shown in Figure. 2, we link $F_N'$ and the viewpoint difference $d_{\text{target}} - d_i$ by mixing the weights with the following weight function:

Similarly, we concatenate the feature vector of the target view to participate in the training and then obtain the weight value of the color. The color value contributed by each adjacent view to this current feature point is obtained by multiplying the weight and color values, and all the color values are summed to obtain the final color. In this case, the color in the current view is used with the same spherical harmonic function as in instant-ngp [63] to get $C_{\text{target}}$, as follows:

where $d(\theta, \phi)$ indicates the line of sight direction, $\theta$ is the pitch angle, and $\phi$ is the yaw angle. The parameter $j$ describes the “order” of the spherical harmonic function, representing the number of ripples of the function on the sphere, and takes a non-negative integer value, $j = 0, 1, 2, \dots, J$. The parameter $m$ describes the variation of the function in azimuth for a given order; $m$ takes any integer value between $-j$ and $j$, $m = -j, -j+1, \dots, 0, \dots, j-1, j$. $P_j^m$ is the $j$-th $m$-order associated Legendre function, defined as follows:

IV.   Experiments

1   Datasets

We first evaluated our model on a publicly available da-taset: 7 scenes (3 outdoor and 4 indoor), each contains a complicated central object or region and a detailed background. Each of these types has about 200-300 or so images. We then similarly validate our algo-rithm on the tidal flats dataset from our earlier work. The main com-parison algorithm ne-facto [64]. In this study, we used UAV tidal flats imagery acquired from a UAV flyover tidal flats environment for perspec-tive fusion. The specific location where the imagery was acquired is along a stretch of thousands of kilometers of Australian coastline, specifically at an altitude of 1000 feet between Smithton and Woolnoth in northwestern Tasmania. It has a diverse intertidal ecosystem that in-cludes a variety of tidal flat types and ecological land-forms. As shown in Figure 3, this data enables a comprehensive assessment of the effectiveness of our algorithms in a tidal flat environment. We further classi-fied the acquired images by intertidal ecosystems, in-cluding “tidal trees”, estuaries, ground texture, plants and deeply watered areas, every scenario is containing 30-90 images that were finely processed and standardized for resolution and color fidelity. A series of standardization steps were used to ensure the consistency and quality of the image data. First, we calibrated the images to remove color deviations and geometric distortions due to light-ing and camera settings. Next, we segmented and filtered each image to exclude low-quality or unsuitable images for analysis.

Figure  3.  Location map of the survey sample area.

DownLoad: Full-Size Img PowerPoint

2   Methodology of the evaluation

Assessment of the accuracy of the quality of the resulting synthesized images and the new views generated, we uti-lize three key metrics: PSNR, SSIM, and LPIPS. These metrics allow for a comprehensive assessment of the structural similarity, luminance contrast, and perceptual differences between the synthesized and real images, en-suring a thorough evaluation of the new views’ quality and similarity. Peak Signal-to-Noise Ratio)(PSNR)is a conventional in-dex for measuring image quality. The formula is as fol-lows:

To assess the accuracy and quality of the resulting synthesized images and new views generated, we utilize three key metrics: PSNR, SSIM, and LPIPS. These metrics allow for a comprehensive assessment of the structural similarity, luminance contrast, and perceptual differences between the synthesized and real images, ensuring a thorough evaluation of the new views’ quality and similarity.

Peak Signal-to-Noise Ratio (PSNR) is a conventional metric for measuring image quality. The formula is as follows:

The PSNR value is positively correlated with the image quality and is usually used to quantify the similarity of the resulting image.

The Structural Similarity Index (SSIM) indicates the structural likeness between the resulting image and the original, including contributions from luminance, contrast, and texture. By calculating the SSIM value, the degree of resemblance between the generated image and the real image can be evaluated:

LPIPS (Learned Perceptual Image Patch Similarity) [65] is a metric for assessing image quality and similarity. Proposed by Richard Zhang et al. in 2018, it is widely used in the fields of image processing and computer vision. LPIPS aims to assess the perceptual similarity between two images, focusing on how the human visual system perceives image quality and similarity rather than pixel-wise differences. LPIPS uses a trained deep learning model (typically a convolutional neural network, CNN) to extract image features. These features capture complex patterns and structures in an image, reflecting human perception more accurately. The metric assesses similarity by comparing localized image chunks (patches) of an image, allowing for a detailed analysis of the local details and textures.

3   Experimental results

As indicated in Table 1, we used a comparison between our method and Nefacto’s results on a public dataset, and we can see that our method outperforms Nefacto in PSNR and SSIM, though it is slightly inferior in LPIPS. As illustrated in Figure 4, our approach involves comparing the images by actually generating them.

Figure  4.  Public dataset comparison results.

DownLoad: Full-Size Img PowerPoint

We likewise validated the specifics of our algorithm on a tidal flats dataset of our own making as shown in Figure 5, where it can be seen that our algorithm achieves much superior results.

Figure  5.  Comparison of renderings of tidal flats environments.

DownLoad: Full-Size Img PowerPoint

Through a contrasting experiment, our approach con-sistently outperforms the conventional NeRF algorithm in both image and accuracy. This result fully demon-strates the higher reliability of our method in modeling and rendering tidal flat environments. Our method is able to capture scene details more accurately and provides a powerful tool for tidal flat research and conservation.

Among the evaluation metrics used, our method is very advantageous.

Table  2.  Comparison results for the borderless scenario dataset

Scene	PSNR		SSIM		LPIPS
	Our Method	Nefacto	Our Method	Nefacto	Our Method	Nefacto
Bicycle	22.74	19.62	0.649	0.497	0.384	0.385
Bonai	27.33	21.40	0.872	0.693	0.252	0.302
Counter	25.51	24.01	0.798	0.744	0.328	0.331
Garden	25.01	23.06	0.745	0.627	0.282	0.284
Kitchen	25.92	26.00	0.789	0.791	0.302	0.299
Room	28.54	22.32	0.891	0.777	0.259	0.303
Stump	15.97	15.96	0.369	0.345	0.736	0.727
Average	24.43	21.77	0.730	0.630	0.363	0.376

 | Show Table

DownLoad: CSV

Table  3.  Comparison results for the borderless scenario dataset

Scene	PSNR		SSIM		LPIPS
	Nerfacto	Ours	Nerfacto	Ours	Nerfacto	Ours
Tidal Trees	20.46	21.77	0.685	0.736	0.231	0.171
River Mouths	25.52	24.90	0.418	0.463	0.189	0.145
Ground Textures	21.65	22.18	0.464	0.411	0.351	0.209
Vegetation	23.39	23.27	0.771	0.787	0.240	0.229
Deep-Water Areas	20.49	20.34	0.429	0.443	0.384	0.383
Average	22.30	22.49	0.553	0.568	0.279	0.227

 | Show Table

DownLoad: CSV

V.   Discussion

In this article, we have compared the proposed algorithm with the most advanced Nerfacto algorithm. At first, we performed comparison trials on a public dataset. The outcome shows that our proposed algorithm is comparable to the state-of-the-art algorithm in most of the indoor and outdoor scenarios. However, in the Stump scene, it is difficult to clearly distinguish between the foreground and background because they are very similar. This may be due to the fact that during the scene formation process, the values are sampled along the light and composited into a single color. Similarly, in the Tidal Flats environment, although our method is able to generate corresponding images in new viewpoints, some subtle features are still lost in some viewpoints. While our method performs well in all train and test images, it also performs well when viewing the content of the scene from an approximately constant range; however, significant aliasing artifacts appear in the rendered scene at some feature points, resulting in overly blurred images at specific viewpoints.

In the follow-up work, we refer to some current stage solutions to related problems. For example, in order to improve rendering, we will let multiple rays pass through each pixel for sampling. However, this is very costly for a neural volume representation technique like NeRF, as hundreds of MLP computations are required to render a single light ray, and reconstructing a scene can take days. Another more efficient approach is to use projection cone techniques, which encode the shape and size of the ray paths and enable multi-scale modeling of the scene by training neural networks.

VI.   Conclusion

In this document, we would like to present a facet of a novel view synthesis approach, especially tailored for tidal flat environments, by introducing a new radial field representation for density and appearance in neural rendering. Our method significantly improves upon the rendering quality and reduces blurring and aliasing artifacts compared to existing techniques such as Nerfacto.

Through extensive experiments on both public datasets and our custom tidal flats dataset, our algorithm demonstrated superior performance in generating realistic new viewpoint images. Despite some challenges in distinguishing foreground from background in highly similar scenes, our method effectively captures the complex geometry and appearance of various tidal flat features.

Future work will explore further optimizations and extensions of our method, such as incorporating more advanced sampling techniques to address remaining challenges and enhance the overall efficiency and quality of the rendering process. By continuing to refine our approach, we aim to provide robust tools for environmental research and conservation efforts, particularly in dynamic and complex ecosystems like tidal flats.