Remote Interference Source Localization: A Multi-UAV-Based Cooperative Framework

I.   Introduction

With the development of the wireless communication technology and Internet of things (IoT), the number of frequency devices such as wireless communication terminals and the intelligent network terminals is accelerating^[1-3]. However, due to the openness of spectrum access^[4], the number of interference sources that can illegally occupy spectrum resources is also accelerating and have brought grave implications to many fields^[5], such as broadcast channels. Therefore, the demand of an efficient and accurate search and localization method to locate interference source is soaring^[6,7]. However, due to the lack of the interference’s priori knowledge and the unknown but dynamic surroundings, e.g. random background noise, the traditional interference source localization methods need fine-grained search in a huge space which result in inefficient localization^[8-10].

Fortunately, unmanned aerial vehicle (UAV) based interference source localization is promising to tackle this issue due to UAVs’ unique characteristics^[11]. As illustrated in Fig.1, compared with ground-based methods, UAVs can be easily deployed almost everywhere and anytime, due to their ability of flexibly move at higher altitude and thereby less affected by the obstacles. Moreover, the localization can be more accurate and reliable since some signal processing devices and sensors, e.g., electronic scanning antennas, carried by UAVs, suffer less multipath interference. Hence, they are capable of performing effective interference source localization. Therefore, in this paper, we consider a scenario where UAVs are applied to locate the interference source. In order to achieve efficient data acquisition, UAVs are equipped with a electronic scanning antenna that can measure power values of received signals in various horizontal and vertical directions.

Figure  1.  Comparisons between the UAV-based interference source localization and the ground-based interference source localization

DownLoad: Full-Size Img PowerPoint

However, UAV-based interference source localization still suffers from several challenges. Firstly, since the UAVs are powered by battery with the energy limitations, single UAV cannot cover a long-range localization and realize complex computations^[12]. Meanwhile, fully autonomous localization method is needed for UAV-based interference source localization. Due to the complex flying environment and dynamic frequency environment, using ground controllers to manually control the UAVs is difficult to adjust UAVs’ flying directions in time. Thirdly, no priori knowledge of the environments and the interference source can be acquired before the localization task. Therefore, it remains challenging to achieve UAV-based interference source localization with a large cover area, a high accuracy and a low energy cost.

Recently, some works have focused on the UAV-based interference source localization. However, they are unsuitable for remote interference source localization since those methods fail to simultaneously consider the demand of a large cover area, a high accuracy and a low energy cost. Meanwhile, many methods requires the UAVs to approach the interference source which may cost energy waste.

Against this background, this paper proposes a novel multi-UAV-based cooperative framework for effective interference source localization. Based on the proposed framework, a novel collaborative search and localization (CSL) scheme is proposed. The framework allows synthesizing individual decisions within a swarm of UAVs to address the mentioned challenges while the CSL scheme uses multimodal reinforcement learning and fast Fourier transformation (FFT) to achieve effective remote localization. The contributions of this paper are listed as follows.

$1)$ We propose a novel multi-UAV-based cooperative framework to achieve accurate and energy efficient interference source localization in a large area. In order to benefit from the cooperate of multiple UAVs, the framework divides the localization problem into two alternatively performed phases: an RL-based searching phase and a localization phase which can achieve remote location prediction. Meanwhile, we formulate the search phase as a novel multimodal Markov decision process (M-MDP), in which we consider the dynamic environment in reality.

$2)$ A novel CSL scheme based on a low-complexity Q-learning algorithm and an FFT based location prediction algorithm is proposed for a swarm of UAVs to locate an interference source. For the search phase, a low-complexity Q-learning algorithm is proposed. During this phase, each UAV in the swarm can decide its own direction of searching through the proposed Q-learning algorithm. In the localization phase, a novel FFT-based location prediction algorithm is proposed to find out the potential location of the interference source, by comprehensively denoising different UAVs’ predictions in different time intervals.

$3)$ In-depth simulation results are presented to demonstrate the performance of the proposed method. The effectiveness of the proposed CSL scheme is confirmed based on the visualization result of UAVs’ trajectories and the predicted locations. Numerical results show that compared to the baseline schemes, the proposed scheme is more time-efficient and can achieve higher accuracy. It is also shown that the proposed scheme can accurately localize the interference source under low signal to noise ratio (SNR).

The remainder of this paper is organized as follows. The recent works are discussed in Section II. Section III presents the system model and problem formulation. In Section IV, we describe the proposed CSL scheme in detail. Simulation results are presented and analyzed in Section V. Conclusions are drawn in Section VI.

II.   Related Work

Recently, UAVs are studied in many scenarios, such as disaster rescue and UAV-based communication^[13,14]. Meanwhile, UAV-based anti-interference technologies are studied due to their unique abilities. In Ref.[14], the authors proposed a interference source localization method which is achieved by a UAV with an angle of arrival (AOA) array antenna and a ground-based beacon transmitter. However, when there’s no reference stations on the ground, unknown changes of antenna element positions will influence the accuracy of localization. Since single UAV cannot cover a long-range localization due to the energy limitations, attempts were made to employ multiple UAVs for a collaborative search and localization of interference sources. In Ref.[15], a received signal strength (RSS) value based localization method using multi-UAV is proposed. However, such a method is applicable only when transmit power and propagation parameters of the interference source are known.

Since the interference source in the real world often remains random, little priori knowledge of the interference source and the environment can be acquired before the localization begins. Therefore, methods that do not require priori knowledge are studied, including the pre-path methods and the reinforcement learning methods. In Ref.[16], the authors proposed a pre-path method where the search area are divided into different cells and the UAV decides the optimal one as the interference source’s location. However, pre-path methods are not suitable for large area search since the interference source only exists in a small area which causes larger number of redundant way points.

Instead of pre-path planning, reinforcement learning^[17,18] is able to exploit samples and function approximation to optimize performance in dynamic environments. Recently, some researches studied reinforcement-learning-based interference source localization with UAVs^[19,20]. In these works, a single UAV is exploited to locate interference source in unknown dynamic environments. Such reinforcement-learning-based searching methods require the UAV to keep searching until it arrives at the target. They can achieve high localization accuracy but will heavily increase energy consumption while degrading time efficiency. Multiple UAVs can enhance the searching range by collaborative search and localization^[21,22].

In Ref.[21], the authors propose a collaborative search and localization based on reinforcement learning and clustering algorithm. Such method can achieve time efficient localization but has lower accuracy compared to single-UAV-based methods. In Ref.[23], the method achieves higher accuracy by proposing a deep reinforcement learning algorithm. However, due to the high computational cost of the deep reinforcement learning algorithm, such method is not suitable for UAVs equipped with limited computing equipment and power source. Therefore, a multi-UAV collaborative search and localization approach which is both accurate and energy-efficient in complex environments is of the focus of this paper^[24].

III.   System Model and Problem Formulation

In this paper, we aim at addressing the problem of multi-UAV-based efficient interference source localization without a priori knowledge of the interference source model or the noise model. As is shown in Fig.2, the multi-UAV-based cooperative framework for remote interference source localization is proposed where $N$ UAVs, denoted as ${\cal{N}} = \{ 1,2,3, \ldots ,N\}$, are employed as a swarm to collaboratively search and locate an interference source on the ground whose location is ${{p_T}} = (x_T,y_T,0)$.

Figure  2.  The multi-UAV-based cooperative framework for remote interference source localization

DownLoad: Full-Size Img PowerPoint

In the proposed framework, each UAV is equipped with an electronic scanning antenna for interference source localization. In order to improve efficiency, the swarm of UAVs conduct reinforcement-learning-based searching and remote localization iteratively. Specifically, the localization process contains $m$ time intervals, denoted ${\cal{M}} = \{ 1,2,3, \ldots ,m\}$ and each time interval contains three slots, namely, data acquisition (slot 1), action selection (slot 2) and location prediction (slot 3). The reinforcement-learning-based searching is achieved in slot 2 and remote localization is achieved in slot 3. In order to achieve remote localization in slot 3, a cluster head UAV is defined to acquire the search results of all other UAVs while compute the localization results and decide whether the localization is succeed.

1   Data acquisition by electronic scanning antenna

In the studied scenario, each UAV is equipped with a three-dimensional electronic scanning antenna to measure power of receive signals from horizontal, as well as vertical, directions. Each electronic scanning antenna is able to measure $u$ horizontal directions $\{\theta_{1},\theta _{2},$$\theta_{3}, \ldots ,\theta_{u - 1},\theta_{u}\}$ and $v$ vertical directions $\{ \varphi_{1},\varphi_{2},\varphi_{3}, \ldots ,$$\varphi_{v-1},\varphi_{v}\}$, as shown in Fig.3.

Figure  3.  The detection range of an electronic scanning antenna

DownLoad: Full-Size Img PowerPoint

Therefore, the power measured by UAV $U_i$ can be defined as:

where $O_{{\theta _k},{\varphi _j}}^{(i)}$ is the raw data directly sensed by the antenna at horizontal angle $\theta_k$ and vertical angle $\varphi_j$, $g(\cdot)$ is a function which turns the raw data $O^{(i)}$ into the preprocessed data $D^{^{(i)}}$ for ease of further inference.

2   Reinforcement-learning-based action selection

The time-varying coordinate of a certain UAV $U_i$ at time interval $j-1$ can be designated as $(x_{j-1}^{(i)}, y_{j-1}^{(i)},$$z_{j-1}^{(i)})$. The initial coordinate of a certain UAV $U_i$ ($i \in {\cal{N}}$) can be written as $(x_{0}^{(i)},y_{0}^{(i)},z_{0}^{(i)})$. We assume that each UAV flies in a straight line between two adjacent time intervals, and stays at a constant altitude $z_0^{(i)}$. Hence, the position of $U_i$ at time interval $j$ can be expressed as

where $l_j^{(i)}$ represents the horizontal distance between horizontal coordinates $(x_{j-1}^{(i)},y_{j-1}^{(i)})$ and $(x_{j}^{(i)},y_{j}^{(i)})$; $\lambda_{j}^{(i)}$ is the angle between the x-axis and the direction UAV $U_i$ flies towards. In this paper, we assume that $l_j^{(i)}$ is equal to $l$ $\forall i,j$.

Therefore, the action each UAV needs to choose is their own flight direction. At time interval $j$, UAV $U_i$ aims at choosing the optimal action based on the received power set ${D}_{j}^{(i)}$, which can be expressed as:

where $a_j^i \in \left[ 1,u \right]$ and $s_j^i \in \left[ 1,u \right]$ represent the action and state of UAV $U_i$, respectively. $\theta _j^i$ is the reinforcement learning method ${f_r}$’s parameters corresponding to $U_i$. The action selects direction of flights from a set of $u$ possible flight directions $\{\theta_{1},\theta_{2},\ldots,\theta_{u} \}$.

3   Location prediction by multiple UAVs

As is shown in Fig.4, since UAVs conduct 3D sensing, the maximum-power direction corresponding to the current flight direction can be achieved. The extension of such direction will intersect with the interference source’s plane. The intersection essentially approximates a possible position of the interference source. In this paper, we study the scenario where there is only one interference source. Intuitively, in the ideal case, the intersection represents the interference source’s location. Therefore, the intersections of different UAVs can be used for remote location prediction.

Figure  4.  Intersection of UAV’s trajectory at time interval $j$

DownLoad: Full-Size Img PowerPoint

After the flight direction $\eta_i$ is chosen at time interval $j$, UAV $U_i$ flies in its direction for a distance of $l$. The maximum power value received on horizontal angle $\eta_i$ can be expressed by $\omega_i$. Moreover, since the intersection and the interference source are in the same plane, the intersection can be expressed in a 2D coordinate form for ease of computation. Therefore, the intersection ${p_{j}^{(i)}}$ of the interference source’s plane and the maximum power’s direction at time interval $j$ can be obtained by

However, due to the noise, multi-path effect, etc., there can be a large number of intersections and may have large deviation from the interference source. In order to reduce the deviation and make precised prediction, intersections of different UAVs’ trajectories and time intervals are considered. Therefore, precised prediction $\hat p$ is achieved by the location set ${\bf{L}}= \left\{p_{j}^{(i)} \right\} (i,k \in$$[1,n],i < k,\forall j)$, given as:

where ${\bf{L}}$ is the intersection set, ${f_l}$ is the prediction process and $l$ is the predicted location.

4   Problem formulation

In the proposed framework, the problem of locating an interference source is divided into two parts: the searching phase and the localization phase. The searching phase aims at finding the optimal trajectories for each UAV, while the location phase aims at achieving the optimal predicted location based on UAVs’ trajectories.

In the search phase, changes in measured power of the UAVs’ received signals can indicate whether the distance between the UAV and the interference source is reduced. Therefore, for each UAV, in order to find out the position of the interference source, the direction of flight in each time interval can be selected according to measured power. Such a search problem can be equivalent to maximizing the expected long-term measured power, which can be formulated as the discounted sum of all future rewards (i.e. measured power) at current time interval $t$. Considering the dynamic environment, such a problem can be modeled as a multimodal Markov decision process (M-MDP) problem, which aims at finding the optimal policy $\pi^{(i)}$ for each UAV $U_i$. Different from the MDP problem which is a tuple $<S,A,r,P, \gamma>$^[25,26], the M-MDP problem consists of six elements $<S,A,r,E,P, \gamma>$ as follows:

$\cdot \ S$ is the state set;

$\cdot \ A$ is the action set;

$\cdot \ r$ is the reward function;

$\cdot \ E$ is the modality set;

$\cdot \ P$ is the transition probability matrix;

$\cdot \ \gamma \in [0,1]$ is a discount factor.

For the interference source localization problem, the UAVs attempt to take more accurate actions when approaching the target for higher RSS. Under this circumstance, the state set and the action set can be adjusted to reduce the redundancy states and actions. Therefore, the modality set is introduced. For certain time interval, the modality can be calculated by the long term state memory $M({S_1}, \ldots ,{S_{t - 1}})$, given as:

where for a given time interval $i$, the UAV’s modality is the maximum probability of transferring to $e$ based on the state memory.

When ${e_t}$ is defined, its corresponding state set and action set can be expressed as $S_{e_t}\in S$ and $A_{e_t} \in A$, respectively. Therefore, as is shown in Fig.5, the search space is reduced and the policy $\pi_e$ has higher possibility to select the optimal action. Meanwhile, the reward function can also be adjusted to $r_e^{(i)}$.

Figure  5.  The reduced search space of the proposed M-MDP

DownLoad: Full-Size Img PowerPoint

The goal of the agent is to maximize the expectation of discounted sum of all future measured power under different modalities, given as

where the policy ${\pi '}^{(i)}$ indicates UAV $U_i$’s mapping from its state space $S^{(i)}$ to action space $A^{i}$ under different modalities, $s_e^{^{(i)}} \in S_{e}^{(i)}$ and $a_e^{^{(i)}} \in A_{e}^{(i)}$ denote the state and action selected at the current modality $e \in E$, respectively. $r_e^{^{(i)}}$ is the reward function constraint to the current modality.

The localization phase can be equivalent to minimize the horizontal distance between the predicted location and the actual location, given as:

where $p_T^{'}$ is the horizontal coordinate of the interference source’s location and $\hat p$ is the predicted location.

IV.   Collaborative Search and Location

Based on the proposed multi-UAV-based remote localization framework, we design the CSL scheme where the search phase is based on a lox-complexity Q-learning algorithm while the localization phase is achieved by a FFT-based location prediction algorithm.

As demonstrated in Fig.6, in the search phase of every iteration (i.e. each time interval), each UAV individually decides the direction of flight in the next time interval by performing reinforcement learning. To find out the potential position of the interference source, the intersections of UAVs’ trajectories are processed by the FFT-based clustering algorithm during the localization phase. The FFT-based clustering algorithm first reduces the noise among the candidate intersections by FFT and calculate the predicted location based on the denoised intersections. The search phase and the localization phases will be implemented until the estimated position meets the terminating condition.

Figure  6.  The flowchart of the proposed collaborative search and localization scheme

DownLoad: Full-Size Img PowerPoint

1   Lox-complexity Q-learning algorithm

In order to achieve efficient reinforcement-learning-based search, a lox-complexity Q-learning algorithm is proposed based on the M-MDP problem. As is shown in Fig.7, the proposed algorithm consists of the data acquisition unit, the modality recognition unit, the reward function unit, the Q-table update unit and the action selection unit. In order to achieve lox-complexity and high-efficient searching, the data acquisition unit and the modality recognition unit determine the current modality $e$ to dynamically adjust the following units.

Figure  7.  The architecture of the proposed lox-complexity Q-learning algorithm

DownLoad: Full-Size Img PowerPoint

1) Data acquisition unit

Considering the random disturbance of power of receive signals, for each direction, the antenna performs sampling for $N$ times and the acquired data can be defined as

where the raw data $O_{{\theta _k},{\varphi _j}}^{(i)}$ acquired by the antenna is normalized as the acquired data by the UAV. $D^{^{(i)}}_{j,k}$ is the acquired data at the horizontal angle ${\theta _k}$ and the vertical angle ${\varphi _j}$ the current state.

2) Modality recognition unit

Based on the acquired data, the modality of each UAV can be determined. Although the environment changes dynamically in the searching phase, the noise always ranges in a fixed interval while signal power of the interference source will get larger as the distance getting smaller. Thus, the classification basis of different modalities are identified by the UAV’s acquired data, given as:

where $\mathbb{E}(\cdot)$ indicates the average function, $\partial(\cdot)$ is the variance function. The current modality $e$ is defined by the threshold function $h$ which maps the number of modality to the variation coefficient of the acquired data (i.e., ${{\partial ({D^{(i)}})} / {\mathbb{E}({D^{(i)}})}}$).

3) States and actions of UAV

In the proposed low-complexity Q-learning algorithm, the action $a^{(i)} \in \{ 1,2,3, \ldots , u\}$ selects direction of flights from a set of $u$ possible flight directions corresponding to the detection directions of the antenna shown Fig.3. The state $s^{(i)}$ is the direction selected by UAV $U_i$ in the previous time interval, e.g., state $s^{(i)}$ is the action ${a'}^{(i)}$ in last time interval, given as

4) Reward function controlled by multimodal recognition unit

In original Q-learning algorithm, the update function of reward table is fixed based on the state and action. However, when the change of modality is considered, adjustments need to be done to the original update function of the reward. Specifically, the update range will be adjusted based on the current modality, given as

where $\mu$ is the update range, ${\varepsilon _e}$ is a discount factor based on the current modality. Therefore, higher efficiency is achieved by partially updating the reward function. Furthermore, in this paper, the improved reward function is given as:

where if the potential action ${a^{(i)}}$ is within the update range, the reward $r_e^{(i)}({s^{(i)}},{a^{(i)}})$ equals to the ratio of the maximum power on the current horizontal direction ${\max (D_{{a^{(i)}},:}^{(i)})}$ to the maximum power of all directions ${{\max (D_{:,:}^{(i)})}}$. The update range is centered by the current state, since it is the most likely direction of the interference source

5) Q-learning update and action selection

In this paper, in order to accelerate the convergence of Q-learning, we simultaneously update action values $Q^{(i)}(s^{(i)},:)$ with various actions for a given state $s^{(i)}$. Hence, different from the conventional Q-learning, the update rule is given as

where $Q^{(i)}(s^{(i)},:)$ collects action values ranging from $Q^{(i)}(s^{(i)},\theta_1)$ to $Q^{(i)}(s^{(i)},\theta_u)$, which are the action values with current state $s^{(i)}$ and all possible actions. Similarly, $Q^{(i)}(s^{'(i)},:)$ collects qualities of actions w.r.t. the previous state $s^{'(i)}$.

Once action values $Q^{(i)}(s^{(i)},:)$ is updated, each UAV selects the optimal action by performing:

where the selection range is corresponding to the update range in (16).

After action $\hat a^{(i)}$ is chosen at time interval $j$, each UAV flies in its direction $\theta_{\hat a^{(i)}}$ for a distance of $l_e$ which is defined by the current modality. The maximum vertical direction $\varphi _j^{(i)}$ can be achieved, given as:

where ${\pi / {2v}}$ is the sampling interval on the vertical angles. The intersections can be achieved based on (6), given as

The intersection will be added into the intersection set ${\bf{L}}= \left\{p_{j}^{(i)} \right\}(i\in [1,n],\forall j)$. The proposed low-complexity Q-learning algorithm is summarized in Algorithm 1.

Algorithm 1　The proposed collaborative localization scheme

1: Initialize the number of UAVs $n$;

2: Initialize flight direction set $d=\left\{1,2,3 \ldots ,u \right\}$;

3:   Initialize learning rate $\alpha$, discount factor $\gamma$ and step length $l$;

4   Set iteration number $j=1$, operation command $flag0=false$;

5: For $i = 1 \to n$

6: Initialize UAV $U_{i}$’s position $\left( x_{0}^{(i)},y_{0}^{(i)},z^{(i)} \right)$;

7: Obtain current environment by Eq.(11);

7: Obtain current modality $e^(i)$ by Eq.(12);

8:    Initialize $Q_e^{(i)}\left(:,: \right)$ and $r_e^{(i)}\left(:,: \right)$ to 0 (i.e. a zero matrix);

8: Select an initial state $s^{(i)}_{0}=randi(d)$;

9: End For

10: Repeat

11: For $i = 1 \to n$

12: $e_0=e^{(i)}$;

13: Obtain current modality $e^(i)$by Eq.(12);

14: If ${e_0} \ne {e^{(i)}}$

15: Update $\mu^{i}$,$l_e^{(i)}$

16: End If

17:      Obtain measured values and update reward table $r_e^{i}$ according to Eq.(15);

18: Update action values according to Eq.(16);

19: Select action by Eq.(17);

20: Update current state $s^{(i)} \leftarrow \hat a^{(i)}$;

21:      Update $U_{i}$’s position according to Eq.(1) and Eq.(2);

22:     Compute $U_{i}$’s intersection according to Eq.(18) and Eq.(19);

23: End For

24: $j=j+1$;

25: Obtain the intersection set ${\bf{L}}$;

26: do FFT-based location prediction (which is shown in Algorithm 2);

27: Obtain operation command $flag0$;

28: Until $flag0=1$

2   FFT-based location prediction algorithm

As is shown in Fig.8, we propose the FFT-based location prediction algorithm to achieve efficient remote localization of the interference source. The FFT-based location prediction algorithm contains three units. The direct localization unit directly presents a prediction based on the current result of the low-complexity Q-learning algorithm. The FFT-based denoising unit aims at reducing the bias in a sequential predictions presented by the former unit. The stability-based stopping strategy aims at deciding whether the proposed CSL scheme terminates.

Figure  8.  The architecture of the proposed FFT-based location prediction algorithm

DownLoad: Full-Size Img PowerPoint

1) Direct localization based on searching result

The proposed FFT-based location prediction algorithm for UAV $U_i$ is performed when its current reward $r_e^{(i)}({s^{(i)}},:)$ satisfies a constraint in the variation coefficient:

Since the reward $r_e^{(i)}({s^{(i)}},{a^{(i)}}) \in [0,1]$, the variation coefficient ${{\partial (r_e^{(i)}({s^{(i)}},:))} / {\mathbb{E}(r_e^{(i)}({s^{(i)}},:))}} \in [0,1]$ holds. Therefore, given a small value of $\lambda \in [0,1]$, a certain $r_e^{(i)}({s^{(i)}},:)$ satisfying (20) means that $r^{(i)}({s_{j}^{(i)}},:)$ suffers less noise or multi-path interference. As a consequence, the direction (i.e. action $\hat a^{(i)}$) of flight obtained by the reinforcement-learning-based searching algorithm can be aligned with the direction of the interference. Therefore, at a certain time interval $j$, only in the case when $r_e^{(i)}({s^{(i)}},{a^{(i)}})$ satisfies (20), can the intersection $p_j^{(i)}$ be reliable for localization.

In each time interval, an estimate of the position of the interference source is obtained. At a certain time interval $j$, the estimate can be achieved by

where $C$ denotes the set of all intersections that satisfy the Eq.(20) at time interval $j$, $p_j$ is the direct predicted location by the swarm of UAVs.

2) FFT-based denoising of the predicted locations

Due to the noise, multi-path effect, etc., the direct prediction is an approximate representation of the interference source’s location which includes certain disturbance and can be expressed as

where $x_{_b}^j$ and $y_{_b}^j$ denote the disturbance on the X axis and Y axis. In this way, predictions can be achieved by reducing the disturbance on the X axis and Y axis. Then, a queue $K = \{{p_{j - \chi +1}}, {p_{j - \chi +2 }}, \ldots ,{p_j}\}$ containing $\chi$ elements is defined and the disturbance is reduced by FFT^[27,28]. Since the disturbance on the X axis and that on the the Y axis are mutually orthogonal, the reduction of disturbance is divided into 2 parts, denoted as the X-axis reduction and the Y-axis reduction, respectively, to get higher accuracy. Each part contains an FFT process to convert the prediction queue from the the time domain to the frequency domain. A cut-off number is defined to filter exceptional data and an Inverse FFT (IFFT) process is conducted to recover the queue from the frequency domain. The FFT process can be expressed as:

where ${{x_{{p_{j - n + 1}}}}}$ denotes the horizontal ordinate of ${p_{j - n + 1}}$, ${{y_{{p_{j - n + 1}}}}}$ denotes the ordinate of ${p_{j - n + 1}}$. $(x_k^F,y_k^F)$ denotes the transformed coordinate. Let ${X^F} = \{ x_1^F, x_2^F,$$\ldots ,x_\chi^F\}$ and ${Y^F} = \{ y_1^F,y_2^F, \ldots ,y_\chi^F\}$, the cut-off numbers of the X-axis reduction and the Y-axis reduction are denoted as $\frac{1}{4}\max (\left| {{X^F}} \right|)$ and $\frac{1}{4}\max (\left| {{Y^F}} \right|)$, respectively. ${X^F}$ and ${Y^F}$ are then filtered by their cut-off number. For $\forall {x_k^F} \in {X^F}$,the filtered data ${x_k^F}'$ can be expressed as

Similarly, for $\forall {y_k^F} \in {Y^F}$,the filtered data ${y_k^F}'$ can be expressed as

Afterwards, the recover process is achieved by IFFT and can be expressed as

where ${p_{j+n-1}}'=({x_{{p_{j - n + 1}}}}',{y_{{p_{j - n + 1}}}}')$ is the denoised coordinate. The denoised intersection queue $K'$ is then achieved, where $K' = \{ {p_{j - \chi + 1}}',{p_{j - \chi +2 }}', \ldots ,{p_j}'\}$. We then compute the centroid of $K'$ by performing

where $p_{j}^{c}$ denotes the center of $K'$ at time interval $j$. Hence, $p_{j}^{c}$ is the predicted location at time interval $j$.

3) The stability-based stopping strategy

As is shown in P2（see Eq.(10)）, the ideal stopping strategy is to estimate the deviation between the predicted location and the location of the interference source. However, since the location of the interference source is unknown, we select $p_{j}^{c}$ as the final predicted location of the interference source when $P_{j}^{T}$ converges over time intervals. In this work, if the standard deviation $\sigma$ of the latest three predicted coordinates $\left\{ {p_j^c , p_{(j - 1)}^c, p_{(j - 2)}^c} \right\}$ satisfies

then the collaborative search and localization terminate, and $p_{j}^{c}$ is output as the predicted location of the interference source. Meanwhile, the stop sign $flag0$ will be set to 1 and broadcast to other UAVs. When the UAVs receives $flag0=1$, they will terminate their search phase and the cluster head UAV will send the predicted location to the ground center.

The proposed FFT-based location prediction is summarized in Algorithm 2.

Algorithm 2　The proposed FFT-based location prediction

1:  Obtain the result of the low-complexity Q-learning: intersection set ${\bf{L}}$;

2:  Obtain current iteration number $j$ and operation command $flag0$;

3: Initialize $flag1=0$, $C = \emptyset$;

4: For $i = 1 \to n$

5: If $p_j^(i)$ satisfies Eq.(20) Then

6: $C=C \cup {i}$;

7: $flag1=1$;

8: End If

9: End For

10: If $flag1$ Then

11: Compute $p_{j}$ by Eq.(21)

12: Pop $p_{j - \chi}$ from $K$ and push $p_{j}$;

13:    Transform $K$ into frequency domain by Eq.(23) and Eq.(24);

14: Filter the transformed data by Eq.(25) and Eq.(26);

15:     Achieve the denoised queue $K'$ by Eq.(27) and Eq.(28);

16: Pop $p_{j - \chi}$ from $K$ and push $p_{j}$;

17: Compute the centroid of $K'$, $p_{j}^{c}$, by Eq.(29);

18: End If

19: If $p_{j}^{c}$ satisfies Eq.(30) Then

20: $flag0=1$;

21: End If

V.   Simulation Results

1   Simulation settings and performance metrics

1) Simulation settings

In the simulations, we consider one interference source located at (5000, 2885, 0) (m) with transmit power of $20$ W and a swarm of three UAVs. Initial positions of the UAVs are set as (0, 0, 200) (m), (5100, 8660, 190) (m) and (10000, 0, 190) (m), respectively. Each UAV is equipped with an electronic scanning antenna which can explore $u=36$ directions and the interval between each direction is ${\pi / {18}}$. The antennas have the same radiation characteristic and is given by

Then, the receive gain $G_{R}(\theta)$ of each electronic scanning antenna for the horizontal angle $\theta$ and elevation angle $\varphi$ is represented as follows:

where $\theta \in [0,2\pi )$ and the antenna efficiency $\beta$ = 1. Thus, received power at each antenna can be obtained as

where $O_{T}$ and $G_{T}$ represent transmit power of the interference source and transmit antenna gain, respectively. The wave length $\lambda$ is set as 3 m; the loss factor $L$ is set as 1; $n^{2}$ stands for power of noise signal, which is a random variable with an average value of −38 dBm.

In Algorithm 1, three modalities are defined $E=\{1,2,3\}$, given as

where ${{\partial (D_0)} / {\mathbb{E}(D_0)}}$ is a constant indicating the maximum initial variation coefficient of the swarm of UAVs. With the increase of $e$, the received signal power of the UAV also increases, indicating that the UAV is approaching the interference source. Therefore, the discount factor of the update range in (14) can be written as

which means that the UAVs explore less direction when getting closer to the interference source in order to balance the searching time and accuracy. Meanwhile, the step length $l_e$ of the UAV is given by

In the FFT-based location prediction, $\lambda$ in Eq.(20) is set as 0.2, and queue $K$’s length $\chi$ is set as 150.

2) Performance metrics

We propose current reward ratio (CRR) in order to measure the convergence of different schemes, given as

where ${\max (r_e^{(i)}({s^{(i)}},:))}$ denotes the max reward value at the current state. Meanwhile, we assume that each UAV in the swarm consumes 1 unit power per meter when flying and consumes 5 unit power per second when conducting inference. The total power ${\cal{P}}_all$ consumed by a UAV is given as

where $l_{all}$ is the total flight length and $t_{in}$ is the total inference time.

2   The benchmark schemes

In this paper, we compare our proposed methods with three benchmark schemes, namely the SCAN scheme^[16], the directional Q-learning scheme^[20] and the RL-TC scheme^[21].

The SCAN method is a SOTA pre-path scheme. As is shown in Fig.9, the SCAN scheme divides the region into equal cells and the center of each cell is considered as a waypoint. The UAV will observe the environment at the waypoints and define whether there exists the interference source in the corresponding cell. In this paper, the SCAN scheme first divides the area into $200 \times 200$ cells. When the UAV finds the optimal cell, the SCAN scheme then divides the cell into $20 \times 20$ sub-cells. The UAV will report the optimal sub-cell as the interference source’s location.

Figure  9.  The SCAN method’s search policy

DownLoad: Full-Size Img PowerPoint

The directional Q-learning scheme achieves single UAV-based interference source localization based on the reinforcement learning. The authors modified the rule for updating the quality of the state-action combinations in the reinforcement learning and the multiple potential flight directions of the UAV can be simultaneously evaluated.

The RL-TC scheme achieves remote interference source localization based on multiple UAVs. The RL-TC scheme uses a low-complexity RSS-based reinforcement learning method at the search phase in order to decrease the computational time. At the localization phase, a two-stage clustering algorithm is proposed in order to reduce the affect of the singular points.

3   Performance comparison

Fig.10 depicts UAVs’ trajectories achieved by different approaches. The SCAN method can locate the interference source but has the longest path. The proposed algorithm and the RL-TC scheme can locate the interference source from a distance. However, the proposed algorithm enables the UAVs to realize localization from a distance of 2.9 km while the RL-TC scheme requires the UAVs to get closer but achieves lower localization accuracy. The directional Q-learning algorithm can achieves higher accuracy at the cost of time efficiency which requires the UAVs to approach the interference source.

Figure  10.  UAVs’ trajectories achieved by different approaches

DownLoad: Full-Size Img PowerPoint

Fig.11 investigates the convergence of different reinforcement-learning-based schemes. Compared with other RL-based methods, the proposed algorithm has the highest convergence speed, which can converge after 600 time intervals and locate the interference source after 640 time intervals. Therefore, it can be concluded that the proposed algorithm selects the optimal action at each time interval more stably since the modality recognition unit enables the UAVs to focus on the most possible directions of the interference source based on their previous knowledge. The directional Q-learning algorithm has similar convergence performance (about 800 time intervals) compared to the RL-TC scheme since the search phase of the RL-TC scheme and the directional Q-learning scheme use similar algorithms. However, the RL-TC scheme takes less time intervals to locate the interference source compared with the directional Q-learning scheme due to the ability of the remote localization. Although the RL-TC scheme and the proposed scheme take similar time intervals to locate the interference source after the RL methods converge, the proposed scheme reduces the computational complexity of the localization scheme from $O(n^2)$^[29] in the RL-TC scheme to $O(n{\rm{log}}(n))$^[30].

Figure  11.  The convergence performance of different schemes

DownLoad: Full-Size Img PowerPoint

As is shown in Fig.12, we evaluate the relationship between the number of the UAVs’ flight directions and the localization accuracy and efficiency. 19 different direction numbers are evaluated, i.e, Ref.[18]. Since the proposed scheme predicts the location by long-term search results from different UAVs, the localization accuracy is not heavily influenced by the change of flight directions and the bias can remain in 30 m. However, the efficiency is heavily influenced by the flight directions. While the UAVs only need 403 time intervals to locate the interference when they can fly in 36 directions, they can consume up to 758 time intervals when the direction number decreases. It can be concluded that, when the search phase is coarse-grained, i.e. the flight directions are fewer, the localization phase needs more search data to achieve accurate localization and therefore consumes more time intervals.

Figure  12.  The influence of the number of the flight directions on the efficiency and accuracy

DownLoad: Full-Size Img PowerPoint

As is shown in Fig.13, the effectiveness of the proposed FFT-based location prediction algorithm is investigated. The singular points in the direct predictions of the UAV swarm are effectively reduced by FFT. The result of the proposed prediction algorithm can achieve the high accuracy and the localization bias is within 25 m.

Figure  13.  The effectiveness of the proposed FFT-based location prediction (the noise is −28 dBm)

DownLoad: Full-Size Img PowerPoint

As is shown in Table 1, the time efficiency and power efficiency of different schemes under a high SNR are compared. Compared with the Rl-based schemes, the SCAN scheme has the fastest inference time at each time interval, which only need 0.01 seconds, but consumes the most power due to the longest flight distance. Due to lack the ability of remote localization, the directional Q-learning scheme takes up to 1267 time intervals to locate the interference source. The RL-TC scheme consumes the most time for inference due to the high computational complexity of the two stage clustering algorithm. However, since the RL-TC scheme can locate the interference source from a distance, the total power consumption is less than that of the directional Q-learning scheme. The proposed scheme needs more inference time than the directional Q-learning scheme but achieves the least total time consumption and power consumption due to lower computational complexity of the FFT-based remote localization algorithm.

Table  1.  The efficiency of different reinforcement-learning-based schemes under the noise of −28 dbm

	Inference time per interval (s)	Total time intervals	Total power consumption
The SCAN scheme	0.010	885	175268.50
The directional Q-learningscheme	0.022	1267	38428.11
The RL-TC scheme	0.041	814	24920.61
The proposed scheme	0.034	633	19957.83

 | Show Table

DownLoad: CSV

We compare the localization accuracy of different schemes under different noise conditions in Fig.14. The SCAN method achieves the most stable performance under all SNRs due to the pre-path planning, while the RL-based methods’ localization biases decrease by the SNR. The directional Q-learning scheme has the highest accuracy since it can approach the interference source to realize the localization. Since the proposed scheme employs the 3D single feature and has better convergence performance, it achieves similar localization accuracy with the directional Q-learning scheme when the noise is below −27 dBm while the accuracy difference increases when the noise exceeds −28 dBm. The RL-TC scheme achieves the lowest accuracy because it achieves remote localization by the 2D trajectories which varies in low SNR condition.

Figure  14.  The localization bias of different schemes under different noise conditions

DownLoad: Full-Size Img PowerPoint

VI.   Conclusions

In this paper, a multi-UAV-based remote localization framework is proposed for interference source localization. In order to achieve higher efficiency, the localization of an interference source is innovatively divided into two procedures. Based on the proposed framework, a CSL scheme is proposed that can simultaneously achieve efficient searching and accurate remote localization. At the searching phase, each UAV can independently explore the environment and efficently search for the interference source by the proposed low-complexity Q-learning algorithm. The results of each UAV are collaboratively analyzed in the localization procedure. At the localization phase, a novel FFT-based location prediction algorithm is presented in order to achieve accurate localization. The simulation results confirm the time efficiency and the localization accuracy of the proposed scheme.

For future work, more improvement will be done about the multi-UAV based collaborative search and localization. The situation of multiple UAV swarms and multiple interference sources will be studied.