TALS image processing

UAV Automated Landing System (TALS) based on optical recognition. Here is model of data processing, simulates video from drone camera. Captured video is viewed to define featured points – markers in runway.

Detected marker is labeled by green rectangle, that will be one of the targets in the video frame. Group of targets defines runway and its spatial position qq663pn. Tracking system traces group of targets during landing and produce control signals for the autopilot.

Model provides different runway moving, including X, Y, X rotation and change distance to camera. Speed of video changes correspond to UAV dynamic capabilities and allows reliable tracking process.

Each target string shows number of label (green rectangle), label size, rotated angle and aspect ratio. These data is used in autopilot tracker for grouping of labels and track every label in time.

For better browsing, use 720p (HD) or 1080p resolution and full-screen mode.

Below you can see landing video preprocessing before tracking this article. Used plane captured video from www.justplanes.com

TALS detects landscape feature points and runway markers. At the approach stage, TALS seeks feature (reference) points – FP, labeled in the video as FP01 … FP04, not all FP marked. FP outputs to TALS tracking subsystem.

At the landing stage, system discovers runway and locked runway image for coordinate processing.

UAV TALS: Tracking Automatic Landing System

TALS (Tracking Automated Landing System) is an automatic landing system addressed to UAV applications. It is based on two independent and cooperative sections whose combined processing allows to feed attitude and position parameters to UAV autopilot.

The first section is based on a AHRS platform integrated with a GNSS/SBAS corrected signal (GPS,GLONASS,GALILEO or any other available satellite positioning system implementing augmentation). It is able to provide the UAV position with accuracy better than 3m.

The second section combines the information coming from a class of electronic sensors with information coming from real-time landing runway image processing, acquired through a properly aimed CMOS HD camera, or any other camera available on the UAV conforming to TALS specifications.

TALS may process real-time runway images in combination with preloaded images of the same runway or execute instant scanning of current runway image to search for landing reference points extracted from UAV maps data base. Reference points may be also received during the mission by remote command and control base, in case of change of final destination for landing. In this case, landing surface may be not necessarily an airport runway, provided that site for landing is identified with the proper reference points.

Both sections operate in automatic mode, creating a single common interface to UAV autopilot. The combined mode implemented in TALS reduces the bias error in calculation of UAV position and speed, allowing a significant increase in guidance parameters refresh frequency. TALS is able to recognize any kind of object present in the preloaded images used as a model; therefore landing scenario are possible also in absence of conventional runway and airfield ground light

The main key factor of TALS is that it’s a totally passive system as it works using runway and Airfield Ground Lighting (AGL) images as they are and/ or geodetic maps reference points . Thanks to TALS, safe landing of UAV will be always possible on runways not equipped with expensive navigation aids on the ground, such as ILS or MLS.

This peculiarity may result of crucial importance in applications where ILS or MLS , even if present on the runway, must be kept off due to security reasons. TALS may be therefore installed in UAV already equipped with ILS/MLS automatic landing systems, allowing the mission operator to decide where and when to use the most appropriate solution according to operational scenario.

Additional TALS advantages come from the invariance of reference images (with or without lights) by environmental conditions , while radio connection established between the ILS/MLS transmitter and UAV on board receiver may be subject to environmental and weather conditions. Ground reflections, in band interferences, presence of water or snow in the sensitive area of ILS/MLS beam, may in fact change the received reference path, producing wrong flight path indications for UAV autopilot.

TALS protects the value of UAV, and related payload, increasing safety during the landing, to be always considered as the most critical phase in the UAV mission. During final approach and landing , UAV may in fact be exposed to sudden changes of wind speed and direction (wind gust) creating serious troubles to attitude control of UAV. In such a scenario, TALS is able to provide accurate and real-time calculations for a better mitigation of turbulence effects.
TALS offers the following main performances during Approach and Landing phases:

  • very accurate acquisition of current glide angle (real angle referenced to gravitational horizon, between UAV and runway touch down zone)
  • calculation of :
    • transversal bank referenced to runway axis;
    • time to touch down;
    • distance to runway threshold;
    • UAV altitude over the runway;
  • High updating frequency of autopilot flight parameters in the final phase of landing to increase the effectiveness of any UAV autopilot during turbulence management
  • video signal of runway available to UAV remote pilot for supervisory purposes. Visual information may be integrated with markers to give the remote pilot additional information about runway and touch down zone position, current glide path angle and time to touch down.

Operational Phases

GNSS/SBAS (GPS,GLONASS) data are used by TALS to initially drive the UAV on the correct glide path angle and to align it to the runway axis.

Lock-on phase

lock1

Camera based auto-landing subsystem works in parallel with the GPS based one. The subsystem searches in the real-time acquired images for peculiar points of landing runway, extracting the information necessary to flight control, both in the horizontal and vertical plane. Such visual information will be fused with the other ones coming from other sensors (accelerometers, gyroscopes, altimeters, inclinometers) to feed in real time the UAV autopilot with requested parameters for approach and landing , with appropriate updating frequency and accuracy.

Tracking phase

lock2

Information from GNSS and camera based subsystem, are compared to check for data reliability. Frequency of UAV autopilot parameters updating is increased to 120 Hz.

Flare and Touch Down phase

targetTALS sensors will provide at 120Hz UAV autopilot (and remote pilot) with following parameters:

  • the UAV altitude over the runway (with 200 mm accuracy) attitude speed and position.

 

TALS operations will stop at the end of touch-down phase.

Technical Details

TALS diagram

TALS diagram

The figure shows the general sensors configuration of TALS.

TALS basically works collecting images in realtime using a CMOS high definition, colour camera, operating in the visible spectrum , with a 30° Field of View (FOV) CMOS HD. Camera is a Gimbal type , to be installed (if not already present in UAV) under the nose of UAV. Landing runway recognition is based on detection of runway image peculiarities (features) and information given by the following AGL lighting systems:

  • PAPI (Precision Approach Path Indicator) bars
  • Approach Lighting Systems
  • Runway Edge
  • Runway Centreline
  • Threshold and Threshold Wings
  • Touch Down Zone (TDZ)

TALS needs a system data base loaded with two images of runway, shot at a close distance in day and night light, or loaded with geodetic references and refernce points of landing site according to UAV flight mission map. When available coordinates of TDZ lights , runway directions and altitude will be also stored in the system data-base.

TALS will properly work even in case of a poor GNSS/SBAS signal. Integration of visual information with data coming from sensors of AHRS platform allows for output of calculated values of UAV attitude and heading, including glide path angle and bank referenced to the runway axis. Data output frequency is maintained to 120 Hz even in case of loss of GNSS/SBAS signal. TALS can be therefore used as a back-up of UAV standard ILS based landing system drastically increasing autolanding capability of UAV.

During landing, video signal of runway may be available to UAV remote pilot for supervisory purposes. Visual information may be integrated with markers to give the remote pilot runway and touch down zone position, current glide path angle and time to touch down.

Below shows parameters and reference values calculated by TALS. Measurement of distance to runway threshold and transversal bank need GNSS/SBAS signal available.

  • Current landing angle [deg]
    Range 0-10 deg
    Dynamic accuracy 0.08 deg
  • Time to touch-down [sec]
    Accuracy: 0.5 sec
  • UAV altitude over the runway
    Range 0-18 km
    Accuracy: 0.2 m (at touch-down phase)
  • Distance to runway threshold
    Range: 0 – 3.5 Nm
    Accuracy: 2.5 m
  • Transversal bank referenced to runway axis [deg]
    Range -10..10 deg
  • UAV roll, pitch, heading angles
    Range -90..90 deg
    Dynamic accuracy 1.0 deg
  • Information data rate
    Up to 120 Hz
  • Video frames with markers output rate
    10 Hz
  • UAV max. speed
    2100 km/h
  • Max. dynamic measurement
    4 g

TALS output signals may be adapted to every UAV command data port. System response time (120 Hz) allows for a prompt reaction to sudden changes (severe wind gust) of UAV attitude in the final phase of flight.

TALS runs on a SBC hosting the CPU, working memory and FPGA section to be devoted to high speed image processing algorithms. CPU collects data from sensors and CMOS HD camera to process and transfer parameters to UAV autopilot for landing. SBC provides for a communication interface to send/receive data to/from remote command and control base.
Processing unit will conform with standard MIL-STD-810F (shock, vibrations, temperature, humidity) and MIL-STD-461E for EMI/EMC.

 

TALS Performances

TALS performances have been compared with ILS ones through a simulation carried out applying Monte Carlo methodology and considering CESSNA 172 flight parameters. The purpose of simulation was to establish a reference for TALS performances in comparison with a known and certified system as ILS is worldwide.

The simulation model was based on aircraft equations and turbulence equations as they are set in international ICAO/FAA standards, assuming the presence of TALS or CAT II ILS system, with ILS errors within limits given in ICAO, Annex 10, Vol. I, Chapter 3.

Typical functions of a conventional autopilot have been included in the model, such as:

  • internal control loops (for pitching, roll, yaw);
  • basic functions (for en- route autopilot, altitude and speed)
  • auto-landing function.

Operational parameters in the simulation model were set to:

  • distance to landing point: 12200 m
  • initial altitude range: 584 – 784 m
  • initial misalignment (referenced to runway axis) range: -200..200 m
  • turbulence zone extended from 9000 m to 400 m from touchdown point
  • turbulence intensity range: 7 – 15 m/s
  • glide path angle: 3°.

 

SIP/VoIP Gateway for the Controller – Pilot Radiolink

VCS, VoIP, SIP, Voice Communication System, ED-136, ED-137, Primaria Ltd, VoIP Gateway

SIP / VoIP gateway is based on the carrier board with a digital bus, on which the bladed interface modules can be installed (see starting from the upper right corner). The photo shows 6 modules installed and two more free space for the left. Interface modules have connectors for external audio devices. In the lower left corner is shown embedded ARM / DSP computer module.

The gateway is designed to provide the functions of digital voice “Voice over IP” (VoIP) and switching media streams via SIP in the air traffic voice system controller – pilot, according to the requirements of the standard ED-136 and ED-137 A / B EUROCAE.

As assembling unit, the gateway contains the base plate, which is set up to 8 interface expansion cards to connect external audio devices. The gateway acts as CWP site, and as a Radio Gateway that is installed on the receiving – transmitting radio center. In this case, the gateway has only one type of interface expansion card, which provides E & M interface for the external radio receiver or transmitter. In this version, a radio gateway can connect up to 8 stations.

Gateway in CWP Site version

Interface cards have the following versions for connecting devices:

  • headset, up to 2 units;
  • gandset with PTT button, up to 2 units;
  • microphone, up to 2 units;
  • footswitch PTT, up to 2;
  • external speakers, up to 8 pcs., 4 W each.

The gateway is connected to:

  • external power supply + 24V;
  • Remote Control and Managment System via Ethernet interface;
  • external documentation system;
  • redundant Ethernet interface to work in the VCS;
  • Ethernet interface for communication with the CWP control panel visit their website.

Optionally in the CWP Site version SIP / VoIP gateway can be connected to the CWP control touch screen panel shown in the figure. This device is also developed by Primaria and shipped separately in different versions.

VCS, VoIP, SIP, Voice Communication System, ED-136, ED-137, Primaria Ltd, VoIP Gateway

CWP touch screen control panel. It contains an open-frame display and capacitive touch glass. It can be equipped with additional built-in microphone and video camera. Contains a built-in two speakers and allows the connection of two redundant handsets

Gateway in Radio version: Radio Gateway

This version is used to connect non – VoIP, legacy radio stations. Radio gateway can also be supplied separately as expand card for a customer radio tranciever to conform ED-136, ED-137 specifications.

In this version Radio Gateway  is installed in a rack with radio equipment together to ensure the following connections:

  • Up to 8 full-duplex E&M radio interfaces (each interface provides PTT and SQ commands);
  • external power + 24V;
  • documentation system;
  • redundant Ethernet interfaces for operation in the VCS.

Gateway Radio version and CWP Site version differs only by expansion interface cards set and software.

Common Gateway Features

Gateway uses an embedded ARM / DSP computer module. On the DSP heavy application is loaded, such as echo cancellation algorithms, speech codecs with audio compression and image processing with built-in video camera.

The gateway contains built Ethernet-Switch Moxa company EOM-104 for the fast detection of the Ethernet connection failure network and switching to a backup.

The gateway G.703 (E1) interface is provided for recording voice signals to an external digital recorder, the type of RJ-45 connector.

Interface expansion cards provide the following connections.

In the CWP SIte version:

  • headset interface, variant 1: two Lemo-type connector for the headset with microphone and PTT button, the handset with the PTT button, microphone button PTT, PTT foot pedal. Microphone supply voltage from 5 to 12V;
  • headset interface variant 2: one connector DSub type for simultaneous connection of a combination of 2 of the following devices: headset with microphone and PTT button, the handset with the PTT button, microphone button PTT, PTT foot pedal;
  • speaker interface: two stereo connector type Jack Ø3.5 mm.

In the Radio Gateway version:

only one type expansion board: E&M module with one RJ-45 connector.

Electrical characteristics:
– Input impedance: 600/1200 ohms ± 20% at a frequency of 1 kHz;
– Output impedance: 600/1200 ohms ± 20% at a frequency of 1 kHz.
The value of the impedance is set with switches on the board.

Power is supplied from one of two independent DC power supplies with voltages from 18 to 32 V. The transition from one power source to another is provided without interruption in the work.

Passive Coherent Locator

The main focus of the company is a Passive Coherent Location (PCL). Primaria developed a PCL prototype named “Prim Ceramic 2700”. Passive radar uses components with a high value of the intellectual property. This IP is based on research and development programs in the field of PCL and Direction Finders signal processing.

The key components of the passive radar “Prim Ceramic 2700” are:

  • antenna system with suppression of the direct path;
  • RF module with high dynamic range, providing a direct conversion radio signal to quadrature I/Q channels;
  • coherent adaptive filter using Gram – Schmid orthogonalization procedure and multiple regression filter;
  • Ambiguity Function module producing Range bins & Dopler bin matrix;
  • plot extractor, based on CFAR algorithm;
  • tracker of the multistatic section that performs targets association.

More about Passive Radar “Prim Ceramic 2700” is described below.

History

Primaria team originates from the 1980s and develop Airport Direction Finders (DF) with a focus on software and hardware signal processing. Starting from 2013, the main project of the company is the development of a Coherent Passive Radar “Prim Ceramic 2700”.

The Direction Finders signal processing was implemented in the following products:

Continue reading

Passive Radar Prim Ceramic 2700: Signal Processing Unit

Prim Ceramic 2700 Signal Processing Unit contains a chain RF Module -> ADC – > FPGA -> DSP. In the terminology of PCL, the unit is a frequency channel, or a bistatic section. Prototype includes two bistatic sections, allow resolve targets due to crossing two ambiguity ellipses.

PCL, пассивный радар, пассивный локатор, LinkWare.RU, LW Ceramic 2700

Passive Padar Signal Processing Unit: RF module, high performance ADC and Signal Processor based on ARM/FPGA cores

On the photo bottom there is the RF module that provides gain and reception signal from the PCL antenna system. RF module is designed and manufactured by Primaria for the product line associated with the radio signal processing, especially for PCL. Output of the module performs two analog I / Q channels. The RF module provides I2C interface for remote control and configuration. The module has a synthesizer and a local oscillator used in the the Master mode. In the Slave mode, the reference frequency received from the other RF module which operates in Master, for coherent reception and processing of multifrequency.

Board features:

  • input frequency range: 50 MHz – 2 GHz
  • synthesizer stability: 10-7
  • input noise level: 6 dB
  • IF bandwidth: 5 – 240 MHz

I/Q components from RF module feed via coaxial cable to the high speed ADC on the mesonin board with high – density FMC connector.

PCL prototype Signal Processor is based on ZedBoard kit (on the photo top) with ARM and FPGA cores. On the right side of the board there is a FMC connector that is connected to a high-speed four-channel ADC board. ZedBoard feeds the parallel ADC samples to FPGA and then processed data direct to the ARM core. Vivado tools are used for the ARM and FPGA cores software development. ARM kernel is running under Linux Embedded.

Then, module BearleBoard provides FFT with DSP for calculation two-dimension ambiguity function.

The signal processor digitizes coming from RF module I/Q signals at a rate of  250 M/samples per second parallel data using 16-bit ADCs. To increase the dynamic range oversampling technology and special digital signal filtering are used.

The signal processor performs the following functions as PCL bistatic section:

  • adaptive filtering based on the Gram – Schmidt orthogonalization procedure and multiple regression filter;
  • continuous calculation Ambiguity Function matrix with the dimension Range bins x Dopler bins;
  • plot extractor based on an CFAR algorithm.

Produced plots are packed into the stream with frame structure and transmit on a prototype multistatic section via Ethernet internal network.

The number of the “Prim Ceramic 2700” prototype Signal Processors corresponds to the number of bistatic radar sections, that is depends on the number of frequencies used.

Multistatic processing is implemented on a multi-core x86 platform and provides the target associations, ambiguity resolution and building targets tracks.

The photo below shows the assembly of the prototype. Numbers denote:

PCL, Passive Coherent Location

Assembled PCL Prototype”Ceramic-2700.

1. Antenna Steering Unit
2. Target Channel (Reflected Signal)
3. Reference Channel (FM Stantion Singnal)
4. Control interface I2C for the RF modules: set frequencies and other parameters
5. IF I/Q output
6. Pass – band filters
7. Reference frequency for coherent two channels
8. Hi speed 2x channel ADC FMC module with 250 MSamples/s, producing 16 – bit I/Q
9. 2x channel adaptive filter based on ARM / FPGA
10. ARM / DSP board calculate ambiguity function and plots with CFAR method
11. Ethernet switch linking modules

Multichannel and tracking processing perform on external PC, using Python OpenGl modules for real-time displaying targets.