Autonomous Drone Software E05: Vision-Based Auto-Landing

This tutorial will give an example of using QR code to guide the landing process of a drone.

Give us a Star 🌟 on Github if you find this tutorial useful.

Help us to understand what you want to learn in the upcoming tutorials by fill out this three-question survey.

---------------------

We suggest developers to install GAAS mirror to use the project:

GAAS v0.7 Release Mirror - x64

Should you have any questions or suggestions to the tutorials or GAAS itself, let us know and join our slack

Previously we have talked about how to set up simulation, how to control your drone with python, how to use SLAM in GPS denied environment for drone positioning and how to avoid obstacles with the help of a pair of stereo cameras. In this tutorial, I will give a basic realization of drone auto landing with the help of a QRCode.

A QR code provides rich visual information for locator, tracker and identifier. It usually consists of many black squares in a grid. For more information, refer to Reading Material [1].

We will be using a python package called pyzbar to detect and decode QRcode in this tutorial.

Fig 1, QRcode

We will mainly be looking at four corners of the QRcode, which are called "Position Pattern". Below is the decoded result of above QRcode and you can see corner positions are given with the help of pyzbar.

The target is placed on a white background of a size 752 x 480 in the x and y direction, note the z direction of the image is perpendicular to x and y direction.

Fig 2, Decoded corner positions of query image

Given four corner positions of the target, as they are in the same plane, we can use homographic matrix together with the help of barometer to calculate relative translation and rotation from current "camera image" to the one put on the floor.

1. Dependency Installation

To setup the environment, we need to install the following dependencies:

sudo apt-get install libzbar0
pip install pyzbar numpy

We assume ROS and opencv are properly installed. Details of the installation can be found in the previous tutorials.

Remember to update GAAS by:

cd GAAS_PATH
git pull origin master

and copy models, worlds and launch files to corresponding folders as depicted in previous tutorials.

2. Homographic Matrix, Rotation and Translation

In computer vision, any two images on the same plane is related by a matrix called homographic matrix, which contains information about rotation and translation between two images. To find the homographic matrix of two images, more than 3 pairs of non-collinear features point correspondences are required. To put it in another way, we need at least 3 pairs of matching points in each image that are not linear to each other to find relative translation and rotation. To find these points, we will use the aforementioned QR Code because each QR Code provides 4 feature points, any two of which are not collinear.

Fig 3, QR Code corner matches between query image and training image

The steps for recovering rotation and translation from homographic matrix are:

1. Given a target image (also known as query image) that contains a QR code, find each corner position;

2. Receive image(also known as training image) from camera and decode image, if the QR code is detected, find each corner position and continue to the third step;

3. Given query image QR code position and training image QR code position, find homographic matrix using a function called getPerspectiveTransform provided by opencv;

4. Decompose homographic matrix to rotation and translation using a function called decomposeHomographyMat provided by opencv. In this step, usually four possible solutions containing relative rotation, translation and planar plane normal will be provided;

5. We can narrow the four solutions down to one final solution, assuming we know the normal of the target plane, which in our case, is (0, -1, 0) in the camera frame.

Fig 4, camera frame illustration (Borrowed from Reading Material [2] )

Recovered landing QR code target using homographic matrix in the third step can be found below:

Fig 4, Recovered landing QR code image from Homographic matrix

All information in terms of QR Code detection, homographic matrix computation and recovering rotation and translation can be found in qr_code.py under tutorial_5.

3. Test Run

Firstly launch the simulation environment by:

roslaunch px4 landing.launch

A gazebo simulation will pop up with a QR Code that is several meters in front of the drone.

Fig 5, simulation world with qrcode as landing target

While running Gazebo, if everything works properly with the exception of the above Err message, the Err message itself DOES NOT affect the performance of Gazebo.

If you happen to know how to resolve this Err message, please submit a PR on GitHub to help us make it go away.

Next, take off the drone by:

python px4_mavros_run.py

Start QR Code detection by:

python sample_fly.py

Now, the drone should be 2 meters above the ground, and QR Code detection is also enabled.

QR Code detection will constantly fail as the landing target is too far from the drone body, so you will have to move the drone forward until the target QR Code lies within the image frame. The forward facing camera does not have a line of sight that is perpendicular to the plane of the QR code, so you might need to move the drone around the target landing QR Code so that the QR Code can be detected properly.

# in the same folder, open a new terminal
ipython2

# in ipython2
from commander import Commander
com = Commander()
com.move(8, 0, 0)

Normally, to achieve a better QR Code detection, the detection camera should be pointing straight downward or at least have a 45 degree angle downward.

Fig 6, train image detection result

We will only be considering translation and neglect rotation here. When the landing target is detected, the drone will be controlled to move towards the target and land. You will find the drone is not going to land right on the target. Instead ,it will land next to it, because the preset query QR Code is in the lower bottom of the image (see Figure 2).

The calculated results will have errors, and the camera position and the body position will also shift.

This tutorial aims to providing a basic realization to auto landing. If you find any bug or need more help, please let us know either by opening an issue.

Give us a Star 🌟 on Github if you find this tutorial useful.

Support us on Patreon.

Help us to understand what you want to learn in the upcoming tutorials by fill out this three-question survey.

4. Reading Materials

1. https://en.wikipedia.org/wiki/Homography_(computer_vision)

2. https://docs.opencv.org/3.4.1/d9/dab/tutorial_homography.html