Comment on page
Autonomous Drone Software E02 : build a 3D model with your drone
In this tutorial, we will talk about a real world use case on how to build a 3D model of a building based on what we have learned in the last tutorial.
Help us to understand what you want to learn in the upcoming tutorials by fill out this three-question survey.
---------------------
We suggest developers to install GAAS mirror to use the project:
Should you have any questions or suggestions to the tutorials or GAAS itself, let us know and join our slack
In the last tutorial we walked through how to control your drone with MAVROS in OFFBOARD mode within Gazebo simulation environment. Now in this tutorial, we will walk you through the process to build a 3D modell using the same API as what we used last time and a website (It's free!) for building models.
This tutorial might be off the topic and have little to do with Autonomous Drone, but building a 3D model with drone has been adopted by many companies, you can check this link:
https://www.archdaily.com/899196/how-to-use-a-drone-to-create-a-detailed-3d-context-model
and by building a 3D model, we have the chance to realize more and more complicated tasks with simple APIs.
Steps required to build such a model are:
- 1.launch Gazebo simulation environment;
- 2.takeoff the drone and record a rosbag;
- 3.command the drone to fly a circle around a typical building;
- 4.land the drone and process images from rosbag;
- 5.upload images to provided website.
Or if you wish to build a model about a real building or an environment:
- 1.takeoff the drone with your controller or you can choose to use provided Python API if you have stable SSH access to your companion computer (take extreme care);
- 2.start recording a rosbag;
- 3.command the drone to fly around the building and make sure the building or environment is thoroughly covered by the camera;
- 4.land the drone and process the images from rosbag;
- 5.upload images to provided website.
We will walk you through each of the above step in Gazebo simulation environment and we will provide an website that you can use to build a model. After this tutorial, you will build something like this all by yourself!
Fig 1, 3D model
Before starting, remember to update GAAS:
# change directory to where you put GAAS and git pull from master branch to update
cd (GAAS_PATH)
git pull origin master
Next, copy files that are required by Gazebo simulation to where you put PX4 firmware by:
cp -r (GAAS_PATH)/simulator/launch/* ~/catkin_ws/src/Firmware/launch/
cp -r (GAAS_PATH)/simulator/models/* ~/catkin_ws/src/Firmware/Tools/sitl_gazebo/models/
cp -r (GAAS_PATH)/simulator/worlds/* ~/catkin_ws/src/Firmware/Tools/sitl_gazebo/worlds/
cp -r (GAAS_PATH)/simulator/posix-config/* ~/catkin_ws/src/Firmware/posix-configs/SITL/init/ekf2/
After completing, you can continue to the next step.
If you followed the last tutorial and the previous step you will have everything we need to launch the simulation. To launch the simulation, in a terminal:
roslaunch px4 sfm.launch
it will launch MAVROS as well as Gazebo simulation at the same time and your drone will be centered at the origin:
Fig 2, Gazebo simulation environment
While running Gazebo, if everything works properly with the exception of the above Err message, the Err message itself DOES NOT affect the performance of Gazebo.
If you happen to know how to resolve this Err message, please submit a PR on GitHub to help us make it go away.
Before you continue, remember to check MAVROS connection status by:
# make sure "connected" is "True"
rostopic echo /mavros/state
When you started simulation, sensor data such as IMU, barometer, camera and GPS etc will be published via MAVROS, a list of available topics can be checked by:
rostopic list
And some of the topics published by MAVROS are:
/gi/simulation/left/image_raw
/gi/simulation/right/image_raw
/mavros/imu/diff_pressure
/mavros/imu/mag
...
What we will be using for reconstructing the 3D model is only the left camera image:
/gi/simulation/left/image_raw
you can view images published from this topic using RVIZ by:
rviz
You will see a window which looks like this:
Fig 3, RVIZ
in the left bottom, select 'Add' and followed by 'by topic' and select '/gi/simulation/left/image_raw' you will see the current left camera image in real time. We will be building a 3D model in this Gazebo environment.
Fig 4, select image topic
Fig 5, image display in Rviz
Rosbag is like a box that wraps data published from ROS/MAVROS, and to build a 3D model, we need a series of images about a building or an environment taken from different angles. To take images and store them, we can use Rosbag.
Since we will only be using the left camera information we only need to subscribe to the left camera topic, and in order to do this, in a terminal:
rosbag record /gi/simulation/left/image_raw -O sfm.bag
Next, takeoff the drone:
python (GAAS_PATH)/demo/tutorial_2/2_Struction_from_Motion/px4_mavros_run.py
The drone will take off to 2 meters high. Now we can start controlling the drone to move around the house to take pictures, while doing this we need to keep the drone headed towards the house so that we can capture detailed information about the house:
python (GAAS_PATH)/demo/tutorial_2/fly_circle.py
The drone will fly around the house at 2 meters high and finally it will arrive at the origin and land on the ground.
While recording the bag, we should keep the following points in mind in order to build a good model of the building:
- 1.try to take images of a building where images are partially "overlapping" with consecutive ones;
- 2.avoid sudden yaw change while taking images;
- 3.try to cover the entire building, sometimes multiple circles are required to cover taller buildings.
You've probably noticed that we commanded the drone to fly a circle rather than a square, which seems more easy as much fewer waypoints would be needed. The reason behind this is a square usually means a big yaw change (90 degrees at each corner) while taking pictures and it is what need to avoid according to the second point mentioned above.
If the drone landed, you should stop rosbag recording using "ctril + c"
Now you will have a rosbag named 'sfm.bag'. Next, let's process this bag into a series of images before starting to build the model.
To build a 3D model, we need images rather than a rosbag. To process the rosbag into a series of images, you can use provided python scripts in the tutorial_2 folder:
python bag2image.py --bag (PATH-TO-YOUR-BAG) --output_path (IMAGE-OUTPUT-FOLDER) --image_topic /gi/simulation/left/image_raw
After this step, you will extract all the images in the bag to your local folder.
Try to limit image number, small scene would usually require less than 100 images. Sometimes you might need to remove redundant images.
We provided a website to build the model, and all you need to do is to upload images. Due to limited computation power, it might take a while before you can get the model.
aviation.giai.tech
Fig 6, 3D reconstruction website
Following is a real world use cases made by the website.
Fig 6, ChangChun JinYing Square built with 76 images (human controlled flight)
After uploading images to the above website (cannot upload image folder), you will need to wait for a while. Once the model is generated successfully you will receive an email and you can fetch the model in the provided website.
If you find any bug or need more help, please let us know either by opening an issue or through our facebook group.
Help us to understand what you want to learn in the upcoming tutorials by fill out this three-question survey.