ArduPilot automatic testing system

By Andrew Tridgell [ http://diydrones.com/profile/AndrewTridgell ]
DIYDrones.com

November 11, 2011

The development team is always trying to find ways to improve our testing and the quality of our code. We've recently add a new automatic testing system that should help a lot with finding bugs and ensuring that the huge range of features that we have added (and plan to add!) to the code keep working.

If you have a look at http://autotest.diydrones.com you'll see the results from 'Tools/autotest', a test system that runs on every change to the ArduPilot and ArduCopter code. The test script currently checks the following:

that both ArduPilot and ArduCopter build on the 1280 and 2560 based APM boards
that both ArduPilot and ArduCopter can build with the 'desktop' build system
that ArduCopter can fly in both manual and automatic (mission) mode
that we can download the flight logs

It also keeps detailed logs [ http://autotest.diydrones.com/autotest-output.txt ], and produces a KML file [ http://autotest.diydrones.com/ArduCopter-test.mavlog.kml ] which you can load in googleearth to look at how the flight went.

The flight itself is controlled using a python scripting system that flies ArduCopter using MAVLInk commands. Currently that script does the following:

starts up the simulator at my local flying field
arms the motors (using virtual transmitter sticks to control the rudder and throttle)
switches the quadcopter to STABILIZE mode
does a manual takeoff using throttle, until the altitude reaches 30 meters
uses the rudder to turn the quad until it is facing north (it starts up facing west)
uses the elevator and aileron sticks to manually fly a square circuit about 50m on a side
switches to LOITER mode, and checks that it holds altitude for 10 seconds
switches back to STABILIZE and decreases throttle, to bring the quad in for a gentle landing
loads a 9 waypoint mission into EEPROM
switches to AUTO mode and flies the mission, which involves a zig-zag route over the field
the last mission step is a RTL, which brings the quad home
then it switches to STABILIZE, brings the quad in for a landing, and disarms the motors

You can see a track of the test flight in the above googleearth screenshot.

Running these test steps every time we make a change to the code is already ensuring that a lot of our code works, but we have plans to greatly expand this test suite. What we'd like to do is have every ArduPlane and ArduCopter feature tested, with code that validates that it actually does the right thing. At the moment the test suite only has very rudimentary testing of whether the flight was actually correct, whereas we should be checking lots of parameters (roll, pitch, altitude, speed etc) throughout the flight.

The other obvious missing piece is ArduPlane testing. The test suite checks that ArduPlane builds, but doesn't yet check that it flies correctly. I hope to fix that this weekend, with ArduPlane flying some simple missions plus some 'manual' (ie. python scripted) flying. I'll probably do this by interfacing with FlightGear, but jsbsim linkage of even a new physics model is a possibility.

Under the covers

This new test system involves quite a few new bits of code, some of which we hope will be used for things other than automatic testing. The quadcopter simulator itself is a simple physics simulator [ http://git.samba.org/?p=tridge/UAV/HILTest.git ] written in python, which Michael has since translated into C# for use in the planner (thats how you can fly quadcopter HIL in the planner now). The communication between the various pieces uses pymavlink [ https://github.com/tridge/pymavlink ], a MAVLink [ http://www.qgroundcontrol.org/mavlink/start ] implementation in python. That uses a bunch of helper scripts that make it fairly easy to script a ArduPilot vehicle. I'm hoping this system will be used for scripting real copters and planes in the future, along the lines of the acrobatic scripting system [ http://diydrones.com/profiles/blogs/adding-acrobatic-scripting-to- ] we discussed earlier in the year.

The other big part of this autotest system is the 'desktop' (also known as SIL for "software in the loop") build of ArduPlane and ArduCopter. This builds the code using a normal C++ compiler, and creates executables which run directly on your desktop computer, rather than on an APM board. The advantage of this is that we can use all the debug tools (such as gdb and valgrind [ http://valgrind.org/ ]) which are normally used for C++ development, but also it runs much faster and doesn't rely on hardware being plugged in. We've already taken advantage of the ability to run ArduCopter under valgrind to find some bugs, and it really increases our confidence in the code to be able to run that sort of sophisticated memory checking on every commit.

If you want to have a look at autotest, then start by grabbing the latest APM code and having a look in Tools/autotest. You might like to start by looking at arducopter.py and seeing how it controls the quad. We're hoping that we will get contributions of new tests. We also plan on putting up a wiki page describing how you can run the autotest system on your own computers (right now it only runs on Linux and MacOS, although Micheal has had some limited success on Windows using cygwin).

How you can help

The autotest system is already quite useful and it has already helped us find some subtle bugs in the QuadCopter code, but we'd love to expand it to cover more aircraft and features. Here are a few things that it would be great to get contributions on if someone has the time:

we need physics simulations for more multi-copter frame types (start with quadcopter.py [ http://git.samba.org/?p=tridge/UAV/HILTest.git;a=blob;f=quadcopter.py;h=60038166ed37155bade47dc7462e383411cb0311;hb=master ], and create ones for hexa, octa etc)
I'd like to produce a nice picture of the flight path from a KML file using a script, so devs don't have to load googleearth to quickly check it looks OK
we need some interesting test missions to look at. I hope to support loading missions in planner format soon to make this easier
we need python code to more carefully check the flight is OK. It may be easier to wait on this until we develop some better helper functions, but if you are a python coder then feel free to jump in now

Hopefully this will be the beginning of a much more rigorous testing effort for ArduPilot. We'll post more information as the system develops!

1:39am

By Chris Anderson [ http://diydrones.com/profile/zlitezlite ]

November 11, 2011

This has been amazingly useful already. Just in the few weeks that we've been using it, Jason has rubbed out a dozen sneaky little bugs in ArduCopter 2.0.50. They would have taken hours each to replicate in the field, if we even knew where to look, and they just pop right out of the test suite.

I don't know how we were able to work without this before! It's a huge time saver and should really accelerate the progress of ArduCopter/ArduPlane in features, performance and reliability. It allows us to focus on adding cool stuff, not trying to track down hard-to-replicate reports of issues.

8:32am