I am a consultant who developed the companion computer/autopilot and software environment for DRL drones used at the AIRR race sponsored by Lockheed in 2019. UZH was the team to beat and I met Davide and Elia there. They almost pulled it off but sports being what they are had a bit of a heartbreak at the end, and the team from Delft ended up pulling through and winning the $1mm prize. Delft then raced against Gab from DRL and iirc came behind by only 6 seconds.
So glad to see this team from UZH continued pushing the envelope and are now beating human champions. If you saw the team and what they managed in under a year, it was clear they were highly talented and human racers had their work cut out for them to stay ahead.
What is the general outline of going from a model of a craft (drone, sailboat, etc) to building a sim that can do reinforcement learning over a physical object interaction with its env?
I want to start playing with models, sims and collected data for sailboat racing- I know the RL/data science stuff, and I assume a good model of your craft takes time to build, and can be improved with collected data. What are some areas to explore when chaining model -> sim -> RL for performance?
I realize this is an extremely complex topic, with several PhDs worth of potential input- if you had to explain to someone technical what it looks like and where to keep digging, what would you say?
Models for drones are actually pretty simple. The races are indoors, so we can assume 0 wind. It’s really basic physics and aerodynamics. We had some sessions in a mocap lab where contestants were able to perform some flights and get some ground truth to tweak their sim -> actual drone dev loop. The simulator itself was based on the DRL simulator, but not the same version on the Steam store.
I did not develop the sim itself but did develop the hardware-in-the-loop portion of it along with things like real-time debugging, and output to the hitl. We had the sim rendering cameras which we output from the workstation to custom HDMI bridges over MIPI that we could treat as real cameras on the NVIDIA Xavier AGX. There was a data channel over Ethernet for IMU data.
I made a custom version of Eclipse that interfaced w GDB for debugging, which also was modified to stay in sync w the sim using PTP, w rewind capability.
As for sailboat modeling, yes it’s more complicated because of the effects of both wind and fluid dynamics. If I were approaching this, I’d probably try and find a physics simulator to start with. Getting ground truth will be difficult, but I imagine you would start w the IMU and GPS data off the boat, but having time synced ground truth for the waves and wind will probably be the hardest part.
Numerical simulation of aircraft is pretty straightforward. The hard part is going from the inertial frame into the aircraft's frame of reference, and back again. Once that part is down, the main procedure is to find the gravity vector in the body frame, calculate thrust, lift and drag vectors in the body frame, calculate any torques, sum everything, and then convert back into the inertial frame to do the numerical integration step. From memory, torques are the hardest part, since you need to work with either the time derivative of the quaternion, or the time derivative of the rotation matrix. But if you stick to standard coordinate frame conventions, it's just applying a formula.
The only difference between fixed wing and multirotors is the different models for forces in the body frame. A fixed wing aircraft would use an aerodynamic model of whatever complexity you like, and a model for the prop. A multirotor would use multiple prop models, and a drag model for the body. Most models for these things are pretty well defined, but you can choose just about anything that "fits" so long as you can justify it. And you can use the same basic setup for any aircraft (and indeed any robot) with a bit of forethought. The data part is mostly specifying things like, say, damping coefficients that you would then want to measure.
The dynamics are not the difficult part of simulating aircraft for RL, it's actually the rest of the environment. E.g. navigation tasks require you to build a navigable environment around your physics model, and specify things like reward functions.
I don't have any expertise in this field, but I expect sailboat training simulations would be much harder than aerial drones because of the underlying physics involved.
I'm sure flight simulations have to deal with some amount of fluid mechanics and turbulence to get an accurate simulation, but I suspect it's fairly simple and you can mostly model it accurately enough using Newtonian physics.
But for sailboats, the entire system relies profoundly on the interaction of two separate fluids with a single body and turbulence and viscosity are deeply interwined in making the boat go. Not to mention that the sail itself is a flexible deformable surface.
As a sailor, you're giving sailors a bit too much credit. Those things all matter, but to say that the average sail boat racer is even optimizing for all those things efficiently at once is a huge stretch. A lot of sail boat racing is strategy and properly performing the basically in a hostile environment. If you could ensure great strategy based on learned data, and immaculate execution based on properly tuned controls, I think you'd win easily.
I don't think GP is saying that sailors take all of those effects into account constantly, so much as he's saying a useful simulation would be monstrously difficult due to the underlying physics.
I've made robot models in the past, but never for anything involving water (although I actually have done physics simulations with water but nothing with robots). I don't know anything about sailing and I'm definitely not an expert at building simulations, but I can grok what GP meant and I agree with his assessment.
To give a rough analogy - it would be like trying to learn how to be a racecar driver on a videogame with simplified physics. There are professional drivers that use simulators and iRacing (or Assetto Corsa or a handful of other games) but none of them are training on Need for Speed, and it's because the difference is so stark you're actually handicapping yourself instead of learning how to drive. You need the simulation to be close enough to reality before it starts to become useful.
A simulation doesn’t necessarily need to be remotely physically accurate to be useful, as long as the simulated behaviour is qualitatively similar to the real thing. For sailing a small boat, I’d imagine the important things would be the shape of the sail and the heel of the boat for a given apparent wind, presented to whatever sensors your sailbot is using. For the water’s surface, random semi-regular perturbation (sum of sine waves) would be fine, and something similar (or maybe a random walk) for wind direction and strength.
For this kind of real-world robotics, you can’t optimise in any academic sense so you just have to go for effective and robust.
Well, for the drones, we did base things off ground truth captured in a mocap lab, and this helped the models significantly. There always will be a jump from simulation to real world, but the smaller you make that jump the quicker you’ll have a functioning vehicle. Without that, you can spend a ton of time (wasted) in simulation when the real world is substantially different. I’m sure a sum of sine waves would get you somewhere, but only so far. To me this is a difficult problem and I think it might be best to do most development in a real world environment.
I was just talking about sailboats specifically, where the external forces are highly unpredictable and similar in magnitude (albeit hopefully a fair bit smaller!) than your control authority over the boat.
The work you did on drones sounds very cool and in that scenario (high powered motors, crazy nonlinear dynamics, relatively small external perturbation, at least from how you described it?) I don't doubt that ground truthing your simulation using mocap data made a huge difference.
> as long as the simulated behaviour is qualitatively similar to the real thing.
My point is that I suspect that unless you have a very sophisticated simulation, it won't be.
> I’d imagine the important things would be the shape of the sail
The shape of a sail is itself a very complex thing to model and simulate. I'm not even a real sailor and I'm already thinking about camber, twist, and luffing. When sailing upwind, the sail is functioning like a wing using lift. When sailing downwind, it's relying on drag. Points in between use a mixture. This implies that turbulence and stall must be simulated at some level of fidelity.
If you've got a mainsail and a jib, then you have to worry about the slot effect and one sail blocking the wind of the other.
> and the heel of the boat for a given apparent wind
Not just heeling, but weather helm, leeway, drag, displacement, and how those interact.
Agree, I imagine just maxing VMG on the fly as conditions change is something a computer could already outperform humans easily.
I've wanted to make a simple sailing game for a while now but found it very hard to actually get something that feels right and compared to other activities there doesn't seem to be great resources on all the different parts of the physical system (in terms of Math).
> If you could ensure great strategy based on learned data
That's my point. If your learned data is coming from a simulation that doesn't model actual sailboat fluid dynamics, you've learned little.
A robot that learned to sail by studying a hollow cube floating in a bathtub full of marbles is not going to be able to make an actual sailboat do anything useful out on real water.
Yes getting timestamped and synchronized ground truth to build/verify your model will be quite challenging for something like this. For the drones we just took them to a mocap lab, and races are inside so we had the luxury of assuming 0 wind. The only remotely tricky part is handling ground effect from the props.
That's a very interest project. As a amateur sailor I have played with some models for sailing path optimizatio, wich can be somehow related to your problem. I started with a very simple model using sailing polar graphs, these map wind speed and course to maximum speed and usually based on empirical data, I found some DBs with this data, I can check if you are interest. Nevertheless for a better simulation you will at least need a simple model that gives you the forces at least especially if you want to simulate the boat acceleration, heeling and leeway. Simulating the sail foil behaviour would be very computational intesive as the airflow around it can get very complicated especially as the sail changes it's shape with it.
Reminds of me of my time 7 years back when i took the robotics courses from Davide. What a legendary professor alongwith Bernstein for AI, Sven for Computational-Econ and Boehlen for DBs.
It is, yes, but also this was done purely by VIO - courses were not mapped ahead of time. Contestants had very limited time on the actual drone, almost all their work had to be done via HITL simulation. The entire stack was done in maybe 6-8 months, then contestants produced their first code for the drone in a couple more months, and were flying shortly after. It was very clear that given more time, 6 seconds was going to be surpassed very quickly. All of this was live events too - contestants were tweaking code up until cameras rolled and spectators were in the stands.
It was mostly VIO. The drone had 4 cameras, two front facing, two aimed more at the sides. There were two IMUs on the companion computer (and more on the flight controller of course), as well as a downward facing rangefinder.
It was a software competition, whoever developed the best AI was the whole point. And we did NOT give them full course data ahead of time - they had to do this based off VIO only (visual inertial odometry).
So glad to see this team from UZH continued pushing the envelope and are now beating human champions. If you saw the team and what they managed in under a year, it was clear they were highly talented and human racers had their work cut out for them to stay ahead.