Yes getting timestamped and synchronized ground truth to build/verify your model will be quite challenging for something like this. For the drones we just took them to a mocap lab, and races are inside so we had the luxury of assuming 0 wind. The only remotely tricky part is handling ground effect from the props.
