In our case it's a hardware h264 encoder in an embedded product, so our method wouldn't be any of any use for you in a more general case. We sussed out an undocumented debug api to get the chip to dma the motion vectors to a debug buffer, and then we can extract and use them. I think those apis are originally intended to be used if a customer is having issues with the product so that the vendor engineers can help troubleshoot.
We were originally looking at analyzing the h264 bitstream in software and getting the vectors out that way. Honestly I don't think that would be extraordinarily difficult, there's already code in ffmpeg to draw the motion vectors as arrows on the decoded frame, so that would be a good place to look if there aren't any preexisting apis or tools to extract them.
