We find it significantly faster to do macOS + Linux VM + Docker than macOS + Docker for Mac. We're using the Remote Containers plugin in VSCode.
I just tried Docker for Mac 4.6.0 and it is still slower than the above setup - though I think this might be down to bind vs. volume mounts.
"To a clever crow, a scarecrow is an opportunity" - Alan Scott
(but he in turn may have been quoting someone else)