Are your Docker images x86_64? On an M1 Mac x86_64 images run under qemu which is very slow, If an ARM64 image is available it should run a bit faster.
Oh man x86 is so slow it's insane. If you're using a non-ARM base image Docker will happily run the x86 variant for you automatically. Unsurprisingly, running an x86 VM inside an ARM VM on a laptop is very very slow