True. I had a somewhat similar experience with the official Alpine-based Python images. The are supposedly leaner than the Debian-based ones, but any advantage is cancelled if you need any PyPI packages that use native libraries.
Now you suddenly need to include a compiler toolchain in the image and compile the native interface every time you build the image.