Rust's compilation speed issues are not due to optimizations. There is some intrinsic slowness, but also some boneheaded decisions that are now baked in. E.g. the unit of compilation being the crate, not file/module, makes it so you hit some bad scaling laws. And the borrow checker is SLOW, and unavoidably so, but borrow checking could have been avoided with different lifetime rules. Etc.
Rust is indeed not designed for quick compilation. Furthermore and unlike what you said, what makes it memory safe has pretty little to do with that. But what makes it fast definitely has to do with that, which is what I said.
I say this as a full-time rust developer btw.