That's because (imo) numbers aren't intrinsic to the physical universe. They are (imo) an abstraction we humans invented to describe certain phenomena. You may not agree but hopefully you can at least see why some people would have this PoV, and especially the more general PoV that mathematics is not about (physical) nature, but about abstract ideas.
