Yes. To begin with, on modern processors, paging is tied with the memory protection system. Memory is protected on a per-page basis. Memory mapped files are implemented through paging. There's also backwards compatibility. Virtual memory allows 32-bit applications to have their physical memory located anywhere. Without that, we'd have to ensure that they reside below 4GB. Programs would start up slower as the executable would have to be relocated every time. Additionally, it's hard to imagine a hypervisor that doesn't use virtual memory. Virtual memory allows hypervisors to present a unified address space to their guests; no matter how fragmented physical memory gets.
What if we don't tie paging with memory protection when designing CPUs?
What if we don't assume, build, or run 32 bit applications?
What if we do actually put a hard physical 4GB (or whatever) limit on processes?
What if we don't need hypervisors?
What if we trade off hardware and software complexity we've built decades ago for limitations we had decades ago, and build something simpler from scratch?