The need for consistent, highly available persistent storage in a distributed system with a heavy write workload vastly complicates engineering effort. You're fighting fundamental issues like the CAP theorem. When you can't "just use Postgres" everything else gets a lot harder. Companies that have this problem tend to have a lot of engineers: Facebook, Google, Uber, etc.
Disks are slow. If you don't need to wait for disk, then you can handle a great deal more req/s per node, which means the point at which you need to become a distributed system is further away. Without the state synchronization problem, converting to a distributed system consists of little more than adding a load balancer.
Even in an environment that needs synchronized state, the services that can be stateless are much easier to deal with (and you try to make as much of your infrastructure stateless as possible).
Storage costs are not relevant compared to the costs of engineering distributed systems.
I agree with the concept but I don't think Uber falls nearly in the same ballpark as something like Twitter on scalability needs. IIRC Uber's huge headcount is due to it having local recruiting/marketing teams pretty much everywhere
It takes engineers to run and utilize all that stuff. You don't need a back-end scaling team you're only throwing around XMPP in Erlang.
Edit: I know b/c I've done that before in $previousjob. Storage is hard. Storage is where the money goes. Durability is expensive. Storing ephemeral stuff in a non-durable KV store is easy, cheap, and trivial to scale. This is why WhatsApp didn't have so many people, not because they were geniuses.
