Which is why we need more competitors in this space.
When you are an underdog, making something "better" even for no direct monetisation path, is an obvious win because it all creates more noise and attracts people to your platform. More noise = more business - but Google don't need more noise, they have all the attention they need, they are basically king, they just need to figure out how to milk it, which means the only intrinsic business force is direct monetisation paths... completely different incentives.
Because indexing the web and organizing all that information is expensive. It’s also a lot of power so we can’t exactly trust that job to a single government or even a group of allied governments. Universities could potentially handle it by pooling a lot of resources, but I can’t really think of who else I would trust to be The Search Engine. Google isn’t ideal but at least it’s priorities are mostly aligned with the average user’s.
Imagine that every Internet user pays just one dollar a month. That's still billions worth of revenue, and it can be segmented further such as basic features available for free and more advanced users (such as large-scale API access) can pay more.
Sadly, that's not realistic. You can get a subset of users to pay, but it's vanishingly small compared to everyone who thinks they are perfectly happy with the "free" status quo. Even a dollar a month, it is infinitely more than zero. I just had this conversation with members of my extended family and almost all of them were of the "I don't care if I'm tracked, I've got nothing interesting to hide" mindset, along with "not a chance in hell I'm going to pay for this, or email, when I get them from Google for free."
How about a company that provides a search engine, but as a part of a greater subscription service whose selling point isn't search? Imagine, for example, that paying for Apple One got you access to an Apple-operated search engine.
It doesn't necessarily have to be a direct payment. Companies can white-label the service or include it as a value-add as part of their offering; for example an ISP or mobile carrier offering this to their customers.
At the moment I'm not aware of any company whose primary service is search. I believe the demand would be there - OS manufacturers currently have to either put up with Google (even if their goals diverge - for example it doesn't work well with Apple's privacy-focused marketing) or Bing.
A third player whose primary business is search (as opposed to using it as a loss-leader to lure people into their ecosystems, like Google and Bing are) could get some use, not to mention that programmatic access to large web index would no doubt be a service many companies would pay for. Think Algolia or hosted ElasticSearch but pre-seeded with a broad index of the web.
It doesn't have to appeal to every internet user. Not even most internet users. It just has to appeal to enough to make money, which is a substantially smaller number than the total number of internet users.
It is now evident Google is not in the business of making its search any better.