Why doesn't Google disrupt itself -- the most concise, cleanest pages that focus on what the user is actually looking for (recipe, not a narrative) should be #1. It's even in Google's interest!
I think their market share encourages them to do nothing. You actually have to offer a good product to grow market share, then slowly run it, then users switch to a new product which originally starts good, and the cycle repeats.
Nobody would start using Google Search in its current format.
It has, and that's mentioned in the article. If you want to be #1 on the search page, buy an ad, because that will always appear above every "organic" search result.
Ads are Google's bread and butter, and has been for the entirety of its existence as a public company.