Hilarious how Google thinks they are now in editorial control of your content to the point where you are on the hook for fixing their bugs. You're being treated as a wayward content provider, rather than that they should be happy to get the benefit of your content to index.
While I agree, opt-in would certainly be better.. I really see no way to transition from the current state, to 100% opt in.
And, I think opt-out was the right choice at the beginning - nobody would have updated their sites to suit the startup search engine.
That said, Google & co could set a date, whereby new content / pages will not be indexed unless its marked as indexable. This would allow historic content to remain indexed, and new content be opt-in at the expense of any new search engines being unable to index historic content. How can they tell if the content has not opted in, or is just no longer actively maintained without having a pre-existing index?
