I would probably work with other ad networks, but that wouldn't be cost effective to monetize my long tail of smaller sites. It would lead to a significant, if not huge, drop in revenue.
You do get in personal contact with people at Google starting at some level of revenue though. So if it ever happened, I guess I would see it coming in some way.
google doesn't care, the company that I work for does around 15x your monthly revenue and was banned from adsense and getting in contact with google was impossible, as was finding a resolution (other than "re-apply in 6 months").
Their huge reach. There's no network with the same amount of advertisers and publishers. Especially with smaller sites, they're the easiest way to monetize, and sometimes to only way at all.