Thanks to a ton of grit by the team, and the insistence of one engineer in particular we built a config management system and started tracking the total percent of our global network config that was managed by our config management system.
That metric was regularly presented at the VP level to hold us accountable to getting the percentage to 100.
It was months and months of boring work to remove inconsistencies and templatize configs. But in the end, I believe it resulted in a much more reliable and ultimately safer network to operate. I'm also happy that my management chain saw the value in this work.
I'm quite proud of the work the team did.
Some side benefits were that once we started going through audits like SOC2, we had a really good story to tell about how we reviewed and pushed changes to production.