I had a coworker working on a data-pipeline sort of thing once. They for some reason wrote their own date parser (Akamai logs) and used a regular expression to find the month. For whatever reason, the RegExp didn't work for March (I think) and so at midnight 1 March the year of first deploy, we started dropping all the logs as un-parseable (causing the NOC to think the traffic had gone to 0 or our system was buggy). We all got paged and eventually figured out it. It had been running in live for several months without problems at that time.
