Hacker News new | comments | ask | show | jobs | submit login

> Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Perhaps the progressive rollout should wait for an affirmative conclusion instead of assuming no news is good news? I'm not being snarky, there may be some reason they don't do this.

Presumably it received a false positive (or it was interpreted as such). This really seems like the root cause, and I suspect a case of happy path engineering striking again.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact