It wasn't one of the monkeys. Most monkeys only run when we have developers who could notice and fix problems. It also happens that our peak usage is while we're home and our quiet time is while we're at the office. That means, in general, the monkeys don't run on a Sunday night.
Unfortunately we're not running 100% on the cloud today. We're working on it, and we could use more help. The latest outage was caused by a component that still runs in our legacy infrastructure where we have no monkeys :)
I suspect they got hacked. Their site always responded, but every time I logged in I was logged out within seconds, and the only way I could log in again was to reset my password, after which I'd be logged out again within seconds, repeat five times over a couple of hours before I gave up.
We've hired lots of folks from yahoo to work on infrastructure and one guy from reddit to be a cloud SRE. but we've yet to hire a security guy from sony.
Downmodded into oblivion. How nice for sharing my personal experience of a topic raised by someone else. I guess I should have known that a perfectly normal outage was responsible why I could log into the site, load pages successfully for thirty seconds, and then be logged out and be unable to log back in without resetting my password. Couldn't have been attackers DOSing Netflix users or Netflix admins trying to squash an attack. The problem was just that their don't-constantly-log-users-out-and-deactivate-their-password service went down because it wasn't on the cloud and I was dumb not to figure that out.
