

A Note on the Recent GitHub Outages - mbrubeck
http://github.com/blog/597-a-note-on-the-recent-outages

======
SlyShy
As much as it sucks when Github goes down, there's a sense in which I look
forward to it happening. It's yet another chance for me to get a peek at their
architecture. As a high school student it's pretty disappointing to me how
hard it is to find fully laid out architectural plans for large websites. So I
appreciate sites like Github and Ravelry being open with that information.

~~~
pavs
Google architecture: <http://highscalability.com/google-architecture>

Google infrastructure: <http://en.wikipedia.org/wiki/Google_platform>

Amazon architecture: <http://highscalability.com/amazon-architecture>

No one will layout the blueprint for you, but most general information
regarding infrastructure are out there.

~~~
steerpike
Wow, thanks for bringing highscalability.com to my attention. Awesome
breakdown of sites.

------
timdorr
Another Yahoo! bot behaving badly? Between them and MSN/Bing's bot, what the
hell is going on with search engine spiders lately?

~~~
jrockway
Imagine you have to write a computer program to visit every page of every
website on the entire Internet.

Sometimes you are going to make a mistake, and someone on the Internet is
going to notice.

~~~
timdorr
Except this has been going on for months, has happened a number of times, and
occurs in violation of standards, like robots.txt, that were created to
prevent it from happening. I'd expect it from an individual or startup, but
not a large company that's been doing spidering for a long time.

------
papaf
From the explanation it looks like github isn't using '5 whys' when things go
wrong. Has anyone tried this? Is it overkill or does it work?

<http://en.wikipedia.org/wiki/5_Whys>

------
sailormoon
If I were GitHub I think I would have mentioned the cause of outage number 3
_after_ I put the rate limiting fixes in place, which is not the impression I
got from the article. It is not hard to write a basic HTTP request spoofer and
it sounds like they're still vulnerable.

