

Github: Major Service Outage - ca136
https://status.github.com/?a

======
zheng
First off, as others have said, disabling port 80 is a great way to handle
this, I don't really care that much if I can see/use github the website for a
few hours, but I'd be much more upset if I couldn't pull my code.

Secondly, I kind of like when big sites go down when I'm not in desperate need
because it means a really nice aftermath write up is on the way. Can't wait to
hear more about this one.

~~~
mst
> "disabling port 80 is a great way to handle this"

It's been a disaster for a bunch of people I know.

git is distributed by nature so they all had extra remotes they could use.

github's issues and other project metadata wasn't distributed, since that's
github's alone.

So all of the friends of mine who are corporate github users who were using
git in a distributed style (a minority of their overall customer base, I would
suspect) are more screwed by the web app's absence than they would have been
by a repository problem.

I suspect github made the right choice for their customer base overall, but I
still find the anecdata interesting.

~~~
kijin
The web app is also available on port 443, which every "corporate github user"
should have been using anyway.

~~~
wfarr
In fact, all HTTP access redirects to HTTPS for just about everything. And
most modern browsers (recent versions of Chrome and Safari) that have accessed
a website over HTTPS once happen to _prefer_ HTTPS by default for that site.

------
creativityhurts
They said they're under a DDoS attack
<https://twitter.com/github/statuses/259029493669310464>

~~~
alook
Who DDOS'es twitter? Mercurial committers? the Bitbucket people?

~~~
fusiongyro
I heard once about a tire shop that drummed up business by strewing nails
along the highway around it. Somewhere outside America; India or Thailand or
somewhere else. Anyway, that kind of business practice wouldn't get you far in
the Western world, so it seems unlikely.

OTOH, if I had a botnet and I wanted to see how powerfully it could DDOS
without drawing a lot of mainstream media attention, github might be good for
target practice. They have better infrastructure than most, and they always do
detailed write-ups afterwards.

~~~
riffraff
OT: the urban legend about tire shops putting nails in the neighborhoods' is
very widespread. I have heard about it in at least three countries, and I
recall my school teacher telling me about it in '91.

I sort of expect it wouldn't be an effective practice anywhere.

~~~
huhtenberg
It's an urban legend until you actually catch a nail and there is a tire shop
conveniently located not 100 yards away. True story, Indian reservation in BC,
Canada. Who cares if it's a tacky business model or an urban legend. It works.

------
Jgrubb
I think it's probably my fault. Github seems to go down every time I start
poking around in someone else's projects.

~~~
zzleeper
By any chance, did you 'poke' with a cluster of LOICs?

~~~
thebigshane
(Never heard that term before...)

    
    
      Low Orbit Ion Cannon (LOIC) is an open source network 
      stress testing and denial-of-service attack application
      [...]
      LOIC performs a denial-of-service (DoS) attack (or when 
      used by multiple individuals, a DDoS attack) on a target 
      site by flooding the server with TCP packets or UDP 
      packets with the intention of disrupting the service of a 
      particular host. People have used LOIC to join voluntary 
      botnets.
    

\-- <http://en.wikipedia.org/wiki/Low_Orbit_Ion_Cannon>

------
FiloSottile
01:41 PM PST

We've temporarily disabled service on port 80 while we investigate the source
of a connection flood. HTTPS, GIT, and SSH service are unaffected.

01:33 PM PST

"We are experiencing issues due to a DDOS attack, working hard to restore
service"

~~~
8ig8
I'm reading over the status updates a few hours after the fact, but for some
reason the 1:33PM stands out because it is wrapped in quotes. Probably minor
and meaningless. Just noted because it's different from the others.

~~~
imbriaco
It's because we update the status site from our chat room via Hubot and
whoever posted that particular update didn't realize that they didn't need to
put quotes around the body of the message.

------
augustl
Cheatsheet for continuing work when Github is down:

    
    
        1) ssh myserver.com
        2) adduser git
        3) sudo su git
        4) cd $HOME
        5) mkdir .ssh
        6) Add people's public keys to .ssh/authorized_keys (in /home/git)
        7) git init --bare myrepo.git
        8) Push and pull to git@myserver.com:myrepo.git
    

Voila!

------
wamatt
/enqueue jokes about decentralized DVCS systems being hosted centrally...

Srsly tho, the relationship between Git and GitHub, might be somewhat
analogous to that of BitTorrent and TPB

~~~
tlrobinson
Fortunately since you have an entire copy of the repository it doesn't really
matter. Your team can just push the latest copy to another remote like
Bitbucket and start using that.

Well, except for Issues, Pull Requests, Wiki, etc, but those aren't part of
Git.

------
FuzzyDunlop
Nothing against Github but this probably highlights the real benefit of DVCS:
setting up multiple remotes for your repo. Manage it probably and when one
service goes down, fall back to Bitbucket or another service.

It would limit the potential damage these attacks could cause, given the
reliance dev teams have on pushing code to a central repo. Taking down a site
like Github has a fairly clear effect on the productivity of a lot of their
users.

~~~
bluesaunders
It'd be glorious if one service (say GitHub) had multiple remote links that
dynamically switched for you when service faced an outage.

~~~
dasil003
If it's one service though there is still always a single point of failure
outside of your control. If you want true redundancy you need to use multiple
versions with no interconnections and develop your own solution to mitigate
partial failure, of course your own solution is also a single point of
failure, but at least it's under your control.

That said, Github has never been down long enough for me to justify a fallback
procedure and policy for my team. The complexity of the solution outweighs the
benefit. We'll cross that bridge when we come to it.

~~~
goldfeld
There will be no crossing if the bridge is down, though.

~~~
dasil003
Of course there will be. We are talking about git. Everyone has a full copy of
the repository. Github's data center gets hit by a meteor tomorrow and it's a
minor setback, nothing more.

------
desbest
Why do people always post on Hacker News whenever Github is offline?

It's not going to make Github go online any quicker. They have a Twitter
account, you know.

~~~
tedivm
It's a site most people here use, and it's interesting. It's not about getting
things back up faster, but letting people know what's going on and giving them
the chance to witness how a large site fails and recovers.

------
hkdobrev
Github status: "We've temporarily disabled service on port 80 while we
investigate the source of a connection flood. HTTPS, GIT, and SSH service are
unaffected."

Great way to keep the git push/pull workflow unaffected.

------
hkdobrev
Github.com is now up <https://github.com/>

<https://status.github.com/>

------
longdivision
<https://twitter.com/AnonymousOwn3r> is taking credit for attacks against
YouTube and GitHub amongst others.

~~~
veeti
Stop giving some idiot script kiddie the attention he wants. It's probably not
him anyway.

~~~
longdivision
It's Likely, but this is someone with a history of claims to this sort of
attack and a large following. I was interested in who was running the large
attacks and what their motivations might be and no sources had been suggested
in the discussion at that point.

------
trebor
GitHub says it's a DDOS attack that's hampering their services.

------
theevocater
It appears that their http endpoints are down, but their git endpoints are
working fine

EDIT: or not anymore. I was able to push but now I am not.

~~~
ca136
Was just able to push/pull okay, overall seems like they're dealing with the
outage really well. Their status page is really helpful, they were on twitter
quickly, and they tried to restore service to people who might need to
push/pull to their repos.

~~~
theevocater
yeah watched their status site off and on for the last few minutes and it
seems their git endpoints are intermittent while the site is out.

------
songgao
HN is much slower here as well. Is HN also targeted?

~~~
FiloSottile
Probably it's only all of us swarming here as GH and YT are down ;)

------
NelsonMinar
Github status is hosted on Github, d'oh. Their Twitter feed posted at 2037Z:
"We are experiencing issues due to a DDOS attack, working hard to restore
service..." <https://twitter.com/github/status/259029493669310464>

~~~
jbredeche
Looks like github status is on heroku. It's loading fine for me:

    
    
      > host status.github.com
      status.github.com is an alias for appid129905herokucom-760859479.us-east-1.elb.amazonaws.com.
      appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 50.19.101.240
      appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 50.19.101.216
      appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 107.22.241.165

~~~
danhunter
Yep after the last major outage they switched to Heroku

~~~
bgentry
They were already running their status site on Heroku prior to the last major
service outage.

~~~
adgar2
This is correct. They needed ~90 dynos to keep up with about 500 qps on their
status page in the last outage.

Source: <http://news.ycombinator.net/item?id=4524489>

~~~
gojomo
Should probably throw a Fastly or CloudFlare cache in front of a single dyno.

~~~
adgar2
That would suggest people in charge know what they're doing.

------
spleeder
The internet is collapsing folks. YouTube is down as well:
<http://news.ycombinator.com/item?id=4670859>

~~~
theevocater
I saw that and was confused. YouTube seems to be working for me

------
brokentone
GitHub, YouTube, I'm having trouble on Wordpress.org, anything else down for
people? Anyone noticing any correlations?

------
xxpor
Does anyone know where GitHub is hosted? I assume not AWS because
status.github is on heroku (and therefore AWS).

~~~
shadowmatter
They're on Rackspace:

[http://techcrunch.com/2012/09/18/patent-complaint-filed-
agai...](http://techcrunch.com/2012/09/18/patent-complaint-filed-against-
rackspace-for-hosting-github/)

<https://github.com/blog/493-github-is-moving-to-rackspace>

------
thechut
Very slow but I was able to pull at 4:55EST

------
jamesjguthrie
Pfft, always down these days. BitBucket for me.

~~~
joshmlewis
Nice try, Bitbucket marketing team. ;)

~~~
piggity
Nice try, LaunchPad marketing team trying to deflect users from Bitbucket!

