
GitHub is getting DDoSed again - mathias
https://status.github.com/messages?
======
jetsnoc
It may be time for GitHub to build out multiple availability data centers and
use BGP as an anycast tool. We do this. I have public facing IPv4 space that
is announced from multiple facilities. Having an IP address hosted from
multiple facilities is a powerful tool. This allows providers to hit our
datacenter through the least amount of ASN routes. We original did this to
minimize latency and create faster regional transaction processing. As an
added benefit - DDoS traffic also gets routed to the nearest facility "load
balancing" a DDoS so that it only affects a single facility or it splits up
the 10gbps of traffic among many facilities if it is coming from many sources.
O'Reilly's BGP book has a great chapter on "Anycast."

From the sounds of it their architecture may not support this. If they had a
SAN solution capable of replication to multiple data centers like HP
LeftHand's product or a multiple master DRBD configuration they may be able to
host github from multiple active datacenters and announce the block equally so
that providers route traffic to them because their ASN is closest.

Who knows, maybe they do all of this?

~~~
mnutt
Thanks for the book recommendation. Is that the one by Iljitsch van Beijnum?

Other than the added complexity, can you share any details about the cost? It
seems like theoretically some managed hosting providers could offer this
assuming they were in multiple datacenters, but I haven't seen any that do.

------
DigitalSea
This happened back in October a couple of days in a row. Who the heck is
targeting Github and why? I wonder if these attacks are related to the Chinese
hacking attacks that have been publicised lately?

~~~
Negitivefrags
I often have to wonder if the DDoS gods roll a dice and pick someone to screw
each day. We have had many DDoS attacks and never once had any indication as
to why.

A few days ago we had a 30 Gbit DDoS. Our server host just blackholed any IP
that was touched by it. They kept moving it around to target different bits of
our infrastructure (unlike previous attacks that just targeted our website).

We lost 6 servers, but thankfully not enough to take us fully offline though
some customers would have experienced problems during that time.

If they had just been a bit more persistent we might have been in serious
trouble.

At that level of DDoS your server host doesn't care about keeping you online.
They want the traffic off their network.

~~~
tomjen3
Get a smaller host, that way they can't afford to cut you of.

~~~
buttscicles
I'd imagine they'd rather cut off somebody than potential performance
degradation for other paying customers.

------
gojomo
There are people who suggest that a DDoS is just a 'digital sit in', a
legitimate way for someone to air a grievance, if they think the targets (or
world) haven't paid them enough attention.

This view makes DDoS seem more normal or even romantic/heroic, and spreads the
tools/know-how more widely. So, pulling off a DDoS becomes a more plausible
and attractive aspiration, for a larger set of surly people with marginal
reasoning skills and destructive impulses.

The DDoS tactic should be rejected as dishonorable censorship and vandalism,
no matter the cause under which it is launched.

~~~
olleicua
Does anybody know what grievance is being aired here?

~~~
blablabla123
Maybe because you have to pay for private projects and Bitbucket UI still
isn't as fresh as Github's.

------
eksith
Are we sure it's a DDoS or is it some sort of massively distributed scrape of
the repos? (Side-effect being DDoS regardless)

I'm starting to think this is some kind of grab for intellectual property;
maybe even a targetting of private repos to somehow gain access.

~~~
jeremymcanally
Nope. It's a straightforward DDoS. No targeting of private repos or anything
like that.

------
naftaliharris
Reading status.github.com over the last few weeks, I found it interesting how
often little things were broken at Github. It's like every few days, a small
part of the site is unavailable or the sysadmins are investigating this or
that connectivity issue. I guess when you're as big as Github, keeping your
site live and operational is completely nontrivial.

~~~
joeblau
I'm sure other services have similar downtimes and issues, but they just don't
give you visibility into their operation. Most companies won't let you know
there is a problem unless you figure it out.

------
sixbrx
I consider it evidence that when some punk figures out how to make a black
hole, we're done for. No reason necessary.

~~~
nwzpaperman
Probably a "sovereign hacker" as non-sovereign-employed programmers are
naturally aligned with the open values and creativity that github exists for.

As far as motive goes, if github can be electronically terrorized, laws to
protect them and everyone from future electronic terrorism only make sense,
right?

<donkey>Eee-ooooo</donkey>

Always do what you can to understand motive!

------
shinuza
Bitbucket at it again.

~~~
hackernewbie
I think this every time.

------
niggler
Is it just me or has github been down a lot in the later months (moreso than a
year ago)? DDoS or otherwise, it doesn't inspire confidence, especially for
paid accounts (which I considered but ultimately decided to go with another
solution)

~~~
obsession
Github has 99.9585% uptime past month. That's like 22 minutes of downtime per
month.

------
yRetsyM
Maybe they should use CloudFlare?

------
alexvr
Good plan. When a site is DDoSed, encourage thousands of HN viewers to check
it out :P

~~~
imjared
Going to go out on a limb here and say that status.github.com is probably
hosted somewhere other than github.com

~~~
brdrak
I was curious about that too. Looks like status.github.com is hosted on AWS,
whereas github.com on Rackspace.

------
babuskov
Hm, just a couple of days later after another potential security exploit is
published... maybe they did not plug all the holes, and someone is trying to
clone all private repositories as soon as possible... hogging the servers in
the process.

~~~
imbriaco
Nope. This was a pretty standard DoS attack.

------
leke
Who would have the motivation to hack GitHub?

~~~
robinh
Honest quess: information and code from private repositories?

------
freddyduarte
Meanwhile at Bitbucket... <http://status.bitbucket.org/>

~~~
windexh8er
I <3 BitBucket over GitHub, but unfortunately they'd fall over in more or less
the same manner under similar circumstances.

------
nixarn
Still not working, trying to load the page of a private repo, keeps loading
and loading.

------
hawkw
Who <i>does</i> that?

------
badgar
This is a pretty typical occurrence for a web service provider of their size.
When is Github going to be able to not fail when targeted?

~~~
irq
When they improve their netops chops. Their recent junior-level mistakes (like
improper spanning tree settings) are an indication of the level of their skill
in this area.

~~~
ghratch
I thought spanning trees were elected/discovered automatically by the routers
themselves instead of being manually set up.

~~~
krunaldo
Yes and no :)

The tree is built up automatically but you can weight the paths and also which
is the start node of the tree. There are also a lot of settings that may or
may not completely fuck you over or fix a problem.

Also you really want to disable STP on ports going to servers as this will 1)
speed up recovery 2) prevent any malicious packets going out from them.

------
martinced
If several countries, distribute across various continents, have managed to
put in place three-strikes and six-strikes (not that I think it's good), it
means that the one and foremost knee-jerking argument saying _"You can't do
anything about DDoS because: [X] It's technically not realist"_ is gone.

Technically now ISPs could throttle the bandwith (or even disallow net access)
to zombies boxen used in DDoS attacks in all the countries applying
"x-strikes" rules.

So there _may_ be light at the end of the tunnel.

It's not exactly as if DDoS was a fatality and nothing could be done about it.

~~~
tatsuke95
> _"Technically now ISPs could throttle the bandwith (or even disallow net
> access)"_

Most ISPs charge for bandwidth. Outside of governmental coercion, is there any
incentive for them to do this?

~~~
nwh
Most charge for incoming but not outgoing, which is what would be used in a
DOS attack.

------
X4
I think GitHub should add hardcore anti-scraping functionality. Even though I
enjoy Opensource repositories, I wouldn't like some bot/govermnent or other
evil to mess with all of our contributions to humanity in a way to defeat us.

~~~
kaoD
If it's open, I can scrape it.

~~~
X4
Not buying it.

