Hacker Newsnew | comments | show | ask | jobs | submitlogin
GitHub is getting DDoSed again (github.com)
87 points by mathias 490 days ago | comments


jetsnoc 490 days ago | link

It may be time for GitHub to build out multiple availability data centers and use BGP as an anycast tool. We do this. I have public facing IPv4 space that is announced from multiple facilities. Having an IP address hosted from multiple facilities is a powerful tool. This allows providers to hit our datacenter through the least amount of ASN routes. We original did this to minimize latency and create faster regional transaction processing. As an added benefit - DDoS traffic also gets routed to the nearest facility "load balancing" a DDoS so that it only affects a single facility or it splits up the 10gbps of traffic among many facilities if it is coming from many sources. O'Reilly's BGP book has a great chapter on "Anycast."

From the sounds of it their architecture may not support this. If they had a SAN solution capable of replication to multiple data centers like HP LeftHand's product or a multiple master DRBD configuration they may be able to host github from multiple active datacenters and announce the block equally so that providers route traffic to them because their ASN is closest.

Who knows, maybe they do all of this?

-----

mnutt 490 days ago | link

Thanks for the book recommendation. Is that the one by Iljitsch van Beijnum?

Other than the added complexity, can you share any details about the cost? It seems like theoretically some managed hosting providers could offer this assuming they were in multiple datacenters, but I haven't seen any that do.

-----

DigitalSea 490 days ago | link

This happened back in October a couple of days in a row. Who the heck is targeting Github and why? I wonder if these attacks are related to the Chinese hacking attacks that have been publicised lately?

-----

Negitivefrags 490 days ago | link

I often have to wonder if the DDoS gods roll a dice and pick someone to screw each day. We have had many DDoS attacks and never once had any indication as to why.

A few days ago we had a 30 Gbit DDoS. Our server host just blackholed any IP that was touched by it. They kept moving it around to target different bits of our infrastructure (unlike previous attacks that just targeted our website).

We lost 6 servers, but thankfully not enough to take us fully offline though some customers would have experienced problems during that time.

If they had just been a bit more persistent we might have been in serious trouble.

At that level of DDoS your server host doesn't care about keeping you online. They want the traffic off their network.

-----

tomjen3 490 days ago | link

Get a smaller host, that way they can't afford to cut you of.

-----

buttscicles 490 days ago | link

I'd imagine they'd rather cut off somebody than potential performance degradation for other paying customers.

-----

martinced 490 days ago | link

If your nickname here can somehow be linked to the company you're working for (e.g. by checking previous messages you posted to HN or on another board) I think it's not very smart to write what you just wrote.

Now the bad guys knows that they should just have tried a bit longer.

Now maybe that because you were DROP'ing / blacklisting IPs maybe they just ran out of zombies but still...

If I was the attacker and read your message and wanted to be bad, I'd just hammer you a bit more to put you in trouble.

Besides that there are some ISPs who do care about both keeping you online and fighting the low-life scums: the ISP XS4ALL from the Netherlands is (was at least) notoriously famous for that.

-----

switch007 490 days ago | link

They know that anyway. The longer and more severe an attack is, the more damage it's likely to do. A host may be able to sustain 30Gbps for a minute or two, but not for 5 minutes. If 5 out of 10 servers are being attacked, you can probably handle traffic at 50% capacity for a while, but not forever.

-----

citricsquid 490 days ago | link

Nowadays it's so easy to execute a large scale ddos that it's just as likely to be an attention starved teenager as it is a group with an agenda. Github is big enough that it's on peoples radars as "that popular website lots of people rely on", the company I work for has websites smaller than Github and we're indiscriminately ddosed every couple of days.

-----

codexon 490 days ago | link

I wish hosts would stop blaming their customers for these attacks.

Every time someone gets ddosed and complains on hosting forums, the #1 reaction is "who did you piss off?".

-----

mratzloff 490 days ago | link

DDoS attempts are just as often business related as not. Maybe not in this case, but it's "surprising" how often these coincide with a business deal in the works. Also, potential partners (usually international) sometimes run DDoS tests against a network to see how resilient it is to attack.

-----

netrus 490 days ago | link

Hm, I don't think it fits the pattern. The most reported Chinese hacking was about espionage, not bringing down websites. Most of the time, DDOS is either related to politics (newspapers, federal agencies etc.) or related to blackmailing (most likely in the case of GitHub. This of course, does not mean that the hacker is NOT sitting in China. But she could be placed everywhere else just as likely.

-----

crag 490 days ago | link

A lot of Chinese (and everyone else) hacking has been for good old fashion profit. But I do agree, I doubt the Chinese or any other "criminal" element is behind this attack.

Github either pissed someone off, or it's about "street cred".

I was interviewing a candidate about a year ago. Who bragged to me that he hacked the North Broward Hospital District.

And I was like, "why would you hack a hospital?"

His answer, "It was there."

I was dumb-founded. I mean I hired him, but still. :)

-----

minopret 490 days ago | link

It's a Monty Python sort of joke?

Interviewer: (sings) Good night, ring-ding-dingy. (shouts) Five, four, three, two, one!

Candidate: (cackles like a chicken)

Interviewer: (writing) Good! Very good, indeed!

-----

bdg 490 days ago | link

> Who the heck is targeting Github and why?

Typically large services are targeted by some some form of cyber-criminal. As an example only, if the Russian Business Network[1]* were attempting to extort money from github they could use a DDOS and go away when paid (for a while).

This happens to some larger media sites during large events such as the Olympics[2].

    [1] http://en.wikipedia.org/wiki/Russian_Business_Network
    [2] http://news.techworld.com/security/3309917/london-olympics-cio-claims-immunity-from-ddos-attacks/
*Their MO is more towards identity theft

-----

smtddr 490 days ago | link

> wonder if these attacks are related to the Chinese hacking attacks that have been publicised lately?

Might be just me, but I don't like this trend. I'm seeing "the chinese" & "hacking" used too much together without proof more often than not. Almost as if they are being made into the next boogeyman to be afraid of.

-----

tatsuke95 490 days ago | link

>"I'm seeing "the chinese" & "hacking" used too much together without proof more often than not."

Get used to it, you're going to be seeing it for what is likely a few decades at least. We need an enemy. The Chinese are the new Red Army; something to blame Western problems on.

-----

philwelch 490 days ago | link

It's a remix of anti-Communism and the type of anxiety we used to feel about Japan's economic power.

-----

darkarmani 490 days ago | link

> something to blame Western problems on.

That sounds really ignorant. DDoS is not a situation of east versus west.

-----

druiid 490 days ago | link

Well, it kind of is a lot of the time, just not in the way the other commenter was thinking. The majority of DDoS traffic I have seen is from Russian, Turkish, Ukraine and Chinese IP address spaces. Basically I think a lot of 'eastern' countries have a lot of people on the internet but not a lot of security. I imagine high rates of pirated windows (and thus lesser counts of security updates) contributes in some of the places.

-----

nwzpaperman 490 days ago | link

Sovereign nations do employ significant signal intelligence resources to defend their government. The Chinese are heavily invested and have been responsible for the majority of attacks on US IT infrastructure for years. It wasn't discussed openly before because the trade relations weren't degraded/ing like they are today.

In case you've missed the memo, they're threatening all of their neighbors outright with kinetic military force. Japan is no trivial country to threaten force against, so the diplomatic climate has changed and more information is being shared publicly in anticipation of an outright military conflict.

There are conflicting interests between individuals that lead to physical fights and the same applies to sovereign nations whose interests run contrary.

-----

blablabla123 490 days ago | link

>It wasn't discussed openly before

It was, at least in some political magazines.

http://www.heise.de/tp/artikel/7/7551/1.html (German article from 2001)

References: http://www.wired.com/politics/law/news/2001/05/43443

And besides Internet, just for the record https://www.fas.org/irp/news/1999/06/990602-275397.htm

-----

onemorepassword 490 days ago | link

Any service hosting 3rd party content on even a moderate scale (say a few hundred various users) is likely to get DDoS-ed and never find out exactly why.

-----

pixl97 490 days ago | link

I had a single IP on a customers 10M-bit fiber line get DDoSed for what seemed like no apparent reason. We had the ISP blackhole just that IP and the issue went away, but it was a real WTF on why it happened, and no it wasn't some ones accidental DNS mistake.

-----

plaguuuuuu 490 days ago | link

One possible motivation might be to knock github offline following a 0day on some software such that the patch/latest version can't be downloaded.

-----

eksith 490 days ago | link

Are we sure it's a DDoS or is it some sort of massively distributed scrape of the repos? (Side-effect being DDoS regardless)

I'm starting to think this is some kind of grab for intellectual property; maybe even a targetting of private repos to somehow gain access.

-----

jeremymcanally 490 days ago | link

Nope. It's a straightforward DDoS. No targeting of private repos or anything like that.

-----

naftaliharris 490 days ago | link

Reading status.github.com over the last few weeks, I found it interesting how often little things were broken at Github. It's like every few days, a small part of the site is unavailable or the sysadmins are investigating this or that connectivity issue. I guess when you're as big as Github, keeping your site live and operational is completely nontrivial.

-----

joeblau 490 days ago | link

I'm sure other services have similar downtimes and issues, but they just don't give you visibility into their operation. Most companies won't let you know there is a problem unless you figure it out.

-----

cheald 490 days ago | link

This is basically SOP at any reasonably large organization. The difference is that Github tells you about it.

-----

sixbrx 490 days ago | link

I consider it evidence that when some punk figures out how to make a black hole, we're done for. No reason necessary.

-----

nwzpaperman 490 days ago | link

Probably a "sovereign hacker" as non-sovereign-employed programmers are naturally aligned with the open values and creativity that github exists for.

As far as motive goes, if github can be electronically terrorized, laws to protect them and everyone from future electronic terrorism only make sense, right?

<donkey>Eee-ooooo</donkey>

Always do what you can to understand motive!

-----

shinuza 490 days ago | link

Bitbucket at it again.

-----

hackernewbie 490 days ago | link

I think this every time.

-----

niggler 490 days ago | link

Is it just me or has github been down a lot in the later months (moreso than a year ago)? DDoS or otherwise, it doesn't inspire confidence, especially for paid accounts (which I considered but ultimately decided to go with another solution)

-----

obsession 490 days ago | link

Github has 99.9585% uptime past month. That's like 22 minutes of downtime per month.

-----

yRetsyM 490 days ago | link

Maybe they should use CloudFlare?

-----

alexvr 490 days ago | link

Good plan. When a site is DDoSed, encourage thousands of HN viewers to check it out :P

-----

imjared 490 days ago | link

Going to go out on a limb here and say that status.github.com is probably hosted somewhere other than github.com

-----

brdrak 490 days ago | link

I was curious about that too. Looks like status.github.com is hosted on AWS, whereas github.com on Rackspace.

-----

alexvr 490 days ago | link

Good point. That would make sense

-----

babuskov 490 days ago | link

Hm, just a couple of days later after another potential security exploit is published... maybe they did not plug all the holes, and someone is trying to clone all private repositories as soon as possible... hogging the servers in the process.

-----

imbriaco 490 days ago | link

Nope. This was a pretty standard DoS attack.

-----

gojomo 490 days ago | link

There are people who suggest that a DDoS is just a 'digital sit in', a legitimate way for someone to air a grievance, if they think the targets (or world) haven't paid them enough attention.

This view makes DDoS seem more normal or even romantic/heroic, and spreads the tools/know-how more widely. So, pulling off a DDoS becomes a more plausible and attractive aspiration, for a larger set of surly people with marginal reasoning skills and destructive impulses.

The DDoS tactic should be rejected as dishonorable censorship and vandalism, no matter the cause under which it is launched.

-----

esrauch 490 days ago | link

The primary association most Americans have with sit ins is the civil rights movement, but it seems likely that almost all cases where someone refuses to leave an establishment after being asked to leave by the manager the person is being an asshole, exactly like the average DDoS'er.

The default assumption when you see someone unwilling to leave a bar is not that they are noble, and neither should it be the default assumption for a DDoS. That said, I don't think it is reasonable to say no DDoS could ever be noble, just that the vast majority of the time it is just someone being an asshole.

-----

olleicua 490 days ago | link

Does anybody know what grievance is being aired here?

-----

blablabla123 490 days ago | link

Maybe because you have to pay for private projects and Bitbucket UI still isn't as fresh as Github's.

-----

gojomo 490 days ago | link

Should anyone care? Once someone launches a DDoS, their argument is invalid. And reporting the grievance might just encourage them.

I don't know anything about any of Github's attackers, but from other incidents I know of at other services, it could be anything. A billing dispute. Anger that something was taken down... or not taken down in response to an unreasonable request. Anger that an account was suspended... or that some feud-rival's account wasn't suspended. People throwing "do it my way or I'll take your site down" tantrums may not make sense to anyone other them themselves.

-----

leke 490 days ago | link

Who would have the motivation to hack GitHub?

-----

robinh 490 days ago | link

Honest quess: information and code from private repositories?

-----

freddyduarte 490 days ago | link

Meanwhile at Bitbucket... http://status.bitbucket.org/

-----

windexh8er 490 days ago | link

I <3 BitBucket over GitHub, but unfortunately they'd fall over in more or less the same manner under similar circumstances.

-----

nixarn 490 days ago | link

Still not working, trying to load the page of a private repo, keeps loading and loading.

-----

hawkw 490 days ago | link

Who <i>does</i> that?

-----

badgar 490 days ago | link

This is a pretty typical occurrence for a web service provider of their size. When is Github going to be able to not fail when targeted?

-----

irq 490 days ago | link

When they improve their netops chops. Their recent junior-level mistakes (like improper spanning tree settings) are an indication of the level of their skill in this area.

-----

ghratch 490 days ago | link

I thought spanning trees were elected/discovered automatically by the routers themselves instead of being manually set up.

-----

krunaldo 490 days ago | link

Yes and no :)

The tree is built up automatically but you can weight the paths and also which is the start node of the tree. There are also a lot of settings that may or may not completely fuck you over or fix a problem.

Also you really want to disable STP on ports going to servers as this will 1) speed up recovery 2) prevent any malicious packets going out from them.

-----

Mahn 490 days ago | link

Not everybody has the Google-like resources to maintain thousands of fallback servers, and failing that often there isn't much one can do against a well distributed DDoS.

-----

cseelus 490 days ago | link

Questions in case of Github remain: Who and why Its not like DDoSing a target of this size is totally 'free'.

-----

martinced 490 days ago | link

We should first now the size of the attack and how many zombies are participating.

With all the exploits out there coming out on a nearly daily basis it's not exactly either as if having an army of a few tens of thousands of zombies was expensive...

-----

martinced 490 days ago | link

If several countries, distribute across various continents, have managed to put in place three-strikes and six-strikes (not that I think it's good), it means that the one and foremost knee-jerking argument saying "You can't do anything about DDoS because: [X] It's technically not realist" is gone.

Technically now ISPs could throttle the bandwith (or even disallow net access) to zombies boxen used in DDoS attacks in all the countries applying "x-strikes" rules.

So there may be light at the end of the tunnel.

It's not exactly as if DDoS was a fatality and nothing could be done about it.

-----

superbaconman 490 days ago | link

Who's the ISP gonna throttle? It's next to impossible to tell the difference between a legitimate request and a zombie. Also not all zombies are knowing contributing to a ddos. Are you gonna kick grandpa off the net cause he doesn't think before he clicks? Make no mistake this is a very technically hard problem to solve. DDoS attacks aren't going anywhere anytime soon.

-----

tatsuke95 490 days ago | link

>"Technically now ISPs could throttle the bandwith (or even disallow net access)"

Most ISPs charge for bandwidth. Outside of governmental coercion, is there any incentive for them to do this?

-----

nwh 490 days ago | link

Most charge for incoming but not outgoing, which is what would be used in a DOS attack.

-----

codexon 490 days ago | link

Most DDoS attacks are not home connections with extremely limited upload speeds anymore. They are now reflected DNS attacks coming from legitimate DNS servers.

-----

X4 490 days ago | link

I think GitHub should add hardcore anti-scraping functionality. Even though I enjoy Opensource repositories, I wouldn't like some bot/govermnent or other evil to mess with all of our contributions to humanity in a way to defeat us.

-----

kaoD 490 days ago | link

If it's open, I can scrape it.

-----

X4 486 days ago | link

Not buying it.

-----




Guidelines | FAQ | Lists | Bookmarklet | DMCA | News News | Bugs and Feature Requests | Y Combinator | Apply | Library | Contact

Search: