
GitHub: About This Week's Availability - remi
https://github.com/blog/1036-about-this-week-s-availability
======
frisco
I remember reading a comment a long time ago -- maybe here or Quora, I think
-- talking about how people don't usually realize how much trouble organized
crime gives small and midsize businesses. You'll start getting DDOSed and then
an email appears a day later: pay us and we'll stop bringing down your site. I
have no idea if extortion is what's happening here versus simple lulzseekers
or a grudge someone holds against them, but I wouldn't be surprised.

~~~
jrockway
"Organized crime" or just regular crime?

~~~
bdg
In some countries that are absolutely poor, it is entirely common to have the
local mob recruiter look for young talented kids that they can bring into
their organization. They make them an offer when the kids are out begging for
change or something like "we'll take care of your mom and make sure she
doesn't break her legs if you come and get an _education_ in our _school_ , we
might even _pay_ you."

The youngster ends up in basically a run-down warehouse full of other kids
each taught how to look for basic exploits like mysql injection or use root
kits, and they sit around and do this all day.

This is why some countries are typically COEs for hacking, such as Russia with
crimeware kits, and until recently Brazil for bank hacking.

This is also why you need coordinated efforts of special agents armed with
assault rifles to clean out some spam-mills sometimes (they're taking them
away from some mob).

What I've described is a bit outmoded, typically this behavior is turning into
other things such as this: [http://abh-news.com/cybercrime-china-hackers-
training-camp-c...](http://abh-news.com/cybercrime-china-hackers-training-
camp-closed-866.html) as well as the makers of exploit scripts are just
getting better and recruiting these kids doesn't make sense any more... you
can intelligently scan for millions of sites using phpmyadmin and
automatically run your exploit with it.

As with any crime you find the trends all over the map: not all robberies are
mob related, not all of them aren't, some of them are more elaborate than
others, etc.

------
tworats
Are there any good guides on how to combat a DDOS? I'm curious how a team of
good technical people without past experience in the area could react.

~~~
zxoq
Fighting a DDOS is very hard if you do not have the resources to purchase more
bandwidth. The DDOS attacks I've dealt with have been almost exclusively UDP-
floods in the 500-2000 mbit range. No amount of server-side configuration will
fix that unfortunately. If the attack is some more creative form, but with
less bandwidth (SYN flood etc.) a cisco router can help out a lot.

Your best bet is to move to a protected hosting provider (dragonara or
ddoshostingsolutions are quite good for Europe / US). Failing that you can
purchase a bunch of VPS servers, round-robin DNS to them and reverse proxy to
your actual server (and keep that IP off the DNS system entirely). This way
only the proxy servers will go down if you are attacked.

Edit: If they are not spoofing they sender address, you can achieve some
success by mailing the hosting provider of the IP performing the attack. In
90% of cases the server has been hijacked and they will shut it down, this
only works for US / European companies though since Chinese / Russian hosting
providers never ever reply in my experience. For maximum success you can
attach a tcpdump of the attack traffic.

~~~
newhouseb
I was poking around a botnet today and saw that the particular one I was
looking at was just DDOSing DNS (port 53). Could you just use an /etc/hosts
file and turn off UDP entirely? My understanding is that part of the problem
is that the reason 53 is so vulnerable is because the OS sits there waiting
for any UDP replies from DNS on it.

~~~
zxoq
If your server is running slow because of the load, and not because of the
bandwidth of the attack that might be a viable route. It is trivial to use
iptables to block all UDP traffic except from your configured DNS server.

Most of the cases though it's as simple as attack is 200 mbit and connection
is 100mbit -> no amount of dropping packets once they reach the server will do
anything, as the connection is entirely saturated.

------
robbiet480
Who would DDOS GitHub and for that matter, why?

~~~
imajes
extortionists.

There's a growing band of pirates (the web kind) who hold sites to ransom via
their massive botnets, in exchange for payment -- 'protection money'.

~~~
pavel_lishin
But why Github? Wouldn't a less tech-savvy business be much more likely to
cave in, and have a better cash flow? (e.g., porn and gambling sites?)

~~~
wahnfrieden
Uptime is very valuable to github. For example, I can't easily deploy to my
site when githubs down. I suspect many others have bought into its
availability too.

~~~
cdelsolar
I had a very slow deploy earlier this week; if it had gotten canceled in the
middle of it that would have been pretty bad. Maybe the deployment script
should copy the repo head to one of my own servers and continue from there?
What do people do?

------
dbecker
If I were a DDoS attacker, I wouldn't want to make enemies with the sort of
people that rely on github.

Though it's easy to imagine that those guys think differently than I do.

------
pron
Very nice. Just one thing: I already know I'm awesome and I don't need my
source revision control provider to boost my self confidence (I'm referring to
the line "...you, our awesome GitHub users"). OTOH, it's clear that it's
GitHub's employees, rather than its users that need constant reassurance of
their coolness. GitHub is a nice company that provides a good, solid and
necessary service - hosted SCM. Then why is it that their _company_ blog
focuses on their drinking habits?

Dear GitHub,

Although your product may sound boring to laymen, you provide a good service
to a very important industry. Your "boring" day job reflects nothing on your
personal and very exciting lives. Your customers, however, like you for what
you provide them with, not for your companionship, and I'm sure your friends
like you for the opposite reasons, as they very well should. In fact,
everybody loves you for many reasons, and you are all very lovable. So please,
keep your extra-curricular activities to your friends, and your work
activities to your customers. You can call your friends "awesome" if that's
your kind of thing, but for various reasons it is better to treat your
customers with proper decorum. If you're unable to keep the two separate,
you're in for some bitter disappointment later in life.

Love, everyone.

------
jvoorhis
I've used Github frequently all week,personally and professionally, and if
they temporarily lost a nine or three, I haven't noticed.

------
AznHisoka
Couldn't they just keep adding IP's to some blacklist (not htaccess, but
something more efficient), and the attackers will eventually run out of IPs to
attack from? There's only a finite # of computers you can really have control
of. So you end up blocking some innocents, but you take care of that after the
attack is over.

~~~
Tobu
Maybe a whitelist could work. My IP has visited in the past week / I have a
legit session cookie, now gimme bandwidth.

~~~
brown9-2
This only works if you decide you want to never accept a new customer again.

~~~
dredmorbius
Not necessarily, though it depends on the complexity you want to throw at the
problem.

If you have known (and proven) legitimate traffic, give it a high QoS.

For never-been-seen IPs, provide a limited initial rate, with training based
on experience.

Use tools such as the ASN Routeviews project to identify contiguous blocks of
IP space (or any other source of BGP routing data, but the RV data are
queryable via DNS and downloadable as zonefiles).

ID bad actors and either block or severely limit them.

Train up or down other traffic as appropriate.

This leaves you tracking large amounts of IP space. Doable in IPv4, somewhat
more difficult under IPv6, though you're still probably going to be able to do
something reasonable.

I'd like to see more of this pushed into the routing/networking layer,
automatically, based on application-layer-based feedback. Maybe someday.

------
IanDrake
I wonder if something like <http://www.cloudflare.com> could help.

------
ellie42
I think if you build your app properly the only problem with DDoS is traffic.
Github partnerships with Rackspace so their cloud traffic can't be expensive.
Correct me if I'm wrong.

~~~
newobj
I suspect you may be oversimplifying the solution or undersimplifying the
nature of these kinds of attacks. What would constitute a 'properly'
infallible defense to you?

------
robbyt
Github is really great, but I'm a cynic.

Indeed, why _would_ anyone want to DDoS github? How can we believe Github that
the outages were due to a DDoS? They're smart people, aren't they using solid
load balancesrs that can mitigate DDoS attacks? Why haven't they issued an
actual statement describing the supposed attacks in better detail?

~~~
jtchang
Load balancers generally don't offer as much mitigation against DDoS as you
think. The load balancer will probably fall over if the backend servers don't
do so first. The sheer volume of seemingly valid traffic is what kills you.

Which is why effective anti-DDoS means working with your upstream provider to
figure out how to differentiate between real traffic and fake traffic. The
best solution to a DDoS is the phone # of a tech that can implement firewall
rules upstream and a good traffic analyzer to tell them exactly how to filter
it.

~~~
swalberg
Often you can't pick apart the good traffic from the bad.

Fortunately bots are usually pretty stupid. If you can outrun them on
bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The
web server or load balancer can send those redirects really fast and it
doesn't take much bandwidth either. I have never seen a bot chase that
redirect.

After a particularly nasty DDOS attack (where our upstream provider just
shrugged their shoulders) I wrote an F5 iRule:

1\. Check for the IAMNOTABOT cookie 2\. If not there, redirect to /cookie-
me?oldpage=the_page_you_were_trying_to_access 3\. Set IAMNOTABOT=true cookie
4\. Redirect to the old page

~~~
jarcoal
Another, similar, technique is checking for a NOBOT cookie and if it doesn't
exist, serve up a page that uses javascript to set the cookie, then reloads.

This requires the bot to interpret javascript, and it can easily be configured
in nginx and other load balancers.

