

GitHub experiencing a large DDoS attack - trevorhartman
https://status.github.com/?date=8-15-2013

======
AnSavvides
I do not understand why GitHub get attacked so often - honestly it's not like
they are doing anything wrong, they are a fabulous service and the go-to place
for most (if not all) developers for all sorts of projects, be it open source
or in an enterprise/corporate context.

~~~
manojlds
Didn't you just answer your question? It is the "go-to" place for most
developers.

~~~
Cakez0r
That doesn't really answer his question. Why would somebody want to
inconvenience developers?

~~~
viraptor
\- Raising awareness of self-hosted alternatives

\- Getting people to use BB instead

\- Fake DDOS to get more companies to pay for github:enterprise

\- Preventing a system upgrade in a company which has deployment relying on
github

There are lots of potential ways to use it...

------
danielsju6
Just a PSA Git is distributed, fire up a quick instance with git daemon.

git daemon --verbose --export-all --base-path=.git --reuseaddr --strict-paths
.git/

~~~
daemon13
>> Just a PSA Git is distributed

Can you please elaborate?

~~~
swede
Public Service Announcement

You can have multiple remotes for your Git repositories, one of them (Github)
being down should not affect your work since you can coordinate a new one with
your team in the manner described above.

~~~
vhost-
Yep! I usually setup a mirror at bitbucket and set my local to push to both
github and bitbucket and only pull from github. It worked well when github was
down and I needed a branch from a remote developer.

~~~
anemator
Nice. I didn't know you could push to multiple remotes (in the same command).
Could you give me an example? Thanks!

~~~
vhost-
The reply right under mine seems to be a reply to you. It has a stackoverflow
link that shows you how to setup two URLs in your origin remote.

------
cheeseprocedure
Can someone with experience mitigating an attack like this describe how it's
done? A known set of hosts/address spaces is fine, but it's the "distributed"
part I don't understand how to deal with.

~~~
pktgen
We don't know anything about the attack, but in general:

1\. All of this requires that you have more bandwidth than the attacker.

2\. ACL drop everything but the ports/protocols you use on your frontend IP.
For github.com that would be TCP 80 and 443. (ACLs are cheap, can be done at
wire-rate on any capable edge router, so they're a good "first step." In this
case it would have the bonus of dropping DNS amplification attacks which are
increasingly common.)

3\. If it's a TCP SYN flood (probably from spoofed IPs) enable SYN cookies.
(Of course now the bottleneck is their load balancer.)

4\. If it's an L7 attack, they'll need to identify patterns in the requests to
identify and drop the traffic. For example, the attacker may be requesting a
single URL only. If it's an L7 attack on a TCP service, they can also
automatically add a drop rule for that source address, because at this point
the client IP will have been validated (via the SYN challenge).

They may also be able to identify certain patterns in the traffic that can be
ACL'd at the edge. For example, IP TTL being identical.

------
cmer
This is getting really problematic. It stops our whole team every time.

Anybody has experience with Gitlab and Gitlab CI? How's the flow compared to
Github, especially for pull requests, commenting and collaboration?

~~~
twerquie
This happens to our team too.

Does anyone else find it ironic that Git is a distributed version control
system, yet we rushed to centralize it and base our entire workflow around
GitHub?

~~~
camus
Github is just a node. IF Github dies , people still have the full repos on
their computers.That's not always the case with SVN... And most people are too
lazy/dont have resources to set up a git server anyway.

~~~
__alexs
Most teams that work on GitHub have none of their own workflow or
infrastructure for sharing code when GH goes down.

You can probably blame those teams for that more than GitHub but people do
seem to have bought into the idea that we can just jam everything up to GitHub
and get it back later.

~~~
briandear
Of course from a risk-management perspective, the likelihood of a long term
outage is statistically low enough to not justify the expenditure for setting
up and maintaining alternative systems (other than simply having repos backed
up.) It's the anti-TSA approach. The TSA spends billions and hassles everyone
despite the relative rareness of air-travel attacks. But, the severity of a
potential attack weighed against the expenditure has led politicians to
believe that the cost is worth it.

The Heroku/AWS outages a few years back had far more of an impact, yet Heroku
still (to my knowledge) relies exclusively on AWS-East, because presumably the
risk-profile doesn't exceed the threshold to justify the expenditures required
to mitigate the risk.

I just wish the attackers would be considerate enough to share the schedule
for these attacks ahead of time, so I can plan a longer lunch. The rudeness of
these attackers is unparalleled. I'm inclined to write them a strongly worded
letter suggesting same.

------
jbrooksuk
Why is this happening to them almost every day? Is someone jealous? Proving a
point? If so, what point?

Surely by now, GitHub must know who's responsible and putting more in place to
mitigate as much as possible before this happens. Right?

~~~
ripter
Everyday? The last attack was August 4th.

[https://status.github.com/messages](https://status.github.com/messages)

~~~
jbrooksuk
I said "almost".

~~~
KurtMueller
You said 'almost every day'. Seems more like 'almost every week'. I'm just
nitpicking though.

------
pulleasy
[http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-commit-
loca...](http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-commit-locally.png)

------
daigoba66
I wonder if there is a way to work with git offline, and then synchronize
later? /sarcasm

------
nonchalance
while git is distributed, Github managed to centralize the open source culture

------
gavingmiller
I ask because I have no plausible answer: What does someone stand to gain by
DDoSing github?

~~~
camus
blackmailing ? "if you dont pay we'll DDOS you" or whatever. Given all these
infected Windows computers all over the world ,it's not going to stop soon.
That's the Microsoft legacy.

~~~
gtaylor
Am I crazy in thinking that there's zero chance Github would pay a "ransom"?
Seems like they'd just work with their vendors/providers and mitigate it to
avoid setting that precedent (making them even more vulnerable to future
attempts).

~~~
bradleyland
The attacker's bet is that they can apply pressure beyond the victim's ability
to mitigate. If they win that bet, the victim may reach a point where paying
out is preferable to the lost business they'd suffer because of the attack.

------
joemaller1
Let me get this straight, the command-line interface for googling code in a
locally cloned GitHub repo is called grep?

~~~
oalders
Or "git grep".

------
thecodemonkey
What a coincidence. We actually just finished migrating all our projects over
to GitLab about an hour before this outage.

I still very much love GitHub, it just ended up not scaling for us (we have a
lot of repositories that seldom needs to be touched, which results in a $20 /
month Linode + Backup being a much better solution)

GitLab also allows us to group repositories and gives a little bit more
flexibility in regards of git server-hooks. Also, server-side branch locking!
(Does anybody know how to lock branches server-side with GitHub?)

------
joemaller1
Snarky comments about distributed version control aside, the bigger problem is
the ecosystems which revolve around GitHub. Two which are immediately screwing
my day up are Composer and Homebrew. Currently, both are totally dependent on
GitHub.

------
gunmetal
Github enterprise works great.

------
ErikAugust
What's with DDoSing GitHub? Special place in Hell...

~~~
sargun
Github.com is a profitable entity that makes a fairly large amount of money.
Additionally, Github.com hosts a lot of content. Content that other people may
not like. Possibilities are: 1) Extortion of some sort i.e. We want your
money, or we'll DDoS you every week 2) Content i.e. Remove repositories a, b,
and c with content we don't like, or we'll DDoS you every week 3) Weapons
demonstration i.e. We want to show off what l33t hax0rs we are, so we'll DDoS
Github.com every week

------
izolate
Let's all take a watercooler break here at HN

------
xnxn
Time to read up on `git send-email`, I suppose.

------
iclelland
15:50 UTC: We are working to mitigate a large DDoS. We will provide an update
once we have more information.

------
Terretta
You git what you pay for.

(Actually, there are long established yet less high profile alternatives with
decent features and better pricing for commercial teams, like beanstalk or
unfuddle or codebasehq. No hip cred, but for example supporting archived
projects that you can read w/o using up your repo count license.)

~~~
gavingmiller
This argument also works the other way. We payed less for beanstalk and got
half the features. Sure it was a remote git repo, but our use case for GitHub
is more than just external storage, and we were able to remove other tools
that we were paying for.

~~~
Terretta
We've been developing custom software for over a decade. GitHub's pricing
model counts every private repo, active or not, while others let you put repos
into R/W or archive mode, or don't count repos at all and only count against
your storage quota.

The difference in annual cost is significant when you have 20 active
independent projects, and hundreds of archived projects, not to mention all
the customers needing accounts and access.

------
alfg
Seems to be a recurring problem for them now.

------
postit
Why in hell is anyone so pissed with GitHub?

------
tlongren
Appears to be loading fine now.

------
briandear
Just a thought, but it would be interesting if there was a DDoS attack on the
github status page. Then what would we do?

~~~
troygoode
find out from @github

------
briandear
Curious though -- yesterday it was the NYTimes, today github, Apple a few
weeks before that. Pretty high profile stuff in my opinion.

------
wavee
So what?

~~~
briandear
The so what is that the majority of people on HN are developers and the
majority of them use Github either personally or with their teams. So a major
outage is costing potentially millions of dollars in potentially lost
developer time. If you calculate an average hourly of just $50 and a developer
loses 20 minutes. Then that outage costs $17 x the number of affected
developers. That's a pretty big loss.

I'm reading HN right now because I can't manage pull requests and do code
reviews of my distributed team's code. So I'm just spinning in my chair.

That's the so-what.

~~~
rblstr
You can't pull code from another node, i.e another developer? Git is
distributed, if you want centralised source control use Perforce or SVN or
something.

~~~
bdcravens
You may want to reread the comment. It's more than being able to commit and
pull via the command line.

