
We are under attack - greatfire
https://en.greatfire.org/blog/2015/mar/we-are-under-attack
======
dreamins
No one likes DDOSes from China. One can plead Amazon as much as one wants. Pay
or get booted, there are probably 2 engineers paid 6 figures a year by Amazon
getting paged for this DDOS, someone must pay for the time they spend tuning
DDOS protection instead of their primary project to make attacked website
accessible for everyone else.

Source: worked for AWS, was oncall during similar attacks. Nasty things with
those they tend to start around 6-7PM (guess when does working day start in
China).

~~~
powertower
Why don't providers just set up a system that creates a country-level null
route for a given destination IP? And have a UI with a checkbox for the user
to do it, for any selected country. It would mitigate the issue, and once it's
over, the user can un-restrict traffic / or just keep blocking if it's a non-
valuable source.

I know you can do this on the server, using many different techniques. But
this does not help as the traffic still reaches you (that you have to pay
for).

You can also do this with Geo DNS (and get much less of a bill).

And the ISPs, datacenters, and anyone with a router can block ASIA or China
allocated IP ranges. Especially if it's not the type of a flood that's
designed to attack the routers (instead of the web-server).

So what's stopping Amazon?

~~~
latj
The point of their website is to make censored content available to Chinese
users.

China is attacking them to prevent Chinese people from reading the website.

Your suggestion is to make the site unavailable to China.

Do you see why it is not a solution? You are basically setting up a market for
censorship-- the attack doesnt ever have to end-- depending on how much China
is willing to pay to keep the website offline.

~~~
themgt
OTOH if the great firewall already blocks this site, wouldn't that mean normal
Chinese citizens would access it through a VPN via another country?

~~~
obituary_latte
If normal citizens had access to a VPN in which to access this site from
another country, it would be quite redundant to use this site then wouldn't
it? Maybe I'm not understanding.

~~~
mod
If normal citizens have access to it without a VPN, then why doesn't China
just block it with the firewall instead of ddosing?

Maybe I'm not understanding.

~~~
obituary_latte
Yeah, no -- my fault. I wrongly assumed it was some type of proxy or way to
get around the great firewall.

------
stevecalifornia
Contact Akamai who recently bought the DDOS mitigation service Prolexic. They
may be able to mitigate the attack and save you bandwidth costs.

Alternatively, call CloudFlare.

Don't just absorb this through Amazon.

~~~
qeorge
CloudFlare is probably not a good choice. They recently blocked access to a
similar service, Lantern, per the linked WSJ article.

 _" CloudFlare, which offers content-delivery network services, said last week
it cut off Lantern’s use of the service, saying it was unauthorized. “We don’t
do anything to thwart the content restrictions in China or other countries,”
said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and
we comply with the law.”"_

[http://www.wsj.com/articles/u-s-cloud-providers-face-
backlas...](http://www.wsj.com/articles/u-s-cloud-providers-face-backlash-
from-chinas-censors-1426541126)

I'm not very impressed. Maybe someone from CloudFlare is around to defend that
position further.

~~~
hackuser
> "We don’t do anything to thwart the content restrictions in China or other
> countries," said Matthew Prince, chief executive of CloudFlare. "We’re a
> tech company and we comply with the law."

There's a popular idea that businesses (and people) have no responsibilities
to anyone but themselves, because what they have is theirs; they built it
themselves. But if you think about it a little, it's obviously false. Here's a
more accurate statement:

 _We 're a tech company whose success is completely dependent on the freedoms
in our nation and many other nations around the world, and on the political
and economic systems, infrastructure, and enormous wealth that blossomed from
them. Without the sacrifices of blood and treasure by our predecessors of
hundreds of years, and of many people today, we would not have these resources
or opportunities today. There are many talented people born in many countries
who, without these benefits, have no opportunity for success._

They can't sacrifice their company for every principle, every time, but
there's a middle ground between that and 'we're just a tech company so we have
no responsibilities'.

~~~
grandalf
> There's a popular idea that businesses (and people) have no responsibilities
> to anyone but themselves.

It's not just a popular idea, it's why they are created as firms instead of
philanthropies. There is a difference and it does matter what the expectations
of the donors/investors are.

> We're a tech company whose success is completely dependent on the freedoms
> in our nation...

This sounds great but how is it reflected in company policies?

> They can't sacrifice their company for every principle, every time, but
> there's a middle ground between that 'we're just a tech company so we have
> no responsibilities'.

A company could easily make a statement to its investors about its moral
stance on issues that it expects might harm the bottom line.

The company does have responsibility to its investors not to go rogue and burn
cash just because it feels good. Most of the time the kind of corporate
behavior that you praise is actually clever PR that costs the companies
little.

~~~
Frondo
Everyone has a responsibility to the world around them.

It may not be coded into law, but it is still a true statement.

~~~
grandalf
Not sure how you thought my sentiments disagree with that.

------
moe
Sitting down and writing a blog-post seems a pretty laid back reaction to
$30k/day in Amazon bills...

First thing I'd have done is _take the site offline_ and call the various DDoS
mitigation services (Incapsula, Cloudflare, etc.).

Pretty surely most of them would gladly pick up the slack here, given the free
PR (possibly even in mainstream press) they get in return.

~~~
pi-err
Not sure that turning the light off is the first thing you want to do when
you're bullied.

And pretty certain that there's zero PR value in those stories.

~~~
chralieboy
They wouldn't just be giving in, it would take less than a day to get up and
running on CloudFlare. That is probably worth avoiding $30k bill, especially
for a non-profit.

As for PR, there absolutely is good will towards organizations that take on
censorship. This is the #1 story on HN, so a company stepping in and saying
"we got this covered, free of charge" not only shows they are good people, but
also gives would be customers an idea of the quick implementation cycle for
their CDN + the load it can handle.

Real world use case + helping to stop censorship = great PR

~~~
loganu
This is exactly what I was thinking. I read the blog post, then came to the
comments. I completely expected something along the lines of ... "Hey, I'm the
VP @ SomethingTech, we like what you're doing and will help you mitigate the
attacks for free if you get in touch with us." ... to be the top comment here.

------
cpncrunch
Move to OVH -- they offer free DDoS protection as standard, and unlimited
bandwidth. I just moved to OVH after getting DDoSed. I'm paying $109/month for
a quad core 3.7Ghz Xeon, 64GB RAM, dual 2TB software RAID. It's a pretty sweet
deal, and I haven't had any problems so far.

~~~
rprime
Unrelated to the story at hand. Have you been recently DDoSed on OVH? I know
that (at least some time ago) they just null-route/deactivate your account on
spot with no notification on anything that looks like a DDoS.

~~~
fapjacks
This may be true. However, they are FAR more inclined to work with a customer
before just flipping a switch like a lot of (for example) American hosting
companies will. I have nothing but good things to say about OVH. They were
helpful during a DDoS I had about a year ago throughout the incident, but I'm
sure they would have nullrouted me had I not been so responsive to their
support guys when things started going badly. "I'm sure" here is pure anecdote
and based on zero evidence, but it's the feeling I got.

------
d0ugie
> Because of the number of requests we are receiving, our bandwidth costs have
> shot up to USD $30,000 per day.

Perhaps the US State Department might be inclined to help?

~~~
xnull6guest
This is the correct answer. Given that this website is part of the 'civil
society' sphere, it likely already gets taxpayer money from the State
Department. They should go to their funders - be they public or private - and
ask for help with their mission - I'm not sure it's appropriate to ask
individuals for help.

~~~
NegatioN
But the funders won't solve this problem though, only mitigate it. (though I
agree it's something they definitely should do ASAP) Putting it out there like
this, seems more likely to overcome the technical difficulties of it.

~~~
e12e
I nice anti-China spin might be part of the mission...?

------
xixixao
Google's Project Shield could help?

[http://www.google.com/ideas/projects/project-
shield/](http://www.google.com/ideas/projects/project-shield/)

~~~
formatjam
Yes, Project Shield is the best solution.

------
ArtDev
This is interesting. Though Hacker News appears to not be blocked, it has been
flagged as "Contradictory" on certain days. Is the Chinese government blocking
certain news items?

Take a look here:
[https://en.greatfire.org/news.ycombinator.com](https://en.greatfire.org/news.ycombinator.com)

~~~
Kronopath
If you click on the day in the calendar to look at the details, the
"Contradictory" status is when some of their test servers work and others
don't. For this site in particular, there are several servers showing a
timeout and no data received. So it's possible that HN is being partially
blocked.

~~~
nitrogen
It looks specifically like cURL's exit value and the downloaded page size
varied on some requests. It would be interesting to know what the
contradictory download size was, and what the curl exit value was.

~~~
Kronopath
All that is there when you click on the date, if you scroll rightwards. The
exit values in the blocked sites are timeouts (CURLE_OPERATION_TIMEDOUT), and
the broken download sizes are 0 bytes.

------
nitinics
You should trace the attackers by tracing back. Work with your upstream
providers and mailing lists (NANOG) and publicly shame these attackers.
Likely, they are spoofing addresses - validate that and make sure you let the
network know where the spoofed traffic is sourcing from to follow BCP38 and
BCP84, defined by RFCs 2827 and 3704.

~~~
Nyr
Assuming it is direct spoofed traffic and not a reflection, naming and shaming
will accomplish nothing. Names of the big ISPs allowing this are not a secret.

------
NextPerception
"We need help to manage this. If you have expertise in this area, please
contact Charlie Smith or ping us via Twitter."

Step 1 : Unleash DDos Attack against target

Step 2 : Wait for the the target to become overwhelmed and ask for public
assistance

Step 3 : Contact the target under the guise of being able to help

Step 4 : Win trust of Target after "mitigating" the attack you are actually in
control of

Step 5 : Repeat until enough access has been gained

------
nerdy
First a 2.6bn request/hour DDoS and then making the front page of HN... talk
about getting flooded with requests.

Hopefully making the front page will at least get them the attention of Amazon
or enough donations to cover the temporarily (absurdly) high operating cost.

~~~
stanmancan
They claim they're able to handle the 2.5bn req/hour right now, and if thats
true, the HN traffic wouldn't even be noticable

~~~
nerdy
Yeah there might've been a smidge of sarcasm in there

------
e40
With a mission such as yours, I would think Cloudflare (or similar) protection
is a must.

------
methou
A number of the DDoSes from China are involuntary induced by DNS Poisoning.
When users query a block dns name, they may receive an IP other than the
website they want to visit. It may used to redirect traffic to make DDoS
without those 'drones' even know about they're involved. Just like for a week
long, whenever I try to access Facebook, I got redirected to some german IP,
and receive a TLS CN mismatch error.

------
abfan1127
Are you using Cloudflare?

~~~
dewey
A lot of CloudFlare IPs are blocked by the chinese firewall, for a site that's
primarily aimed at chinese users probably not an option.

Edit: I don't actually know if this is still true and on what scale, I just
know that's it's true for a website I use according to chinese users.

~~~
mirashii
Even if it is true, getting an enterprise account, which is still very
reasonably priced, gives you dedicated IP addresses to your site, so this
shouldn't be a huge concern. Generally when under fire, Cloudflare is also
happy to get you up and running and talk finer details on billing for the long
term later.

The bigger problem is their stance on Latern mentioned above.

~~~
mirashii
Honestly a bit confused on the downvotes, was just trying to provide
additional information for people who aren't familiar with working with CF.
Anyone care to enlighten me?

------
disjointrevelry
I hope you keep us up to date. I live in the US and I've had accounts suddenly
locked out after their passwords were changed even for criticizing some
Chinese issue. Several email accounts, among other sites. With the amount of
money the Chinese are spending, I think it's not going to be easy in the US.
Maybe you're an American and things will be easier for you, but I've learned
not to trust the natives. Not even their law enforcement.

Maybe the corporations might be more stable and have more integrity, but you
have to realize that China is the largest economy in the world now, and they
can flick Amazon out of their market with their pinky and not even blink.

------
jgroszko
Isn't this title a little sensationalist without specifying who's under
attack? I assumed it was a royal we and after clicking the link realized it
was just this one site.

~~~
rcthompson
It's the title of the blog post that's being linked to. With that title being
transplanted to HN, you need to consciously think about what the context is
supposed to be, which is why HN lists the source domain.

------
richieb
Maybe [http://www.google.com/ideas/projects/project-
shield/](http://www.google.com/ideas/projects/project-shield/) can help?

------
calbear81
State sponsored cyber warfare is something that happens at a scale that normal
private companies and organizations are not well equipped to deal with. I
wonder if there's someone you can alert in the government who can consider
coordinating a counterattack or pass a backchannel note to the right people to
cut it out.

~~~
LLWM
Calling a simple DDoS cyber warfare is a bit hyperbolic when we know there are
actual sophisticated attacks being carried out against far more important
targets.

------
morlockhq
How effective would this solution be:

[http://www.linuxjournal.com/content/back-dead-simple-bash-
co...](http://www.linuxjournal.com/content/back-dead-simple-bash-complex-ddos)

~~~
sah2ed
A similar approach is linked to in the article's comments in
[http://www.inetbase.com/scripts/ddos/ddos.sh](http://www.inetbase.com/scripts/ddos/ddos.sh)

from [http://deflate.medialayer.com/](http://deflate.medialayer.com/)

------
nathanb
Should the Chinese government have the power to shield their citizens from
information and monitor them electronically?

Should a group of people in democratic, Western countries be able to subvert
the will of a world superpower with impunity?

Of the two scary worlds, I guess I'd rather choose the latter. But I don't
even like having to choose.

(I doubt Amazon like being asked to choose even less, and I would be surprised
if they cut you any slack. Sedition is not looked upon favorably, and abetting
those perpetrating it is not either.)

~~~
1ris
>Should a group of people in democratic, Western countries be able to subvert
the will of a world superpower with impunity?

Crazy time we live it that this is even possible.

~~~
logfromblammo
I'd suggest that the ability to perform a subversion of the will of a foreign
state is a necessary adaptation, preventing a nominally democratic state from
sliding towards aristocratic, oligarchic, or plutocratic governance.

The ability to increase freedom in a foreign state is related to the ability
to prevent a decrease of freedom in your own.

Inconvenient websites help uphold the duties of the fourth estate when
mainstream media outlets have seemingly abandoned--or at least heavily de-
prioritized--those duties. It is important for all governments, not just those
with sketchy human rights records, to know that even the mightiest machine can
be taken offline by a single wooden sandal.

------
secfirstmd
For human rights, civil society and media working in difficult DDoS threat
environments check out these guys, they are awesome ---> Equalit.ie
[https://equalit.ie/](https://equalit.ie/) who run a DDoS project called
Deflect - which I think is free for people in those categories of people.
[https://equalit.ie/portfolio/deflect/](https://equalit.ie/portfolio/deflect/)

------
jbrun
Call [http://equalit.ie/](http://equalit.ie/) \- they have a free open source
tool for exactly this!

------
toaskaquestion
Out of curiosity, who pays more -- the attacker or the victim? Purely from a
monetary perspective.

Edit: never mind, figured it's obvious.

Aren't DDoS requests pretty much simple GET requests? Is it not possible to
determine which requests to serve and which ones to ignore?

~~~
Sanddancer
These days, DDoSes are not just lots of GET requests, because, as you said,
they're fairly easy to mitigate. These days, the most common attacks are
various UDP-based attacks, like NTP reflection [1]. You send a spoofed header
to a server that speaks over UDP, and they send a huge amount of traffic to
the victim.

[https://blog.cloudflare.com/understanding-and-mitigating-
ntp...](https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-
ddos-attacks/)

~~~
lucaspiller
Couldn't something like that be blocked at a firewall level with AWS though,
i.e. drop everything except TCP port 80?

------
WhitneyLand
I hate that this is happening, but isn't this something you have to
expect/prepare for when your business involves controversy?

Not to mention when the people who don't like you have the resources of nation
states.

------
dataker
They couldn't wait a day?

[http://gizmodo.com/china-finally-admits-it-has-an-army-of-
ha...](http://gizmodo.com/china-finally-admits-it-has-an-army-of-hackers-for-
cybe-1692188006)

~~~
LLWM
Perhaps next week they'll admit they have a navy.

------
tlrobinson
It would be interesting to see which IP space the bulk of the traffic is
coming from. Seems like it would be trivial for the Chinese government to
spoof traffic from any IP within China...

~~~
Tossrock
For a DDoS, you can spoof your IP to anything, because you don't care about
actually receiving response packets. This is standard in a SYN flood.

------
Animats
Post the IP addresses from which you're getting attacked. Others can analyze
them by ISP, and public pressure on the worst ISPs might help.

~~~
Consultant32452
The bad ISP in question is the People's Republic of China.

~~~
Animats
I mean post the whole IP address list for public analysis.

~~~
Consultant32452
Honestly most companies I've worked for have blocked the entire Chinese IP
block. Obviously Chinese hackers can use proxies but it honestly does cut out
a TON of problems. The downside of course is you will never get a Chinese
customer/viewer. In preparing to respond to this post I googled "China IP
block" and pretty much every result was about how to configure .htaccess or
iptables to block the entire country.

------
infinitnet
[https://news.ycombinator.com/item?id=9242710](https://news.ycombinator.com/item?id=9242710)

------
rtpg
I wonder if the US gov't wouldn't mind donating $11m/year to this org to deal
with the increased costs...

------
anonbanker
The more I watch the security theater, the more this all looks to be set-up. I
really don't fear the chinese, as the NSA is far more competent and organized,
and is much more dangerous to the world at large, not just the citizens of the
United States.

anyone remember that story a few days ago where China said "Hey, we want NSA-
level backdoors in hardware, too!"? can someone help me with a link?

------
EGreg
Isn't this what CloudFlare has built a reputation for defending against?

------
gnarbarian
when stuff like this happens it would be nice if we could simply refuse to
route all traffic originating in the offending countries until the attack is
over.

------
datashovel
It seems a little hypocritical to me that AWS will create a service for every
technology known to man, but will not create a service to help companies who
rely on their infrastructure to deal with DDoS.

------
haosdent
Could we block this in ISP?

------
me1010
yawnnnn...

wake me when the clickbate is over.

------
stevejones
This thread brought to you by the CloudFare PR Agency.

------
Xeoncross
\- Nginx instead of Apache

\- Use memcached+nginx to load the drupal content instead of calling up PHP
each request (PHP saves the page in memcached, nginx reads it from there).

\- put it all behind CloudFlare

This works for Wordpress too.

------
ramigb
I think they need to use something similar to this ...

[http://en.wikipedia.org/wiki/Coral_Content_Distribution_Netw...](http://en.wikipedia.org/wiki/Coral_Content_Distribution_Network)

~~~
ramigb
I would appreciate it if people who down voted my comment explained to me if
it's because i'm wrong or because they don't understand what i am trying to
say or just for the heck of it :).

~~~
lucaspiller
The issue is your comment doesn't really add any value. Anyone can paste a
random link saying "you should use this" but it takes effort to explain why it
would be useful to them. HN comments are about fostering discussion, so say
something to be discussed :-)

~~~
ramigb
Thank you for your input, but 1. this is not a random link, 2. the explanation
is all inside the link i posted, so why to be redundant? anyways, if i didn't
care for a discussion i wouldn't post it at the first place and i wouldn't
later ask why was i downvoted, thank you again for your opinion lucaspiller.

------
revelation
If you're paying $30000 for a site that is pretty much all static in bandwidth
costs a day, you're probably paying about $29990 too much.

~~~
random_rr
2.6 billion requests per hour will do that to you

~~~
brador
Route them through a cached captcha page. Or just call cloudflare.

~~~
eli
Serving a captcha page is _more_ work than serving a static page.

~~~
monort
You can block with firewall IPs of users, who didn't solve captcha. You will
get only SYNs from incoming connection requests then.

~~~
mikey_p
You don't really understand the distributed part of DDoS. I've had services
taken down my attacks and when doing a post mortem we could see the increase
in traffic, but when accounting for frequency, our office was still the main
user.

