
What happens when DNS in China is poisoned and pointed at your server - aaronbrethorst
http://furbo.org/2015/01/22/fear-china/
======
methou
A Chinese here, I'm deeply ashamed by this malicious action at a national
level. though DNS poisoning wasn't new for Chinese netizens but it was always
poisoning A records to varies foreign military sites like DoD or some Korea
sites (which also makes no sense at all). At scale of this big, those
shameless bastards running state censorship machines has been outrageously
attacking on random innocent site owner, this is far beyond unacceptable.

And starting from like a month ago, a lot of people on the Chinese internet
started to decorate the GFW, and concludes with "for your own good" shit. I
can see from those fragments of thoughts, China is not going to stop the GFW,
and as Moore's Law advancing, it's going to be even worse.

Sorry for swearing, the feeling of "there's nothing anyone can do to stop this
BS" just leads to greater despair.

~~~
fallous
I have seriously contemplated a special request handler for Chinese traffic
that is being directed to the wrong host on our servers. Something along the
lines of "Chinese government censors broke your internet" or the like.

I figure it either helps inform the citizens as to why this link isn't working
or the message itself causes the GFW to ban our IP as well.

------
pilif
Nitpick:

 _> One thing I learned is that Apache can have problems figuring out which
virtual host to use in some cases:

>>If no ServerName is specified, then the server attempts to deduce the
hostname by performing a reverse lookup on the IP address._

ServerName is the name Apache reports in Error messages and using for
redirects. It has nothing to do with the Host-Header sent by the client. If it
did, a reverse lookup definitely wouldn't be of any help either.

His remedy was still ok though - just for the wrong reasons.

~~~
chockenberry
I'm the author of the post linked above: thanks for the clarification about
ServerName. I spent a lot of time reading that documentation and got pretty
confused about how it was actually being used.

~~~
marcosdumay
Complementing it, when there is no Host HTTP header, apache displays the
default virtual server for the port used. Unless you've set it, it's the first
entry on your config file.

------
noonespecial
Not particularly feasible but it would be awesome to simply proxy all of the
requests to their intended destination.

~~~
IgorPartola
Oh man. That would be pretty cool. Somebody like CloudFlare could do that
since they proxy for lots of smaller sites. Then again, HTTPS will add even
more complexity.

~~~
hobs
Even if it likely impossible for anyone smaller than CF, it might even mean
they would move the attack off sooner since it wasnt doing anything :)

------
fallous
We were the lucky targets of China's DNS poisoning on Monday the 19th. Any
requests from clients in China to Edgecast's CDN was instead thrown at our
public IP. After deducing what was going on, we ended up blacklisting large
(class A in some cases) APNIC address blocks assigned to China.

Interestingly, it wasn't our webservers that were overwhelmed but instead the
Cisco firewall that sits in front of them. 25K concurrent connections made it
decidedly unhappy.

~~~
jlmorton
Same here. We received well over 150mbit/s, also on Monday the 19th, also from
China. We also saw occassional spikes starting on January 9th. We saw the same
BitTorrent /announce traffic, and lots of other random traffic.

We ended up mitigating it by moving our IP address on that host, and blocking
all input on the old address.

~~~
themoogle
you are lucky you only got 150mbit/s

Here is what I had to deal with on the 9nth Ramming my server...

[http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png](http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png)

Yes that is almost 1000mbit/s All coming from china...

------
kesor
I wrote a blog post about a DDoS on a website by the Chinese at
[http://dvps.me/ddos-attack-by-torrent](http://dvps.me/ddos-attack-by-
torrent). Falsely I assumed that it was torrent tracker injection, while in
fact this looks like a spike in the affect of DNS poisoning in China.

[http://serverfault.com/questions/656093/mod-security-
block-r...](http://serverfault.com/questions/656093/mod-security-block-
requests-by-http-host-header/) describes the problem as well.

[https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat...](https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepiratebayorg+Resolving+to+Various+Hosts/19175/)
also describes the same thing.

As well as
[http://www.webhostingtalk.com/showthread.php?p=9351951](http://www.webhostingtalk.com/showthread.php?p=9351951)

------
acdha
This is why I really like having Varnish in front of anything Internet-facing:
for something like returning a 400 for an un-approved Host header, you're
going to be effectively bottlenecked by the network interface.

------
19890604
A Chinese here. I would suggest, if possible, use 301 to redirect the
malicious requests(which are without recognized hostnames in HTTP header) to
<www.gov.cn>, which is the government's face.

~~~
smilekzs
Good idea!

------
smilekzs
They (the GFW) _intend_ to have every server owner ban China IPs so that they
could claim "you guys are also banning us, what position are you in to blame
us for GFW?"

~~~
pavel_lishin
Big ol' [citation needed] on that one.

~~~
smilekzs
/* That's what he said */

Well, do you know what the Communist Party thinks?

------
WillHuxtable
Maybe China are trying a new approach to censorship. Instead of blocking
things themselves, perhaps they're trying to get sysadmins to block all of
China. I imagine it would save them a lot of work.

------
thaumaturgy
That's cute, the IP address for his demo curl command is ... daringfireball's
web server.

~~~
Anthony-G
I noticed that as well. However, they seem to be friends / colleagues /
associates. A web search for "Craig Hockenberry daringfireball" shows that
Craig Hockenberry is (or has been) a contributor to John Gruber's show and
John Gruber has recommended one of Craig's apps [1].

[1] [http://safetylightapp.com/](http://safetylightapp.com/)

------
emiunet
Thanks a lot for the post. What suprised me the most was the fact that Google
is serving 40K (now 47K in live statistic) requests per second. That was way
below what I thought. Each Apache server without real optimization running on
a typical physical machine would be able to serve 40-50 requests/second; so
they only need a thousand of servers to do 50K requests/second. Or am I
missing something?

~~~
Narkov
Yeah, the actual search bit.

Serving a static file is pretty simple at 40-50 req/sec but searching an
entire index of the Internet @ 40-50 req/sec is harder.

------
pdoconnell
We started seeing this around the 9th I believe. We were not the only people
as well, based on a somewhat late ISC post[1].

We thought it was a new form of intelligent blackholing. Instead of sending
traffic to IPs that could easily be blacklisted by tools to get around the
firewall, the Great Firewall would start sending them to random "good" IPs for
the same result. Others seem to think the same thing[2].

[1]
[https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat...](https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepiratebayorg+Resolving+to+Various+Hosts/19175/)
[2] [https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-
visi...](https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visitors-
blocked-sites-redirected-porn)

------
sneakest
I had one of my clients have the same thing happen to them 3 nights in a row
about 10 days ago. All of the sudden they got dumped a ton of traffic out of
no where. Torrent tracker updates and what seemed like legitimate traffic
routed to the wrong ip. Then after about an hour most of it stopped. During
the attack and after we changed some firewall settings and clamped down our
request per second and connection per ip limits on the web servers. One big
change was having the web servers respond 444 to any host request that was not
configured (i.e. default). So if someone came to the ip looking for say
google.com they would get a 444 response as it is obviously misrouted traffic.

Then the next day the same thing happened but much more traffic. The 444
change helped some but there was just too much traffic for the web servers to
handle quickly so they bogged down to a crawl. Luckily we were able to figure
out through severfault and some other searches that it appears to be DNS
poisoning coming from China. We ended up banning the entire country.

The third day the same thing happened but because we had blocked the traffic
from China at our outside firewall the servers were unaffected. Since then we
have seen some flashes of traffic being blocked but not nearly as much as
before.

In all cases the increased volume of traffic only lasted about an hour. The
only thing I can surmise about the length of time is DNS records only being
cached for about an hour. So after that time the poisoned cached DNS records
were replaced by the real resolving ip address. In the case of the article's
author their servers suffered a much longer attack than we did. I am not sure
if it is the Chinese government doing it or an attack for hire scheme using
holes in many Chinese based DNS servers.

TLDR; Remove default server settings from your web servers and have the
default server block respond 444 or 404. This may help stave off the attack
until your incoming traffic takes up all available resources of the web
servers. Of course you could always block all traffic based in China like we
did.

Here is info on the status codes incase you are curious:
[http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_C...](http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error)

------
userbinator
Seeing /announce reminded me of this:

[https://torrentfreak.com/bittorrent-users-ddos-websites-
with...](https://torrentfreak.com/bittorrent-users-ddos-websites-without-
knowing-120919/)

Using DHT is also a particularly effective way of getting massive amounts of
UDP traffic directed at you, which does not stop even after stopping the
torrents. It subsides eventually, but can be quite irritating.

------
kesor
One possibility is to block requests coming with a different 'Host:' header
than your website. Or just block the URLs like described in dvps.me/ddos-
attack-by-torrent

------
antihero
> thousands of Apache child processes

Were you still using preforking MPM or something? Surely switching to threaded
or even better, event MPM would have mitigated this to a degree...

~~~
cesarb
Or put a nginx in front of the Apache, disable keepalives in the Apache server
(letting nginx deal with keepalive from the clients), and reduce MaxClients so
even if they're all active the server does not go into swap.

(I've done exactly that for a friend, it made a huge difference in how much
traffic his box can handle).

------
beachstartup
why is 'out' inbound, and 'in' outbound?

~~~
faster
Because it's from the perspective of the router; traffic out of the router is
TO his network.

~~~
beachstartup
if he owns/operates the router, it still doesn't make any sense.

in our network, 'out' means outbound from the edge, and 'in' means in from the
edge, in all cases.

~~~
faster
The bandwidth chart I get from my colo (where I don't own the router that
measures my bandwidth usage) works the way I described. The article mentions a
single server, and nothing about owning the router also. If they did, that's
probably where they'd null-route the traffic they don't want.

It seemed backwards to me too, when I first saw it.

------
kesor
[http://en.wikipedia.org/wiki/Golden_Shield_Project](http://en.wikipedia.org/wiki/Golden_Shield_Project)

~~~
avinassh
and also:
[http://www.youtube.com/watch?v=rH8UitdFb2o](http://www.youtube.com/watch?v=rH8UitdFb2o)

------
thspimpolds
This is why i cloud front personal hosts, and fastly work hosts. I never open
my server to the worlds with cdn' being so cheap or free now

------
nl
Surely at this level of traffic it would make sense to get in contact with
your provider and see if they can block it upstream?

------
bastih
New @ GFW: Now providing infrastructure stress testing for the western world.

------
coldcode
I wonder if this could really be harnessed in China as a weapon.

~~~
brazzledazzle
That's the only reason I can imagine they'd respond with anything else besides
127.0.0.1 or some internal web server with warnings to obey their overlords.

------
ChrisArchitect
for those wondering, extended title could be along the lines of "Experiencing
the effects of a DNS poisoning attack"

------
bombless
I live in China. We have a department named 工信部, it blocks a site by polluting
the DNS and point the domain name to random foreign IP. I think this explains
what happened here.

~~~
zaroth
A random foreign IP... Wow, yeah, that just makes perfect sense.

Clearly all requests from China with an incorrect host header should just be
served a Goatse-style 'hello.jpg'.

