
Ask HN: A botnet may be attacking our site right now, how should we respond? - panabee
It seems like our ecommerce site is under attack from a botnet, causing our site to go down. We&#x27;re receiving 50-100 requests per second (far surpasses normal traffic). Some requests are for outdated URLs not even normally accessible from the site.<p>Here&#x27;s IP data related to requests: https:&#x2F;&#x2F;gist.github.com&#x2F;panabee&#x2F;9595411. Can post more if more is needed for diagnosis.<p>Questions:<p>1) How do we confirm if the site is under attack?<p>2) If the site is under attack, how can we ward off the attack and prevent future ones?<p>3) We seem to have a ton of sleeping MySQL connections that continue to crop up even after we try killing them. Is this symptomatic of a botnet attack?<p>We appreciate any help or guidance anyone can offer.<p>Thanks!
======
patio11
First step: get in touch with your hosting provider. They're better at this
than you are, and may have suggestions or knobs they can twiddle on their end.

You might just be getting crawled by an overly aggressive search engine, by
the way. (Probably not one of the big guys.) From your perspective that is
isomorphic to being hit with a botnet, but the nice thing about that is the
search engine won't attempt to get around the mitigation steps you put in
place. (Well, typically.)

~~~
panabee
Thanks. We spent half the day with them and are awaiting a confirmation from
their network guys that this is, indeed, a botnet attack. Unfortunately, the
network guys aren't in until the morning -- only the general support people
are -- prompting the post on HN. So much for the 24/7 support they claim ...

We also suspect it might be an aggressive search engine. Our next step is to
pull some IP data. If it turns out to be a search engine, what can we do
beyond editing robots.txt?

If it's not a search engine, how can we confirm a botnet attack ourselves. We
don't want to wait for these network guys to wake up and drink their morning
coffee.

Thanks again!

~~~
patio11
robots.txt is purely advisory and many of the people who code a crawler which
could get in a state where it hits you hundreds of times a minute might not be
great about actually respecting it. If it is a crawler, though, it is probably
operating from a single C block, likely on AWS or what have you. You'd
probably just drop them at the firewall. Simple, no fuss.

I'm not a sysadmin, and I have nothing but respect for you and your team with
the following recommendation: if it isn't obviously how to implement "drop
that C block at the firewall" then find yourself a competent contract
sysadmin, pay him for one hour of work today, then pay him for 10 hours of
work over the next few weeks beefing up your infrastructure. It will be cheap
at the price.

If you're absolutely determined to do it yourself the magic word is "iptables"
and may God have mercy on your soul, because iptables is the most obtuse piece
of software I've ever had the misfortune of trying to configure.

In the alternative, if you're actually facing a botnet, the expected behavior
is lots of requests from widely distributed IPs. You'd be able to verify this
with your access logs (which record IPs and timestamps) and a 10 or 20 line
shell script to put a histogram of unique requests per IP. If it's a few
requests from thousands of IPs per hour, then you have a bit of a challenge to
address it on your end of things. My first totally-not-a-sysadmin advice would
be to dial down dynamic content within the app for non-logged-in users and
start aggressively caching pages, which might or might not be enough for you
to weather the storm.

Best of luck and skill.

~~~
panabee
ok thanks for the suggestions! one other data point: we seem to have a ton of
sleeping mysql connections that continue to crop up even after we try killing
them. is this symptomatic of a botnet attack?

~~~
thaumaturgy
If you're really actually in a bind, drop me a line (email's in profile). I'll
give you a hand.

If you're looking for advice on how to mitigate this yourself:

\- apachetop is a useful tool for this sort of thing, and since the activity
is occurring on a single site, it should be easy to use. (Assuming of course
you use apache.)

\- If it's an actual botnet, you'll need to pick some honeypot URLs -- links
that regular customers aren't hitting but the botnet is -- and then use
something like Fail2Ban to pipe accesses to those URLs into iptables bans.

\- If it's a crawler, it should be from a limited set of IPs and a quick
iptables rule addition will fix you up. For reference, here are two easy-to-
use rules:

    
    
        iptables -A INPUT -s <source_ip_address> -j DROP
    
        iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP
    

The first rule will add a "drop" command to your INPUT table for any traffic
coming from <source_ip_address>. The second rule will allow you to quickly
drop a range of IP addresses; <source_ip_range> should look something like
"10.0.0.1-10.0.1.255", and "iprange" is a literal.

~~~
obituary_latte
Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for
sharing.

~~~
euroclydon
Just be careful not to ban Googlebot. It's pretty good at following new links.

~~~
obituary_latte
Good point. It seems the bot is also good at minding robots.txt so that might
be an option. Could also have the honeypot script to do a quick whois on the
ip before passing along to fail2ban.

------
Udo
_> 1) How do we confirm if the site is under attack?_

It _could_ be some kind of attack, but it's so weak it barely qualifies.
50-100 requests is so low it's probably a search engine spider or a log-
spamming bot. If request rates like these bring your site down, you have a
serious software problem. You really should be able to handle at least a few
hundred requests per second, especially if they're bogus requests going to
non-existent URLs.

 _> 2) If the site is under attack, how can we ward off the attack and prevent
future ones?_

If this was a real attack, you'd be dealing with thousands of requests per
second and a real spike in data volume.

As a basic protection against things like these, get behind a reverse proxy
_right now_. Cloudflare offers a good free plan, but it doesn't include SSL.
I've had a good experience with X4B as a DDoS-protection reverse proxy
recently. Their plans start at a few bucks per month, check them out.

 _> 3) We seem to have a ton of sleeping MySQL connections that continue to
crop up even after we try killing them. Is this symptomatic of a botnet
attack?_

This question can't be meaningfully answered without knowing more about your
setup. Generally, sleeping connections are fine - most web servers create a
persistent connection pool. What's "a ton" in this case?

I think the only really valuable advice to give here is: hire someone to
review your setup, your software, to make some changes, and above all to give
you a meaningful briefing. Contacting your provider isn't going to help.

~~~
panabee
Agreed, the site is terribly antiquated and poorly coded. I helped to clean
some of the code, but there is regrettably still a lot of ugly, ugly, heinous,
ugly, ugly code. There are definite issues that require addressing.

If this helps with diagnosis, here's IP data related to requests:
[https://gist.github.com/panabee/9595411](https://gist.github.com/panabee/9595411).
Can post more if more is needed.

Thanks so much for your help!

~~~
Udo
It looks more like an attempt at spamming, but without the full log entries
(containing the referrer and the destination URL) it's impossible to be sure.
In the end it doesn't really matter though.

Anyway, you have identified the main issue: some more code cleanup is in
order. If you want to do this yourself: keep in mind that elegance in code is
not the primary goal, runtime behavior is. Make sure you cache your results as
much as possible, eliminate unnecessary DB calls.

On the sysadmin side, assuming your app is PHP the following is mandatory:
make sure you're using PHP-FPM, and enable APC or Opcache. Optionally,
consider installing Memcached and start using it from the app to cache data
that is expensive to generate. If you're running on Apache, switch to Nginx.

Finally: reverse proxy service. They'll filter out most malicious requests for
you. When the day comes where you experience a real attack, you'll have no
chance of keeping the server up without using such a service.

------
thaumaturgy
I took a quick look at the IP data you just released. Extracted IPs then
unique-sorted them and took a few random samples. Everything was Eastern
Europe, Russia, or Nigeria. There are some broadband IPs and some pppoe (!)
IPs in there too, which might explain why it's not a very heavy attack.

There are 500+ IPs listed. I've permabanned a few million IPs on my servers,
and I bet there's some overlap. Gimme a sec to see if I can put together a
quick script for you.

------
cheald
I'm surprised I haven't heard it mentioned yet, but Varnish can ward off an
awful lot of request floods. If these pages aren't changing frequently
(particularly if they are outdated or shouldn't be accessible), then Varnish
should be able to cache and respond with them trivially. This will take a
tremendous amount of load off of your actual app servers.

Additionally, Varnish has ACLs which make it possible to reject requests based
on whatever patterns you choose (source IP, request structure, etc). While it
puts you into an arms race, if you can identify a pattern in the bad traffic,
you can short-circuit it before it ever becomes an actual problem. Black-
holing poorly-behaved clients is very easy, and can let you absorb very high
request concurrency with extremely little effort.

Varnish also makes it very easy to locate URLs and IPs being used in these
sorts of things, via varnishtop and varnishlog. For example, `varnishlog -i
RxURL` gets you a constantly-updated list of the URLs being hit most
frequently, or `varnishlog -i SessionOpen` will get you a list of the most
common client IPs. With that kind of information, doing reactive filtering
becomes very easy.

------
mdb31
50-100 requests per second doesn't sound like an attack (or at least not a
very scary one, depending on request size...), more like a crawler or
prefetcher gone haywire.

Anyway, best way to figure out what's going on, is to generate a distinct list
of IPs originating the bad traffic from your logs, and block those addresses
at your firewall. If this doesn't stop the strange requests in 15 minutes or
so, you may want to get some expert help.

Your hosting provider may be able to be of some assistance here, but that
greatly depends on their level of clue: many budget providers will just drop
you at the first sign of denial-of-service related trouble, so do keep that in
mind.

Signing up for a specialized service like Cloudflare is probably your best
bet, and should be pretty affordable as well (their web site lists their
business plan with "advanced denial of service attack mitigation" as
$200/month, and they suggest that if you call them they can have you protected
pretty much immediately).

~~~
panabee
thanks for the suggestion! we're using singlehop ... definitely not a budget
hosting provider. these requests have been ongoing since yesterday morning
(~36 hours). we looked at cloudflare earlier today. do you have experience
with them? does the solution really work?

~~~
mbijon
Cloudflare's plans that include DDoS protection are as good or better than
typical hosting resellers. May not know enough about your site to make the
decisions a good datacenter team could though. On the up-side Cloudflare
really is there 24/7.

Unfortunately the advanced DDoS support is only in their upper $$ plans. Might
be worth it to keep your site revenue going though.

------
caleb23
To answer your first question, like people said already, pull a log from your
servers with the IP's. If it is a lot of different IP addresses then that is a
good sign of a DDOS attack of some kind. If it is a lot of requests from the
same IP address or only a few IP addresses it probably is a search engine or
something like which is causing the problem. You can also look up the IP
addresses by simply Googling them to see who owns the IP.

To fight off the attack immediately I would recommend switching your DNS over
to CloudFlare's Pro or Business plan (that depends a lot though on your site's
current configuration, the size of your budget, etc.) with certain settings,
then configuring Sucuri CloudProxy with CloudFlare, and lastly implementing
some additional security for your server on Single Hop.

In terms of preventing future ones, you basically want to be proactive as
possible in terms of preventing them, but that won't stop everything. The
configuration I recommended above should be pretty good for most sites, but
you might want to consider other DDOS mitigation companies as well and you
could need something completely different depending on your business etc. Then
you want to have a plan and several other things in place in case it was to
get past the infrastructure you implemented to prevent it, so you can minimize
the downtime you have etc.

It is hard to tell whether that is a possible issue just based on the
information you gave in terms of the sleeping MySQL connections.

Lastly, depending on what the issue exactly is, it probably would be a good
idea to make your site as static as possible, but that is difficult to do
since your site is down right now. Once you get the site back up though and
stable, you might consider doing this for a little bit. Lastly, having a
separate site that you can deploy in the event of an attack is a good idea as
well.

If you want more help, shoot me an email at caleb.lane4@gmail.com. I hope that
helped and you get this solved quickly.

------
jbaudanza
What does your stack look like? If you're running a ruby app, check out rack-
attack: [https://github.com/kickstarter/rack-
attack](https://github.com/kickstarter/rack-attack)

It was created by kickstarter to fend off these kind of attacks.

------
User8712
You could always try to improve the performance of your server and database
configuration to handle 50-100 requests per second, and render the attack
useless.

A. Check the number of connections per IP. If any have an unusually high
number, block them.

B. Why is your site down? Database can't keep up? If so, why not, is it
something simple like indexes not setup correctly?

C. Did you actually check the load on your server? Double check it's actually
being overloaded, and you don't have a configuration issue with server
connections set too low, while your hardware is practically idling.

D. How many pages are being accessed by the _attack_? Would you be able to
cache them?

E. Check server logs, and see if you can identify anything that separates
these requests from your typical traffic.

------
issa
My only advice is to hire a consultant who can give you some advice. If your
site is getting taken out by 50-100 requests per second (that is a very low
number), you have some basic structural flaws.

A few hours of time from a developer experienced with scaling will save you a
lot of grief. Specifically, it sounds like you need a caching strategy
(probably for all layers of your app, including the database).

Super-general overview: Use memcached in front of your DB. Use a reverse proxy
like Varnish and/or a CDN in front of your entire site. I don't want to make
the obvious guess that this is a PHP site, but if it is, make sure you are
using apc or xcache as well.

------
taylorhou
cloudflare.com? - I believe you could get everything swapped over to their DNS
and start benefiting from their ability to ward off attackers within a couple
of hours.

~~~
panabee
we looked at them earlier today. do you have experience with them?

~~~
spicyj
They're reputable. HN itself uses Cloudflare.

~~~
dknecht
I am from CloudFlare. If you need help getting setup feel free to email me
(dane AT cloudflare.com).

------
tluyben2
While running one of the largest free hosting providers we had many of these
smaller attacks; usually some netstat & iptables incantations will ward them
off. Something like

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

will show how many connections per IP for instance; with smaller attacks,
usually, to actually make a dent in your webserver, the attackers will open
many connections per IP. That helps you identify the 'bad' IPs. As someone
else said; a lot of them are from eastern EU/Russia; you can use that
information to be a bit more sure about what makes a bad IP a bad IP before
you block it. Also check the Agent strings; often they all have the same Agent
which is another way to filter.

We used to have tons of scripts running to try to identify the 'enemies'; in
the end the simplest thing was the best; block everything with a lot of
connections and open them up 24 hours later.

------
gojomo
As others have noted, it may not even be a malicious, targeted attack - just
the side effect of other spamming or careless crawling/scraping. Reviewing
your full logs, and comparing current traffic/behavior with 'normal' periods,
may help further characterize what's happening.

There's a presentation called "Running At 99 Percent: Surviving an Application
DoS" by Ryan Huber that may be helpful. (Slides and video from a number of
conference presentations can be found via a search on that title.) It
highlights some of the ways even a trickle of the wrong kind of app-specific
traffic can be enough to bottleneck weak points in your app... and talks about
mitigation ideas, both as a matter of good design and emergency request-
filtering via server-side tools.

------
makmanalp
I wonder if this could be related to someone trying to mass-scrape you to get
to your data cheap ... So instead of having to do tons of domain queries
themselves, instead they use you to make the queries and just scrape the
results into a large database. At least a 100 to 1 savings.

------
SchizoDuckie
Have you checked your webserver logs for funny looking http requests?

Have you considered that you're being SQLMapped by someone using the Tor
Network?

( [http://sqlmap.org](http://sqlmap.org) )

------
enneff
100 requests per second does not sound like a botnet attack. If it were you
probably wouldn't be able to access your servers on their public interfaces
for all the traffic.

------
BorisMelnik
something that low it could just be a spam program testing to see if you have
certain pages available. for instance some programs look for "signup.php" or
"register.html" and if = exists will proceed with a registration bot.

contact your host. sounds like normal normal shit.

------
good_guy
Can you tell us the site URL so we can take a look ?

~~~
panabee
sure, but the site is down. :) do you have an email? don't want people to
think we're doing this for free publicity...

~~~
good_guy
Oh.Yeh, you really shouldn't post it in here becasue it might cause it to get
DDOSed by HN traffic.

------
skimmas
shutdown servers. go outside. frustrate attackers by don't caring.

