
FastMail under DDoS Attack - aroman
http://blog.fastmail.com/2015/11/11/ddos-attack-may-lead-to-potential-service-disruption-this-week/
======
mmaunder
We recently were hit by multiple DDoS attacks over a weekend. We have our own
servers in a data center with 5 redundant 1 Gbps links. The DDoS was 20Gbps
according to the upstream providers.

Our upstream implemented layer 7 mitigation which did an unbelievably
effective job at stopping the attack in it's tracks. I don't know the tech
that they used, but it performs deep packet inspection up to the application
layer and they charge a modest additional fee for passing our traffic through
that system.

The effect was that our traffic dropped to very slightly below normal levels
during the attack, which would indicate that there were probably a few false
positives, but we didn't have a single customer complaint.

~~~
Illniyar
With such a positive feedback you should really tell us the name of the
upstream

~~~
junto
Ditto. Would you care to share?

------
kevinSuttle
And Protonmail. [https://protonmail.com/blog/protonmail-ddos-
attacks/](https://protonmail.com/blog/protonmail-ddos-attacks/)

Fastmail, pro tip: don't pay any ransom.

~~~
robn_fastmail
Not a chance :)

~~~
PuffinBlue
Thanks for taking this stance. I'm a customer (I think we actually talked
together recently about Apple/Google vCard implementation) and I appreciate
you guys making a stand against this criminal behaviour.

It could seem to some people that such a stance is easy, but no matter the
strength of principle, when you see your business go offline and customers
start banging at the doors for you to sort it out then the situation becomes
more complex.

So from this customer - keep up this stance, keep being transparent and I for
one will stick by you guys without hesitation!

~~~
Nyr
Indeed, I don't want them to fund criminals with my money like those Swiss
guys did.

DDoS mitigation isn't that expensive antway considering the size of FastMail.

------
andrem
DDoS attacks today are such a commodity. It takes next to nothing to launch
them.

You can get upwards of 200Gbps for 1/6 of a bitcoin. It is very easy to setup
and you can DDoS your favourite site in a matter of minutes.

These are not very smart attacks and can be mitigated even using the free tier
of cloudflare.

I don't have the background on the mail provider attacks but 6.5k ransom seems
to come from attackers who use easily available booters. Hushmail switched to
Cloudflare throughout their attacks and that seemed to have helped, not sure
what fastmail will do.

But any public web service should not be in a position where they are
vulnerable to off the shelf DDoS attacks.

DDoS attack as suffered by for example Githuh with heavy coordination and
nation states behind them require more specialised defenses. There are
commercial alternatives out there that go from anywhere between 9k-40k per
month depending on bandwidth and technology - see Imperva, Prolexic, Neustar,
Nexusguard, Blacklotus, Incapsula, etc..

Apart from the initial setup which is more involved than Cloudflare's there is
not much to do apart from throwing money at it. Quite the money making
business really :)

~~~
brongondwana
If we were pure web we would have ducked behind Cloudflare immediately. Since
we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly)
solution.

This is our theory for why they're currently attacking email providers. We're
not "just a web site", and attackers realise that the situation is more
complex for email sites, we can't just hide behind Cloudflare.

We're not sure who's actually attacking us, the ransom note comes from a
freemail provider and a connection from a tor exit node. We can only guess at
their total capabilities.

~~~
cat-dev-null
0\. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent
proxies in front of everything (eg at the edge on the other side of HA network
gear with either 2 (or 3, if deploying a private, admin network too) NIC teams
for isolating traffic). This will let you do raw L3 traffic measurements on
each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network
banning (if you don't own/admin the network gear or don't want to touch it in
production). These are super cheap and only need ~128 MiB RAM each and very
little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD
partition if possible). (Your public IP(s) should point to these boxen.)

1\. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or
haproxy.

nginx: Compile it from source if that's all you need, and reduce your attack
surface.
[http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html](http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html)

haproxy: [http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-
infr...](http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-
infrastructure-with-postfix-and-load-balancers/)

2\. Setup something like fail2ban:
[https://rtcamp.com/tutorials/nginx/fail2ban/](https://rtcamp.com/tutorials/nginx/fail2ban/)

3\. There are many other tweaks and there are some appliancized VMs for anti-
spam and DDoS that can be dropped behind the trusted network-side. (I would
advise against Cloudflare-like services for most mature and non-web apps
because they are add'l points of failure and increase latency, and they
duplicate what good sys/netadmins implement routinely, especially if you're
already deployed to multiple DCs servicing multiple continents and/or geodns.)

Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin
whom sold out and became SRE manager and then a consultant. I used to maintain
multiple deployments of commercial Zimbra (from m&a activities) for clients
including hi-ed, non-profits, VIP individuals, and enterprises.

~~~
pyvpx
this is the first I've heard of being able to block traffic in excess of
10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like
services adding latency.

~~~
brongondwana
Yeah, it's small-time advice. Good advice for protection against complexity
attacks, not so much for protection against tens of Gbps of random junk that
fills your entire pipe.

(we do run nginx on out frontend machines for both web and mail protocols,
protecting the Cyrus servers behind it from complexity attacks and providing
fan-out connection routing)

We dropped all the DDoS packets at our edge firewall quite comfortably - users
wouldn't have even noticed except that it filled up our incoming links, so
packets started dropping.

I'm really quite impressed at the tech which the big DDoS protection providers
have for packet inspection and cleaning the feed before it reaches the end
host.

It does lower the overall egalitarianism of the internet to have to deploy
defenses - we lower our overall routability to put these mega filters in front
of incoming packets - but that's the reality of a world where fiends can
control tens of thousands of boxes and have them spew traffic at any random
network address. You need to filter out at the boundary.

Nothing short of filtering beforehand can stop a channel from being filled if
it gets more than its capacity per second of incoming packets.

------
herbig
I've just started checking out Fastmail as a result of this thread. I hadn't
heard of it before but I'm really into it.

I've been trying out different email services lately in an effort to untether
myself from Google, and this one seems like a really good choice.

I'm not into the iOS themed non native Android app though, but being an
Android developer I'm more averse to that nonsense. It seems to work great and
is really fast though. Does anyone know if there is a native app planned?

I'd be willing to pay a subscription for email if it's worth it. I'd love to
hear others' experience with them.

~~~
bratsche
I have a lifetime email alias from my university, and when I used it with
Gmail it would always appear to people as "From me@gmail.com on behalf of
me@wherever.edu". I wanted my .edu to be my _actual_ email address, but my
gmail address is what was ending up in people's address books.

So I tried out Fastmail with my alias and had no trouble getting it working
the way I wanted it to work. So I've been with Fastmail ever since.

~~~
danieldk
_when I used it with Gmail it would always appear to people as "From
me@gmail.com on behalf of me@wherever.edu"._

This doesn't happen if you configure an external SMTP server in Google Mail
for the alias.

[http://gmailblog.blogspot.de/2009/07/send-mail-from-
another-...](http://gmailblog.blogspot.de/2009/07/send-mail-from-another-
address-without.html)

[https://support.google.com/mail/answer/22370?hl=en](https://support.google.com/mail/answer/22370?hl=en)

Moreover, you don't want to use a different 'from' address without sending
through the appropriate SMTP server. If the domain has SPF or DKIM set up, a
receiving server might reject your mail if it wasn't sent from an expected
SMTP server.

------
benmac
Given that so many of us are now hosting on AWS, I'd like to ask the question
- who has been hit with a DDOS attack / extortion letter while is hosting on
AWS? It would seem that there's many old-tech companies hosting in data
centers that would seem to be far more vulnerable to non-TCP attack vectors
than AWS-hosted systems. Is that who is generally targeted here? Are there any
stories, anecdotal or otherwise, about people getting hit with DDOS attacks
while using AWS. Here's a talk by AWS on their measures against attacks -
[https://www.youtube.com/watch?v=Ys0gG1koqJA](https://www.youtube.com/watch?v=Ys0gG1koqJA).
The only thing short of Silverline etc defense that they seem to be lacking is
the reporting dashboard indicating when they've defended against DDOS attacks.
So has anyone received a letter from DD4BC and other miscreants whilst hosting
their domains on AWS?

------
athenot
It's nice to see this published proactively. At the very least, transparency
like this helps users understand what the circumstances are ahead of time.

Best of luck to the Fastmail team, I hope they are able to weather the storm
out.

------
duncan_bayne
And they've very clearly refused to pay the Danegeld demanded by their
attackers. Smart, principled and yet another reason I'm a happy customer of
theirs.

------
touchofevil
Runbox was also recently under ddos attack
[https://blog.runbox.com/2015/11/ddos-attacks-on-
runbox/](https://blog.runbox.com/2015/11/ddos-attacks-on-runbox/)

------
jayess
I ditched gmail for fastmail two years ago and don't regret it for a second. I
haven't noticed any disruption at all. Keep up the good work, guys.

------
alastair
Zoho Mail also attacked recently: [https://www.zoho.com/service-
updates/blog/zoho-services-unde...](https://www.zoho.com/service-
updates/blog/zoho-services-under-criminal-attack.html)

------
bsder
It looks like it's a general attack on email providers because they can't duck
behind Cloudflare (or similar).

------
1rae
Hushmail recently had it as well.
[https://help.hushmail.com/entries/107539976](https://help.hushmail.com/entries/107539976)

------
Swinx43
I have not noticed any service interruption at all. Well done in handling this
attack! This is just one more reason why I will remain a loyal FastMail
customer.

------
rdl
Wow, seriously, fuck these people. There has to be a technical solution to
this, since it's infeasible to find/fix/finish the Armada Collective.

SMTP/IMAP/etc. are pretty crappy protocols in a lot of ways, but they're what
everyone has deployed. They can be proxied like HTTP/HTTPS. There are
spam/reputation issues with outgoing traffic, too, which makes this even more
annoying.

~~~
ppoint
There is a SilverLine by F5 Networks:
[https://f5.com/products/platforms/silverline/f5-silverline-d...](https://f5.com/products/platforms/silverline/f5-silverline-
ddos-protection).

------
blowski
We're a FastMail reseller, and so far we haven't experienced any problems -
either on delivery, or the front end interface. So good job defending against
it.

------
alessioalex
What I love about FastMail is that I can neatly organize my emails into
folders based by categories (via rules for subdomains), such as: newsletters
folder (service_name@newsletters.mydomain.com), social
(service_name@social.mydomain.com) etc. Everything is so clean and I don't
even see any spam, they're top notch.

------
pluma
As a happy FastMail customer: thank you for standing up to them and announcing
this in advance. More power to you!

------
awqrre
DDoS is a type of attack that is so old that I wonder why it is still possible
to exploit it. It is at least 15 years old? (edit: I am not blaming FastMail,
this is just a general assumption)

~~~
Analemma_
It's still possible to exploit because "DDoS" isn't one attack, but a
category: people keep finding new ways to slam a target with tons of traffic.

For a while, the most common way was with botnets of compromised PCs. They
still exist, but big attacks with them are less common since Microsoft has
gotten better at securing people's computers. The big thing now is
"amplification attacks": basically, finding a way to send a small amount of
data and get some other host to flood your target with a huge amount of data
in response. Search "NTP amplification attack" for details. More recently,
China has weaponized the Great Firewall to be yet another DDoS vector: they
inject JavaScript into pages that people visit, and that JS floods a target
with requests.

As long as there is some way to point a lot of requests somewhere you want,
DDoS attacks will be a thing.

~~~
awqrre
not necessarily... you could use AI to detect attacks or some other methods
that I am not aware of.

~~~
toast0
Most attacks are trivial to detect, you don't need AI. It's just hard to get
useful work done when all your incoming interfaces are overloaded with easily
detectable abuse.

~~~
toomuchtodo
The "best" way would be to have application logic to detect non-legitimate
requests, and make an API call out of band to upstream networking gear to
insert a null route for that IP (so as to drop the traffic at the edge before
any real "work" takes place on it).

In a previous life, I ran physical datacenters, and while the gear wasn't
terribly powerful then (we're still worried about running out of memory on
core routers, hence why IP blocks don't get sliced up and piecemealed out with
the exhaustion of IPv4 space), I'd expect newer hardware to be able to keep
up.

The network can remain irrational longer than you can stay online.

~~~
Karunamon
Problem is, beyond a certain volume, even the upstream gear is gonna get
saturated just reading the header on the bogus packets and directing them into
the bit bucket. It's not unheard of for the larger attacks to take down entire
ISPs.

~~~
zo1
Unless you keep propagating "upstream" and the message gets to the "source"
ISP, and they block the actual misbehaving user/account. For all we know, they
can kick them off the network after sufficient transgression, and ban their
account at the hardware ADSL level (assuming that's what it is). This also
presumes the ISP is willing to implement such a feature, and kick-off their
paying (albeit infected) users.

I don't know much about this stuff, so I'm extrapolating and pseudo-solving.

~~~
toast0
Usually there's a lot of diversity in the immediate source of the traffic. If
it's a volumetric attack, the immediate source is the misconfigured servers
that spoofed packets are being sent to. If it's an in band attack, the
immediate source is usually botnet members, but occasionally regular browsers
being served bad scripts by a compromised or mitmed site.

You could work to notify the network owners, but it's whack-a-mole; even with
strong efforts there are enough DNS and ntp servers out there configured to
generate a pretty big reflection.

------
copperx
Why do we always hear of DDoS attacks to mail services, but never to Gmail?
are they more DDoS resistant? are the attacks not public?

~~~
detaro
One would assume DDoSing the biggest service around, run by a company that has
experience running giant internet facing services, a lot of security
experience and tons of cash would be a lot harder than a provider with in
comparison small connectivity and a small number of locations, yes. Pure size
helps, + if you are at the scale of Google your upstreams probably have more
interest in helping you if it were necessary.

It would be interesting to see how often people try though.

~~~
PuffinBlue
It may even been simpler than that and they might not even need 'upstream' any
more. They may not even have upstream for many things, given their scale now,
they probably just peer to Tier 1, if they even need that any more! (they were
actually just as big as Tier 1 folks back in 2010![0].

Google has been buying up 'dark fiber' for years and has thousands of miles of
cable connecting their data centers.

They can certainly handle petabit/s levels of traffic inside the
datacenter[1], it's not that much of a stretch to think that they can handle
double digit terabit/s through their collective external fiber links.

Also, just think about their normal level of operation. Even just all the
Android devices feeding data back and forth, let alone analytics, maps, gmail,
search etc etc. They've got 36 data centers and co-locate in more than 60
public exchanges (and that was in 2010!), not to mention the Google Global
Cache (GGC) servers inside consumer networks across the globe.

Their scale is ridiculously large. I suspect that they actually can't be
DDoS'd in the normal 'chuck traffic at them' sense.

[0]
[http://www.theregister.co.uk/2010/03/17/the_size_of_the_goog...](http://www.theregister.co.uk/2010/03/17/the_size_of_the_googlenet/)

[1] [http://googlecloudplatform.blogspot.co.uk/2015/06/A-Look-
Ins...](http://googlecloudplatform.blogspot.co.uk/2015/06/A-Look-Inside-
Googles-Data-Center-Networks.html)

------
DyslexicAtheist
last week protonmail, then zoho (their services were offline for 6 days) ... a
new one every week it seems

