
Mitigating a DDoS on Mastodon - dredmorbius
https://coffee-and-dreams.uk/security/2019/10/20/mitigating-a-ddos.html
======
pjc50
Decentralisation fans take note: despite wanting to remain independent, the
only effective solution was in this case to re-insert a giant global
intermediary (Cloudflare) _and_ block all the anonymous unaccountable Tor
users.

If a decentralised system is to stay decentralised, it needs to consider
spammy bad actors.

~~~
generalpass
How come private contract clauses can't be initiated to protect from malicious
actors?

What if I own a server and connect it to an ISP under an agreement where the
ISP is accountable for clearly malicious behavior coming from its connection
(regardless of origin)?

Then, that ISP requires the same agreement from me, and everyone connecting to
that ISP, and on down the chain.

Wouldn't we all be very active in policing bad actors in the networks we
manage?

~~~
Analemma_
1) This doesn't deal with botnets and other compromised devices. Would you
want your ISP to terminate your service if you (or worse, your roommate) got a
virus?

2) This would require ISPs to do even more invasive monitoring of all traffic
to be in compliance. They'd essentially have to DPI everything, or even break
TLS between you and your destination, to know if your traffic was malicious.
No thank you.

3) Many ISPs simply don't care. A lot of malicious traffic comes from
countries where ISPs will just look the other way for a bit of cash. I suppose
we could come up with a system that depeers bad ISPs, but this would have tons
of collateral damage to innocents as well as reintroducing the exact
centralization we're trying to avoid (where's the "master list" of bad ISPs to
depeer?)

Whatever the solution to bad actors online is, it isn't ISPs.

~~~
generalpass
> _1) This doesn 't deal with botnets and other compromised devices. Would you
> want your ISP to terminate your service if you (or worse, your roommate) got
> a virus?

2) This would require ISPs to do even more invasive monitoring of all traffic
to be in compliance. They'd essentially have to DPI everything, or even break
TLS between you and your destination, to know if your traffic was malicious.
No thank you.

3) Many ISPs simply don't care. A lot of malicious traffic comes from
countries where ISPs will just look the other way for a bit of cash. I suppose
we could come up with a system that depeers bad ISPs, but this would have tons
of collateral damage to innocents as well as reintroducing the exact
centralization we're trying to avoid (where's the "master list" of bad ISPs to
depeer?)

Whatever the solution to bad actors online is, it isn't ISPs._

Yes, I would like it if I had something that unbeknownst to me is harming
others (beyond some de minimis) through their service, and per their contract,
they certainly have the right refuse my service until the condition is
rectified. Anyone relying on my service will either suffer or be owed
something, by me. Note that this isn't some arbitrary shuttering of some
service. This is a harmful activity being blocked from harming and is spelled
out in the contract clauses.

You make it sound as if this stuff is so hard, yet here we are discussing this
in a comment section of a post by a person who doesn't seem to be employing
highly-sophisticated tools in identifying the bad behaviors. All he would have
to do in my dream world is show this behavior to his (contracted) service
providers, and them on up the chain.

But notice that this option is not available, thus the only option is to use a
centralized provider that is effectively big enough to completely absorb a
huge percentage of bad activity. He even comments that owners of networks are
only voluntarily providing responses and actions to these activities. They
could just as well not be bothered and what then?

If the ISPs don't care and people who don't want this traffic on their
networks disconnect from them, this is bad? And, yes, whole countries may have
problems connecting anywhere. Mind you, even those countries had some reason
to connect to the World Wide Web (itself with a mountain of even just protocol
requirements) in the first place, and it likely has to do with some minimal
amount of trade with the outside world. To continue this trade communications
they will have to provide a service that others are willing to connect to.

~~~
hombre_fatal
Aside: I think your quoting strategy is most confusing of all since you bother
to use ">" once but then never again. Just use it for each paragraph you quote
and don't worry about italicizing.

It wasn't until this post that I realized the italics of your upstream post
wasn't your original content. I don't find it nice to squint at the text to
see when it stops being italicized to know when you've started your post. But
a final ">" is easy to see.

------
koroshitekure
hey, i'm the author of the article

really... surprised it got submitted here

incidentally i'm running pleroma, not mastodon. minor detail but you know

~~~
dredmorbius
I'm quite active on Mastodon and HN, thought this might be of interest.

Would you prefer the title were modified? The mods can do that. I thought that
specifying _what_ the DDoS mitigation was applied to would be helpful, though
my presumption of Mastodon was in error, apologies.

~~~
koroshitekure
I'm not too bothered about correcting it, just thought it good to note

------
ekimekim
On the subject of the IP leaking: Note that IPv4 only has 2^32 addresses, and
people can and do mass scan all of them (see here shodan.io). If your service
is exposing any identifiable information (ie. if it's not completely blocking
all non-cloudflare IPs) then it's fairly easy to find even if it's
"unguessable".

~~~
buro9
Cloudflare EM for DDoS Protection here.

If a customer wants to hide their IP then the best way to do it:

1\. Onboard onto Cloudflare

2\. Audit your app and ensure you aren't leaking your IP (are you sending
email directly? making web calls directly? - make adjustments to use APIs of
other providers, i.e. send emails via Sendgrid API, etc)

3\. Change your IP (it was previously public knowledge in your DNS records)

At this point your IP should be unknown, so...

4\. Use `cloudflared` and [https://www.cloudflare.com/en-gb/products/argo-
tunnel/](https://www.cloudflare.com/en-gb/products/argo-tunnel/) to have your
server call us, rather than us call you (via DNS A / AAAA records)

Because this connects a tunnel from your server, you can configure iptables
and your firewall to close everything :)

Here's the help info: [https://developers.cloudflare.com/argo-
tunnel/quickstart/](https://developers.cloudflare.com/argo-tunnel/quickstart/)

PS: to the OP I tried to contact you via keybase, feel free to ping my email.
We are working to improve the DDoS protection for attacks in the range you
were impacted by and the product manager would enjoy your feedback if you're
willing to share them in the new year.

~~~
koroshitekure
hey, OP here

I'm no longer on keybase, deleted it a few days ago - but I'm more than happy
to share what I found if you want

pretty sure it's nothing groundbreaking though

other contact methods are listed on my profile

(edit: by OP I mean article author)

~~~
jabot
On an unrelated note: why are you no longer on keybase? Are there any problems
with the service?

~~~
koroshitekure
the guy that was attacking me attempted to dox me, so I shut down all non-
essential accounts to prevent anything similar happening in the future

------
rehasu
Is a federated system like Mastodon not setup in a way that users have access
lists and if one server is down they simply connect to the next? I would
expect to just limit the access to my server in a way that no illegal content
is added to its storage and I don't have to pay horrendous fees for the
network traffic and then just let the DDOS happen. At some point it needs to
stop since the DDOSer will find nicer targets and the source of your problem
will not have enough funds left. And then your service simply continues.

That's at least how I think a federated system should work. Not sure if
reality matches that.

~~~
dredmorbius
As others have noted: accounts are instance-bound.

Other Fediverse protocols -- I believe either Friendica or ... I _think_
Hubzilla? -- have some level of account portability.

There's a fairly long-standing request for Mastodon to support this. For now,
you can have accounts on other instances forwarded to your primary.

While you can export and import your own follows, _followers of your account_
won't automatically redirect to the new home.

Masodon _content_ however will syndicate across the Fediverse, and even some
of my posts from a now-dead instance can (occasionally) be found.

~~~
gargron
> While you can export and import your own follows, _followers of your
> account_ won't automatically redirect to the new home.

This information is outdated since October 11, 2019. You can move followers
from one account to another in Mastodon 3.0.

~~~
dredmorbius
TIL, thanks!

(For others: Gargron is Mastodon's creator.)

------
pferde
It just boggles my mind how petty and vindictive people can be.

~~~
dredmorbius
Also: how much _just one asshole_ can screw things up for everyone.

Unfortunately, this is probabalistically likely as any community grows.
Equivalent raging occured fairly early in the life of other social networks
such as Usenet and The WELL.

------
aeyes
> Cloudflare logs really suck for non-enterprise customers.

Clouflare logs also suck for enterprise customers

~~~
jgrahamc
In what way do they suck? We have incredibly detailed logs available to our
customers.

[https://developers.cloudflare.com/logs/about/](https://developers.cloudflare.com/logs/about/)

------
wlkr
Great write-up. As someone unfamiliar with the legal side of this, would it be
worth the author contacting law enforcement of some kind?

~~~
philpem
Unlikely. From past experience (a 4-year-long online harassment and dirty-
tricks campaign!) the police are generally clueless to online matters. The
most common responses I've had were "we don't police the Internet", "there's
no evidence" or "there's nothing we can do".

Unless you're, yanno, EMI Records, Sky TV or someone with political sway.

The most productive (best outcome) way of handling it tends to be to turn your
OPSEC up to eleven and put all your XP into defence. Again, based on
experience.

~~~
wlkr
Thank you for your insight. As you point out, it's no alternative to a robust
defence, I was just curious as to whether it would ever actually be
investigated. I would probably report it still just for accountability (e.g.
insurance). Especially now you can report online.

------
ryencoke
Another option since you already have OSSEC running.. Create a script for
OSSEC that interacts with the cloudflare api to block the flooding IPs. This
way the flood doesn't get to your server.

------
tyingq
Also a good idea to have more than one DNS server, from separate providers
denoted as authoritative for your domain. Assuming your CDN will let you.

~~~
judge2020
Cloudflare I believe will remove the site if other nameservers are added.
Simply using a non-CF registrar and having a backup authoritative nameserver
will suffice.

------
qxnqd
on my own instance of Mastodon

------
adventskalender
Ever since they injected 70kb of JavaScript into my site, I don't trust
Cloudflare anymore. I just checked, and other CDNs seem to be very expensive.

I wonder if there are any other ways to defend against DDoS?

Maybe looking for a host that helps with such matters? But then they will
probably be more expensive, too?

