
Taking Over DigitalOcean Domains via a Lax Domain Import System - infosecau
https://thehackerblog.com/floating-domains-taking-over-20k-digitalocean-domains-via-a-lax-domain-import-system/index.html
======
flexd
This doesn't help my impression of Digital Ocean at all (even if I am a paying
customer currently). A few years ago you could impersonate Digital Ocean staff
on their support pages with no effort. They grabbed the username from your
email, so whatever you put in front of the @ becamse your username on the
forums, visible to everyone. And the avatar came from one of those
email->avatar services where you can sign up and set it to anything. So when I
signed up with a username like digitalocean@mydomain.com, I ended up being
called "digitalocean" on the support forums, and if I had wanted I could just
change the avatar to the Digital Ocean logo and impersonate DO or anyone else.

I tried reporting it but got pretty much the same answer as this guy (though I
did not get banned). Luckily they fixed it like a year later.

Great write-up, and interesting problem! I wonder if more hosting providers
are vulnerable to the same problem.

~~~
jsmthrowaway
I can think of at least Cloudflare (somewhat), Linode, and Hurricane Electric
off the top of my head. Anybody who operates a well-known ns1 type of
resolver. It's more a problem with zone hygiene than hosts, honestly.

~~~
ilogik
Cloudflare does something similar to AWS. Each user gets different nameservers

~~~
electroly
Is that true? The nameserver I was given by CloudFlare is
"nelly.ns.cloudflare.com". From some cursory searches, it seems like a large
number of domains have that same nameserver. AWS nameserver hostnames have all
kinds of numbers in them that seem a lot more like they're generated per user.

~~~
dividuum
While you can generate unique hostnames of name servers per user you can't
assign all of them unique IP addresses (at least in IPv4) as dns resolving
doesn't have anything like vhosts (why should it?). So there will be overlap.
Which is not a problem as long as the set of name servers for one users
doesn't overlap the set of name servers for another user for a single domain.

So you confirm that a set of name server belongs to a user by verifying that
its domain record has those name servers configured. You then don't allow
another user to control the same name servers for that domain as long as the
domain record still points to them. New users can then freely create new name
servers but the domain still uses the servers assigned to by the domain owner.

What DO does is that it allows any user to control the name server for a
domain if the previous owner gives up control of them. It should only do that,
once the domain records start pointing anywhere else.

------
adanto6840
This same thing happens with CloudFlare & is being actively exploited. We
reported it to them within the last two weeks and we were told that it's
expected behaviour and that they weren't going to do anything about it.

I asked them to, at the absolute least, send an email notification to the
prior-CloudFlare owner letting them know that the domain "your CF account used
to control is now being controlled by a new CF account". Better yet, implement
a domain ownership validation scheme.

They told us that they wouldn't be making any changes.

FWIW, on CloudFlare what happened to us was: we were moving registrars for
~100 domains, from GoDaddy to Route53. During this transition, the NS for the
domains temporarily became blank; at this point CF automatically removed the
domains from our CF account. The NS were then re-added to the domains on the
Route53 side (<4 hours of 'no nameserver' time).

Apparently there are people out there that are looking for domains that are
pointed to CF and then attempting to add them to their own CF account
(automated I'm sure) -- which CF lets them do without any verification once
they've been auto-removed from your [the original CF account] account.

Interestingly, the original account must be still stored in their system with
the domain because we were able to re-add the domain to our original CF
account without any verification; effectively "stealing the domains back" to
our CF account, away from the thieve's CF account.

In this case, the "attackers" (perhaps more appropriate, I call them
'malicious actors') were able to commandeer ~100 of our domains for ~2 months,
for free; they redirected them to Russian websites, torrent sites, affiliate
sites, etc.

Again, this is being _actively exploited_ on CloudFlare, at the direct expense
of CF customers -- but, according to CF, it's not an issue...?!

~~~
yoo1I
> according to CF, it's not an issue...?!

According to CloudFlare, they are are a reverse proxy, and they are not
responsible for anything. This has been their response to _every_ issue that
I've tried to bring up with them over any channel, including here on HN.

CloudFlare just doesn't care.

~~~
xxdesmus
not at all accurate. If you find something abusive or malicious report it:
cloudflare.com/abuse -- every report filed there is reviewed by a human.

~~~
adanto6840
For what it's worth, in the support ticket exchange I did indeed offer to
submit details of the attack vector along with proof that it's currently being
actively exploited, for financial gain & at the direct expense of your
customers.

I was basically told "you will be wasting your time"...

~~~
jc4p
To be fair, he did say "is reviewed by a human", not "we care" or "we will act
on it"

------
arkadiyt
I will never stop being infuriated by responses like this from companies - how
many more megaleaks have to happen before they realize that they need to
embrace white hats, not ban their accounts, not sue them, not swat them / have
them arrested, not silence them.

Great find / writeup.

~~~
IanCal
Why, when the author realised this was likely to be possible, didn't they get
in touch with DO? Or try with a domain they knew was OK to use? Or at _least_
just try with one domain.

They took almost twenty thousand, sent all the requests to their own server
and logged them.

That's surely not your first step.

~~~
arkadiyt
Domains can only be added if they are not in another account, so he added only
domains which were previously deleted but still pointed to DO nameservers
(hence almost certainly not being used).

And if DO was really concerned about the damage then they would have removed
the added A records, but they didn't, they banned the author and continued to
let all traffic go to his server.

~~~
IanCal
> (hence almost certainly not being used)

Well they were still being accessed by real people.

> And if DO was really concerned about the damage then they would have removed
> the added A records, but they didn't, they banned the author and continued
> to let all traffic go to his server.

Both seem reasonable. And it does sound like the security team are actually
doing something about it.

> I reached out to DigitalOcean’s team to see if they can assist in deleting
> the domains from my account (sadly leaving them vulnerable again) or
> sinkholing the DNS to 127.0.0.1. I received a very helpful response from
> someone on the security team and it appears they will look into it.

I assume these things happened from different departments. Banning an account
because you saw it make 20k requests to your API adding domains seems pretty
reasonable, and it was a few hours before they reached out. If you saw that
activity, would you ban the account or leave it open hoping that they'd be
doing something nice and reach out?

~~~
wongarsu
> Banning an account because you saw it make 20k requests to your API adding
> domains seems pretty reasonable, and it was a few hours before they reached
> out. If you saw that activity, would you ban the account or leave it open
> hoping that they'd be doing something nice and reach out?

If I was a domain reseller, adding my 20k domains to digital ocean just to get
banned without warning, explaination or option for reconsideration I would be
rightfully upset. If they didn't want people adding large numbers of domains
they could just have limited the feature instead of banning people who reach
some arbitrary threshold.

If on the other hand the department that executed the ban knew that the
registrations weren't made by the domain owners, they should be discussing
such a huge incident with the security team. That discussion would naturally
lead to them knowing about the specifics of this case, unless this case wasn't
widely shared in the security team.

So the options are:

1\. Digital Ocean bans legitimate customers without warning or option for
reconsideration; for no obvious reason

2\. Big security incidents don't get reported to the security team

3\. The responsible people thought that this was not a big security incident

4\. This incident wasn't discussed in the security team

5\. They knowingly banned a white hat hacker (who may or may not have gone too
far)

Of all those options, the last one is by far the one that looks best for
Digital Ocean.

~~~
IanCal
Or 6, they spoke to the security team in the interim period of several hours
between the addition of all the domains and when the author reported the issue
to the security team.

Neither of us really know what's happened here, but the author at least thinks
DOs actions were justified so I'm reasonably happy to leave it at that.

------
jarland
Hey Matthew,

I just wanted to let you know that I really appreciate your feedback, as well
as the feedback from the other commenters here.

I understand that many here are concerned that banning the account seems, from
this perspective, to have been an unjustified action. I do believe that there
is a bit of a misunderstanding on the timeline of events here, as well as the
source of the decision. To be clear, Cash supported the decision that I had
made to ban the account in question, and there had been no communication
between us and Matthew at this point. We began receiving a significant number
of support requests to remove domains from this account, and I authorized the
shutting down of this account as it was clear to me what was happening. I have
been working with our engineers to see to the removal of the domains from the
account as well.

I apologize if our actions seemed at any point rude or inappropriate, it was
definitely not my intention. I want nothing more than to look out for the
safety and wellbeing of our customers, and I chose what I believed to be the
best action. I do want you to know that if I was aware that a security
researcher had been working on testing a theory, I might have acted
differently. That can, however, impact the reason behind a white hat test. You
generally want the company to see you as normal user, so that you can see how
they act in return. We do shut down users who are intentionally causing
problems for other users, and I do think that was made evident here.

I do understand that opening a line of communication with Matthew may have
been appropriate, and I consider that valuable feedback moving forward.

<3 Jarland

~~~
mandatory
Hey Jarland thanks for the thoughtful response. I don't think your actions
were rude or inappropriate (what host would see this behavior and not act
similarly?). I apologize that the discussion on the topic became so negative
towards DigitalOcean (this might be my fault, I was merely trying to point out
why I hadn't been able to delete the domains - not say that DigitalOcean was a
bad company/etc). The disclaimer I added early on I thought would mitigate
some of this negativity but apparently not quite.

At the very least I'm happy that some people have seen this more as a study of
this pattern of functionality being an issue instead of this being only a
DigitalOcean problem. I've seen (and have been reached out by) multiple people
realizing this affects their company as well which has been awesome to watch.

~~~
jarland
No worries! You're always welcome at DigitalOcean :)

------
diegorbaquero
Great article! I'm saddened by DO's response and further wronging a white hat
by banning you.

Let's remember Linode offers 2x the RAM.

~~~
Twirrim
Linode has an _awful_ track record for security.

~~~
sneak
So does DigitalOcean.

------
DangerousPie
Something similar happened to me a few months ago with Cloudflare. I set up a
new domain to use Cloudflare's nameservers but did not immediately get around
to setting it up on the admin panel. By the time I wanted to add the domain,
someone else had already grabbed it and set up some sort of spam page.

Took a few emails to Cloudflare support to resolve this one. They also didn't
seem to care much about the security implications when I questioned them about
it.

So this is far from a DO-specific issue...

~~~
xxdesmus
"They also didn't seem to care much about the security implications when I
questioned them about it." no one ever said that was the case.

------
ultramancool
Wonder what else might be vulnerable to this... CloudFlare seems like it may,
they only have a handful of nameservers in any case.

Sort of hard to call it a vulnerability on DO's part though - more of an issue
with the admins. I think most DNS services operate in this way, really,
route53 may be the exception, not the rule.

------
anilgulecha
TO: ANY DIGITAL-OCEAN USER,

This is an absolutely terrible response from DO. If I had anything hosted
here, I'd move away ASAP. Seriously, do it.

~~~
zatkin
Do you know of an alternative that can host an instance of FreeBSD?

~~~
cyphar
I believe Vultr offers FreeBSD (and custom ISO) hosting at about the same
price and offers storage servers too. Their documentation and remote console
tools leave a lot to be desired though (I wanted to install openSUSE and ended
up resorting to manually entering the iPXE commands at a console to get the
damn thing installed).

~~~
po1nter
Are Vuktr and DigitalOcean related in any way? their websites look really
similar down to the animation that shows you how to create a new
droplet/server.

~~~
nik736
No, Vultr is a spin off of choopa. DO and them have nothing to do with each
other.

------
V8OaSsoA
this post raises questions:

Was there a realization into how legitimate users may be affected by this
action? Was there a plan to remove those domains from their account after
making and disclosing their proof of concept?

Why not stop at 10 or 20, and then alert DO to the findings?

20 thousand was unnecessary.

~~~
mandatory
Fair point, my relucatance to stop was mainly due to companies usually
disreguarding reports unless I have strong proof. Stopping short of the full
scope would've left it up to speculation as to the full amount of vulnerable
domains.

It was my plan to delete the domains (or at least null route them so others
couldn't take them over with more malicious intent). However my account was
banned before I could do so.

~~~
tombrossman
Have you had any contact with DO's support or security team prior to this? You
are correct that companies frequently ignore or downplay reports but it's nice
to at least make that first attempt for each new find (per company). This is
only ever done as a courtesy, I'm not saying you must or even that you should
have here.

Remember, it's just another nerd out there somewhere who receives your report.
Sometimes they are super stoked to have it and will be very responsive. This
can be more satisfying than a public disclosure without first contacting the
company.

------
mixedbit
Amazon S3 has similar problems. To host static website you need use your
domain name as the S3 bucket name. Amazon does not verify ownership of your
domain, and bucket names use global namespace.

Someone can easily block you from using S3 static website hosting by adding a
bucket with your domain name before you do. Also if you delete a bucket and do
not change your DNS, someone can recreate the bucket and will be serving files
from your domain.

~~~
athrun
As a best practice, use a 'normal' name for your S3 bucket and then put
Cloudfront in front of your S3 website.

This way you are not limited by bucket names and you also avoid any SSL
validation errors.

Note: you can set Cloudfront's TTL to 0 if you don't need any caching.

~~~
mortar
This is certainly good practice as long as anyone contemplating this considers
the cost implications versus serving from S3 directly.

~~~
athrun
Good point. It's actually generally cheaper to serve content through
Cloudfront instead of S3 directly as bandwidth is less expensive with
Cloudfront.

However you are going to pay for Cloudfront requests in addition to S3's. In
the vast majority of cases, it's insignificant compared to the savings you can
achieve on the bandwidth side. (And at a certain level you can ask AWS to
waive CF request fees entirely).

------
nowprovision
Bye bye digitalocean - account deletion request submitted 1178917. When you
have reckless people like Cashan Stine (trust & safety specialist - WTF is
that title? sounds like a road safety officer?) that close accounts due to a
security report then it won't win any business from me or my clients.

~~~
dumbsecreport
I've reported multiple vulnerabilities to DigitalOcean before and they've
fixed them rapidly, credited me for the effort, and gave me free time on their
services.

The difference is I didn't exploit 20 thousand domains to make flashy
headlines and prove a point about something that isn't even a serious bug.

~~~
jsmthrowaway
More throwaway astroturfing? You say "20 thousand" the same way as your other
likely throwaway account, V8OaSsoA (that is to say: somewhat identifiably)
_and_ complained about someone ripping off DigitalOcean's Web design on this
account.

I'm doing math on the throwaways that are oddly attracted to this thread. You
are making it very obvious that you are almost certainly a DigitalOcean
employee across the two throwaways you've created so far, and _that 's_ giving
me a whole lot of pause on DigitalOcean that I didn't have from reading the
incident itself. If you're an employee or, less likely, a superfan, are you
sure this is the type of sustained attack you want to levy? It's not making
_anybody_ look good.

~~~
hatsix
Eh, I used '20 thousand' in one of my responses, does that mean I'm the real
identity of the throwaways?

Probably not.

~~~
dumbsecreport
Yeah, if it wasn't obvious enough already, I used that number because it's the
one in the headline of the article........

------
tszming
I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check
the authority of DNS due to the chicken-and-egg problem. The AWS way to handle
this issue is definitely awesome but the infrastructure required is not worth
for those companies who are providing "free DNS service" as an add-on to their
existing customers. Anyway, IMO, it is your fault if you point to a nameserver
but not utilizing it.

~~~
jsmthrowaway
The random nameservers are only accidentally a defense against this attack.
They're avoiding SPOFs, including TLDs -- you never receive nameservers in the
same TLD for example. It's a reliability and scaling consideration with this
accidental benefit.

Most admins don't think about a complete TLD failure. Amazon did.

~~~
tszming
>> accidental benefit.

Agree

>> Most admins don't think about a complete TLD failure. Amazon did.

I think companies such as Google or Facebook did think that before, but I am
not sure why they didn't follow this _trick_.

------
supersan
Banning his account was totally unjustified since he approached them first
with the issue. A less ethical person could have tried to make money or sold
this off on the back market. People like him should be rewarded not have their
accounts banned. For all we know he just saved DO a lot of headache in sorting
this issue had it gone wrong. I really wish the response from DO on this was
different.

~~~
corobo
Adding 20k domains to your account is probably enough to flag as abuse even if
you own the domains. Next time the author should probably try just the one or
two. Bonus points if they're their own domains.

~~~
AgentME
>Bonus points if they're their own domains.

If the service doesn't understand the issue at all, then when you explain that
they're your domains, then they'll probably just tell you it's working as
intended and that users should be able to add their own domains.

~~~
corobo
Arseuming makes an arse out of you and me. Give the techies on the other end
of security@ the benefit of the doubt on the first go.

When they fail to understand that then feel free to go ahead and pick one
that's not yours to prove the point. One though, not 20 thousand.

------
nvigier
Thank you all for the conversation around this!

The security team at DigitalOcean has been working to ensure that DO is a safe
place for security researchers to identify issues on the Internet as well as
at DigitalOcean - security is in everyone's interest. We encourage researchers
to contact us when they want to use our platform for this type of work
specifically so that we can avoid the types of pain that Matthew encountered
while doing his experimentation.

Feel free to reach out to security@digitalocean.com and we will be happy to
help.

Nick, DigitalOcean Security Director

------
jbb555
"I was walking down the street and I noticed your house wasn't locked very
well. So I stole all your stuff and put it in my own house.

Now I'm in prison because of this so it's really hard for me to put it back."

The article writer is an idiot. He deliberately stole accounts because he
could. Just because he then decided to blame the provider because he was able
to do this does't make it any more defensible.

If I mug you in the street, should I then post that because I was able to do
so it's all your fault? No. I'd go to prison....

~~~
TheCowboy
Comparisons of events like this to violent crime often seem inaccurate.

A better comparison is removing 10 people's lunch money from their school
lockers, maybe due to a careless sequential scheme of creating combinations,
and giving the money back. And doing this before talking to the principal or
any teacher.

Either way, he still should have contacted them before.

~~~
T2_t2
A better analogy is "I found a dollar on the ground". I'm not sure that is
really a crime!

------
djhworld
Very interesting read, thanks. I'm surprised at the response from Digital
Ocean, did you adequately explain what you had done?

The first person that replied looks like he just skim read your email or
didn't understand the fact you had sinkholed a lot of traffic.

~~~
mandatory
I believe I was clear about it. However sometimes my writing can be unclear so
perhaps it wasn't properly understood (I assume you're talking about their
security team's response and not Trust & Safety?). Kind of sad about the
massive amount of hate for DigitalOcean in this thread as their security team
really seemed quite nice. Their support was just acting on an anomaly they had
seen so _shrugs_.

~~~
djhworld
It was the guy who said "Thank you for sending this in. This is a known
workflow within our platform. We are committed to always improving our
customer’s experience and have been examining ways of minimizing the type of
behavior you are describing."

It sounds to me like he read your email about the vulnerability, dismissed it
as "on the backlog" and skipped the bit about the fact you had sinkholed lots
of traffic.

We only have one side of the story though, so who knows. Lessons can be
learnt.

------
devrelm
I actually suggested this was possible on security.stackexchange.com a while
back and was basically met with "meh".

[http://security.stackexchange.com/questions/49612/how-
does-d...](http://security.stackexchange.com/questions/49612/how-does-
digitalocean-dns-verify-the-owner-of-a-domain)

------
i_have_to_speak
TL;DR - If you own example.com and use DO as your nameserver, then anyone with
a DO account can add DNS records for example.com.

~~~
ultramancool
Not quite, only one person can, so if you add first, it's yours. Typically
they have you add the domain and then you switch the DNS and in that
configuration you wouldn't be vulnerable at all.

~~~
MichaelGG
It sounds like this only impacts people that weren't using their domain, eh?
If I leave my NS pointing to some random provider and delete my account, I'm
sorta handing off control...

At first I thought it was an issue where DO was using their own DNS servers
but letting you add domains. So if Microsoft.com wasn't registered, you'd
register it then all DO customer traffic would get your DNS records.

Some telcos have this. Sign up, say you're porting a number. Provide number of
local bank and invalid port info. Account gets set up, telco routes their own
calls to you. Port is rejected and customer service is asking for info. All
while you are getting the bank's calls and forwarding them. Perfect MITM.

~~~
samuellb
Yes, security of number porting is too lax. My number was once ported by a
telco (accidentally in this case). They did call me and I had to confirm that
I would get a new SIM card and which size I needed, _but since I was expecting
a new SIM card from the very same telco_ I didn't understand that they were in
fact porting my number. They forgot to say that.

Luckily I was able to get my number back, after many calls to customer support
(they managed to undo the porting when I wanted to check how it was going) and
a couple of weeks of waiting.

------
dotBen
An additional vector for this kind of attack is to create a zonefile for a
subdomain off of a working, live domain administered by the same DNS server.

EG if foo.com is a working site on your DNS provider, try creating a zonefile
for bar.foo.com and see if you can create an A record to point to your own
server.

This used to be something shared web hosting services running CPanel/WHM were
particularly susceptible to. Clearly, the risks here are both
phishing/identity and cookie credential stealing.

------
fmichlick
I don't think that setting up custom DNS for everyone as suggested by the
author is quite as simple as it sounds.

It's not enough to just come up with the custom nameservers. In order to use
them in most TLDs they also need to be "registered" with the registry that
operates the TLD.

So let's say you have myDNSdomain.com. You get a new customer who owns
NewCustomer.com and wants to you your DNS, so you create these nameservers for
them:

ns237.myDNSdomain.com ns2323.myDNSdomain.com

In order for your new customer to be able to use those on their
NewCustomer.com domain, you will need to go to your registrar and set up these
nameservers. The registrar will then create the corresponding nameserver
records with Verisign, the registry. Only then, the customer will be able to
use the nameservers on his domain.

~~~
iancarroll
Well, yeah. That process already happens when you set up your domain to use
DigitalOcean, but the nameserver used during the setup isn't unique.

------
perlgeek
On the topic of having to pay for traffic to the sinkhole server: how about
just closing ports 80 and 443? Then you only get a SYN, and answer with a
NACK, that's far less traffic than processing a complete HTTP(s) request.

------
gengkev
Did anybody else click CrashChrome.com (or equivalent) in the sidebar?

~~~
zubi
I clicked on it even before reading the article. Not only the chrome crashed,
but also my notebook! I had to reboot the computer.

------
ben_jones
I find myself asking WW$D where $ is any large tech company with a "good"
reputation. What would Google have done? Lyft? Spotify? Blizzard? Use some
imagination to apply a similarly dangerous security breach to these companies.

I feel like this question yields better context to ethical arguments because
it makes us aware of the cognitive biases and view things from a more abstract
perspective..

EDIT: Is there a way to include plain asterisks in HN posts?

------
wrren
As an aside, you can set up a security group in AWS that blocks inbound
traffic on port 80 if you'd like to neuter the incoming requests.

~~~
perlgeek
Or, like, simply shut down the web server.

~~~
irth
He'd still have to pay. By neutering it he means that nobody with bad
intensions should be able to use the domains. It is neutered now, but if he
was allowed to change DNS entry to point to localhost, he wouldn't have to pay
Amazon.

------
chinathrow
Am I reading this right:

The only defence AWS has against this type of attack is the random (?)
grouping of four different NS?

~~~
breakingcups
It's effective.

------
20yrs_no_equity
I made the mistake of applying for a job there once. I was discriminated
against. Despite being in a protected class, I was so surprised to be so
obviously discriminated against by them. (I've interviewed a lot, I don't
always get a follow up, this isn't sour grapes, this was very different.) But
of course the HR people are careful to not say things that are overtly
discriminatory. But when a company insists on a VIDEO call rather than a phone
call (despite asking them to do the first one by phone since I was not in a
location with good bandwidth at the time they wanted the call)... and then
visibly reacts to your image when they first see it, and then pretty much
blows you off, despite being well qualified for the position... yeah, it's not
what they say.

------
rasz_pl
Same thing happened to me after reporting SQL injection (in 2015!) on Vivaldi
website. Polite email and blocked account.

Some companies do seem to prefer to learn about vulnerabilities from pastebin
database dump.

