Hacker News new | past | comments | ask | show | jobs | submit login
Taking Over DigitalOcean Domains via a Lax Domain Import System (thehackerblog.com)
385 points by infosecau on Aug 26, 2016 | hide | past | web | favorite | 170 comments

This doesn't help my impression of Digital Ocean at all (even if I am a paying customer currently). A few years ago you could impersonate Digital Ocean staff on their support pages with no effort. They grabbed the username from your email, so whatever you put in front of the @ becamse your username on the forums, visible to everyone. And the avatar came from one of those email->avatar services where you can sign up and set it to anything. So when I signed up with a username like digitalocean@mydomain.com, I ended up being called "digitalocean" on the support forums, and if I had wanted I could just change the avatar to the Digital Ocean logo and impersonate DO or anyone else.

I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.

Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.

And the ban reminds me of the recent case, where DO invalidated the credits of many people within of 2 weeks with a simple TOS change.

I had to pay 5€ to even be able to add the 100$ credit from the GitHub students pack to my account (for "verification purposes"), and then they – illegally – delete it just like that? (I never got to use any of it)

DO is one of the shadiest hosters I know.

On one hand, the policy change was made a year prior. On the other hand, we didn't communicate it as well as we should have. I apologize for that. I consider myself as much responsible as anyone else on that. If you ever want that account credit back, please let me know. I'm an easy find on Google, or you can open a support ticket anytime.

For what it's worth, we posted on our blog about just this. https://www.digitalocean.com/company/blog/details-on-expirin...

I was affected by this - I lost some credit that I had remaining from the Github Student Pack due to it expiring with relatively short notice. If I open a support ticket, will I be able to ask to get that credit back?

Absolutely, please ask for me if you run into any trouble. You should not have any problem.

"Was made a year prior", that’s wrong.

I got the ToS change notification, and merely 2 weeks later my credit was invalidated.

I had had the credit on my account for over 2 years by that time, but hadn’t used it.

And I had opened a support ticket, and was advised to take legal action if I believed it was invalid or wanted it back.

No thanks, I’ve had enough.

The change was made a year prior. It was not communicated via email. That email you received was a reminder. We only put up a notification in the control panel. That was the communication failure, and I apologize for it.

At no point should anyone be advising you to take legal action to retain your credit, and I would be grateful for the opportunity to review the ticket in question.

> Why did I not receive any notification? A change in contract of that magnitude should have led to me being directly notified per email or other communication methods, so that I’d actually have a chance at still using the credit.

I agree with you completely. That is precisely why I refer to it as a communication failure. I had a year to ask "Why didn't we email about this?" I didn't ask that question. I failed. I'm sorry.

I understand that you're not interested in helping, but my offer stands, and I appreciate you and your feedback. We're all just people here trying to do what we think is right. I know I fall short, I make mistakes, but all I want is to own them and make them right.

> The change was made a year prior.

Why did I not receive any notification? A change in contract of that magnitude should have led to me being directly notified per email or other communication methods, so that I’d actually have a chance at still using the credit.

This way, you notified me right when the credit ran out, which was extremely shady, and I’ve met hundreds of others with the same issue in IRC when you invalidated it.

I can’t and won’t be able to assist you with an investigation to improve your public image, though, as I’m quite busy at the moment, and my time is better spent contributing to open source projects than discussing the wrongs of a hoster that won’t change anyway on HN.

>I had had the credit on my account for over 2 years by that time, but hadn’t used it.

>A change in contract of that magnitude should have led to me being directly notified per email

It seems to me the credit is a form of charity for students, and you had not bothered to redeem it for 2 years. Now, you seem to be implying that DO is acting in an evil manner for only alerting you to the change via the dashboard (which the DO rep in this thread admitted was poor execution on their part). I don't think you are being charitable, and I think the underlying assumption you are making almost sounds conspiratorial.

I'm not a DO customer, and don't have any affiliation.

Yes, I had not redeemed it, because, as a student, I was trying to save it up knowing I’d want to do a project where I’d require servers during summer break between 2nd and 3rd year of my undergrad degree.

The credit was invalidated 3 months before that.

With the legal 3 year validity period of promotional credit and coupons that German law guarantees, I had expected being able to do that.

They changed their ToS a year before, you got a notice to review them. What's illegal about revoking promotional credit?


No, they changed their ToS in March, and invalidated all coupons older than April of the year before by April.

That left me just 2 weeks.

And what’s illegal is that in Germany, promotional credit or coupons given without a time limit is valid for at least 3 years.

I can think of at least Cloudflare (somewhat), Linode, and Hurricane Electric off the top of my head. Anybody who operates a well-known ns1 type of resolver. It's more a problem with zone hygiene than hosts, honestly.

Cloudflare does something similar to AWS. Each user gets different nameservers

Is that true? The nameserver I was given by CloudFlare is "nelly.ns.cloudflare.com". From some cursory searches, it seems like a large number of domains have that same nameserver. AWS nameserver hostnames have all kinds of numbers in them that seem a lot more like they're generated per user.

While you can generate unique hostnames of name servers per user you can't assign all of them unique IP addresses (at least in IPv4) as dns resolving doesn't have anything like vhosts (why should it?). So there will be overlap. Which is not a problem as long as the set of name servers for one users doesn't overlap the set of name servers for another user for a single domain.

So you confirm that a set of name server belongs to a user by verifying that its domain record has those name servers configured. You then don't allow another user to control the same name servers for that domain as long as the domain record still points to them. New users can then freely create new name servers but the domain still uses the servers assigned to by the domain owner.

What DO does is that it allows any user to control the name server for a domain if the previous owner gives up control of them. It should only do that, once the domain records start pointing anywhere else.

It's not true at all, no. I got "dan" on two or three different accounts.

This seems like a great candidate for an ACME style protocol.

I would totally support the ACME spec being expanded to cover this, and various other domain verification related issues

Or just claim your domain if you point it at a DNS service..

This same thing happens with CloudFlare & is being actively exploited. We reported it to them within the last two weeks and we were told that it's expected behaviour and that they weren't going to do anything about it.

I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme.

They told us that they wouldn't be making any changes.

FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time).

Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account.

Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account.

In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc.

Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!

> according to CF, it's not an issue...?!

According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN.

CloudFlare just doesn't care.

But I read on their blog that they are saving the internet, like, everyday and twice on Sundays!

not at all accurate. If you find something abusive or malicious report it: cloudflare.com/abuse -- every report filed there is reviewed by a human.

It seems that this is getting downvoted, and I want to explain why I agree with the downvotes:

The policy on that web page has two egregious issues.

1) It does not have any provisions for SPAM. I regularly get e-mail SPAM that has CloudFlare-protected links. That abuse page does not even apply. Effectively, CloudFlare offers spammers a 'pink contract'.

2) CloudFlare has a reputation for ignoring abuse, and that page effectively says nothing about whether abuse will be stopped; that page only offers to transmit the personal information of someone who reports abuse off to an abuser. This is not a theoretical; this has happened in the past, and the personal information of someone who reported a site that contained child pornography was then posted all over the net for its users to begin harassment with.

So if you find something abusive or malicious, your best bet really is not to report it. CloudFlare won't act, other than to put you at risk.

For what it's worth, in the support ticket exchange I did indeed offer to submit details of the attack vector along with proof that it's currently being actively exploited, for financial gain & at the direct expense of your customers.

I was basically told "you will be wasting your time"...

To be fair, he did say "is reviewed by a human", not "we care" or "we will act on it"

I don't care whether it's reviewed by a human or not, the fact remains that CloudFlare does not act on abuse reports because they claim "reverse proxy = not responsible" in every abuse report I've sent them even on matters that do not pertain to their reverse proxy functionality at all.

They just don't want to be responsible, and unlike any other large network service provider that I sent abuse reports to, they just try and deflect and keep servicing spamming and phishing operations instead of booting them from their systems.

> according to CF, it's not an issue...?!

Definitely not the case.

I work at CloudFlare, not in DNS or on this code, and have mentioned this incident in the all-company chat. There is a healthy conversation happening there and it turns out a fix was already in the works for the underlying issue.

adanto6840 has supplied the support ticket number (thank you), and this specific incident is also being reviewed.

If only it was possible for them to get proper support outside of hoping an engineer sees their HN comment!

I will never stop being infuriated by responses like this from companies - how many more megaleaks have to happen before they realize that they need to embrace white hats, not ban their accounts, not sue them, not swat them / have them arrested, not silence them.

Great find / writeup.

Why, when the author realised this was likely to be possible, didn't they get in touch with DO? Or try with a domain they knew was OK to use? Or at least just try with one domain.

They took almost twenty thousand, sent all the requests to their own server and logged them.

That's surely not your first step.

Domains can only be added if they are not in another account, so he added only domains which were previously deleted but still pointed to DO nameservers (hence almost certainly not being used).

And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.

> (hence almost certainly not being used)

Well they were still being accessed by real people.

> And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.

Both seem reasonable. And it does sound like the security team are actually doing something about it.

> I reached out to DigitalOcean’s team to see if they can assist in deleting the domains from my account (sadly leaving them vulnerable again) or sinkholing the DNS to I received a very helpful response from someone on the security team and it appears they will look into it.

I assume these things happened from different departments. Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?

> Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?

If I was a domain reseller, adding my 20k domains to digital ocean just to get banned without warning, explaination or option for reconsideration I would be rightfully upset. If they didn't want people adding large numbers of domains they could just have limited the feature instead of banning people who reach some arbitrary threshold.

If on the other hand the department that executed the ban knew that the registrations weren't made by the domain owners, they should be discussing such a huge incident with the security team. That discussion would naturally lead to them knowing about the specifics of this case, unless this case wasn't widely shared in the security team.

So the options are:

1. Digital Ocean bans legitimate customers without warning or option for reconsideration; for no obvious reason

2. Big security incidents don't get reported to the security team

3. The responsible people thought that this was not a big security incident

4. This incident wasn't discussed in the security team

5. They knowingly banned a white hat hacker (who may or may not have gone too far)

Of all those options, the last one is by far the one that looks best for Digital Ocean.

Or 6, they spoke to the security team in the interim period of several hours between the addition of all the domains and when the author reported the issue to the security team.

Neither of us really know what's happened here, but the author at least thinks DOs actions were justified so I'm reasonably happy to leave it at that.

See my response below and the logs were secure removed shortly after (as stated in the blog post).

Not that I don't believe you, but you already lost white hat status in this case the moment you log traffic on 20k domains. Just do an attack on another domain you own is white hat, 20k domains is not.

You're probably right about the logging being a bit too far, it was mainly my curiosity getting the best of me. One of my big assumptions was that all of these domains were just owned by one domain broker and this wasn't actually a systemic problem with the implemented importation methodology. I also thought it would be mild because if they had been deleted from an account they were likely no longer used (or so I had wrongfully assumed).

I certainly don't attribute malice on your part, and I'm sorry if my comment came across this way. My point was (at least intended to be) that I'm not really surprised at DO banning you.

I would be careful about doing something like this on a large scale, I would actually be surprised if this doesn't technically violate some laws. Please be careful, you don't want to end up trying to explain nameservers to a judge. [edit - and the rest of us don't want that either, even just selfishly I would like as many security researchers practicing their craft as possible, but also I don't want people being punished for trying to do the right thing]

I wouldn't call redirecting 10s of thousands of accounts 'white-hat' hacking. He could have just done like, I dunno, 3, to prove his point.

Because it can still cause real damage, even if the intentions were good.

Hey Matthew,

I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here.

I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well.

I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here.

I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward.

<3 Jarland

Hey Jarland thanks for the thoughtful response. I don't think your actions were rude or inappropriate (what host would see this behavior and not act similarly?). I apologize that the discussion on the topic became so negative towards DigitalOcean (this might be my fault, I was merely trying to point out why I hadn't been able to delete the domains - not say that DigitalOcean was a bad company/etc). The disclaimer I added early on I thought would mitigate some of this negativity but apparently not quite.

At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.

No worries! You're always welcome at DigitalOcean :)

I think most of the "outrage" is with the complete lack of validation and the "This is a known workflow within our platform" response more than the ban. Any plans to address the issue other than reactive bans?

I think it's absolutely appropriate to sit down and talk about ways that we can prevent this, both in the short term and long term. I don't think it's as simple of a problem as some might suggest, because you've got to balance expected behavior, a reasonable expectation of convenience, and added security measures.

I don't think you can make a proper decision where you're only looking out for protection, or only looking out for convenience, or only looking out for expected behavior. I think you have to mesh all of these items together and make a change that addresses each item, and I don't think that's necessarily a one day discussion. Certainly security does come at a cost of convenience, and that is okay, but it is important not to toss convenience aside as something not worthy of consideration.

So yeah not trying to be vague, my position is not with engineering or security but with the support team. I do think we need to talk about this, and conversations are taking place, but I don't honestly have the ability to say "Yes we're going to implement _____ within ____ days" or something like that. At least not today.

Great article! I'm saddened by DO's response and further wronging a white hat by banning you.

Let's remember Linode offers 2x the RAM.

Linode also does not verify domain ownership before you add a domain to their DNS. And they also use the same nameservers for all accounts. So I think they're in the same boat as DO as far as this "vulnerability" is concerned.

I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.

Linode has had _far_ larger issues than DO ever has. That's why I switched to DO in the first place.

Same here. Please stay away from Linode. They used to be good. Now they are unprofessional and unethical.

I'm a new Linode user (as of this week) and have been happy so far. What are these problems you talk about? I'd like to know before I commit further.

There have been several critical issues. Search for 'linode hacked' on hn.algolia.com and see for yourself.

No one remembers how Linode tried to cover up getting hacked?

>white hat

Gray hat. (Still doesn't justify the idiotic response from DO security team)

Linode has an awful track record for security.

So does DigitalOcean.


Then you've been through their back end being compromised and customer details leaked several times. I'd been using them since about 2009ish and used them until earlier this year.

Something similar happened to me a few months ago with Cloudflare. I set up a new domain to use Cloudflare's nameservers but did not immediately get around to setting it up on the admin panel. By the time I wanted to add the domain, someone else had already grabbed it and set up some sort of spam page.

Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it.

So this is far from a DO-specific issue...

"They also didn't seem to care much about the security implications when I questioned them about it." no one ever said that was the case.

Wonder what else might be vulnerable to this... CloudFlare seems like it may, they only have a handful of nameservers in any case.

Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.


This is an absolutely terrible response from DO. If I had anything hosted here, I'd move away ASAP. Seriously, do it.

I have an account with DigitalOcean (and several competitors) and I'm not going anywhere or moving any sites around because of this. Sure, they could have handled things better and the security researcher could have too. I don't see any malice or incompetence here, nor do I see a reason to make the effort to switch to another provider.

Where are you going to run off to? How is their security better over there? How many hours of work does that involve?

Allowing adding a domain without verifying ownership, and banning someone who reported a security issue isn't incompetence?

I think you're looking more kindly on their security practices than most folks in the security world would.

Banning someone who added 20k domains to their account. There's a difference between reporting a security issue and blatantly abusing their services.

It's pretty understandable actually. Most DNS providers do similar, it's really just the admins fault for not switching their DNS, likely because they were abandoned and unused.

I should leave the most reliable host I had to date because someone was messing around with things he shouldn't be?

DO knows people can do this, but they don't want people to do it.

Remembering 2 recurring DNS servers is easier for bulk management than a bunch of different ones.

You don't test services like that, it can negatively affect other users.

Response wasn't perfect but it was reasonable.

> Response wasn't perfect but it was reasonable

I'm sorry, "We aware that we make it easy for about 20k domains to be directed to a malicious host, but we're not going to do anything about it" is reasonable?

But of course, it's because Matt "was messing around with things he shouldn't be". It's all solved - we just need everyone to stop doing things that DO "don't want people to do".

Meh the owners of the domains gave up control by pointing to someone else's nameservers.

You've just condemned 99% of domains. You really think that's reasonable?

I do think it's reasonable.

If the domain is added to the account there is no PoC, it's only for domains that have been removed from accounts but still have the nameserver values(meaning the domain is not being used at this point, there's no zone file if it isn't added to an account).

So this is mostly only going to affect currently derelict domains. I'm not saying it isn't something to worry about, but I do think it's a reasonable solution.

What you just described as reasonable is not the scenario I asked about, which is just "pointing to someone else's nameservers".

If I buy a domain from a registrar, I can point the registrar at Digital Ocean's nameservers (or AWS, CloudFlare, etc) for my domain (by adding NS records at the registrar). Then I need to go to Digital Ocean and add my domain and records to their nameservers (via their control panel or api).

If I remove my domain from Digital Ocean, it's my responsibility to then go to the registrar and point the registrar away from DigitalOcean's nameservers. (I own the domain, so I'm the only one the registrar allows to do this. Digital Ocean cannot do this.)

Now, your suggestion is that Digital Ocean goes and verifies that I'm the one who owns that domain. But how would they do this (legitimate question)? I imagine manual verification of ownership of every domain upon creation isn't feasible for their scale. Digital Ocean could query DNS, and see NS records pointing to Digital Ocean, but this only tells them someone configured the nameservers for that domain - it doesn't imply ownership. Digital Ocean can check Whois for the owner of the domain. Checking Whois might work for many cases, but at least some registrars have the option of obscuring Whois data.

It seems simpler to put the onus of security on the owner of the domain. I should cleanup my registrar's NS records before removing my domain from Digital Ocean to ensure nobody hijacks it. I would be satisfied as long as Digital Ocean maintained a simple eviction policy (I don't know if they do) as a way for legitimate owners to add their domains to Digital Ocean's nameservers.

Huh? 99% of domains point to some nameserver they aren't contracting service? That is, 99% have invalid NS?

(Obviously when I say somebody else's NS I mean a NS they have zero reason to think would respond with correct records. Obviously not talking about outsourcing DNS hosting.)

Except a (theoretical?) attacker isn't going to stick with the things they're "meant" to do.

I'd much rather know about this vulnerability via a researcher than when a high profile company loses control of their accounts / domains.

Except this isn't even a vulnerability, this is expected behavior given the design I think, to encounter an issue with it, you'd need to leave your domain with DO's nameservers, but delete it from your account. This would only happen if it's something you're not using anymore, or if you're a severely terrible admin.

> this isn't even a vulnerability, this is expected behavior given the design

Those aren't mutually exclusive. It's certainly possible to be broken by design.

> This would only happen if it's something you're not using anymore

From the writeup it seems like it would also happen if you registered a new domain and assigned the DO name-servers but didn't immediately point it to something.

Typically when registering a new domain you add it on your panel first and then change the NS as instructed. If you didn't follow that order, well, that's on you. It's the same design just about everyone providing DNS uses (CloudFlare, XName, FreeDNS, probably others).

But DO instructs you to do it in the reverse order: https://www.digitalocean.com/community/tutorials/how-to-set-...

That's a community tutorial, however it does appear their panel is very sparse in instructions, when you add the domain you see a nameservers list. They don't even tell you that it's not pointed at them and that you should now configure your nameservers as other providers (like CloudFlare) do. I think you're right, some improved documentation would definitely be good here.

However it's the way I've always done it, because otherwise somebody else could add the domain first... it just made sense to me I guess.

Do you know of an alternative that can host an instance of FreeBSD?

I believe Vultr offers FreeBSD (and custom ISO) hosting at about the same price and offers storage servers too. Their documentation and remote console tools leave a lot to be desired though (I wanted to install openSUSE and ended up resorting to manually entering the iPXE commands at a console to get the damn thing installed).

Their support also leaves a lot to be desired. Had a persistent network problem due to a misconfiguration of their system and it took two weeks of back and forth before they really looked into it.

Are Vuktr and DigitalOcean related in any way? their websites look really similar down to the animation that shows you how to create a new droplet/server.

No, Vultr is a spin off of choopa. DO and them have nothing to do with each other.

No, but they've always ripped off the site like that, either on purpose or not. It was worse when the company just came out.

Looking at their DNS setup workflow[1] and API functions[2] I don't see any step where you would have to verify domain ownership - which is this whole thing is about, isn't it?

[1] https://serverpilot.io/community/articles/how-to-configure-d...

[2] https://www.vultr.com/api/#dns

I was providing an example of a host that provides FreeBSD support. To be clear, I don't think Vultr is a good host so I'm not really sure why I mentioned them to be honest...

I'm familiar only with AWS, and it supports FreeBSD.

"Supports" is a strong phrase. Colin Percival creates FreeBSD AMIs. They work mmmmmostly as you might expect, except there's no cloud-init (instead it uses a gizmo that cperciva wrote), but AWS doesn't support it--that's a community thing.

TransIP is similar to DO (except they've had large storage for years) and they support FreeBSD. I've been a happy customer for a couple of years now, never had any problems.


Yeah me too.. However, transip only has a dutch dc.

Sure, Bytemark Cloud https://www.bytemark.co.uk/cloud will let you install from a CD image if you need to, and we FreeBSD works fine.

Meh, it was just mild incompetence. I expect the only providers (hell, large companies in general) who never responded this way are the ones who haven't been around long enough.

Already thinking about it.

this post raises questions:

Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept?

Why not stop at 10 or 20, and then alert DO to the findings?

20 thousand was unnecessary.

Fair point, my relucatance to stop was mainly due to companies usually disreguarding reports unless I have strong proof. Stopping short of the full scope would've left it up to speculation as to the full amount of vulnerable domains.

It was my plan to delete the domains (or at least null route them so others couldn't take them over with more malicious intent). However my account was banned before I could do so.

Have you had any contact with DO's support or security team prior to this? You are correct that companies frequently ignore or downplay reports but it's nice to at least make that first attempt for each new find (per company). This is only ever done as a courtesy, I'm not saying you must or even that you should have here.

Remember, it's just another nerd out there somewhere who receives your report. Sometimes they are super stoked to have it and will be very responsive. This can be more satisfying than a public disclosure without first contacting the company.

That turned out to indeed be the case after their first response :/

How would it have ended if you hadn't reached out to one of their security people after being banned? They'd have given you the traffic, locked your account, and congratulated themselves on a job well done?

How does making a domain that does not resolve suddenly resolve negatively impact a user, again? The domain could not have possibly been operational before the circumstances that brought about this scenario, and there is no legitimate traffic they could possibly be receiving. DigitalOcean could certainly improve authentication here but there are dozens of authoritative services that do not, and this is not a new problem (but is a valuable reminder). I can think of four free providers off the top of my head that this trick would work against.

Put another way, the domains were entirely unresolved and offline. Then they weren't. If anything, this is a nice lesson about keeping your zone and delegations clean, and I'm glad I read it. Nobody got hacked, nobody lost traffic, nobody was impacted. If there was sensitive traffic going to a non-resolving domain, I have more questions than answers. I agree adding the full set was probably a bit much, but you can make that point without misplaced concern for alleged harm.

I'm not impressed with signing up for HN to hit someone like this and your far worse and flagged followup, particularly since it really looks like astroturfing. I guess take solace that the OP pretty much guaranteed he won't get the zones again, since they're for research.

(Nice ninja edit and deletion.)

> How does making a domain that does not resolve suddenly resolve negatively impact a user, again?

Adding 20k domains to your nameservers at once might cause issues in resolution while it catches up with itself which will affect other users using the DNS service. It shouldn't on DO's scale, but it might. This is why the whole thing isn't a "white hat" situation, the author tested without thinking of potential consequences.

What was his/her flagged follow up if you don't mind me asking?

dang would ask me not to import the content to the discussion, and in this case I agree. Suffice to say it was a snarky comment directed at you, deleted after being flagged.

Why run a loop 20 times when you can run it 20000 times. I guess the thoughts of the developer were 'check domain, redirect if not used, repeat till end', not 'stop after 20'.

Amazon S3 has similar problems. To host static website you need use your domain name as the S3 bucket name. Amazon does not verify ownership of your domain, and bucket names use global namespace.

Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.

As a best practice, use a 'normal' name for your S3 bucket and then put Cloudfront in front of your S3 website.

This way you are not limited by bucket names and you also avoid any SSL validation errors.

Note: you can set Cloudfront's TTL to 0 if you don't need any caching.

This is certainly good practice as long as anyone contemplating this considers the cost implications versus serving from S3 directly.

Good point. It's actually generally cheaper to serve content through Cloudfront instead of S3 directly as bandwidth is less expensive with Cloudfront.

However you are going to pay for Cloudfront requests in addition to S3's. In the vast majority of cases, it's insignificant compared to the savings you can achieve on the bandwidth side. (And at a certain level you can ask AWS to waive CF request fees entirely).

EDIT: Just tested it and looks like I'm wrong. Proxying with CloudFlare doesn't help either... Looks like I may have done this with CloudFront instead?

That's not correct. The S3 bucket name is always prefixed. The format is: bucketname.region.amazonaws.com.


To clarify, you're going to have to add a DNS record either way. Doesn't matter what you call your bucket. And no, you don't need to put CloudFlare in front of it for this.

You can't point DNS to any bucket, the bucket name must match the domain name.

Oh wow you're right! Just tested it.

Bye bye digitalocean - account deletion request submitted 1178917. When you have reckless people like Cashan Stine (trust & safety specialist - WTF is that title? sounds like a road safety officer?) that close accounts due to a security report then it won't win any business from me or my clients.

That's not why his account was closed. His account was closed not for discovering a vulnerability, but for exploiting it.

While his intentions might have been good (and I expect that they were!), that kind of behavior isn't.

He did not exploit it, he just provided proof. He did not make any money from the traffic and visitors just saw a white page.

That is exploiting the bug. That is literally exploiting the bug. In the same paragraph where you say he did not exploit the bug, you describe the peripherals of his exploit of the bug.

If he was operating responsibly, he would have applied it to a domain he controlled and provided that as a proof of concept. Instead, he ganked twenty thousand domains. That is at best irresponsible and at worst malicious and DigitalOcean (who I am no fan of, for what it's worth) has no obligation to figure out, or even care, which is which before showing him the door.

Doing it once, proof. Doing it 20,000 times... Exploit.

If you'd just straight up cancel an account that fast I don't believe you had any service or clients hosted on DigitalOcean. Idle threats belong on Facebook and Twitter, not Hacker News.

You don't need to make a deletion request, you can deactivate your account from your account settings, and it offers to delete everything for you. That's what I did when I wanted to close my account recently (I wasn't using the droplets I had, and liked my other VPS better anyway.)

>and liked my other VPS better anyway

Which ones?

Ramnode. Their prices are good and their support is phenomenal. I've always gotten replies to my tickets within minutes, often from Nick Adams himself (Ramnode's CEO.)

A few examples:

- Whenever a new release of OpenBSD comes out, I send a request for them to add the new install media, and they make it available ASAP.

- I switched from OpenVZ to KVM after a couple months and having paid for a year of the OpenVZ service, and they put my remaining credit toward the new service.

- I completely hosed my machine one time and had them restore from backup. Took a while because they wanted to be double-extra-sure of what I wanted, but it was overall a quick and painless process.

I don't know how things will change as they grow, but right now they just seem like a good company, run by people who care about the quality of their product and the satisfaction of their customers.

I've reported multiple vulnerabilities to DigitalOcean before and they've fixed them rapidly, credited me for the effort, and gave me free time on their services.

The difference is I didn't exploit 20 thousand domains to make flashy headlines and prove a point about something that isn't even a serious bug.

You're coming across as a shill either to make DO seem like an infalliable company or to blatantly astroturf enough to get people to hate DO (See jsmthrowaway's sibling comment).

I'm less inclined to believe the latter. I'm looking forward to transferring my domains on DO to elsewhere tonight.

I'm just sharing my experience of what happens when you disclose bugs responsibly.

I'd be happy to share bug tickets with anyone who isn't on some silly chill hunting crusade, for sure, despite my trollish throwaway name.

More throwaway astroturfing? You say "20 thousand" the same way as your other likely throwaway account, V8OaSsoA (that is to say: somewhat identifiably) and complained about someone ripping off DigitalOcean's Web design on this account.

I'm doing math on the throwaways that are oddly attracted to this thread. You are making it very obvious that you are almost certainly a DigitalOcean employee across the two throwaways you've created so far, and that's giving me a whole lot of pause on DigitalOcean that I didn't have from reading the incident itself. If you're an employee or, less likely, a superfan, are you sure this is the type of sustained attack you want to levy? It's not making anybody look good.

Eh, I used '20 thousand' in one of my responses, does that mean I'm the real identity of the throwaways?

Probably not.

Yeah, if it wasn't obvious enough already, I used that number because it's the one in the headline of the article........

I really doubt DO would hire me. And any admin could look up what you said and disprove it. I'm not using any proxies or VPN.

Mmmmm, smell that plastic grass.

I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check the authority of DNS due to the chicken-and-egg problem. The AWS way to handle this issue is definitely awesome but the infrastructure required is not worth for those companies who are providing "free DNS service" as an add-on to their existing customers. Anyway, IMO, it is your fault if you point to a nameserver but not utilizing it.

The random nameservers are only accidentally a defense against this attack. They're avoiding SPOFs, including TLDs -- you never receive nameservers in the same TLD for example. It's a reliability and scaling consideration with this accidental benefit.

Most admins don't think about a complete TLD failure. Amazon did.

>> accidental benefit.


>> Most admins don't think about a complete TLD failure. Amazon did.

I think companies such as Google or Facebook did think that before, but I am not sure why they didn't follow this trick.

Banning his account was totally unjustified since he approached them first with the issue. A less ethical person could have tried to make money or sold this off on the back market. People like him should be rewarded not have their accounts banned. For all we know he just saved DO a lot of headache in sorting this issue had it gone wrong. I really wish the response from DO on this was different.

Adding 20k domains to your account is probably enough to flag as abuse even if you own the domains. Next time the author should probably try just the one or two. Bonus points if they're their own domains.

>Bonus points if they're their own domains.

If the service doesn't understand the issue at all, then when you explain that they're your domains, then they'll probably just tell you it's working as intended and that users should be able to add their own domains.

Arseuming makes an arse out of you and me. Give the techies on the other end of security@ the benefit of the doubt on the first go.

When they fail to understand that then feel free to go ahead and pick one that's not yours to prove the point. One though, not 20 thousand.

> The main reason I did not reach out with the theory instead of the proof-of-concept was because I believed that it would be ignored due to lack of evidence (as is my experience with past disclosures)

I think this was his mistake.

Thank you all for the conversation around this!

The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation.

Feel free to reach out to security@digitalocean.com and we will be happy to help.

Nick, DigitalOcean Security Director

"I was walking down the street and I noticed your house wasn't locked very well. So I stole all your stuff and put it in my own house.

Now I'm in prison because of this so it's really hard for me to put it back."

The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible.

If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....

Comparisons of events like this to violent crime often seem inaccurate.

A better comparison is removing 10 people's lunch money from their school lockers, maybe due to a careless sequential scheme of creating combinations, and giving the money back. And doing this before talking to the principal or any teacher.

Either way, he still should have contacted them before.

A better analogy is "I found a dollar on the ground". I'm not sure that is really a crime!

> Theft is defined as the physical removal of an object that is capable of being stolen without the consent of the owner and with the intention of depriving the owner of it permanently.

This is a really bad comparison. He didn't deny anyone access to anything or take anything at all. The real-world equivalent would be if he found a bunch of empty community/neighborhood whiteboards and drew a (trivially erased) line on them. The signs weren't there for him to use but they were empty (he didn't erase them) and the change he made was absolutely minimal.

He just configured a bunch of deleted/unconfigured domains to point them to a blank web page running on his own server. The point is that he could have done all sorts of nefarious things with that redirection but he didn't. He made a harmless change to demonstrate the vulnerability. That's what white hats do. It's what they're supposed to do. We should thank him for his efforts not lambaste him for actually doing something about the problem.

It's not very re-assuring for users of DigitalOcean to know that issues like this can put our domains at risk to "idiots". I don't care whether the guy was malicious or a nice guy or what. I care that a system I may be trusting my domain name to is trivially exploitable like this.

He didn't steal anything at all.

All of those domain owners are free to change their nameservers at any time.

Very interesting read, thanks. I'm surprised at the response from Digital Ocean, did you adequately explain what you had done?

The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.

I believe I was clear about it. However sometimes my writing can be unclear so perhaps it wasn't properly understood (I assume you're talking about their security team's response and not Trust & Safety?). Kind of sad about the massive amount of hate for DigitalOcean in this thread as their security team really seemed quite nice. Their support was just acting on an anomaly they had seen so shrugs.

It was the guy who said "Thank you for sending this in. This is a known workflow within our platform. We are committed to always improving our customer’s experience and have been examining ways of minimizing the type of behavior you are describing."

It sounds to me like he read your email about the vulnerability, dismissed it as "on the backlog" and skipped the bit about the fact you had sinkholed lots of traffic.

We only have one side of the story though, so who knows. Lessons can be learnt.

I actually suggested this was possible on security.stackexchange.com a while back and was basically met with "meh".


TL;DR - If you own example.com and use DO as your nameserver, then anyone with a DO account can add DNS records for example.com.

Not quite, only one person can, so if you add first, it's yours. Typically they have you add the domain and then you switch the DNS and in that configuration you wouldn't be vulnerable at all.

It sounds like this only impacts people that weren't using their domain, eh? If I leave my NS pointing to some random provider and delete my account, I'm sorta handing off control...

At first I thought it was an issue where DO was using their own DNS servers but letting you add domains. So if Microsoft.com wasn't registered, you'd register it then all DO customer traffic would get your DNS records.

Some telcos have this. Sign up, say you're porting a number. Provide number of local bank and invalid port info. Account gets set up, telco routes their own calls to you. Port is rejected and customer service is asking for info. All while you are getting the bank's calls and forwarding them. Perfect MITM.

Yes, security of number porting is too lax. My number was once ported by a telco (accidentally in this case). They did call me and I had to confirm that I would get a new SIM card and which size I needed, but since I was expecting a new SIM card from the very same telco I didn't understand that they were in fact porting my number. They forgot to say that.

Luckily I was able to get my number back, after many calls to customer support (they managed to undo the porting when I wanted to check how it was going) and a couple of weeks of waiting.

An additional vector for this kind of attack is to create a zonefile for a subdomain off of a working, live domain administered by the same DNS server.

EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server.

This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.

I don't think that setting up custom DNS for everyone as suggested by the author is quite as simple as it sounds.

It's not enough to just come up with the custom nameservers. In order to use them in most TLDs they also need to be "registered" with the registry that operates the TLD.

So let's say you have myDNSdomain.com. You get a new customer who owns NewCustomer.com and wants to you your DNS, so you create these nameservers for them:

ns237.myDNSdomain.com ns2323.myDNSdomain.com

In order for your new customer to be able to use those on their NewCustomer.com domain, you will need to go to your registrar and set up these nameservers. The registrar will then create the corresponding nameserver records with Verisign, the registry. Only then, the customer will be able to use the nameservers on his domain.

Well, yeah. That process already happens when you set up your domain to use DigitalOcean, but the nameserver used during the setup isn't unique.

On the topic of having to pay for traffic to the sinkhole server: how about just closing ports 80 and 443? Then you only get a SYN, and answer with a NACK, that's far less traffic than processing a complete HTTP(s) request.

Did anybody else click CrashChrome.com (or equivalent) in the sidebar?

I clicked on it even before reading the article. Not only the chrome crashed, but also my notebook! I had to reboot the computer.

I was wondering the same thing! I did, and it crashed. Currently running 53.0.2785.80 beta (64-bit) on OS X.

Vivaldi doesnt crash, but it is a resource starvation dos ( several threads go 100% cpu).

I saw it but due to security concerns I did not click.

I find myself asking WW$D where $ is any large tech company with a "good" reputation. What would Google have done? Lyft? Spotify? Blizzard? Use some imagination to apply a similarly dangerous security breach to these companies.

I feel like this question yields better context to ethical arguments because it makes us aware of the cognitive biases and view things from a more abstract perspective..

EDIT: Is there a way to include plain asterisks in HN posts?

As an aside, you can set up a security group in AWS that blocks inbound traffic on port 80 if you'd like to neuter the incoming requests.

Or, like, simply shut down the web server.

He'd still have to pay. By neutering it he means that nobody with bad intensions should be able to use the domains. It is neutered now, but if he was allowed to change DNS entry to point to localhost, he wouldn't have to pay Amazon.

Am I reading this right:

The only defence AWS has against this type of attack is the random (?) grouping of four different NS?

It's effective.

I made the mistake of applying for a job there once. I was discriminated against. Despite being in a protected class, I was so surprised to be so obviously discriminated against by them. (I've interviewed a lot, I don't always get a follow up, this isn't sour grapes, this was very different.) But of course the HR people are careful to not say things that are overtly discriminatory. But when a company insists on a VIDEO call rather than a phone call (despite asking them to do the first one by phone since I was not in a location with good bandwidth at the time they wanted the call)... and then visibly reacts to your image when they first see it, and then pretty much blows you off, despite being well qualified for the position... yeah, it's not what they say.

Same thing happened to me after reporting SQL injection (in 2015!) on Vivaldi website. Polite email and blocked account.

Some companies do seem to prefer to learn about vulnerabilities from pastebin database dump.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact