Hacker News new | comments | ask | show | jobs | submit login
How-to take down any DigitalOcean-hosted website.
31 points by pjbrunet on Apr 28, 2014 | hide | past | web | favorite | 11 comments
Simply send an abuse complaint. Digital Ocean doesn't even read or verify the complaint. They automatically power down the server (within about 30 mins) whether or not the complaint refers to actual URLs on the server.

This just happened to one of my clients today (hour ago) and Digital Ocean is not responding to support tickets and not responding on Twitter either.


Thanks for inquiring, but this hackernews title is very misleading. We take customer satisfaction very seriously at DigitalOcean. It wasn't one complaint, but 76, and it wasn't immediate, but 4 days with no responses that led to action being taken.

Of those, about 50 of them were done on the 25th, for which we received no response.

If we do not receive a response for 50+ abuse complaints we unfortunately have to take action, which was done in this case.

Hopefully that clears this issue up.

Thanks, Moisey Cofounder DigitalOcean

So the images were deleted and everything was resolved on my end that same afternoon of the 25th.

Not four days with no response, more like a few hours to resolve the problem with productive back-and-forth communication.

However, DigitalOcean again threatened to shut down the server later that evening of the 25th (automated email) because DigitalOcean support had not closed all the abuse tickets. Somehow it's my job to close the ticking timebomb tickets, apparently these tickets countdown to auto-shutdown of your Droplet if they're not all closed within an unspecified timeframe. Luckily I was online to click delete-refresh delete-refresh delete-refresh on the rest of the tickets or no doubt the server would have been powered off again.

The 26th and 27th, no problems.

Today (28th) DigitalOcean gets a pile more rapid-fire complaints and shuts off the server again, within 30 minutes of receiving the complaints. This time the complaints are bogus because I already deleted everything on the 25th.

What does that tell me? DigitalOcean did not check if the complaints were legit before powering off the server. All the URLs in the complaints are 404. It wasn't until Zach got involved that I was given some assurance the server wouldn't be auto-shutdown again in the event of more bogus complaints.

Moisey, I suggest you review what actually happened. It was not 4 days of no response.

According to your own ticket system, the first abuse complaint came in "04/25/14 at 16:10" and the 50 complaints (I did not count) came in rapid-fire succession, several per minute. Do you want a screenshot? I did not see the auto-generated emails you sent till after the client's server was shut down.

I became aware of the problem Fri, Apr 25, 2014 at 12:12 PM thanks to Hyperspin, which is when you shut down the server. Not sure of the timezone difference but where are you getting 4 days? Did you even read the tickets?

You can see my tweet on the 25th here:

"@digitalocean turns out, nothing to do w/ DNS. Account locked due to abuse." 12:43 PM - 25 Apr 2014


I checked your status website which said you were having DNS issues. This client recently dropped your DNS servers due to errors so that wasn't the problem.

Within a few minutes I found out the account was locked due to some images automatically pulled from "open graph" and Twitter meta tags. (In other words, images that were intended to be shared, but whatever that's another tangent.)

Fri, Apr 25, 2014 at 12:44 PM told the client you allowed me to power on the server to delete the images. Then I had to upgrade MySQL because you didn't shut down the server properly.

Fri, Apr 25, 2014 at 1:47 PM spoke with the client and he wants to delete the entire category of images so this would never happen again. All the images were in the same directory "rm -rf" was all I needed. So within about one hour of discovering there was a problem it was taken care of.

Posted on 04/25/14 at 18:44 (support ticket time) "Thank you for responding to this manner quickly. Please let us know if there is anything else we can do to help."

So everything was fine, until (To be continued...)

Hi PJ,

This is Zach from DigitalOcean. I'm very sorry that this was your experience with us. We've replied to your ticket (and Tweet) and fully unlocked your account.

We take abuse (DCMA complaints), account security and platform security seriously. Sometimes it's a tough balance, and when accounts are incorrectly locked we take quick action to resolve.

Please feel free to let me know if you have any further thoughts: zach at digitalocean.com.

Thank you, Zach DigitalOcean Director of Support

I feel bad posting this to Hacker News because I like Digital Ocean. But I take downtime seriously. It's not my website that was shut down and I'm saying this on my own, not on behalf of the client/owner. So I can't really comment on the details.

I responded to your support ticket already, thank you.

This is just my opinion. I already posted some of these suggestions on Twitter.

- 30 mins notice before you power down a server sounds extreme. Why not a 24-hour notice? An emergency SMS, tweet or phone call would have been awesome.

- Shutting down a server is a big deal. That particular server has several background processes running all day. Maybe I'm dreaming, but if you could have just shut down http (what the complaint was about) and not the entire server, that would have been great. That way I'd still have ssh access to delete files, etc.

- I don't understand your policy of (paraphrasing) "Your account will continue to lock and re-lock as long as you have open abuse tickets." Even if the problem is resolved and the account is unlocked by support and everyone is happy, your system continues to re-lock the account until all abuse tickets are closed. The company complaining sends rapid-fire complaints (can't be that uncommon) and clicking "delete ticket" 100 times (waiting for refresh) is annoying. Seems like you could easily fix this by merging all the complaints into one ticket or with some kind of "checkbox" option.

- I don't get the impression you have someone at DigitalOcean familiar with abuse complaints. Maybe you do, I don't know. I get the impression you don't read or filter the complaints. Seems like you just press the "off switch" willy nilly.

The account is unlocked now. Just downloaded a backup in case this happens again. Still no response to why they didn't bother reading the abuse complaint. (Edit) DigitalOcean wrote back a nice response. I'll follow up with them before commenting futher here.

It is troubling to hear that digital-ocean takes arbitrary action to shut down servers without giving atleast 24 hours notice.

It is worrisome for me because I am a linode user and only recently created a digital-ocean droplet and after playing with it, I liked their service and was planning to host a new app with them instead of Linode while the existing one's remain with linode.

But now I am not sure?

Can any digital-ocean staff please provide a link that shows your policy for shutting down servers. I need to be sure this high handedness is not actually in your policy.

This is the email you get 30 mins before they shut you down:

"Oh no! We've found an issue with your account and issued you a new ticket that needs to be addressed as soon as possible.

Please login to view the ticket: https://www.digitalocean.com/support

Thanks so much, DigitalOcean"

After 30 mins you get this:

"Account Locked by DigitalOcean Staff We are sorry to inform you that your account has been locked by DigitalOcean Staff. None of your virtual servers have been destroyed. We have opened a support ticket with you regarding the cause of this lock. Please navigate to the Support Page to resolve the issue that has caused your account to be locked. Thank you, DigitalOcean Staff"

And this is why I keep recommending Linode to all my clients !

annnnd this made me close the tab about using ansible to create a droplet. Back to the pyrax docs!

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact