
Stop Using Digital Ocean Now: The Aftermath - sdogruyol
http://serdardogruyol.com/?p=137
======
jjoe
Sorry but I must be blunt... What do people expect for < $8/mo service? Should
DO spend a few man hours to fix the issue at hand (abuse or not) and very much
burn the 7 months worth of service income from this account? This is an
unreasonable expectation.

Something has to give. Yes we've all built and sold products and believe in
providing an impeccable service worth far greater than the sum of its parts.
Because we're in it to please everyone and build a reputation. But hosting is
different. There are real costs for _not_ taking action (upstream null
routing, blacklisting, chargeback fees, fraud, abuse, etc).

I'm definitely playing devil's advocate and yes I go above and beyond for my
clients. Never have I kicked a client to the curb for abuse they haven't
originated. But DO is unmanaged and acquires clients by the shovel. Something
has to give.

For the record, I did publicly state here in HN that DO's business plan
doesn't add up and this is one of the side effects.

~~~
nwh
I expected a budget service, but they've been anything but that. Their support
staff aren't just reading off a script, their instances are fast and downtime
non existent. In this case the user fucked up and was made part of a botnet,
and DO were forced to protect their network. I'm happier running my services
on DO than on any other host.

------
slig
On the other hand, here's my experience with Linode.

Another host company claimed that one of my machines was doing port scan on
their network. Linode opened a ticket and preemptively blocked all outgoing
connections to the SSH port from my machine.

I had enough time to see what's going on and chatted with a very responsive
support. The aftermath is that I moved all my data to a new linode, waited to
for the DNS propagation and killed the old linode. No service disruption and
no all-nighters.

I seriously can't recommend they enough. And yes, I'm aware of the security
problems they had months ago, but I bet that they don't want any more damage
to their brand and they're working very hard to no let that happen again.

~~~
veemjeem
Not all VPS service providers are treated alike. Linode is probably a mid-tier
VPS. Their cheapest plan is $20, which is 4x more expensive than Digital
Ocean's cheapest plan of $5 a month. I would guess that most of the ultra
cheap VPS plans probably won't come with a network engineer to help you.

~~~
slig
Agreed. If I'm trying something new, or just playing with, I could try a new
and cheaper host ($5/m is less than a starbucks coffee where I live).

But if I'm running something that pays me money, there's no doubt that I'll
choose the best provider. And the best provider, for my requirements, is only
$15 more expensive.

------
jrochkind1
> _The things which i still don’t understand.... Is my privacy is more
> important than my user experience or happiness of the service ? Even if i
> want them to tell what really happened ?_

Well, yes. It is completely appropriate that they aren't going to discuss your
logs and other private details in public without your permission. You don't
understand that, for real?

They have no _obligation_ to discuss your case in public at all, although it
might be wise for them to do so and explain themselves, if you are generating
lots of bad feelings for them.

(They do, I'd agree, have an obligation to discuss your case _with you_ and
tell you why they closed your account. It's not really clear to me if they did
so; it kind of seems like they did so, something to do with DDoS.)

------
JeremyBanks
OP, can you please confirm whether or not you made this post quoting a reply
from Digital Ocean in the previous thread, as [1] claims you did?

 _Now i 've received an answer from DO. I seriously dont know how i did a
DDos._

 _Here it 's._

 _Greetings,_

 _Based upon the tcpdump results, I have again confirmed that your droplet was
indeed performing a Denial of Service attack._

 _With this information, we are unable to restore services to your account._

If this is true, it is disingenuous not to mention this reply in your post.

[1]:
[https://news.ycombinator.com/item?id=6439501](https://news.ycombinator.com/item?id=6439501)

~~~
sdogruyol
The thing is that they wanted to stay private when i posted this so i
immediately removed and now waiting for them. By the way i received that
response after 2 hours of the topic opening. They weren't answering me at all
before HN post

~~~
veemjeem
So did your server get owned? Or was the DDOS attack a result of a bug in your
code?

Also, why do you delete your posts on HN? Do you only keep posts that make
your viewpoint look favorable?

~~~
sdogruyol
I don't know the first question's answer and that's what i am trying to learn.
Second it's my first time participating in a HN topic that much i didn't know
the etiquette here sorry for that.

~~~
veemjeem
I'm guessing that most people who run a server may not even realize when they
get hacked. These people (you included) probably should not run their own
servers and stick to PaaS solutions like Heroku or Google App Engine. It
happens all the time to guys who think they can install & maintain Wordpress
themselves.

You probably should have analyzed the issue before making a blog post about
it. I have had servers hacked into in the past, but I wasn't about to defame a
company over my own mistake of securing the server.

~~~
sdogruyol
You miss the point here. It's not about getting hacked or so. It's their way
of handling it. Like i said they kill it first and then tell you the reason
why. What's the point in it ?

~~~
thaumaturgy
The point is that your system may be actively attacking another system, and
it's their responsibility to immediately stop the attack first and then
contact you after.

If they _don 't_ do this, then they run the risk of having their netblock(s)
blackholed by upstream providers or other networks, which is bad for all of
their customers.

They aren't responsible for making sure your system is secure, you are. You're
a sysadmin now; it's not just a toy, there's responsibility too.

------
meritt
His server was probably broken into, someone used it for DDOS'ing, DO shut it
down and customer has no clue what's going on.

------
zagi
Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the
document and will discuss the matter publicly.

\-----

All times are UTC.

Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08
00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55
roughly 7 minutes later.

The customer informed us that it was a script that was crawling in the
background.

We informed the customer that it may be a good idea to check through the
virtual server to see if there were any signs of a compromise just in case.

The droplet was unlocked at this time.

A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was
opened 2013-09-24 12:27:14 to request more information from the customer.

Because this was already a second occurrence we had to do a more thorough
follow up. Discussing the matter with the customer, he informed us that it was
a mysql db dump script that was pushing data to dropbox.

He provided us a link to a github project that he wrote, we asked further
questions. Specifically if you are writing a mysql dump remotely why are the
packets being sent as UDP? Additionally if the final destination is dropbox
that would be an SSL encrypted connection and again why would that transfer go
over UDP?

We reviewed the code of the dump-to-cloud project and it was using the dropbox
sdk, here is where the file transfer is initiated:

    
    
       def upload_file(file_name)
    
            client = DropboxClient.new(@access_token)
    
            file = open(file_name)
    
            puts 'Uploading file!! Please wait.'
    
            response = client.put_file("/#{file_name}", file)
    
            puts "uploaded:", response.inspect
    
        end
    

From the dropbox SDK here is where it sets the destination for the file
transfer:

    
    
       def build_url(url, params=nil, content_server=false) # :nodoc:
    
            port = 443
    
            host = content_server ? Dropbox::API_CONTENT_SERVER : Dropbox::API_SERVER
    
            versioned_url = "/#{Dropbox::API_VERSION}#{url}"
    
    
    
            target = URI::Generic.new("https", nil, host, port, nil, versioned_url, nil, nil, nil)
    
    
    
            #add a locale param if we have one
    
            #initialize a params object is we don't have one
    
            if @locale
    
                (params ||= {})['locale']=@locale
    
            end
    
    
    
            if params
    
                target.query = params.collect {|k,v|
    
                    CGI.escape(k) + "=" + CGI.escape(v)
    
                }.join("&")
    
            end
    
    
    
            target.to_s
    
        end
    
    
    

The code that actually transfers the file from the dropbox sdk:

    
    
        def do_put(url, headers=nil, body=nil)  # :nodoc:
    
            assert_authorized
    
            uri = URI.parse(url)
    
            do_http_with_body(uri, Net::HTTP::Put.new(uri.request_uri, headers), body)
    
        end
    
    

The file is transferred via HTTPS since it is going to a secure service and
HTTPS would rely on TCP for the data transfer, again to ensure that all
packets are delivered.

Given that it was the second incident that a UDP traffic pattern was observed
in less than 30 days and that the information the customer provided regarding
the traffic did not match up, we made a determination that in fact it couldn't
be this script that was generating the traffic.

All of this information was relayed to the customer that we did not believe
that the traffic in question was related to this script because it would not
rely on UDP, an insecure protocol to deliver files to a secure endpoint where
data integrity was of the utmost importance.

Unfortunately, we could not unlock the account at this time because the
information we received was not clear and we already had two incidents of
outbound UDP traffic that appeared to be disruptive and abusive in nature
totaling 1Gbps as if it were a denial of service attack, typically associated
with UDP packets.

~~~
sdogruyol
Hello Ben, thanks for the response. Fırst of all at first ticket i told that
the only possibility of having an UDP outgoing is that script that i wrote.

Other than that i've no other activity or script that can generate that much
traffic. Haven't you even considered that my droplet may be compromised or
being attacked ?

Instead of letting me know what exactly happened or which processes were
running at that time you just locked the account and accused me.

Couldn't you even look at the access logs or so to see which IPs login into
the droplet and then take your action later instead of closing it instantly?

~~~
sdogruyol
I was also tremendously happy with DO and their service. But what if you get
your production apps down without even any notification and proper reasoning ?
That's the thing which makes you feel insecure.

~~~
nav1
They did notify you.

>Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08
00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55
roughly 7 minutes later.

Also, you should do a better job securing your server. It seems like the
server was compromised.

------
skizm
I use DO for a messing around on a small web app I'm developing. Within 24
hours of having my droplet up the root password was guessed and my machine was
used for some DDoS. Granted I was an idiot for not changing the password
immediately but I definitely felt like DO should just use ssh key validation
like AWS does right off the bat. That deters attackers from even trying brute
force attacks in the first place.

Anyway, I checked the logs and pretty much the minute my machine was deployed
a script was guessing my password (lots of failed login attempts for "root"
and "oracle"). This probably means someone knows DO's IP addresses and their
automatically generated password scheme (all lower case alpha characters of a
fixed length).

I reported the incident and destroyed my droplet since there was nothing
important on it. When I heard back from DO I basically got (paraphrasing here)
"you should install fail2ban next time". Case closed. I'm not a big customer
or anything so I don't expect premium support or anything but I feel like
someone should have looked into the attack a bit more. Seems like a lot of
people are experiencing the same thing.

I guess what I am saying is you get what you pay for (it is only 5 bucks after
all).

EDIT: Still using DO. I was just a bit more careful next time I deployed a
droplet.

------
adamlj
About a week ago I got to experience Digital Oceans very tight suspension
policy first hand. What happened was that one of the accounts I manage was
suspended. I had to go through a very long and detailed validation process
before they understood that they had done wrong (they admitted to doing an
error and apologized). But when my account was activated again my droplet had
been destroyed..

After some more time they managed to resolve this and I'm again a happy DO
user but I wish that they take a look at their policies. Just the fear of
knowing that they can shut you down by mistake for a day or two is bad enough
to not use them. They should have a policy where they at least call you and
talk to you before they do anything.

~~~
erichocean
These kinds of seemingly random suspensions at DO are starting to concern me.
I've got a service with ~10K active users I'm migrating to DO as we speak, but
man! I'm starting to feel like it might be professionally negligent to do so
if they're trigger-happy with shutting down instances.

It'd be nice if DO had some way to communicate "hey, this VM matters to my
business, PLEASE don't do anything stupid/automated without contacting me
first", but that's probably too much to ask for the cost.

Gah. Back to AWS. This sucks. :/

------
antr
Some months ago a tried to open an account with DO, but my account was blocked
by DO before creating a droplet because my dad, who has the same name as I do,
had already used the service. DO wanted me to verify my identity, address,
etc.

I can't stand such friction, so I stayed with AWS, who have never had an issue
with having a customer with very similar or equal name to another. Clearly,
there is something wrong in DO's identity/fraud detection process, and even
more wrong is the fact that they are locking user's production accounts
without any warning at all.

~~~
jd007
I ran into the same problem just last week when I tried to make an account to
try it out.

I added a credit card to my account without a promotion code, but later on
found out about the new promotion they are doing (free $10). I realized that
you cannot enter a promo code if you didn't do it the first time you add
payment to an account, even if the account is otherwise still new (I didn't do
anything with it after adding the payment card).

So I de-activated my account, and made another one, entered the same payment
info but with the promo code this time. Then my account was instantly locked
for using the same credit card, and I received an email shortly after citing
section 2.6 of their ToS which prohibited users from using the same payment
card on multiple accounts (even though my previous account was de-activated).

Their trial/promo code system really needs to be revamped as I spent way more
time figuring out how to put the code in than actually trying out their
product. Also their fraud detection system is perhaps too sensitive? I mean I
should be able to re-use payment cards if my old account was de-activated...

~~~
arthulia
They have really quick customer service; they probably would have added the
promotion to your account if you had contacted them.

~~~
jd007
Probably. But at the time I thought it was a problem I could've taken care of
in less than a minute, basically de-activate my account then create another
one and add the code. In this case I chose not to contact them, because as
fast as their support can be it mostly likely won't be faster than I can de-
activate and make a new account. But I was wrong as my account was locked :(

Another thing is their ToS. I admit I didn't read their ToS prior to
registration (most people don't I assume, but I won't use that as an excuse),
but I did go back and read it afterwards, and it said:

"Users are restricted from registering multiple accounts with the same billing
details without first notifying DigitalOcean of that intent to ensure that
accounts aren't automatically flagged as possibly fraudulent and without
notification accounts may be treated as abuse and/or fraudulent which would
lead to suspension of service."

And I thought, even if I did read this I would've reasonably assumed that
since my other account has been de-activated, I'm free to make a new one with
the same billing information. Unfortunately I was wrong.

------
mechinn
Wait... did I miss something or did you send snailmail and expect a response
within 10 hours?

Also I am nowhere close to being a lawyer but I'm pretty sure if they
disclosed your private information on here without your written permission you
could sue them probably for more than either of these 2 posts are worth
against them.

~~~
sp332
I think he scanned the letter and emailed it.

------
ogghead
Sad to say, the same thing happened to me and the support messages were
unhelpful to say the least. They have several copy-and-paste answers they use
far too eagerly.

And it's asinine to ask someone to update billing information when they've
locked your account from accessing the billing info page. Two of us have seen
that with DO. Makes them look quite sloppy.

------
csomar
_And now after nearly 10 hours or so i still haven’t heart from DO._

There is a problem on the Internet is that people demand things NOW. Really?
How long will it take for stuff to happen in real life (especially if you are
dealing with government).

Some stuff does happen immediately (like registering or purchasing something),
but stuff which requires human intervention is obviously slow. And it requires
time.

Yes, you can have 24/7 response. But only if you hire someone (probably 3
persons) working for you round the clock which will mean paying thousands of
$$ per month and not $5/month.

~~~
arthulia
10 hours is unreasonably long for someone who is running an application that
other users need to access. People are not going to be happy if Farmville is
suddenly not remembering all the cows they milked last time they were logged
in. This is about keeping customers happy down the chain.

That said, I actually really like DO and I've only had prompt, helpful
responses from their customer service. I don't build apps but it is an
_excellent_ personal vps, and I still recommend them to my friends when they
are looking for one.

~~~
veemjeem
If you're paying $5 a month, don't expect to have support turn around times
under 1 day. If you're running Farmville, I would hope that you're paying more
than $5 a month for your server.

------
robomartin
> Is it fair to tyrannically close someone’s account up and accuse him or
> further treat him as a liar ?

Isn't this what Google does on a daily basis to users of their various
services? There seem to be no consequences for them and less than zero
interest in improving any of it.

There are companies that can cause your business untold financial damage
through these kinds of actions. For some reason they continue to evade
responsibility in financial, moral and ethical terms. It sucks.

~~~
josh2600
The difference is: you are the product for google and not the customer. Trust
me, google doesn't treat advertisers the same way it treats free gmail users.

~~~
objclxt
Google often shuts down advertiser accounts with little or no notice. You only
have to search for "adsense" or "adwords" along with "banned". Some of those
are with good reason - others are not.

------
PeterisP
There may be all kinds of valid reasons to validate acounts, check documents
and ownership of cards, etc.

None of those reasons excuse putting services offline while this validation is
happening - first try validation, give it some reasonable timeframe, and only
then cut or don't cut the service depending on results. Is it really so
complicated?

------
txutxu
OK, after read the two blog posts, and all the comments here, this is my
tough:

I've a poor's men backup solution too, from my personal droplet to gmail.

I use it for backup /etc, /root, /usr/local/(s)bin and /html, with excludes
and rotations (about 15MB total), encrypted 4 passes with 4 different
algorithms.

I did write two scripts, one for backup-restore (and encrypt/decrypt, secure
deletion, html email generation, etc) and other to list-retrieve remote
backups using IMAP.

On gmail I did put a filter to send all to the trash, so I get 30 days backup
rotation. More complex rotations could be implemented using different cron
tasks, with different config files, pointing to different accounts (srv-month-
number@...), but for my personal VPS I don't need that, 30 days is enough.

But.

Now, I go to the digital ocean panel, to the "backups" tab, and I read:

    
    
       "Pricing is set at 20% of the Droplet's monthly cost (e.g. It will cost $1/mo. to enable backups for a 512MB Droplet)."
    

And I can only think: facepalm.

It's not only the price of the implementation time, that me and this other
person with ruby/dropbox have spend...

Even if it looks like "you get a _gratis_ backup solution", really it may be
more expensive (because of all the network bandwitch)

Well, at a side of the fun of implementing your own backup/restore scripts,
you get:

* provider independence (if all DO is down, you still can restore from gmail or dropbox)

* no periodical costs (but remember the implementation cost maybe bigger the first time, then is just reuse and edit a few variables).

* Security (you control how and where the data goes, otherwise you can save and restore from the provider, but implementation, management, internal policy, budget, team or technology changes, etc around your data is up to you).

But, I will never call again my solution a "poor's man" solution, because
counting implementation time and network traffic, is much more expensive than
$12/YEAR.

Edit: formatting a list.

------
Adirael
Previous post was submitted here:
[https://news.ycombinator.com/item?id=6438761](https://news.ycombinator.com/item?id=6438761)

------
jensenbox
I left them long ago after super crap support.

No thanks and good luck.

~~~
veemjeem
Most online services that are under $5 a month rarely come with support. I
feel bad for startups that charge so little per month because usually the less
you charge, the more demanding the customer is. Virtualized servers tend to
require more support than say, online accounting or email products.

People who go to Digital Ocean are going for cheap VPS solutions. If they had
money to blow, they'd probably be with amazon or rackspace.

------
j_baker
> And now after nearly 10 hours or so i still haven’t heart from DO. Also now
> that the HN topic is not on the front pages no one is getting updated about
> the situation. That’s why i wanted to write this post.

I won't speak for everyone, but I can personally live without having updates
on the OP's Digital Ocean drama on my HN homepage at all times.

------
outworlder
> DigitalOcean requested me to send a paper which authorizes them to disclose
> the information in public. It was midnight here in Turkey and i was asleep.

> And now after nearly 10 hours or so i still haven’t heart from DO

What about timezone differences? You were asleep, they could be, too.

~~~
sdogruyol
That's a possibility. But at this time yesterday they were really really
active.

------
moneyrich2
that's rough

they obviously only care about conversion and not validating the user before
hand in (any) way, so brutal.

From verizon I got a similar answer they couldn't discuss my own account with
me after me giving them my social and all that - wtf. i hung up and tried
another service rep and they gave me too much of my own personal info with out
verifying any of my personal details and anyone elses' on my plans, which is
kind of scary (Wtf)

also i really really like the idea of daily backups to dropbox, with like 1TB
of backups i need a solution, and this seems like a great one. thanks for
coming up with that

~~~
phildini
you're going to back up 1TB of data to dropbox?

------
pekk
Funny that DO's public response was to demand permission to discuss the case
_publicly_ , when apparently they had not discussed it with the customer
concerned!

~~~
veemjeem
I think they have. The customer in question simply didn't put it in his blog
post. I feel as though the customer may be presenting the facts in such a way
to make DO look bad. I don't see anywhere in his blog post about his DDOS
attack. His server was shutdown because it was running a DDOS attack. My guess
is that his server got owned...

People get disproportionally mad when their server gets shutdown, for a
service that charges $5 a month for a VPS. Most people probably wouldn't get
angry at Netflix if they cancelled their $7 a month account, but shutting down
a VPS feels like a personal attack.

If you pay $5 a month for a service, don't expect a customer representative to
hold your hand and debug your server troubles.

------
bliti
I've also had a bad experience with DO in production. Moved to Webfaction and
have never looked back. They are just excellent in every which way.

------
outworlder
By the way, someone should watch this thread and start saving all posts by the
OP. Last thread he deleted a lot of them.

------
adrianlmm
I'm a DO user, I created my account 2 months ago and I'm a happy customer.

------
orenbarzilai
Isn't that a major violation of their SLA? Have you consulted with a lawyer?

~~~
veemjeem
Even if they violated their SLA, the most DO could do is offer a $5 refund,
which isn't that big a deal. It's probably too costly to hire a lawyer over a
$5 server.

Also, I don't think the SLA would include things like getting hacked, or
running illegal services.

------
dgdkfjghkdfhkhg
Never heard on DO but will use them for a project soon - looks great

~~~
debian3
Indeed, never though a $5 provider would go that far for a noob on their
network. Looks like their monitoring system is good too, took them only a
couple of minute to take him down, which is good.

------
kbar13
is this... a joke?

