
Ask HN: Is Digital Ocean safe enough for production? - iamadmin
I am in the late stage of developing application using Firebase, Google App Engine and some Compute for my application backend, primarily because it is less pain than managing own servers. But the post yesterday &quot;Firebase Costs Increased by 7,000%!&quot;[1] made me rethink my decision as I can not afford any surprise bills now. So the cheaper alternative would be to use Digital Ocean, but is it production ready and if anyone using it successfully for running a large production app ?
 [1]https:&#x2F;&#x2F;medium.com&#x2F;@contact_16315&#x2F;firebase-costs-increased-by-7-000-81dc0a27271d
======
cdnsteve
Keep in mind Digital Ocean doesn't offer managed database instances or
services. So all the OS patching, security updates, deployment management, web
server + DB configuration nuts and bolts and scaling will be on you, taking
more time. You may save some cash but you will lose time.

If you're very comfortable managing servers and your goal is to keep costs
very low then it go for it. If time is more important (likely) then stick to
what you have, then migrate only when required and you have a real business
case for the move.

~~~
infecto
This is the one thing that keeps me using AWS for personal projects. Sure I
might not be getting as powerful of an instance for the cost in some cases BUT
I really really love the power of RDS and some of the other services AWS
provides.

I know DBs are not the hardest things to setup but in the back of my mind I am
always concerned about the security of my DB setup along with any tuning.

------
txutxu
Hello, I'm using it for not too big stuff.

Your biggest concerns, maybe:

\+ Bandwidth limitations (the $5 instance has 200m/s I think)

\+ Lack of additional services you get in other clouds (email sending with AWS
SES, for example). It's ok if you just need instances, shared storage, shared
IPs, availability zones, backups and an API.

\+ Response capacity. I'm not sure if it will be better or worse, but I think
it's not the same team, architecture, core infrastructure, etc compared to
your current providers... this may require some research to understand if it's
a concern for you.

\+ The "private" IP of the instances, so you can talk with instances in the
same region, is in a shared network with other clients (or at least, it was
the last time I did check).

\+ Some services (balancers, ipv6) have been released recently, so maybe they
are not so "mature" as in other providers. Didn't test any of them here,
sorry.

\+ Some times, the kernel version in the repos, and the kernel version
available for the instance from the API or web interface, are out of sync (at
least in Debian instances).

Said that, the instances and the service, perform as expected.

You also have "cloud init" and an API in digital ocean, so you can hook your
configuration management system, to automate actions, scaling-up, scaling-
down, etc.

They notify for planed maintenance.

As in every cloud provider, sometimes you spin up an instance, and you see lot
of old traffic and requests (crawlers, bad dns's, etc). This is no different
here.

I think you should make a POC with a pre-production environment... this is
cheap. How big is your app?

~~~
iamadmin
Thanks for the great insight. App would serve close to 25k customers initially
and will grow eventually.

------
invisiblea
I use DO for several one or two box production sites (LEMP and RoR stacks),
and have minimal complaints. I'm a big fan of scaling up the CPU without
having to also increase HD size (which is a Linode requirement).

The biggest issue we've faced was their use of Google DNS. A couple of months
back, Google DNS had an outage, and all our calls to external services semi-
silently failed. Because this wasn't technically a DO service outage, it
wasn't reported by them, and because we didn't know they used Google DNS we
didn't trace the issue to the outage. We could have done more, but they could
have also been more transparent.

Overall, we are starting to move more projects to DO from Linode, because I
feel Linode's previously excellent support has slipped dramatically (customer
for at least eight years). We also leverage a lot of AWS services
(particularly S3 and SES).

~~~
iDemonix
What have you done to mitigate the DNS risk?

~~~
stevekemp
Simplest thing there is to split zones across two providers - e.g. Route53 +
DNS Made Easy, etc.

(DO do have their own DNS API, but it is terrible to program against.)

------
BJanecke
What does that mean?

Go read
[https://www.digitalocean.com/business](https://www.digitalocean.com/business)
and [https://status.digitalocean.com/](https://status.digitalocean.com/) if
that doesn't put you at ease send them an email.

However it shouldn't matter. Droplets are essentially just on demand "vps's"
most of the operations work is done by your operations team(in this case I
imagine this is you). Where your hosts(machines, containers, vps's whatever
you want to call them) are shouldn't matter very much, what is important is
that you have repeatable infrastructure, so that when you spin up a new VPS
anywhere(AWS, DO, Vulcan, Your Garage) you can get up and running again fairly
quickly.

[edit] Added status page

------
WrtCdEvrydy
If you're going to deploy on Digital Ocean, I recommend you rebuild your app
within Flynn (so you can migrate if anything goes wrong with your bill). Take
a backup of your cluster (or the individual apps) after you are done and if
you get a surprise, just rebuild your cluster somewhere else from a backup
file.

Your software is too precious and your small startup is too precious to be
married to someone who just thinks of you as "a customer". DO has never done
me wrong (I have used their platform for years) but having a plan is handy.

~~~
tyingq
This is great advice. I'd go one further and select some DO competitor to
deliver some part of your app. Even if it's just the backend for static assets
that you're pointing a CDN at.

Then, if you run into issues with either DO or the competitor, you already
have an account and familiarity with how things work.

Also, don't host the DNS records at either place. Put them somewhere else so
that your DNS records aren't held hostage if there's an account issue.

~~~
mintplant
I can recommend [https://dns.he.net](https://dns.he.net) for this. Free, no
fuss, and backed by a major Internet backbone provider.

------
paukiatwee
Not specific to DO but here are some checklist might helpful to you.

1.1. Elastic IP - Allow you to remap IP to another instance, this is very
important for HA where you can bring up another server and remap IP to new
instance.

1.2. Alternative to elastic IP is load balancer

2\. Bock storage - Allow you to use persistent storage with redundancy.
Important to keep data safe when some instances down. You can bring up new
instance and start using existing block storage.

DO support all of the above so DO is good enough for production usage. If you
need latency sensitive services like RDS, maybe you can consider AWS. I know
some people use RDS with compute on other provider like DO.

------
marcus_holmes
> primarily because it is less pain than managing own servers

I would take a good, long, hard look at any decisions you've made on this
basis.

You haven't avoided the pain, you've deferred it. Would it be better to deal
with that pain now when you have no users and no critical data, or later when
scheduled downtime is a major issue?

~~~
corobo
Another way to look at this might be - do you want to deal with this now as a
sole dev/ops/etc person of many hats or later, when you can hire someone to
look after this?

~~~
codefined
Also, for lots of things there is really no need to host your own servers.
It's going to cost more and be less reliable unless you get to a much larger
stage or have a lot of computational requirements.

------
thomk
We are using it for a series of sites that get about 5K unique visitors/day if
that helps.

------
gabemart
I have hosted my side projects at DO for four years, and am very happy with
their service. I've served half a petabyte of data to 5 million users in that
time. I have consistently been happy with their documentation and API. I
haven't required much customer service, but the customer service I have had
has been adequate.

When I've done the calculations for my use cases, I've found it would be
significantly cheaper for me to have a second, redundant copy of my entire
infrastructure at another provider as a failover in the event of a prolonged
DO downtime than it would be to host my side projects at Amazon or Google.

------
jaredandrews
I recently deployed a wordpress site, using Trellis[0], on to Digital Ocean.
The site has a pretty low traffic volume, but so far it has been working
great. Considering switching all future sites I work on to DO but I'll be
interested in seeing how this first one turns out.

[0] [https://roots.io/trellis/](https://roots.io/trellis/) \- If you have to
work with Wordpress at all I strongly recommend you give Trellis and the other
Roots.io projects a try. Coming from more "sophisticated" tools to Wordpress
was a real struggle until I found roots.

------
noir_lord
It is a lot more reliable than it used to be (back in the days of "welp, we
are taking a router down in the middle of the day again").

I still run all my production stuff on Linode because I've _never_ had any
unplanned downtime since 2009 (when I started using them), they've had issues
over the years but always stuff away from the actual core thing I value which
is _keep this online_.

Since they underwent a massive DDoS a while back they've really hardened their
systems up and put in their own links to the backbone all that jazz.

~~~
charlieegan3
So you weren't effected by the DDoS in January 2016?

~~~
noir_lord
No but more out of luck than anything, I'm UK based so my linodes are all in
London region, it didn't get slammed the way the US ones did.

------
cookiecaper
It's all relative, of course. I have only used DigitalOcean for side projects
that served at most thousands of human users per day, and I didn't have any
real issue with their stability. In the general case, I would say that DO
works fine.

Since it sounds like you haven't even launched yet, DO should have no issue
getting you started.

------
JohnTHaller
Digital Ocean's internal support infrastructure has caused us a couple
headaches over the years. They appear to have fixed some of these
organizational shortcomings over the last 2 years. Still, you should not
expect the level of support you'd get from an expensive managed provider from
a self-serve more-automated startup.

About 3 years ago, our instance got shut down over the weekend because a
spammer in Germany reported it for serving an 'infected' file while attempting
to advertise their MX scanning service. Said file had a false positive in
ClamAV (common for Windows EXEs) and only ClamAV. Digital Ocean shut down the
production instance with no warning and then told us about it via ticket. It
took hours to get a response and when we finally did, the instance could not
be restarted (DO at the time had a bug where the kernel indicated in their
admin had to match the kernel in the image or it could fail to start). Total
downtime was 2 days if I recall correctly. The security spammer took out our
instance 1 or 2 more times before DO finally either blacklisted them or
correctly noted that it was a likely false positive. I got a few months credit
for this issue at the time.

About 2 years ago, DO had an issue where they shutdown networking to our
production instance because they detected a "Brute Force attack" originating
from that instance. The detection was due to an increase in downloads because
we'd released a new version of our software coupled with, yet again, a false
positive in ClamAV (and only in ClamAV)... though in a separate copy of
software on the same server that had been released a month prior. This also
happened on a weekend and took hours and then a couple days to resolve due to
the aforementioned bug at the time of DO instances failing to start due to the
kernel in use within the Droplet not matching the kernel indicated in the
admin.

In both of the above instances, a single support person killed the instance
without pinging the customer first. This is to be expected from any mostly-
automated lower cost provider. Contrast this to a Rackspace managed cloud
instance where they will first contact the customer when a detection occurs or
a false positive shows up in an antivirus scan. Of course, that's comparing
our managed cloud instance at Rackspace to a self-serve low-end droplet at
Digital Ocean. Why the difference? Likely because you'll pay between 3 and 5
times as much for the same level of server between the two providers.

As a result of the above, I kept our production servers at Rackspace but
continued to use Digital Ocean for secondary services. We have backups for
those services on DO and the ability for our product to fail-over to the
backups automatically should an issue like this occur again. We have been
considering moving more services to Digital Ocean as they appear to have
ironed out the kinks of their first few years of operation now.

~~~
nitrix
It's worth noting that when a similar issue happened to me and went on their
"unofficial" IRC channel to raise awareness and ask questions; I was faced
with arrogant staff members that didn't want to restart the servers or
mitigate the DDoS attack. I was banned on the spot. Then the Customer Service
tickets were ignored in a similar manner too. I had a growing startup at the
time. Needless to say I've been doing everything I can to destroy their
reputation ever since.

------
tmaly
I have used DO for project for several years now. You do have to do all the
maintenance yourself. Upgrading between Ubuntu 14.04 to 16.04 had some issues
that forced me to take the site down for half a day. Other than that, it has
been smooth sailing.

If you know how to do all the sysadmin stuff, I would say your fine in terms
of using DO for production.

------
awinter-py
I think GCE has better support for encryption at rest than DO.

On the plus side, DO will make you jump through fewer hoops than GCE to get
started. GCE had UX and logic bugs in their automated billing system when I
used them last that ended up affecting uptime.

------
charlieegan3
I've had an side-project instance running for over 2 years without issue.

------
aibottle
No, they cut your service when you get too much traffic!!!

~~~
pmontra
Some references?

------
sandGorgon
the easiest way to do this is using Docker Swarm mode. all you need is a
docker-compose.yml and your dockerified application. you can have a quorum of
master/managers so that even if a DO instance dies, you will have your
requests routed to the failover.

------
notlambda
DigitalOcean is quite expensive compared to GCE AWS and Vultur, but to answer
your question yes it is. I ran a website with 2M monthly active users on their
5$ VPS

~~~
ek750
That's a bit surprising, can you expand on the difference in price between the
$5 droplet and GCE? Is it other costs not included in the monthly base?

~~~
notlambda
DO is more expensive with their higher end VPS. GCE is cheaper if you don't
use a lot of bandwidth

------
thiagooffm
Of course it is. They provide you hosting, since the very beginning of the
internet there has been companies providing this kind of service with success
and if you wanted and had enough time and funds you could even do it
yourself...

Now there's only virtualization on top of that.

~~~
lloydjatkinson
I'm 100% sure OP was looking for a yes no answer relating to uptime, how
reliable they are, SLA's, policies, security. Not an explanation of hosting
companies.

"Is this new make of car being manufactured safe to drive? How reliable is
it?"

"Yes of course cars are safe, cars have existed for over a century. Many cars
have been used since the beginning of cars being invented. You could make a
car too if you had the money".

See how ridiculous and unhelpful your comment was?

------
iDemonix
It's as safe as you are. I've been using DO for quite a while now, running
droplets for DB, Web, cache (mem/redis) and plenty of other stuff, no issues -
all downtime (maintenance) was minor and was pre-communicated effectively.

You can manage your own SSH keys, software updates, and you can enable 2FA on
your DO account. It's definitely safe enough, if you're safe enough.

Happy to answer any questions, but tbh their support/docs section is some of
the best on the web - I often find myself referring to some of their docs for
non-DO stuff in my day job!

~~~
riffic
It's all about operational practices. DO is a DIY IaaS platform, so if you
practice good operational hygiene you'll be successful. If your operational
expertise is lacking, you may have a bad experience with the product.

