
Show HN: VPS Comparison – Automated tests to compare VPS by yourself - joedicastro
https://github.com/joedicastro/vps-comparison
======
smcleod
I have servers on Vultr, AWS, Azure, physical hardware in data centres and a
few others. The physical servers are by far the fastest obviously, but when
looking at the VPS / 'cloud' services Vultr is by _far_ the fastest and most
reliable servers out of the lot, a close second would be Sitehost (AU/NZ
Only). On the other end of the scale, the least reliable services we've
experienced is easily those hosted on Azure where constant network issues,
unreliable performance and random unexplained outages are frequent. Rackspace
used to be good, going back say 2 or so years ago, then it felt like they were
struggling although their technology far exceeded that of AWS - Amazon's
marketing and targeting of management level sales seemed to outweigh and
caused them to buckle.

~~~
joedicastro
Thanks for share this!

You know, this is the kind of of knowledge that we all like to have, but even
then, all we have is your word (I don't have any reason to doubt it), what I
tried to share here is a way to back all of that (at least performance,
reliability is a different kind of monster) with numbers.

~~~
jazoom
Let me add my word. Vultr is powerful and reliable. Recently they cut their
prices roughly in HALF. They were the best (IMHO) before that. Now it's not
even a competition.

~~~
h1d
Linode seems to provide pretty much exact same plans, but their control panel
is a bit too geeky (you need a few extra steps to launch a OS because
launching an instance means powering on an empty machine without a disk),
touching Vultr control panel feels pretty easy.

I just hope they both get block storage deployed on all their regions.

~~~
joedicastro
Linode seems to be working in a new version of their control panel, I hope
that they do a great work, because for too much people the current one is a
entry barrier, even when they do a very good work.

------
Elect2
What I cared the most about is CPU limit. On all these vps if you keep 100%
CPU usage they will treat you as a noisy neighbor. But none of them clearly
tell you the baseline of CPU resource that "safe" to use.

~~~
throwasehasdwi
This is why I hate using AWS and similar virtual servers. I use them for quick
dev stuff then get a non shared server. Massive datacenter scaling doesn't
save that much money. The money in cloud is made from huge over-provisioning.

Most providers will artificially limit you below the published limits if you
use "too much" CPU or internal network bandwidth. You think that 1Gbps link
between your machines isn't shared? See what happens when you try to use it
full tilt for a couple days.

~~~
extra88
> The money in cloud is made from huge over-provisioning.

Did you mean "overselling," selling more than your infrastructure could
support if every customer used all you sold them?

------
notamy
> I have serious doubts about the OVH’s and Scaleway’s unlimited traffic,
> seems more marketing strategy than real to me (joe di castro).

Anecdotal, of course, but I have an application on OVH (on the USD $3.50/month
plan) that pushes/pulls >10TB/month - coming close to saturating the 100/100
connection at times - and I've not heard anything from OVH about it being an
issue.

~~~
zzzcpan
Yeah, they have no problem with that. But I do recall years ago when a few
customers were running hundreds of seedboxes OVH changed ToS and asked them to
leave.

~~~
dawnerd
It seems to depend on if you generate complaints. If you're not causing
problems for them they won't likely care. Goes for most seedboxes, actually.

~~~
joedicastro
That, I no like it. I understand it, but not like it.

It's like being at their mercy (or other clients) or like if the rules were
written in the shore's sand, you're well until the tide rises. I would prefer
clear lines into this kind of things.

~~~
tw04
So you'd prefer them to what? Go to court so people can keep seeding illegal
torrents? They'd never win in court and it would be business suicide.

~~~
joedicastro
I was not referring here about the "activity" per se, I was talking about
quotas and percentages of use. No matter for what kind of activity you are
going to use the service, I would like to have clear from the beginning what
limits of use I have.

Another thing completely different is the permissibility about certain illegal
activities or loopholes in their servers. That's not my concern, that's only a
question of moral/ethics/legal between the client, the provider and the
authorities.

~~~
tw04
There aren't any limits. I've run my port at 100% for a full month and they've
never batted an eye. The only time I've heard of anyone getting the banhammer
is for repeatedly getting abuse notifications due to illegal torrenting.

------
Sami_Lehtinen
I've been very keen with [https://UpCloud.com](https://UpCloud.com) at first
their servers don't seem to be cheap. But when you factor in the actual
performance, those do provide a good on bang ratio. So far I don't have any
complaints about UpCloud. But it's not the same story with OVH. UpCloud didn't
throttle the disk I/O, even it was truly insane, due to a software bug. That's
what I recommend, if you're looking for good instead of just cheap. UpCloud
doesn't oversell resources, so full CPU / RAM utilization is totally
acceptable. Also VPS resources are guaranteed and performance is pretty much
constant.

Scaleway is actually quite good when you factor in the performance / price
ratio with it. (VC1S) But that's not something I would recommend for
'production use'. Yet, it's very nice for hobby projects.

All systems are under constant monitoring. CPU, RAM, Disk I/O & Networking.
Unfortunately the data collected isn't something I can publish.

Btw. OVH SSD VPS 1 does provide IPv6, yet it's not being tested. Why? - Yes,
you'll need to manually add the routing once to enable it.

~~~
joedicastro
Thanks for the information, it's very helpful.

Yes, I know about the OVH plan and IPv6, but in the hurry and in the middle of
all the tasks to accomplish I left behind a few things. I'll keep improving
it. Thanks!

------
lvh
This is a great set of tests. Once you move out of the side-project phase and
start turning stuff into a business, you probably care about security features
too. Unfortunately, most VPS providers don't have a great track record there.
With the big Cloud providers, you have tools like AWS' VPCs and security
groups to help fence off parts of your app from the rest of the world.

In a perfect world, you wouldn't care so much about network reachability and
care a lot more about, say, mTLS or at-rest crypto. But as a young company,
managing that probably isn't the best thing to spend your resources on right
now. Much like Google's BeyondCorp: just because you can expose some apps to
the internet with years of work and millions in resources doesn't mean that a
VPN or a SG isn't a pretty good idea for now -- if nothing else, it'll do a
pretty decent job protecting you from the background radiation of the Internet
:)

~~~
joedicastro
Thanks! Great comment, I agree.

------
vldx
I was asked recently (2017) by Linode to provide scanned my credit card (after
paying, then eventually refunded). The reason - fraud prevention, whatever
that means. Certainly leaves bad flavor and decided to go w/ DO.

~~~
scrps
I encountered the same thing (also in 2017) with the addition that they wanted
front and back scans of a government-issued ID as well.

Having to pay with a credit card before you can use paypal didn't earn them
any points either. Suffice to say I did not go with Linode.

~~~
JohnTHaller
PayPal's fraud detection sucks, so I can definitely understand companies
wanting to verify/charge a credit card directly before accepting PayPal.

~~~
joedicastro
Yes, probably is one of the reasons why they do those checks. I suppose also
that depends a lot on the kind of plan that are you going to use, probably
they do more checks for more expensive plans.

------
sjs382
Half-baked startup idea: because VPS performance is so dependent on the VPS's
neighbors, create a service that spins up a bunch and then compares them on
demand. Keep the highest performing one. Charge your users to perform these
tests.

~~~
NKCSS
You mean the 'Netflix Model'? This is what they do all the time on AWS; when
needed, spin up a bunch of VM's; test performance, drop the ones that perform
worse and use the better ones.

~~~
tuananh
do you have any source for this?

------
RX14
I'm a little confused as to why comparing single-threaded performance (for
scaleway) is more "fair". Surely you should compare both, to allow people to
compare for single-threaded and multi-threaded workloads.

~~~
joedicastro
I understand you, but is more fair in the sense of comparing apples to apples
if you are going to do one test of a kind. Currently all of this tests as a
batch job take several hours, doing more tests, means more time and more job
for me. Specially that when I took that decision I didn't gather all the data
automatically and I have to collect all of that by hand and put it on the
tables. There is a reason for all. Now that I have (almost, I have to add 1
more) all the tests automated and I collect all the data with a python script,
I can start to think to do all kind of "alchemy" and improve the comparison.

Thanks for you feedback. I'll add this as an issue into the repository to
improve the comparison.

------
neom
I am not easily impressed when it comes to vps and cloud stuff, but this is
very impressive. Kudos. Curious why you didn't include ec2?

~~~
merb
ec2 is different however lightsail would be comparable. sadly there is still
no instance outside US.

~~~
neom
Why is ec2 different?

~~~
merb
you can combine different network adapters/storage options and instance types
and also create your own virtual network, with various other options. while
these VPS providers only allow a specific amount of storage for a specific
type of instance and also have a included traffic option while ec2 does not
have that. I mean more and more VPS providers have an API, but they still
don't have the customizability and services like a cloud has, yet they are
preferable in many use cases.

~~~
h1d
DO has block storage, Vultr has it on a couple regions and Linode seems to be
preparing for that too.

------
porker
Would love to see Joyent included. Silly pricing, hopefully performance that
justifies it.

[https://www.joyent.com/pricing/cloud/compute](https://www.joyent.com/pricing/cloud/compute)

~~~
joedicastro
I understand you, but as I said in another comments, I would like to expand
the comparison, but in the end is a question of time and costs. If I find a
way to sustain the costs, or the providers itself offer me a way to test their
machines, I have no problem to expand the comparison. But in the meantime, I
share the tools because I knew from the beginning that I only have limited
resources. And after all, reproducibility, independence and transparency are
the keys IMHO to have a trustful comparison.

------
tedmiston
Curious to know if anyone uses Vultr whether they actually hit 100% uptime, or
if that's more marketing and they just issue credits for the SLA regularly.

~~~
lima
100% uptime is an unrealistical claim. There are so many things that can go
wrong with a VPS (that is, by design, only ever running on one node at the
time). The probability of that node failing or a messed up route taking
offline all routers at once is low, but it's not 0%.

In order to even come close to 100%, you need a multi-region distributed
system and even then, so many things can go wrong.

Look at their outage credits
([https://www.vultr.com/sla/](https://www.vultr.com/sla/)) if you want to get
realistical percentages.

Even Google Cloud Platform - which has multiple layers of redundancy at every
level - does not claim 100% uptime:
[https://cloud.google.com/compute/sla](https://cloud.google.com/compute/sla)

Disclaimer: I work for a competitor who would never advertise 100% uptime -
it's a known fact that shit happens

~~~
joedicastro
I totally agree with this, after all is common sense. Even if all the hardware
were perfect, and all the software were bugless, you would still have to deal
with physics and with those imperfect and error-prone machines that set up all
the things called humans.

------
thinkMOAR
Oh that is a nice overview, perhaps regarding IP reputation, include an
overview which of those providers has the most ip addresses on RBL lists too?

Performance is nice, but if you are already on (a) blacklist (s) from your
start.. :) since most of those providers, have a lovely reputation in regards
of compromised machines being abused in botnet and other malicious
activities... i think it might be an interesting index.

~~~
joedicastro
Interesting idea... it didn't though of that for sure. I thought of talking
about how permissive the providers are with certain type of contents (like
legal adult content), but it is an information not easy to collect and so I
went with the official legal terms/ToS instead.

I'll have to think about that, and how collect that information. Thanks!

------
vorpalhex
Extremely useful! I'm currently on DO (and I love them) but I've been eyeing
vultr and this definitely pushed me to give them a go.

~~~
joedicastro
Thanks! I'm glad that is helpful to you.

------
mysterydip
This is just what I needed! I'm downsizing from a colo and looking for just
this sort of data. Vultr seems really appealing especially with the ability to
upload my own iso (My preferred linux distro isn't one of the common host
offerings). I would love to hear people's experiences good and bad with them
as a provider.

~~~
h1d
Out of curiosity, which distro do you need to use?

~~~
mysterydip
Tiny core. I use debian or ubuntu where required, but prefer the features and
minimal footprint of tiny.

------
dijit
There's a VPS provider whom I am infatuated with[0] and I'd love an easy
objective way to compare them with Amazon/Linode.

The only concern is that they only have 2 datacenters and they're both in the
Netherlands.

[0]: [https://tilaa.com](https://tilaa.com)

~~~
NKCSS
Just wondering; why do you prefer them over TransIP for instance?

~~~
dijit
Ease of access to engineers, transparency when it matters.

And I've never had a node go down, that helps.

------
HugoDaniel
It would be nice to include hetzner :)

~~~
joedicastro
I put you here the same reply that I gave to another user, because the answer
is essentially the same. Anyway, thanks for you feedback.

 _I understand you, but as I said in another comments, I would like to expand
the comparison, but in the end is a question of time and costs. If I find a
way to sustain the costs, or the providers itself offer me a way to test their
machines, I have no problem to expand the comparison. But in the meantime, I
share the tools because I knew from the beginning that I only have limited
resources. And after all, reproducibility, independence and transparency are
the keys IMHO to have a trustful comparison._

------
corford
Very nice. One other datapoint that could be useful for some: Terraform
provider support (and how capable the provider is). Terraform has providers
for Digital Ocean, Scaleway and Lightsail but nothing for Linode or Vultr.

~~~
joedicastro
That's a good idea, I suppose that would have to do the same with another
tools like Apache libcloud or Ansible itself.

I'll add this suggestion as an issue to the project. Thanks!

~~~
tpetry
A terraform provider for Vultr is the really missing part for seitching from
DO to Vultr.

~~~
joedicastro
I suppose that is the same with another tools and providers. The community
size and the work that the providers put on this makes a big difference IMHO.
Another reason to share this in the comparison. Thanks!

------
rkv
Would be nice if the "Notes" section referenced the matching table entry.
Example:

>Send root password by email Yes No[1] No

>Notes:

>1\. DigitalOcean sends you the passwords only if you don’t use SSH keys, in
plain text.

Otherwise, great comparisons.

~~~
joedicastro
Thanks!

Yeah, I agree, that would be nice, but as I automate the creation of those
tables and cut/paste them every time that I run a set of new tests, to do that
would require a lot of manual job to me. That's the reason why I don't do
that, to keep it manageable.

------
mtw
Thanks - I found the framework performance towards the bottom very
interesting.

What about including other popular frameworks? Ruby on Rails, Laravel,
ASP.NET, symfony, express.js

~~~
joedicastro
Thanks to you!

Yes, there is always that possibility, is a question only of time, more tests
implies more time to complete the full batch. And of course, more time to
implement it.

I have the intention to keep improving it, and thanks to the Ansible tags, the
tests can be run individually and independently (I tried from the beginning to
make them as atomic as possible for this same purpose). Thus is very easy to
extend.

------
SmellTheGlove
Everything I've wanted to know but been too lazy to do on my own. This is
excellent, thank you for sharing.

~~~
joedicastro
You're welcome! Thanks!

------
proyb2
I saw Lynis is use for audit and harden VPS, have anyone use it? I haven't
harden my Linode.

~~~
joedicastro
Well, it's very useful, and in continue development.

In the end you have to know what you are doing, but it will point you to very
useful tips and show some warnings that you can look to improve the security
of you server. But in this, I still don't know two SysAdmins that concur on
the best way to harden a server. There is a lot of things to consider and
there are very strong opinions on how to do certain things.

But without doubt, it's a very useful tool, even if you know what you are
doing you can use it like a checklist to know if you are forgetting something.
It's not perfect (nothing is, and less in this field) but I can only recommend
it.

------
nodomain
Very impressive reference. Would be interested in ec2 as well. And I think
"datacenter in the EU" does not necessarily mean "bound to EU law" since the
parent company is in the US, right?

~~~
efdee
If the data is stored in the EU, it is bound to EU law.

------
halfeatenpie
Great start to a comparison review.

I've used all the service providers available and these opinions include my
historical experience with them (I've used all providers for many years, my
account with DO is since 2013). Most of my service uses were focused in those
service provider's North America locations, Asia locations, and Oceania
locations. Note I did do a short comparison between DigitalOcean and Vultr
back in 2014 here: [https://vpsboard.com/threads/digitalocean-vs-vultr-the-
asian...](https://vpsboard.com/threads/digitalocean-vs-vultr-the-asian-face-
off.3688/)

Most service providers here are fairly decent.

From my experience, OVH's KVM VPSes weren't worth the time, but their VMWare
VMs were absolutely top notch.

Linode has had two major security events over the last few years, one of which
was in early 2016 which included release of credit card information. However,
they also offer paypal payments now and are fairly top notch in terms of
performance. (They've taken approaches to fix their problems from 2016 with
this blog post here: [https://blog.linode.com/2016/02/19/security-
investigation-re...](https://blog.linode.com/2016/02/19/security-
investigation-retrospective/) ).

DigitalOcean is fairly reliable. Their Singapore location early on had regular
packet loss events, but they've been shaping up pretty nicely over time.

Vultr has been a major problem for me. Vultr for a while (on their storage
instances in Japan) had terrible I/O which took a solid two hours just to
install Ubuntu 14.04. When contacted support, they stated they "stopped a
noisy neighbor and to retry" but no real improvements were seen.

We had a service running on a VC2 that irregularly required a large amount of
CPU power (higher than 0.5 load on a 1 CPU instance for a period of 1 hour a
night). The same task was completed faster with VMs from DigitalOcean and
Linode. But the biggest problem I've experienced is their network.

For the longest time their Japan location was single-homed and routing was
funky. We'd regularly see any traffic going outside of Japan being routed back
through their Los Angeles location, so adding 300+ ms latency for any of our
users in Korea, China, and anywhere in South-East Asia. Today it's much better
with 40 ms latency from their Japan location to our Korea client, and a solid
80 ms latency to our Singapore client.

A colleague of ours had very important clients who required 100% uptime (which
Vultr agreed to) but over a period of a month had regular packet loss events
or network loss (regularly 40 minutes per event and around one event every few
days monitored via external monitoring systems such as uptimerobot,
statuscake, and a GCP server running Icinga2). Didn't receive much help from
Vultr's engineers nor could they find the issue. They ended up moving to
Google Cloud Platform. They did end up getting SLA credit from them but no
reason to have SLA credit when you're not on their platform anymore.

I regularly return to Vultr every few months to every year or so to see how
it's going and they've come a long way with the services they're offering. I'd
recommend them for a dev environment and testing over a period of an hour a
day or so. However, I would caution with production systems. Your mileage may
vary, but I'd suggest reviewing the quality of the product in addition to the
price tag per resource.

~~~
lilbobbytables
It's truly helpful to hear experiences first hand across different providers -
especially so specifically.

I'm really interested in your experience with OVH - how long ago did you last
use it, and what problems did you have?

I've used OVH successfully for various unimportant things for a while now. I'm
about to setup a prod environment on OVH for a side project that we anticipate
getting some good initial traction. So, not trying to spend a ton, but willing
to forego the conveniences of AWS the best cost to performance ratio.

OVH has made a big Openstack push, it seems. "VPS Cloud n" series is KVM
(Openstack), with NVMe drives, using Ceph (Openstack) storage, and a moderate
99.99% uptime (same as DO).

It's quite attractive at their prices - so I'm trying to gather as much
relevant intel as possible on others' experience with them.

~~~
skrowl
I have multiple dedis (not VPSs) at OVH Canada (Windows on bare metal as well
as Proxmox with Linux and Windows VMs). You'll want multiple dedis there too,
if you use them.

Occasionally a server will hard lock and when you got into the management web
app and ask for a reboot it will say that the system is not responding and
they have to send a tech to go physically power cycle it. When that happens
(and it will, it's happened to ALL of my dedis there at some point) you're
down for 30-90 minutes.

They provide remote reboot and a java-based ILO which can solve most weird
issues, but the hard lock stuff continues to confound us.

~~~
lilbobbytables
Yikes. That's what I want to avoid. Since I'm going VPS, I'm looking at
RamNode now instead.

The CEO is actually really nice and accessible, and their customers seem to
really vouch for them.

------
hathym
Nice, I recommand that you include the 1€ arubacloud vps.

~~~
joedicastro
Thanks! I would take into account.

------
mxuribe
Really helpful details; great job on this!

~~~
joedicastro
Thanks!

------
solotronics
hey if you would like a Softlayer / IBM Bluemix instance to test I would be
happy to provide

gene.gaddy@ibm.com

~~~
joedicastro
Thanks for the offer. I'll take into account.

------
raybb
How long did it take you to write this up?

~~~
joedicastro
More than two months in my spare time. To collect the "static" information,
and create the Ansible roles and python scripts to collect the "dynamic" info
(the results from the benchmarks) and finally compose the final document.

I lost a lot of time waiting for the benchmark tests to finish ;-)

------
gketuma
Nicely done.

~~~
joedicastro
Thanks! Very kind of you.

------
slitaz
Nice and thorough work!

~~~
joedicastro
Thanks! Very nice of you.

------
id122015
I thought linode and digitalocean were clouds not VPS

~~~
vxxzy
cloud is just a fancy word for Virtualization.

~~~
neom
That's not strictly true, most folks agree that cloud services typically have
an "on the fly" type elastic nature.

~~~
vxxzy
At the end of the day there exists a hypervisor... Virtualization.

~~~
Danihan
Yep, afaik Amazon ec2 is still Xen.

------
throwit2mewillU
Strasboug (DE) - not for quite some time.

~~~
joedicastro
You're right! another people pointed to same error as well and was fixed in
the repository with a PR.

I'm sorry for that, was a huge mistake, I hope that nobody have been offended
by that.

