
Digital Ocean Private Networking Is Not Private - jstanley
http://incoherency.co.uk/blog/stories/digital-ocean-private-network.html
======
ztnewman
I was about to express my outrage when I decided to actually do some research.
Their posts pretty clearly indicate it is shared private networking -
[https://www.digitalocean.com/company/blog/introducing-
privat...](https://www.digitalocean.com/company/blog/introducing-private-
networking/)

~~~
Someone1234
What a terrible name. So it is non-private private networking, great
distinction there DO. I wonder why people are confused..?

What about "Shared Intranet" instead. Why has the word "intranet" fallen out
of style? And why use the word private to describe something that is
inherently non-private by design?

I actually think the addition of this free inter-droplet channel is amazing
(and a potential massive cost saver), but the name sucks.

~~~
jtymes
Technically, I think they're correct with their definition[1]. However, they
should be more explicit with the "Shared" part and have some better
documentation rather than an announcement blog post[2], a community guide[3],
and a moderator comment on that community guide [4]. Especially in today's
privacy-conscious world.

Having said that, people who are concerned about privacy should always be
validating their setups themselves. The maxim of "trust but verify" comes to
mind.

[1]:
[https://en.wikipedia.org/wiki/Private_network](https://en.wikipedia.org/wiki/Private_network)

[2]: [https://www.digitalocean.com/company/blog/introducing-
privat...](https://www.digitalocean.com/company/blog/introducing-private-
networking/)

[3]: [https://www.digitalocean.com/community/tutorials/how-to-
set-...](https://www.digitalocean.com/community/tutorials/how-to-set-up-and-
use-digitalocean-private-networking)

[4]: [https://www.digitalocean.com/community/tutorials/how-to-
set-...](https://www.digitalocean.com/community/tutorials/how-to-set-up-and-
use-digitalocean-private-networking?comment=4486)

~~~
Someone1234
> Technically, I think they're correct with their definition.

I never said they weren't "technically correct," I said the name sucked and
confused people [0].

They could also just use "Data Center LAN." No confusion there.

[0]
[https://en.wikipedia.org/wiki/Privacy](https://en.wikipedia.org/wiki/Privacy)

------
jgwest
When setting up multiple VPSs connected by "private networking" with a company
like Linode, or Digital Ocean, or what-have-you, you need to assume that the
inter-VPS links are not secure. It's a little piece of knowledge that comes
with experience, hard for newbies to realize.

The first time you set up one of these clusters, you might follow one of
Linode or DigitalOcean's handy guides, where they might suggest i.e. a reverse
proxy server receiving (and decrypting e.g. HTTPS) inbound traffic, routing it
out to multiple worker machines, and a single backend database system. Linode
sells dedicated Load Balancers for the front end of exactly this sort of set-
up.

These guides almost _always_ fail to mention that the data is observable as
cleartext in the internal network. They ought to be reedited with big bold
warnings starting that these links ought to be secured. (Can Linode's Load
Balancers even secure these links?) Besides other eavesdropping customers,
there could potentially be little magic government agency plugs installed --
or the eavesdropping customers could be government security agencies
themselves. (Sorry, tinfoil, I know...)

OpenVPN connections are a lightweight, efficient solution. They're also
transparent once you change IPs from those of the virtual network interfaces
to those of the secure virtual network interfaces. Such a configuration is
still non-trivial, though, for someone configuring their VPS via a control
panel rather than the command line.

~~~
nyir
In your experience, is a single OpenVPN server sufficient on such a network,
or is it possible to have a fallback (server)?

I previously tried using a distributed VPN setup without a single main server;
that didn't work out so well however, mostly because the software was somewhat
unreliable.

------
scolson
Most data centers actively want the default private networking to work across
the datacenter with everyone. A better term would be non-routing. Rackspace
calls theirs "service net". These non-routing networks encourage people to use
cheap internal bandwidth (rather than bounce of an expensive leased line just
to come back in) and possibly offer a special service just to that data
center's customers. There are tons of companies offering sql and nosql
databases as a service, and any number of other offerings.

Most cloud providers recommend you configure your firewall rules keeping in
mind that other people could be on your internal network connection. This is
really a security best-practice anyway.

------
beering
This is basically a billing feature, so that you can use "free" internal
networking capacity instead of metered external networking usage. They are
just following what Linode and other VPS providers do.

I think part of the problem is that people confuse DO, who just rent out cheap
boxes, with more "full service" cloud providers. DO is partly to blame because
they keep marketing themselves as a cloud platform when really they're yet
another company who rents out servers with a few extra features.

~~~
nixgeek
Not that DigitalOcean currently bills for bandwidth over the external network,
nor have they done at any point from conception through to today.

[https://www.digitalocean.com/community/questions/questions-a...](https://www.digitalocean.com/community/questions/questions-
about-bandwidth-transfer-limits-and-billing?comment=27465)

------
cweagans
It's "private" in the sense that it's not going over the public network,
rather than private to your account. This was pretty clear when the feature
was announced. I don't know why it's a surprise now.

You still have to secure your server, even if it's only communicating over the
"private" interface. I usually use this feature for database servers, and lock
down the firewall so that only the web servers can communicate with the db
servers, and only via the "private" interface.

------
lbotos
This is the same way Linode's works as well. I'm curious, how many people here
thought that Private networks were walled to only droplets/nodes on the same
account?

~~~
vertex-four
Amazon's private network is private by default, I believe Google's is as well,
and I believe Azure's is. Digital Ocean is acting much more like a traditional
VPS company than a modern cloud computing company here.

~~~
res0nat0r
Well what you are describing is VPC (which for new accounts for a while now is
enabled by default), but VPC is software isolation which creates private
networks, but EC2 Classic is technically shared private IP space between all
customers. There is a big difference between the two.

~~~
justinsb
EC2 (and OpenStack) uses security groups to give you the best of both worlds:
shared address space so you _can_ route to other customers; but the firewall
means you have to explicitly allow the traffic.

~~~
res0nat0r
Yup EC2 security groups are great for ease of management. Group to Group
communication for something like load balancers to auto scaling back-end app
servers is great as you don't have to worry at all about tracking IP
addresses.

------
rubiquity
I don't understand the outrage. There's a clear distinction between "Private
Network" and "Private To Just My Servers Network." I've been a DO user for
years and I've never been confused about this. Anyone I've ever recommended DO
to has never been confused about it either.

I just automated the firewall and SSH tunnels/TLS certificates setup to
restrict access and encrypt communication between my droplets when necessary.

~~~
nirvdrum
I see a lot of people get tripped up by this via my work on rubber [1]. It's
probably selection bias, but most of them are switching to DO as a cheaper
alternative to EC2 and in EC2 your private network is "private to just my
servers network" unless you go well out of your way to change that. It also
seems like a pretty sensible default for a company that has a lot to lose if
malware is able to propagate through a datacenter by anyone willing to invest
$5 for the month.

[1] -- [https://rubber.io/](https://rubber.io/)

------
wongarsu
>In Digital Ocean's defence, nowhere do they state that their Private
Networking mode provides privacy

The word "private" is kind of in the name, I would have expected it to be
private (to my droplets) too.

------
artursapek
A private network has always meant "a network not accessible through the
public internet". If people are using something they don't understand their
outrage should not be taken seriously. Learn iptables, it's not hard to use!

~~~
kylec
Actually, I'd say it is, at least relative to the low barrier of powering up a
droplet and apt-getting a few packages.

------
sneak
It continues to amaze me how many people blame the user for "doing it wrong"
or being stupid or something when a company is found to be blatantly lying to
reduce their costs.

It's like the whole industry has Stockholm Syndrome or something from decades
of vendor abuse.

People, really: private means private. They lied. There's no two ways about
it. We didn't misunderstand because we're stupid, we misunderstood because
they were cutting corners and being intentionally misleading by using words
that have definitions different than the thing that was actually happening.

This isn't new with Digital Ocean. Their contempt for their customers is
palpable. A year or two ago I caught them leaking customer data and they made
a huge handwavey blog post full of lies about how nothing was leaked.

Don't do business with liars.

------
copsarebastards
This is a complete non-issue; anyone really concerned with privacy wouldn't
rely on DO's security.

Anyone who was using DO's private network for security reasons and complains
is just being hypocritical. They don't care about security: if they really
cared about security they would spend the time to implement it themselves.

~~~
nirvdrum
You may as well written "anyone really concerned with privacy wouldn't use a
VPS." You don't have to outsource security to screw this up. You're given a
non-routable IP. It's called "private networking". And you've just been sold
on moving from EC2, where you're effectively on your own VLAN. It's very easy
to mess this up. And adding a firewall rule doesn't work because you probably
added the subnet anyway. Even if you were diligent enough to detect that,
you'd have to propagate rules per droplet every time a new droplet is created
or another destroyed (wouldn't want a new account to get your old IP). Doing
that in a fault-tolerant way can be a lot of fun. Oh, and your iptables rules
look great, but Ubuntu won't restore them for you upon reboot. So even more
chances to mess up.

This is deceptive and it shouldn't be acceptable. Maybe you shouldn't be a
sysadmin without knowing all this, but I don't know how you learn it without
getting burned by it every step upon the way. It's not like the defaults are
sensible in most cases.

~~~
justizin
> And you've just been sold on moving from EC2, where you're effectively on
> your own VLAN.

Mrh, not quite. If you are a relatively new EC2 customer, then you are a VPC
customer, which might effectively be your own VLAN.

EC2 has also been this way for ages, but it has also always had security
groups. If you open something up on EC2 Classic to 10.0.0.0/0, any other
customer within that region will be able to access your services.

Sometimes this is desirable for collaboration sake, but bridging VPCs is
probably a better solution.

> I don't know how you learn it without getting burned by it every step upon
> the way. It's not like the defaults are sensible in most cases.

Welcome to #SysadminLife ;)

I do agree that everyone could be clearer about this, Rackspace, Linode,
Digital Ocean, and Amazon. A substantial motivation is to keep your inter-VM
traffic off the edge routers.

~~~
nirvdrum
> EC2 has also been this way for ages, but it has also always had security
> groups. If you open something up on EC2 Classic to 10.0.0.0/0, any other
> customer within that region will be able to access your services.

Except that's not how anyone did it. You could define rules that used a
security group as the source. And since every instance belonged to a default
security group, locking down access to other machines you owned was quite
trivial -- you only added rules to allow access from that default security
group. If you wanted to add subnet rules, that was usually limited to special
cases you would have to actively think about.

> Welcome to #SysadminLife ;)

> I do agree that everyone could be clearer about this, Rackspace, Linode,
> Digital Ocean, and Amazon. A substantial motivation is to keep your inter-VM
> traffic off the edge routers.

My source of issue was with the grandparent blaming people that get burned by
it. As long as the only way to learn this stuff is to get burned by it, it's
counterintuitive to blame people for getting burned by it. And in this case,
there are measure DigitalOcean could take to make things easier on everyone.
E.g., configure the droplets to save and restore iptables rules on
shutdown/power-on and lock the instances down by default to only have port 22
publicly exposed.

~~~
justizin
> Except that's not how anyone did it.

I can affirmatively tell you that more than one of my employers, successful
startups several years in, _did_ do this when I started.

Anyway, I don't blame people that get burned by this, very little in hosting
is obvious.

That's a great example of why people need skilled ops help. Understanding how
one computer connected to a cablemodem works and understanding what happens
inside of a datacenter are very different things.

------
dogma1138
When you buy a VPS you should assume that it's not private.

There have been plenty of POC's of side channel attacks against
sharded/virtual infrastructure including retrieval of encryption keys from a
co-hosted VM. On top of that you inherit the risk of potential vulnerabilities
within the hypervisor itself which could be potentially exploited to execute
code on the host or to compromise co-hosted machines.

There is no mode for any virtual infrastructure to give you neither truly
private networking (since at best it's going to be on a virtual switch) or a
private host for that matter. Unless you control everything including the
datacenter it self any true sense of privacy should not be considered,
especially not in a virtualized multi-tenant environment.

------
shaggy
When a PaaS/Cloud provider says private networking, they mean RFC1918 address
space. It's not publicly routed IP space which is private. So saying they
offer private networking is accurate against that long standard definition.
The technology and effort involved in providing the equivalent of AWS' VPC is
huge and most smaller providers don't have the scale, man power, or resources
to do it. It's a non-trivial solution to deploy.

Not understanding the platforms you choose to use is not an excuse to write
something like this article. It's also not an excuse to not understand how to
manage the platform(s) you run your services on.

~~~
sneak
You're moving the goalposts. They didn't offer "rfc1918 networking", they
offered private networking. Private is a word with a meaning.

They were caught lying. Why is it important to you to to cover for them?

------
webnanners
I agree with shaggy. I don't know how you would expect to have a VPN
accessible only by your account's droplets for $5. A private network means an
internal network. So it's just using the internal infrastructure of the data
center to communicate with other droplets in the data center. It's more
performant than using the external network. Usually companies' advertise a
fully private account infrastructure as a "Private Cloud" or something of that
nature.

~~~
sneak
You would expect it because it has "private" in the name and because words
mean things.

The "you should have known they were lying because $5 is obviously too cheap
to provide the service they are claiming" argument doesn't hold water.

~~~
webnanners
"Private Network" is often analogous with "Internal Network". They aren't
lying,it's private as in it's using private IP assignment within their own
infrastructure. Thus, private.

[http://en.wikipedia.org/wiki/Private_network](http://en.wikipedia.org/wiki/Private_network)

This is purely a case of not knowing industry terminology.

------
chrismarlow9
I would have thought the fact that you can't set static internal IP's would
have implied this. If you're looking for something that's stupid easy to setup
and super flexible checkout tincd. A co-worker turned me on to it and it's
fantastic so far, but requires a bit of re-thinking how you want to layout
your VPN (if you want to REALLY use it the way it should be, otherwise you can
just set it up like a normal VPN).

------
lobster_johnson
As a secure alternative, would something like ZeroTier [1] or cjdns [2] work
to create a private cloud? Or is there a simpler solution that would give you
seamlessly encrypted networking between nodes (even across data centers)?

[1] [https://www.zerotier.com](https://www.zerotier.com)

[2] [https://github.com/cjdelisle/cjdns](https://github.com/cjdelisle/cjdns)

------
seunosewa
It seems to me that would be quite easy for virtualization software to prevent
individual VPSs from reading networks packets not meant for them, thus
providing decent privacy. What's the technical hurdle that prevents this in
practice?

~~~
viraptor
It's tricky because the routing is completely dynamic. You also can't use the
typical hardware tricks at scale - for example providers will have more
customers than available vlan numbers.

So you have to have a dedicated solution running something comparable to
openvswitch in software, or interface with your hardware really fast and
really often. Usually the hardware is not really built to be reconfigured
constantly every second (or more often).

Once you start adding more than address filtering, you get an explosion of
rules. For example in openstack (neutron), it's pretty common to just put a
hard limit on the number of entries - otherwise you end up with one vm
spawning and suddenly every router has to add another entry for each of the
ports to each of the possible 1000s of other vms. And then clean up and keep
everything consistent when vms are killed again. It really does get hard
beyond the bare basics.

------
brandon272
It's amazing to me how people can not do their research, not understand the
service they are choosing to sign up for, and proceed to publicly malign a
company because of their (the customer's) ignorance about how the product
works.

~~~
sneak
One has a reasonable expectation that services called "private" will be. It
requires no further research.

Why is it amazing to you that people take a company's statements at face
value? Why blame the user when the other party is actively misleading them to
cut corners and reduce costs?

------
kuon
If you need real privacy, I highly recommend tincd to setup a mesh VPN
network.

------
a2tech
This is news to no one. Every datacenter I've ever done business with offered
'internal transfer' for free and used basically identical verbiage to explain
it.

------
ripter
Someone needs to send them a dictionary.

Private: belonging to or for the use of one particular person or group of
people only. "all bedrooms have private facilities" synonyms: personal, own,
individual, special, exclusive, privately owned "his private plane"

~~~
eropple
Is a country club to which you pay membership dues a _private_ club?

~~~
qeorge
Yes, in the US at least, "a private club for members and their guests" is a
common phrase, and would likely apply to a country club.

This would fit Digital Ocean's interpretation of private - not accessible to
the general public, but accessible to other members.

~~~
eropple
That was the idea, yes. =)

~~~
qeorge
I'm a little slow. Blast you and your Socratic method!

