
Don't build private clouds - mhausenblas
https://m.subbu.org/dont-build-private-clouds-9a54b3d30c8b
======
anon1253
Yeah so we have a different story. We build Natural Language Processing and
statistics software. We tried the cloud first, but it was just unworkable.
Limited IOPS, outrageous prices for even a modest amount of RAM, slow
virtualized CPUs, limited GPU options with terrible performance. So one day I
hired half a rack, bought some machines with fast SSDs, 128GB RAM each, dual
CPU with high clockspeed, and even some boxes with GPUs. The difference was
incredible. Not just in performance, which was orders of magnitudes faster
(going from barely workable to "pleasant user experience" without any software
changes), but also in terms of cost. If we were to provision these machines on
Azure or E2 we'd spend in a month what these machines cost, and our current
current costs are maybe $500 in electricity + rack rental. Sure, it's a bit of
a workout to put stuff in racks, but I actually quite enjoyed doing it. Sure
stuff will break and things will go down. We can take the downtime at this
point. And seriously, cost and performance wise there is no competition
between cloud and rolling your own hardware (if you know how to do it). Also
I'm perpetually surprised by how overstated the complexity of this is, yes
it's tricky. But so is CSS, and Javascript these days. And unless you go "full
cloud" (never go full cloud), you still have to manage
fallover/redudancy/CI/provisioning yourself. It's not much different on
"actual" hardware (there is no cloud, it's just somebody else's computer). If
my firewall or switch catches fire, that might be problematic. But if that
happens at an Amazon or Azure data center, that's also problematic. At the
very least you'll suffer terrible latency because it needs to move the VMs
dynamically.

~~~
cyberpunk
Rock on anon' \-- we feel the same way!

Once you get above a few boxes though, you might need some pricey geeks for a
month or two to rig up cobbler, ceph, openstack, proxmox, or whatever-takes-ya
simply because it's quicker than getting up on all that yourself right away --
but really there is no magic involved here and it's just so you can have it
totally manageable as you grow.

The tooling is much more friendly these days; and to be totally honest --
provisioning _probably_ isn't the problem you're trying to solve anyway (more
likely packaging/deployment/ci is the pain after you got a bit of breathing
space on perf). I've seen loads of folks coming back on prem spend months
rigging up crazy vm provisioning things when their estate is basically static
because they didn't realise they were solving the wrong problem.

If your ops time is spent building machines, then sort that out. Probably and
more likely it's spent on getting apps running and dealing with moving traffic
about and deployments and such; so that's the real thing to work on.

I think the 'complexity' argument seem so loud on HN as the opsy people are
the minority here and it's not exactly a sexy/exciting/fun thing to pick up,
and when it goes wrong it's brutal, especially when you can be nailing
features instead.

All I'd say is -- regardless of what you're doing, getting into containers at
this point makes your life quite a lot easier; you get a standard deployable
unit of code you can throw into aws, or on prem depending on your workload,
and it removes a lot of the anti-patters of config mgt.

------
Freak_NL
If you are handling data that is subject to national regulations concerning
personal data (e.g., health care) using a cloud provider for all your needs is
risky and might not be legal. If you use one of the big-name (and therefore
affordable) providers such as Amazon, your data may very well fall under
America's Patriot Act — there is a lot of legal uncertainty about this. If I
go with a local party instead, there is also no guarantee that this company in
turn won't branch out to the US, or get bought up by an American company,
again putting citizen's private data in reach of (in my case) foreign
governments.

~~~
rcarmo
Erm, no. All major cloud providers have clear statements regarding data
sovereignty, compliance and whatnot. There is no uncertainty, only lack of
awareness (but I'll freely grant that it is not a simple matter).

Look for the equivalent of [https://www.microsoft.com/en-
us/TrustCenter/Transparency#Tra...](https://www.microsoft.com/en-
us/TrustCenter/Transparency#Transparent)

(Full disclosure: I work at Microsoft)

~~~
cyberpunk
Do you honesty believe that?

Maybe I'm too cynical these days, but I find it pretty laughable that a
statement regarding 'data sovereignty' has any real meaning at all these days.

Unless you control the hardware, you have no idea what the hypervisors are
doing on your rented vm's. The idea that national actors don't / wouldn't pull
anything they want from them seems comically naive to me. Christ, we can
barely trust our own hardware anymore with backdoored firmware crap, IME and
such..

Realistically though, most companies don't care so it's not a problem if you
keep the blinkers on; their compliance folks just want the document which says
it's not happening and they'll never know when it does so who cares. Their ops
folks can barely use the APIs probably, let alone contemplate the impact of
someone reading the memory of my databases from the hypervisors.

As an individual and someone who cares for my data or that of my clients
though, that's not enough for me, and I'd never trust tnese folks with any
data, only mindless compute and serving public content...

~~~
kesor
Don't you share this data over networks from time to time? Is all your data
siloed and bolted down to one place never to move anywhere? How do you trust
the data to not take a hike in some random direction you don't fully trust?

~~~
cryptarch
By encrypting anything you transmit over the Internet?

~~~
arpa
and anything you have on hardware you don't own. And on hardware you own.

------
kkapelon
The article does not address important topics such as privacy and security.

For some organizations using a private cloud is not a technical decision but
instead a political/ethics one.

As an example I would bet that there are several companies out there who would
not care if all services were down for a full day (in their private cloud), as
long as they were certain that no government could easily access them.

~~~
erelde
GovernmentS can't host their services on Google's or Amazon's cloud for
example.

(the US are not counting since this is THE government many others are not
trusting and the home of the public clouds)

~~~
kesor
The FUD is strong, but that was exactly the point of that article. Having old-
fashioned people in goverments who disagree that they should ditch their
datacenters and start using clouds, is backwards thinking.

Just like goverments find ways to allow private companies to run their
cellular networks, phone networks, tv cable networks, power grids - the same
reasoning should allow private companies (public clouds) who do it much better
than goverment employees, to run their compute services.

~~~
cyberpunk
I'm not so sure.

There's a lot of gov.uk stuff running on non-gvt gear these days thanks to
GDS; (aws, skyscape, others on gcloud) -- the problem with how that's gone
though is while they're actually running in "the cloud" and tryig to do things
more 'modern' the architecture has to be accredited by a bunch of GCHQ folks
who apparently are totally insane yet you can never actually talk to. And so
you end up only being able to build what looks like a 2000's datacentre arch
with 500 /28 vlans and weird IDS appliances and so on, otherwise CESG won't
let you hook up to any of the data-sources you probably need.

In that style arch, where you can't just provision things and changing even a
netmask requires your whole architecture to be re-accredited otherwise you'll
lose access to the PSN; what possible benefits are there of running in the
cloud?

It's more expensive, needs constant oversight from accreditors and generally
sucks.

The legacy gov DCs are a shitshow too; but that's down to people like
capgemini and ibm or whoever and well, fuck them.

~~~
kesor
You mean the IBM that is doing $10m contracts and then paying $30m for
screwing these up? [http://www.abc.net.au/news/2016-11-25/ibm-to-pay-
over-$30m-i...](http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-in-
compensation-for-census-fail/8057240)

------
cyberpunk
Okay, I wrote out a whole breakdown of everything that's wrong with this post,
but after I re-read it, it came across as so harsh that I've just deleted the
lot.

TBH; the only people who would have agreed with me knew my points anyway so
bother?

I felt like as this is my specific area, that I have a kind of responsibility
to people who read misinformation like this to help see through such stuff.
It's hard to do for me without getting kind of angry though; opinions like
this is why I end up having to deal with 800k/month aws bills and 3 hour
deploys..

Either way; regardles of me -- please, people -- if you think that the
'typical journey' of implementing your own infrastructure and migrating a
legacy application to run on it is _ANYTHING_ like the description in this
blog post, then please take it from me as someone who moves horrible legacy
codebases into CD for a living, that you should perhaps seek some alternative
advice.

Suubu; I'm sorry, I'm not trying to be mean but your typical' private cloud
journey is complete and utter horseshit.

~~~
oarsinsync
> TBH; the only people who would have agreed with me knew my points anyway so
> bother?

Can you please actually elaborate, or it defeats the point of your entire
post. While the rant may have helped you, I don't know you, I don't your
history, and this post doesn't encourage me to go through your post history to
find it.

At the same time, OP's post filled me with a slight sense of dread as I'm
about to go into a private cloud environment and wondered if I was making a
mistake. Your post has certainly calmed those nerves. Actually having some
meat with the mustard would help more.

Thanks in advance!

~~~
cyberpunk
Well, I don't want to pick the whole thing apart because I've been a bit too
mean already, but the 'typical' process of private cloud in that post is
bullshit.

Lets make some assumptions on this gig you're starting then, we'll imagine
you're going into some job at a place which has a horrible legacy codebase
(lets' go medieval assume struts and oracle 10, deployed with some ancient
bamboo/rpm concoction which takes forever and always dies and the existing ops
team barely keep the lights on, let alone are capable of keeping up with the
rate dev is trying to let the business move..).

So you land. They've got a fresh budget, a load of bearded scala devs who are
rewriting the whole thing as microservices and words like "ISB" and "ASYNC"
are being thrown around. Hell, someone even mentioned elixir in the bar on
your welcome drinkies.

You're going to drag this horrible monster into the new world, but to get
there and to a better arch you're going to have to replace it a bit at a time,
and while that happens the code that's making all the money needs to be
maintained, and it needs to change faster. And you're going to do this on
prem.

So. You land. On prem, ooooeh baby. Realise: the pain you're going to have is
not going to be how you instantiate vms to run the apps or spin up
loadbalancer IPs or whatever the arse else infra wise -- AWS will do all that
right away for you for sure, while working out how to get a bunch of tin to do
it will take longer, but it's like getting into first gear slightly faster
than the guy beside you on a F1 race -- in the long run, this isn't where the
time is going to be spent.

Lets say you're the ops dude instead, how do you move a few hundred hand built
centos and java boxes to some kind of api based provisioning thingy with no
config management? Yeah well.. There are a lot of options, if you're the one
building it out, tbh the best choice for the infra part of the provisioning is
just to pay someone to make this their problem. Redhat. Vmware. Whoever does
Proxmox. Whatever. You'll lose E-Penis points but who cares your job is to
help the business work faster. Even if you have do it yourself, there is a
large ecosystem around things such as openstack or proxmox or smartos or
whatever cobbler and so on, and this stuff is pretty friendly, it won't take
you _THAT_ long. Hire a contractor for a bit. Blammo.

The post talks about doing 'compute' then 'storage' and 'network' then
'stateless monoliths' and so on. Whatever. That's not reality. Why is the
business paying for this? They want to be able to iterate quicker! CD is a
tough goal with a legacy codebase, but they got used to agile and hell if it's
not going to get faster -- the job is to enable that in any way possible, the
tricky part of that is not going to be how you deal with infra, it's how you
get the code through a pipeline and deploying painlessy.

Infra is a pain point. Legacy environments are fragile as hell and good
operations folks are few and far between, and most of them are undervalued and
a bit weird. -- but you personally, oarsinsync, are going to better off on
prem for a few simple reasons in that world:

1) Total Ownership: Something goes titsup? Yeah, your team owns the entire
show up until your ISP. Network borks? Storage slows down? It's all yours. Dig
in. AWS develops cross-az latency? Log a ticket and 2 days later when they
respond you'll have already forgotten it happened. Explain that to the
business.

2) Perf: Don't believe the rubes; you'll be quicker on your own gear if only
because you can move your noisy VMs to sep machines, set various
limits/quotes/throttles that you can't on a public cloud, and you're not
sharing spindles with 300 mongo vms.

3) Expierence: It's not exactly a business goal, but you'll learn more when
you're having to stace some jvm at 3:00am and crying into a toilet than if
you're just bouncing instances and getting away with shit code. If you just
outsource all your problems forever you'll never be able to have an opinion on
them, even if that opinion is just "I never want to use RAC again" or "mongodb
is death" or whatever..

------
a_imho
I don't quite agree with the cost analysis. You either spend on building your
infrastructure or integrating public cloud services. You will pay your devops
team anyway. Owning stuff is usually cheaper than renting, there are many
cases when it can make sense to have your own hardware.

------
snarfy
> Phase 3: Then deal with the stateful monoliths. These are your large
> monolithic databases. This is usually where private cloud journey hits the
> wall due to the risk and complexity in making such monoliths cloud native.

We are tasked with moving to the cloud, and this is where everything stops.
Over the years, instead of fixing the monolithic database design, the company
instead kept beefing up the hardware for the database. A bigger cpu and more
ram is cheaper up front than re-engineering the entire company infrastructure.

And now we are paying for it. Our db hardware now is the highest end hardware
the db software supports. None of the cloud providers db support can handle
it. We are forced to fix the monolithic design if we are to move to the cloud.
It will be a multi-year effort.

~~~
mooreds
> It will be a multi-year effort

And the company is willing to go through it because the public cloud offers so
much?

Just curious if this was going to stop the overall project or not.

~~~
snarfy
We were bought out by investors and this is their major initiative. I assume
it is to reduce long term operating costs.

------
micheljansen
This last point trumps all others:

"[...] on-premise infrastructure on the other-hand brews a culture of
mistrust, centralization, dependency and control. [...] These difference
between on-premise data centers and public clouds influence how teams think,
plan and execute. These are nothing but attributes of culture."

I'm currently working with a large financial services organisation that is
moving to the cloud. It's incredibly painful, because of all the challenges
involved in guaranteeing the safety and security of private data etc., but
once these hurdles are taken once, it's incredibly liberating. The move is not
just a technical one though, it requires a change of culture as well.

~~~
kesor
Unfrotuantely, all too often when a company is moving some workloads to a
public cloud, they take along the same culture they use in their "private"
datacenter.

~~~
micheljansen
Absolutely. Fortunately, in this case, we are actually moving _because_ we
want to change culture.

------
alrs
Private clouds wouldn't be the recurring disaster that they are if the people
who built them had used AWS before. Instead of "don't build private clouds,"
I'd offer "If you think Openstack means 'free VMWare,' you've already failed."

------
sfifs
If you store your data with a third party cloud provider, government agencies
can legally compel your cloud provider to give them your data while preventing
them from telling you. If your data is on-premises, they'll have to come and
take it from you. If you are a multinational company with entities in multiple
geographies and related party transactions, you know what you'd prefer.

~~~
kesor
The Pirate Bay

------
kitd
I was musing yesterday whether a technologically-advanced nation state of the
future might provide cloud infrastructure like it provides
roads/rail/healthcare/etc.

Obviously, this only makes sense in a world where individuals as well as corps
need to present themselves online as a matter of course, but it might enable
eg personal ID or data privacy tools that address many of the problems people
have 'donating' their private data to commercial orgs.

~~~
adrianN
A wet dream for police states. Distribute dumb terminals to government-run
cloud servers for free and soon enough the only people who don't make their
whole life transparent to the state are the criminals.

------
pasta
What about privacy? Our company doesn't use a cloud provider for that exact
reason.

~~~
kesor
Have you investigated the issues with privacy in public clouds, or do you just
"assume" that it is much worse than your own mismanaged hardware without doing
any due diligence?

From exprience, most organisations manage the privacy of their customers much
much worse on their own hardware than they would if they used a public cloud
provider.

Want to know how "privacy" really looks like? Take a look here -
[http://www.informationisbeautiful.net/visualizations/worlds-...](http://www.informationisbeautiful.net/visualizations/worlds-
biggest-data-breaches-hacks/)

~~~
cyberpunk
The difference is simply though that they own the entire scope of the privacy
concerns on prem; even if they manage them poorly.

In the public cloud, as an engineer/employee/government I can just login to
any given XEN/KVM/SmartOS/Whatever host that's running your instances and
simply read your secrets and even if your opsec is perfect there's nothing you
can do about it.

'privacy' in the cloud is not about mismanagement/good practice -- it's about
who is able to act on it.

There is no way that anything you have running on infra you don't completely
own and control will ever be private; to even a casual observer it's obvious
that there is absolutely no defense or privacy possible in such an
environment.

------
cyberpunk
Am I missing something here?

Is your point that everyone should move to the public cloud because you think
it's "difficult" to build a "private" one?

Sorry to be blunt, but are you joking, an idiot or just completely unaware of
the reality our industry outside of startup land?

Forgetting the compliance reasons, or that I have zero confidence that a
combination of giant American corporations have any interest in protecting my
data and that of my users, forgetting also that the performance differences
are incredibly significant (take 8 15k sas's in raid 10 vs.. what? EBS with 2k
'provisioned IOPS' on a shared san over iscsi?), the cost savings are much
more significant than you claim here.

I recently bought a half rack load of dell boxes which have like 380GB of ram
each and 52 threads, and they cost around £4k each.

Even with a whole army of overpriced ops dudes to install openstack or
kubernetes on all that, the cost we spent would be spanked in a month on AWS
for similar capacity, and mine isn't a recurring one.

Why does it have to be so black and white for you folks?

A "private cloud" is absolutely the correct choice for a lot of people. If I
were you I'd quietly delete that post, because I've absolutely judged you --
and probably unfairly -- on the content. On that, I wouldn't hire you.

Listen, both these things have a place, but they're NOT competing in the minds
of those of us running serious infra. AWS allows us to prototype apps rapidly,
to put project work on a credit card associated with a specific project budget
instead of spending the shared budget, and for sure there are some nice
services which do a bit of heavy lifting for us which helps speed that up.

After that phase though, very, very few people things actually need or make
sense to stay there once it becomes a real product. Your scaling concerns are
not being well served by 1000s of ec2 instances. Moving back in house is
usually the smarter option.

If you're pumping loads of data into the internet, then you should be behind a
CDN and the origin shouldn't be smashed to the point where you're dependent on
throwing compute into the void to cope. Akamai or CloudFlare are cheaper than
deploying 100s of ec2's because some shitty code can't handle your traffic.

If data is coming in, read a book on queues; you'd be pretty surprised what
you can do with even two 1U boxes and a good software architecture. We could
handle 100k+/sec of messages back in the java 1.5 and activemq days with 1U
sparc boxes; how much do you recon it costs to do that today in AWS?

tl;dr - urgh. lets' talk in 10 years.

</rant>

~~~
davidjgraph
> If I were you I'd quietly delete that post, because I've absolutely judged
> you -- and probably unfairly -- on the content. On that, I wouldn't hire
> you.

Based on your reply, I suspect most people wouldn't want to be hired by you,
either...

~~~
FatalBaboon
Well he makes a really good point, did we really get to the point where we
have so many unskilled workers in the force that recommendations about NOT
doing something because it requires skilled labor to do is legitimate?

I share the sentiment that amazon & co are way too expensive beyond a simple
proof of concept for example, even though for widely varying workload you may
still need a public cloud.

~~~
davidjgraph
I'm not talking about his point, I'm talking about the kind of tone calling a
stranger an idiot.

~~~
cyberpunk
Should I have danced around my view of his post to more subtly imply how I
view it without _actually_ using the words instead of just straight out saying
it like I did?

I'm not attacking the person; calling someone an idiot does appear to be a
personal attack, but it was asked as a question -- I was trying to convey my
bewilderment at how anyone could share his views ("you like pepsi over coke?!
what are you some kind of hippy?! -- etc!)

Come on, we're grown up and this is our area of expertise, why should we
censor ourselves -- I'm not sure how else I could have expressed my view on
the whole thing so succinctly tbh. You knew what I meant right off didn't you?

Are you offended on the authors behalf? Do you think that person, who is
clearly an experienced technologist and who appears to have spoken at a
conference, who has views on what is a pretty insider issue and has spawned a
debate on the whole thing on hackernews is going to be offended because
someone on the internet thinks it's stupid?

If he is; then I'd like him to know that it wasn't my intention to offend but
also -- this is the internet and we all benefit from frank honest discussions,
which I'd like to have and if you have issues with people questioning what
you've written a bit aggressively then perhaps you shouldn't have written it.

It's not like I'm going full youtube or /b/ here... This is a forum, and no
personal attacks happened.

------
whyagaindavid
We are in the same boat. We tried opencloud and has severe issues with our
unified login - mix of AIX,samba,ladp,AD systems. I wonder how to convince
universities who have legacy data and staff (aka Prof) who are unwilling to
change to cloud especially in these days of data leak. Any pointers?

~~~
Freak_NL
> […] who are unwilling to change to cloud especially in these days of data
> leak.

I would start by addressing that concern, but it may come down to a matter of
beliefs; i.e., you trust that 'the cloud' will be secure this next iteration
(or the next, or the next), and they remain sceptical. Can't blame them.

------
jtwaleson
What is the ratio between public and private cloud data centers is in terms of
server count / data center count etc?

I think the amount of enterprise DC's / colocation facilities is pretty stable
but that almost all of the growth is in public cloud. Don't have any data to
back this up though.

~~~
discodave
In terms of existing infra, private DC capacity is surely an order of
magnitude more than public cloud. New capacity getting installed might be a
bit more evenly split. How you differentiate between any private infra and
private cloud I don't know.

Public clouds often use colo facilities, especially for new/small regions. So
colo != private.

~~~
kesor
Depends on the public cloud probider, but the leading providers are actively
building their own buildings for expansion purposes.

For example MSFT investing $15B in its datacenter infrastructure.
[https://azure.microsoft.com/en-
us/overview/datacenters/](https://azure.microsoft.com/en-
us/overview/datacenters/).

GOOG naturally don't need to colo anything, they have their own
[https://www.google.com/about/datacenters/inside/](https://www.google.com/about/datacenters/inside/).

This article shows an analysis of DC vs. Public/Private Cloud IT
Infrastructure:
[http://www.datacenterknowledge.com/archives/2016/07/06/cloud...](http://www.datacenterknowledge.com/archives/2016/07/06/cloud-
providers-will-spend-23b-on-data-center-hardware-in-2016-report/)

------
Jaruzel
Can anyone tell me what typeface is being used on this graph?

[https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5B...](https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5Bf8_s6sIi_9A.png)

~~~
coroxout
Looks like Apple's "Chalkduster":
[http://www.identifont.com/find?font=Chalkduster&q=Go](http://www.identifont.com/find?font=Chalkduster&q=Go)

~~~
Jaruzel
It does! Thank you. :)

