
Ask HN: Is your company sticking to on-premise servers? Why? - aspyct
I&#x27;ve been managing servers for quite some time now. At home, on prem, in the cloud...<p>The more I learn, the more I believe cloud is the only competitive solution today, even for sensitive industries like banking or medical.<p>I honestly fail to see any good reason not to use the cloud anymore, at least for business. Cost-wise, security-wise, whatever-wise.<p>What&#x27;s a good reason to stick to on-prem today for new projects? To be clear, this is not some troll question. I&#x27;m curious: am I missing something?
======
AgentK20
Like many others have pointed out: Cost.

I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who
operates about 700 rented dedicated machines to service 70k-100k concurrent
players. We push about 4PB/mo in egress bandwidth, something along the lines
of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us
_an order of magnitude_ more than our entire fleet's cost....JUST in bandwidth
costs. Even if we bring our own ISPs and cross-connect to just use cloud's
compute capacity, they still charge stupid high costs to egress to our
carriers.

Even if bandwidth were completely free, at any timescale above 1-2 years
purchasing your own hardware, LTO-ing, or even just renting will be cheaper.

Cloud is great if your workload is variable and erratic and you're unable to
reasonably commit to year+ terms, or if your team is so small that you don't
have the resources to manage infrastructure yourself, but at a team size of
>10 your sysadmins running on bare metal will pay their own salaries in cloud
savings.

~~~
mmmBacon
A few years ago I was trying to start a company and get it off the ground. We
had to make decisions on our tech stack and whether we were going to use AWS
and build around their infra. Our business was very data heavy and required
transferring large datasets from outside to our databases. Even in our early
prototypes, we realized that we couldn’t scale cost-effectively on AWS. I
figured out that we could colocate and rent racks, install HW, hire people to
maintain, etc... for way less than we could use the cloud for. I was shocked
at the difference. I remember saying to my cofounder why does anyone use AWS,
you can do this on your own way cheaper.

Later I worked at a FAANG and remember when Snap filed their S1 when they were
going public they disclosed that they were paying Google $5B and we were
totally shocked at the cost compared to our own spend on significantly larger
infra.

I think people don’t realize this is doable and it’s great to hear stories
like yours showing the possibilities.

~~~
RachelF
Dropbox did the same thing a few years back - moved everything from Amazon S3
to their own storage.

My guess is they did it for cost reasons.

~~~
llarsson
That S3 is eventually consistent with object updates (HTTP PUT) might also
screw up things for a company whose core value is synchronized storage.

~~~
dfsegoat
I don't mean to sound daft, just clarifying my own understanding, but isn't
Dropbox eventually consistent (as a system)?

~~~
llarsson
Oh, sure, but when they think they have written something to S3 and got a
successful HTTP response back from the API, perhaps they want to be able to
tell clients to go fetch the new data from the bucket. But those clients may
not get the new data then, due to eventual consistency.

~~~
lozenge
S3 is immediately consistent for new objects unless the service received a GET
on the object before it was created. It's easy to use this to make an
immediately consistent system.

~~~
ozkatz
S3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS
[1] and S3Guard [2] mitigate this for data processing use cases.

[1] -
[https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-f...](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-
fs.html) [2] -
[https://blog.cloudera.com/introducing-s3guard-s3-consistency...](https://blog.cloudera.com/introducing-s3guard-s3-consistency-
for-apache-hadoop/)

------
tgamblin
I work in Livermore Computing at LLNL.

We manage upwards of 30 different compute clusters (many listed here:
[https://hpc.llnl.gov/hardware/platforms](https://hpc.llnl.gov/hardware/platforms)).
You can read about the machine slated to hit the floor in 2022/2023 here:
[https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-
capita...](https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capitan-
projected-worlds-fastest-supercomputer).

All the machines are highly utilized, and they have fast Infiniband/OmniPath
networks that you simply cannot get in the cloud. For our workloads on
"commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd
pay for equivalent cloud nodes, and for the really high end systems like
Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life
of the machine.

The way machines are procured here is different from what smaller shops might
be used to. For example, the El Capitan machine mentioned above was procured
via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We
write a 100+ page statement of work describing what the machine must do, and
we release a set of benchmarks characterizing our workload. Vendors submit
proposals for how they could meet our requirements, along with performance
numbers and test results for the benchmarks. Then we pick the best proposal.
We do something similar with LANL and SNL for the so-called commodity clusters
(see [https://hpc.llnl.gov/cts-2-rfi](https://hpc.llnl.gov/cts-2-rfi) for the
latest one). As part of these processes, we learn a lot about what vendors are
planning to offer 5 years out, so we're not picking off the shelf stuff --
we're getting large volumes of the latest hardware.

In addition to the cost savings from running on-prem, it's our job to stay on
the bleeding edge, and I'm not sure how we would do that without working with
vendors through these procurements and running our own systems.

~~~
zozbot234
> All the machines are highly utilized, and they have fast Infiniband/OmniPath
> networks that you simply cannot get in the cloud.

It's weird that these networking technologies are not used more in "plain"
datacentre settings, since networking latency and throughput has to be a
significant challenge to scaling up non-trivial workloads and achieving true
datacentre-scale computing. We hear a lot about how to "scale out", but that's
only really feasible for relatively simple workloads where you just seek to do
away with the whole issue of keeping different nodes in sync on a real-time
basis, and accept the resulting compromises. In many cases, that's just not
going to be enough.

~~~
alexpotato
There are a lot of people from the National Lab super computer world who end
up in High Frequency Trading for just the reason you describe.

Specifically, how do you optimize a large cluster of computers to operate at
the lowest possible latency. For the National Labs, those computers could be
in the lab or with other labs around the world. For the HFT folks, the
machines could be in an exchange or spread across multiple exchanges around
the world.

Source: I used to be head of Global Latency Monitoring for a HFT.

~~~
j88439h84
I'm curious why you moved to LLNL from HFT?

~~~
eyegor
Money is a safe guess. Research pay scales aren't even close to private
sector, especially not finance.

~~~
dwohnitmok
It sounds like the opposite direction happened here.

------
centimeter
We are a 1000-2000 person company and we have probably on the order of $100M
of servers and data centers and whatnot, and I think we spend about 2/3rds of
that every year on power/maintenance/rent/upgrades/etc.

We don't generally trust cloud providers to meet our requirements for:

* uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots of fancy redundant infrastructure that we can't rely on from cloud companies)

* latency (this is a big one)

* security, to some degree

* if something crazy is happening, that's when we need hardware, and that's when hardware is hard to get. Consider how Azure was running out of space during the last few months. It would have cost us an insane amount of money if we couldn't grow our data centers during Corona! We probably have at least 20-30% free hot capacity in our datacenters, so we can grow quickly.

We also have a number of machines with specs that would be hard to get e.g. on
AWS.

We have some machines on external cloud services, but probably less than 1% of
our deployed boxes.

We move a _lot_ of bandwidth internally (tens of terabytes a day at least,
hundreds some days), and I'm not sure we could do that cheaply on AWS (maybe
you could).

We do use <insert big cloud provider> for backup, but that's the only thing
we've thought it was economical to really use them for.

~~~
H8crilA
Hundreds of terabytes a day is really not that much, depends on what latency
can you accept. I often run computations over datasets that are petabytes in
size, just for my own needs. A big data move would be at least tens of
petabytes or more like hundreds, or thousands.

Also surprised about latency, latency from what to what? Big cloud providers
have excellent globally spanning networks. Long distance networking is crazy
expensive, though, compared to the peanuts it costs to transfer data within a
data center.

Reliability - again, not sure I buy it. Reliability is "solved" at low levels
(such as data storage), most failures occur directly at service levels,
regardless of whether you have the service in house or in the cloud.

The rest of your points make sense.

~~~
centimeter
> Hundreds of terabytes a day is really not that much

How much would it cost to move this across boxes in EC2? I actually don't
know, that's not a rhetorical question. A lot of our servers have 10-40gbit
links that we saturate for minutes/hours at a time, which I suspect would be
expensive without the kind of topology optimization we do in our datacenters.

> Also surprised about latency

We've spent a surprising amount of money reducing latency :) We're not a high
frequency trading firm or anything, but an extra 1ms (say) between datacenters
is generally bad for us and measurably reduces performance of some systems.

> Reliability is "solved" at low levels

To whatever extent this may be true, it's certainly not true for cloud
providers. One obvious example is that EC2 has "scheduled maintenance events"
where they force you to reboot your box. This would cost us a lot of money
(mostly in dev time, to work around it).

Also, multi-second network dropouts in big cloud datacenters are not uncommon
(in my limited experience), but that would be really bad for us. We have
millisecond-scale failover with 2x or 3x redundancy on important systems.

~~~
tstrimple
> How much would it cost to move this across boxes in EC2?

Nothing. You generally only pay for data going out of cloud providers. Not
data going in or data being transferred within the same region.

> One obvious example is that EC2 has "scheduled maintenance events" where
> they force you to reboot your box. This would cost us a lot of money (mostly
> in dev time, to work around it).

You're not going to have a successful cloud experience unless you build your
applications in a cloud suitable way. This means not all legacy applications
are a good fit for the public cloud. Most companies really embracing the cloud
are mitigating those risks by distributing workloads across multiple instances
so you don't care if any one needs to be restarted, especially within a
planned window.

> Also, multi-second network dropouts in big cloud datacenters are not
> uncommon (in my limited experience), but that would be really bad for us. We
> have millisecond-scale failover with 2x or 3x redundancy on important
> systems.

Are these inter-region network dropouts or between the internet and the cloud
data center? You're not going to be relying on a public internet connection to
the cloud for critical workloads.

All that being said, there are plenty of workloads which I don't think fit
well in the cloud operating model. You may very well have one of them.

~~~
iampims
You pay for cross-AZ Traffic in AWS, and that adds up really fast.

~~~
Wintereise
Yep. Got bitten HARD by this recently, $1.5k inter-az transfer charges that we
never saw coming.

Our fault, I suppose -- but multi-az is prohibitively expensive if you need to
run anything data heavy distributed.

~~~
resonator
I'm working on reducing a $50K per month bill for Inter-AZ traffic at the
moment.

> but multi-az is prohibitively expensive if you need to run anything data
> heavy distributed.

If you communicate between your AZs via ALBs, multi-az is effectively free.
Our bill is so high because within our Kubernetes cluster, our mesh isn't
locality aware; it randomly routes to any available pod. 2/3rds of our traffic
crosses AZs.

------
horsawlarway
I'm slowly coming to the complete opposite opinion you seem to have.

I've worked almost entirely for companies that run services in various cloud
infrastructures - Azure/Heroku/Aws/GCP/Other.

I recently started a tiny 1 man dev shop in my spare time. Given my experience
with cloud services it seemed like a no brainer to throw something up in the
cloud and run with it.

Except after a few months I realized I'm in an industry that's not going to
see drastic and unplanned demand (I'm not selling ads, and I don't need to
drive eyeballs to my site to generate revenue).

So while in theory the scaling aspect of the cloud sounds nice, the reality
was simple - I was overpaying for EVERYTHING.

I reduced costs by nearly 90% by throwing several of my old personal machines
at the problem and hosting things myself.

So long story short - Cost. I'm happy to exchange some scaling and some uptime
in favor of cutting costs. Backups are still offsite, so if my place burns I'm
just out on uptime. The product supports offline, so while no one is thrilled
if I lose power, my customers can still use the product.

Basically - cost, Cost, COST. I have sunk costs in old hardware, it's dumb to
rent an asset I already own.

There might well be a point when I scale into a point where the cloud makes
sense. That day is not today.

~~~
tjbiddle
What's the time trade-off?

I've been drawing out my plans lately for a hobby project, all 100% on AWS.

Being able to spin up my entire infrastructure with Terraform, build out
images with Packer, setup rules for off-site backups, ensure everything is
secure to the level I want it, etc. - It takes me next to no time at all.

I can't imagine buying hardware, ensuring my home is setup with proper
Internet, configuring everything here, and then still needing off-site backups
anyway.

Now, keep in mind - I'm definitely coming in from a Millennial point of view.
My entire career was built on cloud. I've never touched hardware apart from
building a computer back when I was 15 or something. I understand virtual.

But being able to build up and tear down an entire setup, having it completely
self-restore in minutes. Can't beat that.

Napkin math has me at ~$50/mo: Full VPC, private/public isolated subnets,
secure NACLs and security groups, infinitely extendable block storage and
flat-file storage, near-instant backups with syncing to a different continent,
5 servers, DNS configurations, etc.

All depends what you're doing too - of course. But for me, just the trade-off
of working with what I know and not needing to leave my cafe of choice, still
not breaking the bank - and if I do, having instant tear down and restore.
Bam.

~~~
tigerstripe
What kind of setup did you have for 5 servers at $50/mo on AWS? Interested to
know - our EC2 instances that are about 1/4 as powerful as a laptop cost
$60+/mo

~~~
tjbiddle
Certainly nothing powerful :-) I can get away with t3.nano and t3.micro for
what I'm doing at the moment. But the beauty of cloud, is that I can scale up
when I eventually need it.

5x t3.nano will be ~$25/mo 5x t3.micro will be ~$50/mo

All of my AMIs are EBS optimized and require a minimum of 8GB for the root
drive (Although they only use ~1.6GB. Not bothering to hack around this to
save a buck.) So that'll be 40GB EBS block storage. Plus I want ~20GB spread
across 3 of the machines.

So EBS should be ~$6/mo.

I only need the volumes of those last 3, the others are good to go with their
base AMI or user-data init script. So I only need snapshot backups of ~20GB.
Being priced incrementally and having minimal changes, I'll only be charged
~$1/mo for that + off-site another $1/mo

So, currently experimenting with the t3.nano - Cost is ~$36/mo. One of these
servers will be used as a personal VPN, and I expect ~75GB/mo coming from my
laptop. So bandwidth charges at $9/mo.

Total $45/mo - For what I have planned now, at least.

~~~
tarasmatsyk
That's exactly the reason I gave up on AWS, I need an accountant to do the
math every month :D

Now I rent a 4GB Linux box for 5$/m with no Dockers or whatsoever and happy
that it just works

~~~
Gravyness
I also hate this complexity with a passion. I love cloud, but pricing can be a
real nightmare.

I don't use AWS specifically but when I needed to know the price of some cloud
service or group of services I spin up the service (or services) in a brand
new project and let it run for 24 hours under similar working environment to
see the impact, then after checking the results (the breakdown of each
service's price in that day) I just close the project entirely, no left overs.

So I tend to successfully avoid these strange, terribly organized, cloud-
specific, service-specific calculator where I can easily forget one aspect of
the service that might cost a lot of money absolutely randomly.

Obviously it is a bad strategy if things are expected to reach $200/month
and/or you do 'price evaluation' frequently, but otherwise it is stupid easy.
I barely spent $50 each year doing this (small company and sporadic system
changes)

But the best part is that the final daily price of your system is as precise
as it can possibly be and that is worth something.

------
reacharavindh
University research group here.

Simply, _cost_

Our compute servers crunch numbers and data at > 80% util.

Our servers are optimized for the work we have.

They run 24/7 picking jobs from queue. Cloud burst is often irrelevant here.

They deal with Terabytes or even Petabytes of moving data. I’d cry paying for
bandwidth costs if charged €/GB.

Sysadmin(yours truly) would be needed even if it were to be run in the cloud.

We run our machines beyond 4 years if they are still good at purpose.

We control the infra and data. So, a little more peace and self-reliance.

No surprise bills because some bot pounded on a S3 dataset.

Our heavy users are connected to the machines at a single hop :-) No need to
go across WAN for work.

~~~
dathinab
In germany it's a pretty common think for universities to have some servers
for themself.

1\. Their use case is kinda different. The servers mostly run heavy CS
research related stuff. E.g. they might have heavy CPU load and heavy traffic
between they servers but they have less often heavy traffic to the "normal
internet" (if they have heavy traffic to the outside it's normally to other
research institutes which not seldom have dedicated wire connections).

2\. They might run target specific optimized CPU or GPU heavy compute tasks
going on for weeks at a time. This is really expansive in the cloud which is
mostly focused in thinks like web services.

3\. When they don't run such tasks in the research groups they want to allow
their juniors to run their research tasks "for free". Which wouldn't work with
a payment model as done in the cloud.

4\. They don't want to relay on some external company.

Also I'm not sure are there even (affordable) cloud systems with compatible
spec? (like with 4+TB of _RAM_ , I'm not kidding this is a requirement for
some kind of tasks or they will take way to long and requires additional
complexity by using special data structures which support partial offline data
_in the right way_ , which can be very costly in dev time)??

~~~
RockIslandLine
It's not just CS. The computational chemistry and materials science
crystallography folks can have jobs that run for days or weeks too.

~~~
veddox
I'm at a center for computational biology - our genomics guys have been known
to use 90% of our university's HPC capacity ;-) My own work (ecological
modelling) is not as heavy, but when I run a full experiment, that takes a 32
core machine about two weeks to complete.

------
Groxx
Meta-comment:

cost:
[https://news.ycombinator.com/item?id=23098576](https://news.ycombinator.com/item?id=23098576)

cost:
[https://news.ycombinator.com/item?id=23097812](https://news.ycombinator.com/item?id=23097812)

cost:
[https://news.ycombinator.com/item?id=23098658](https://news.ycombinator.com/item?id=23098658)

abilities / guarantees:
[https://news.ycombinator.com/item?id=23097213](https://news.ycombinator.com/item?id=23097213)

cost:
[https://news.ycombinator.com/item?id=23090325](https://news.ycombinator.com/item?id=23090325)

cost:
[https://news.ycombinator.com/item?id=23097737](https://news.ycombinator.com/item?id=23097737)

threat model:
[https://news.ycombinator.com/item?id=23098612](https://news.ycombinator.com/item?id=23098612)

cost:
[https://news.ycombinator.com/item?id=23097896](https://news.ycombinator.com/item?id=23097896)

cost:
[https://news.ycombinator.com/item?id=23098297](https://news.ycombinator.com/item?id=23098297)

cost:
[https://news.ycombinator.com/item?id=23097215](https://news.ycombinator.com/item?id=23097215)

That's just the in-order top comments I'm seeing right now. (please do read
and upvote them / others too, they're widely varying in their details and are
interesting)

The answer's the same as it has always been. Cloud is more expensive, unless
you're small enough to not pay for a sysadmin, or need to swing between truly
extreme scale differences. And a few exceptions for other reasons.

~~~
cortesoft
There is also another answer... I work for a CDN, so we can't really use the
cloud when in many ways we ARE the cloud.

Although we do often make jokes about "what if we just move the CDN to AWS?"

~~~
mcny
It is a pity East Dakota won’t make jokes like these. Can you imagine
cloudflare running on aws? What happens when someone tries to denial of
service them while on aws?

On a different note, Netflix still runs “on the cloud”, right? I mean what
does it really mean? Dropbox can still have most of its stuff on aws and do
the expensive part on premises if cost is a concern?

The truly bizarre stuff happens at hybrid cloud.

~~~
ddorian43
Netflix runs it's own bandwidth/cdn. Sometimes it actually has a pop/box
INSIDE your ISP
[https://openconnect.netflix.com/](https://openconnect.netflix.com/).

~~~
mcny
My understanding is the website, the user services such as authentication,
heart beat (not sure what is the proper technical term but the thing that says
where I am in a particular episode). That and internal apps like project
tracker not to mention dev/test.

At least in my imagination. At my work, I'm not even worth throwing an SSD at
my work computer. My manager is powerless to help as the company has some kind
of deal to only buy from HP? No idea what kind of glue procurement is sniffing
at this company...

------
bcrosby95
We have around 20 servers in a colo center down the street.

At this number of servers we can still host websites that have millions of
users (but not tens of millions). They are not exotic servers either. In fact
by now they are, on average, around 11 years old. And costed anywhere from 2k
to 8k at the time of purchase. Some are as old as 19 years. Hell, when we
bought some of them - with 32GB of memory each - AWS had no concept of "high
memory" instances and you had to completely pay out your ass for a 32GB
server, despite ram being fairly cheap at the time.

We have no dedicated hardware person. Between myself and the CTO, we average
maybe a day per month thinking about or managing the hardware. If we need
something special setup that we have no experience in, we have a person we
know that we contract, and he walks us through how and why he set it up as he
did. We've used him twice in the last 13 years.

The last time one of us had to visit the colocation center was months ago. The
last time one of us had to go there in an emergency was years ago. It's a 5
minute drive from each of our homes.

So, why exactly should we use the cloud? We have servers we already paid for.
We rent 3 cabinets - I don't recall the exact cost, but I think its around $1k
per month. We spend practically no time managing them. In our time being
hosted in a colo center - the past 19 years - we've had a total of 3 outages
that were the fault of our colo center. They all lasted on the order of
minutes.

~~~
dahfizz
I think people who have no experience managing servers dramatically
overestimate how much time it takes to manage servers. Depending on your team,
it can definitely be easier to manage your own hardware than to manage your
cloud infrastructure.

~~~
wooly_bully
In my experience, it's not the time required but that a lot of development
teams don't have a sysadmin or ops skillset.

~~~
jcrawfordor
I live in a software engineering world professionally but my background is in
traditional "neckbeard" Linux system administration. This ends up making me
"DevOps" but honestly a lot of what I've ended up doing in my career is basic
sysadmin for organizations that get a remarkably long ways before realizing
they need it - things like telephony and video surveillance become really
unreasonably expensive when you end up relying on a cloud service because you
don't have the skillset to manage them in-house.

This is purely my opinion, but I think that 1) there is a strange shortage of
IT professionals (people who are _not_ software engineers but instead
understand _systems_ ) in much of the industry today, and 2) a lot of tech
companies, even those that are currently well functioning, might be able to
save a lot of money if they hired someone with a conventional IT background.
This is a little self-serving of course, but it really does astound me when I
see the bills that some companies are paying cloud services to do something
that is traditionally done in-house by an IT department. And not everything
can readily be outsourced to some "aaS" provider, so on top of that you end up
with things like software companies with multi-million budgets running an
office network that consists of a consumer WiFi router someone picked up at
Fry's - not realizing that they are losing a lot of time to dealing with how
poorly that ends up working.

I think part of the problem rests in academia - at least in my area a lot of
universities seem to have really backed off on IT programs in favor of CS. I
went through an undergraduate program that involved project management,
decision analysis, and finance courses because these were considered by the
college (I would say accurately) critical skills for the IT field. But that
program had an incredible two students and was widely considered inferior to
the CS program with hundreds.

Another part of the problem though seems to rest in industry. The salary
differential between "DevOps Engineer" and "IT Analyst" is incredible when in
practice they end up doing mostly the same thing in a lot of small orgs. So I
end up walking sort of an odd line of "I have a long background in IaC but I
also know about conference room equipment." And I'm not saying that everything
with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up
costing just as much and seem to be less reliable - not surprising for a
platform which, by practical necessity of the lack of IT support in many orgs,
is built on the Silicon Valley time-tested architecture of "five apple
consumer products taped together."

~~~
chillfox
From my experience, large enterprises sabotage the effectiveness of internal
IT with bureaucracy and politics in a misguided attempt to eliminate all
possibility of mistakes being made.

It's usually done with the "let's pretend it is ITIL" process.

Let me give two examples where if I had been the client then I would
absolutely have sprinted for the cloud if I could, or at the very least start
talking it up as much better.

1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me
sitting in front of my computer with screens open showing the problem and
waiting for various managers/decision-makers to fly by and take a look as they
were ping-ponging around the office panicking about what would be impacted by
the fix. Everything that was going to get impacted was already impacted by the
system not working, and I had to explain that to them multiple times. Towards
the end of the day, I eventually got the go-ahead to do the 3 minutes of work
to fix the system. This system being down had prevented another team from
doing any work for the entire afternoon.

2) Two full days of politics and paperwork to get approval to do 30 minutes of
work, all while the client was impatiently asking "is it done yet" every few
hours.

------
burnte
Yes. Why? Cost, availability, flexibility, bandwidth. For a lot of companies,
on-prem servers are the best solution for efficiency and cost.

One great example. We were paying $45k/yr for a hosted MS Dynamics GP
solution. For $26k we brought it in house with only a $4k/yr maintenance fee.
We bought a rackmount Dell, put on VMWare, have an app VM and a DB VM. My team
can handle basic maintenance. In the past 11 months we haven't had to touch
that server once. We have an automated backup, pulls VMs out daily and sends
them off to Backblaze. Even if we need to call our GP partner for some
specialized problem, it's not $45k/yr in consulting costs.

We had a bunch of Azure servers for Active Directory and a few other things.
When I came in 2 years ago I set up new on-prem DC VMs, and killed out absurd
Azure monthly bill, we were saving money by month three. A meteor could take
out Atlanta and the DCs are our satellite offices would handle the load just
fine until we restored from backups and we'd STILL save money. We've had MORE
uptime and reliability since then too.

If I have a server go down, we have staff to get on it immediately, no toll
free number to dial, no web chat to a level 1 person in India, etc.

Our EMR is hosted, because that's big enough that I want to pay someone to be
in control over it, and someone to blame. However, there have been many times
where I'm frustrated with how they handle problems, and jumping from one EMR
to another is not easy. And in the end they're all bad anyway. Sometimes I DO
wish we were self hosted.

The Cloud is just someone else's computer. If they're running those machines
more cheaply than you are, they're cutting out some cost. The question is, do
you need what they're cutting?

~~~
jedberg
> The Cloud is just someone else's computer. If they're running those machines
> more cheaply than you are, they're cutting out some cost. The question is,
> do you need what they're cutting?

They're cutting overhead and getting better deals on hardware than you could
ever get.

Their efficiency is their profit margin.

~~~
Slartie
> Their efficiency is their profit margin.

Last time I checked, AWS had a profit margin in the 40%-50% ballpark.

Sorry, but the semiconductor industry doesn't operate with any kind of markup
that would allow such profit margins from "getting better deals on hardware".
The only one able to make that kind of profit used to be Intel on high-end
server CPUs, and even they are now pressured by AMD and custom ARM silicon
options. Anything else needed for a server, RAM or flash chips or whatever, is
usually selling on thin single-digit margins.

Cloud provider profit margin is perfectly logical and explainable through
lock-in effects keeping their customers paying big markups to stay in AWS
infrastructure. Be it software that was built against AWS proprietary
services, be it having the necessary engineering skills to manage AWS
infrastructure in the team but lacking the skills to manage on-prem hardware,
be it the enterprise sales teams of cloud operators schmoozing CTOs of big
corporations and making them jump on a "going into cloud" strategy as some
kind of magic bullet to future-proof their corporations' IT, be it the
psychological effect that makes "using the cloud" apparently a mandatory thing
to be "cool" in todays' silicon valley culture, and therefore by extension the
whole worlds' IT engineering culture.

The most ironical of them all is this weird effect that drives people to
rationalize these things, writing comments like yours, because nobody likes to
admit they've painted themselves into a corner of lock-in effects. And of
course there's the irony of this all being history repeating itself: anyone
still remembering when IBM dominated the IT industry?

~~~
krageon
> that would allow such profit margins

The percentages don't _quite_ hit that amount of discount, but they are much
much higher than (I at least) expected.

------
throwaway6845
Mostly, headspace. If I run my own server, I just need to apply my existing
Ubuntu sysadmin knowledge. If I use AWS, I have to learn a whole load of AWS-
specific domain knowledge, starting with their utterly baffling product names.
My time is more valuable than that.

Also, sheer cost. Literally everyone I know in my particular part of the
industry uses Hetzner boxes. For what I do, it’s orders of magnitude cheaper
than AWS.

~~~
HatchedLake721
That’s how you get old, when your time is more valuable than a massive shift
in technology.

~~~
henriquez
Nah, we already did mainframes in the 1970s. Renting CPU time only makes sense
if you don’t need CPU time or you like wasting money.

------
shockinglytrue
Try running any service with an average egress exceeding 10 Mbit/s then tell
me cloud still makes sense. By the time you reach 1 Gbit/s the very idea of it
is enough to elicit a primitive biological defensive response.

We don't do on-prem but we do make heavy use of colo. The thought of cloud
growth and DC space consolidation some day pushing out traditional flat rate
providers absolutely terrifies me.

At some point those cloud premiums will trickle down through the supply chain,
and eventually it could become hard to find reasonably priced colo space
because the big guys with huge cash-flush pockets are buying up any available
space with a significant premium attached. I don't know if this is ever
likely, but growth of cloud could conceivably put pressure on available
physical colo space.

Similar deal with Internet peering. There may be a critical point after which
cloud vendors, through their sheer size will be able to change how these
agreements are structured for everyone.

~~~
jedberg
Netflix runs on the cloud and does 30% of all internet traffic.

That being said, 99% of that traffic is served from servers in colos now, but
10 years ago it was all served from CDN providers like Akamai, which is just a
specialized cloud.

~~~
toomuchtodo
This is kind of a big caveat (“Netflix is in the cloud but almost none of the
work is done there”), and something I have to mention to non tech decision
makers when they say “but Netflix!”. I even have a slide for presentations
just for this (“You Are Not Netflix”).

~~~
jedberg
99% of the work is done on the cloud. What comes off of those colo servers is
literally just bits streaming from disk to network. There is no transformation
or anything. No authentication, no user accounts, no database. Nothing.

Just static files served efficiently.

~~~
enneff
> What comes off of those colo servers is literally just bits streaming from
> disk to network

So... the core of their business?

~~~
jedberg
Not at all. It was so “not core” that it was outsourced.

The core of their business is recommendations, encoding, and authentication.
All of those are done 100% on the cloud.

~~~
sidibe
Sure but probably 99% of the

> 30% of all internet traffic.

is outside of the cloud.

~~~
anshumania
Anyone know the bill Netflix has for running on the cloud ?

------
catlas3r
Why stay on premise?

Cost. On-prem is roughly on-par in an average case, in my experience, but
we've got many cases where we've optimized against hardware configurations
that are significantly cheaper to create on-prem. And sunk costs are real.
It's much easier to get approval for instances that don't add to the bottom
line. But for that matter, we try to get our on-prem at close to 100%
utilization, which keeps costs well below cloud. If I've got bursty loads,
those can go to the cloud.

Lock-in. I don't trust any of the big cloud providers not to jack my rates up.
I don't trust my engineers not to make use of proprietary APIs that get me
stuck there.

Related to cost, but also its own issue, data transfer. Both latency and
throughput. Yeah, it's buzzwordy, but the edge is a thing. I have many clients
where getting processing in the same location where the data is being
generated saves ungodly amounts of money in bandwidth, or where it wouldn't
even been feasible to transfer the data off-site. Financial sector clients
also tend to appreciate shaving off milliseconds.

Also, regulatory compliance. And, let's be honest, corporate and actual
politics.

Inertia.

Trust.

Risk.

Interoperability with existing systems.

Few decisions about where to stick your compute and storage are trivial; few
times is one answer always right. But there are many, many factors to
consider, and they may not be the obvious ones that make the decision for you.

------
strags
Cost and Latency.

My team and I run the servers for a number of very big videogames. For a high-
cpu workload, if you look around at static on-prem hosting and actually do
some real performance bencharking, you will find that cloud machines - though
convenient - generally cost at least 2x as much per unit performance. Not only
that, but cloud will absolutely gouge you on egress bandwidth - leading to a
cost multiplier that's closer to 4x, depending on the balance between compute
and outbound bandwidth.

That's not to say we don't use the cloud - in fact we use it extensively.

Since you have to pay for static capacity 24/7 - even when your regional
players are asleep and the machines are idle, there are some gains to be had
by using the right blend of static/elastic - don't plan to cover peaks with
100% static - and spin up the elastic machines when your static capacity is
fully consumed. This holds true for anything that results in more usage - a
busy weekend, an in-game event, a new piece of downloadable content, etc...
It's also a great way to deal with not knowing exactly how many players are
going to show up on day 1.

Regarding latency, we have machines in many smaller datacenters around the
world. We can generally get players far closer to one of our machines than to
AWS/GCP/Azure, resulting in better in-game ping, which is super important to
us. This will change over time as more and more cloud DCs spring up, but for
now we're pretty happy with the blend.

------
hprotagonist
AI compute is so much cheaper on-prem that it's not even in question.

And there are clients that demand it.

And researchers, in general, like to do totally wacky things, and it's often
easier/cheaper to let us if you have physical access.

~~~
sdan
+1 on this. Get a nice server with some GPUs and you'd save a lot more than
paying the super expensive costs on cloud.

------
jabroni_salad
I'm in rural iowa and you really can't bank on a solid internet connection.
One of my clients decided to iaas-ify all their servers and it works great
except when it is windy out. They're on fixed wireless and the remote mast has
some sway to it. 3-4 times a year they get struck by lightning and have a
total work stoppage until all their outside gear can get replaced. Even their
VDI is remote so all the thin clients just disconnect and they are done for
the day.

Also, my clients aren't software development firms. They are banks and
factories. They buy a software based on features and we figure out how to make
it work, and most of the vendors in this space are doing on-prem non-saas
products. A few do all their stuff in IAAS or colo but a lot of these places
are single-rack operations and they really don't care as long as it all works.

A lot of people in small/midsize banks feel like they are being left out. They
go to conferences and hear about all the cool stuff in the industry but the
established players are not bringing that to them. If you can stomach the
regulatory overhead, someone with drive could replace
finastra/fiserv/jackhenry. Or get purchased by them and get turned into yet
another forever-maintenancemode graveyard app.

------
sdan
Founder of a growing startup:

Started with a cluster of Raspberry Pis and expanded onto an old desktop.
Primarily did this for cost (raspberry pis alone were more powerful than a GCP
$35/mo instance). Everything was fine until I needed GPUs/handling more
traffic than those Raspberrys could handle. So I expanded by including cloud
instances in my Docker Swarm cluster (tidbit: Using Traefik and WireGuard)

So half on-prem half in the cloud. Honestly just scared GCP might one day
cancel my account and I'll lose all my data unless I meet their demands (has
happened in the past) so that half on-prem stores most of the data.

~~~
chickenpotpie
At $35/month though GCP would only have to save you a half an hour of
maintenance for it to be worth it though.

~~~
sdan
Well, given that I am using Docker it doesn't really matter much... but the
bigger issue is: GCP in the past has completely blocked access from accounts
when they detect random things.

Unless I meet their demands, my entire infra is gone/down for days, which I
can't deal with.

------
XCSme
I use a $5/mo DigitalOcean VPS droplet instead of AWS or other "cloud"
service. I only have to host an analytics dashboard (
[https://usertrack.net/](https://usertrack.net/) ), I don't need scaling and
this way I know exactly how much I will pay. The resources are more than
enough for my needs and I don't think it could be much cheaper even on the
most optimized pay-per-minute of use cloud platforms.

I also have some other APIs hosted in the same way (eg. website thumbnail
generation API), for the very low traffic I have and no chance of getting
burst traffic I think the use case of a VPS or dedicated server is perfect.

~~~
chickenpotpie
Whenever I need to host something small and I’m trying to decide between DO
and AWS I always ask myself. Would I rather be surprised by the bill or my
website crashing from too much traffic? I almost always pick DO because I
don’t want to mess something up and lose a few hundred dollars.

~~~
jackson1442
Wholeheartedly agree. I think AWS is moving in the right direction with
Lightsail[0], which is a service very similar to DO droplets and includes
transfer. Nice if you want to use AWS for like one or two other services, but
I tend to still go with DO for small things.

[0]: [https://aws.amazon.com/lightsail/](https://aws.amazon.com/lightsail/)

~~~
XCSme
That sounds interesting. By "moving in the right direction" do you mean that
it's still in beta or not released yet? Or that it's just the first step of
many to come?

~~~
adventured
Lightsail works well now. It's a little over three years old. In the first
year or so after it was released, they were notorious for being slow in most
regards compared to their peers (it launched using rebranded instances from
AWS, and used spinning disks, going up against SSDs their competitors were all
using). They've largely caught up on performance with DigitalOcean, Linode,
Vultr and similar.

That said, I've stuck with DigitalOcean even though Lightsail tests fine. I've
had a great experience over the years with DO and see no reason to leave.

------
michaelt
On-prem makes your cost control proactive, rather than reactive. Nobody gets a
new server without first having a purchase order approved - and the burden to
get that approval falls on the person who wants the server.

In the cloud, at least the way it's generally used, cost control is reactive:
You get a bill from AWS every month, and _if you 're lucky_ you'll be able to
attribute the costs to different projects.

This is both a strength and a weakness: on-premise assets will end up at much
higher utilisation, because people will be keen to share servers and dodge the
bureaucracy and costs of adding more. But if you consider isolation a virtue,
you might prefer having 100 CPUs spread across 100 SQL DBs instead of 50 CPUs
across two mega-databases.

------
doctor_eval
Lots of great insights here, which fully accord with my experience, even in
the small end of town.

About a year ago, I was in a meeting with my new CEO (who had acquired my
company). My side of the business had kept hardware in-house, his was in AWS.
We had broadly similar businesses in the same industry and with the same kind
of customers.

My side of the business needed to upgrade our 5+ year old hardware. The quote
came to $100K; the CEO freaked out. I asked him how much he spent on AWS?

The answer was that they spent $30K __per month __on AWS.

The kicker is that we managed 10x as many customers as they did, our devops
team was half the size, and we were rolling out continuous deployment while
they were still struggling to automated upgrades. Our deployment environment
is also far less complicated than theirs because there isn't a complex
infrastructure stack sitting in front of our deployment stack.

There was literally no dimension on which AWS was better than our on-prem
deployment, and as far as I was able to tell before I quit, the only reason
they used AWS was because everyone else was doing it.

~~~
pickle-wizard
With all the job hopping that goes on in tech, there is a lot of Resume Driven
Development. People want to use AWS because it will help them get their next
job.

I'm finally in a job that I'm happy with and can see myself staying here until
retirement. I have noticed that has changed my technology recommendations. For
example we recently started looking at configuration management tools. Ansible
is the obvious choice from a resume perspective as it is very popular. I ended
recommending Powershell DSC. Why, because our environment is mostly windows,
the team is familiar with Powershell, and for our use case is much faster.
Powershell DSC is not as popular so it won't help me get another job. When it
comes time to expand the team, I can hire someone who understands
configuration management tools or powershell, and get them up to speed in a
day or two.

------
adreamingsoul
Personally, I'd rather have capex than opex.

My observations from working with, and in the "cloud":

The "cloud" does benefit from it's scale in many ways. It has more engineers
to improve, fix, watch, and page. It has more resources to handle spikes,
whales, and demand. Almost everything is scale tested and the actual physical
limits are known. It is damn right impressive to see what kind of traffic the
cloud can handle.

Everything in the "cloud" is abstracted which increases complexity.
Knowledgeable engineers are few and far between. As an engineer you assume
something will break, and with every deployment you hope that you have the
right metrics in place and alarms on the right metrics.

The "cloud" is best suited for whales. From special pricing to resource
provisioning, they get the best. The rest is trickled down.

Most services are cost-centers. Very few can actually pay for the team and the
cost of its dependencies.

It's insane how much VC money is spent building whatever the latest trend of
application architecture is. Very few actually hit their utilization
projections.

------
erulabs
We hear from our customers mostly what has been said here: cost and mental
overhead. There is a bit of a paradox - companies that plan to grow
aggressively are wary of AWS bills chopping their runway in half - they're
very aware of _why_ cloud providers give out a year for free to most startups
- they recoup that loss very fast once the cash faucet opens up.

What really gets me is that most cloud providers promise scalability, but
offer no guard-rails - for example diagnosing performance issues in RDS - the
goal for most cloud providers is to ride the line between your time cost and
their service charges. Sure you can reduce RDS spend, but you'll have to spend
a week to do it - so bust out the calculator or just sign the checks. No one
will stop you from creating a single point of failure - but they'd happily
charge for consulting fees to fix it. There is a conflict on interest - they
profit from poor design.

In my opinion, the internet is missing a platform that encourages developers
to build things in a reproducible way. Develop and host at home until you get
your first customers, then move to a hosting provider down the line. Today,
this most appeals to AI/ML startups - they're painfully aware of their idle
GPUs in their gaming desktops and their insane bill from Major Cloud Provider.
It also appeals to engineers who just want to host a blog or a wedding
website, etc.

This is a tooling problem that I'm convinced can be solved. We need a
ubiquitous, open-source, cloud-like platform that developers can use to get
started on day 1, hosting from home if desired. That software platform should
not have to change when the company needs increased reliability or better air
conditioning for their servers. If its a Wordpress blog or a minecraft server
or a petabyte SQL database - the Vendor should be a secondary choice to making
things.

~~~
sbrother
I've found that Kubernetes mostly solves this problem. I say mostly because
for AI/ML workloads that require GPUs, we still rely on running things on bare
metal locally, and deploying with GKE's magic annotations and Deep Learning
images. But for anything else, I haven't had an issue going all in on k8s at
the beginning, even with very small teams.

~~~
erulabs
Yep! My startup is [https://kubesail.com](https://kubesail.com), so I agree :)

As for ML on Kube, I agree, there have been and still are some rough edges.
The kernel drivers alone make a lot of out-of-the-box Kubernetes solutions
unusable. That said, we've had a lot of success helping people move entirely
onto kube - the mental gain alone from ditching the bash scripts or ansible
playbooks (etC) alone is pretty freeing.

------
dogecoinbase
Yes. Three major reasons:

\- Cost. It's vastly cheaper to run your own infra (like, 10-100x -- really!).
The reason to run in cloud is not to save money, it's to shift from capex to
opex and artificially couple client acquisition to expenditure in a way that
juices your sheets for VCs.

\- Principle. You can't do business in the cloud without paying people who
also work to assemble lists of citizens to hand over to fascist governments.

\- Control. Cloud providers will happily turn your systems off if asked by the
government, a higher-up VP, or a sufficiently large partner.

EDIT: I should add. Cloud is great for something -- moving very fast with
minimal staffing. That said, unless you get large enough to renegotiate you
will get wedged into a cost deadend where your costs would be vastly reduced
by going in-house, but you cannot afford to do so in the short term.
Particularly for the HN audience, take care to notice who your accelerator is
directing you to use for cloud services -- they are typically co-invested.

~~~
PaulWaldman
Regarding the shift in CapEx to OpEx, on-prem servers can also be leased,
keeping their costs in OpEx.

------
reilly3000
I’ve been studying like a fiend to get AWS certs and thoroughly understand the
cloud value proposition, especially for ephemeral workloads and compliance
needs. I’m all for cloud solutions that make sense and love when
serverless/usage-only systems can’t be deployed. That said, I recently started
work on a friend’s system that he has had running in colo for a long time.
It’s absolutely insane how long his systems have been up. There are processes
that have been alive since 2015, with some hosts having uptime linger than
that. He’s got a nice HA configuration but hasn’t had any incidents that have
triggered failover. He recently built a rack for his home with 384gb ram and
gobs of cpu across 3 nodes, with rack, nice switch and UPS for just shy of
$2500 ( he is quite the bargain hunter... ). I did some quick math and found a
similarly equipped cluster (of just VMs, not dedicated hosts) has a 1.1 month
break-even with on-demand costs, no bandwidth considered. Sure, maybe a 1 year
reservation could make it a 2-3 month break even instead, but why? Those
machines can easily give him 3-5 years of performance without paying another
dime.

If you can feasibly run workloads onpremise or colo and have a warm failover
to AWS you could probably have the best of all worlds.

~~~
whatsmyusername
If he has processes that have been up since 2015 how is he patching? That's
one of my biggest gripes with on-prem, it's easy to leave something that works
alone... until it gets popped by a 5 year old vuln.

In cloud I'm constantly looking at what we have because I have good billing
tools in place to see what we're paying for.

------
grantlmiller
I always find it important to separate "cloud" into 2 categories:

1\. IaaS - Which I mainly define as the raw programmable resources provided by
"hypercloud" providers (AWS, GCP, Azure). Yes, it seems that using an IaaS
provider with a VPC can provide many benefits over traditional on-prem data
centers (racking & stacking, dual power supply, physical security, elasticity,
programmability, locations etc).

2\. SaaS - I lump all of the other applications by the hundreds of thousands
of vendors into this category. I find it hard to trust these vendors the same
way that I trust IaaS providers and am much more cautious of using these
applications (vs OSS or "on-prem software" versions of these apps). They just
don't have the same level of security controls in place as the largest IaaS
providers can & do (plus the data is structured in a way that is more easily
analyzed, consumed by prying eyes).

~~~
opportune
What about first-party SaaS? Those can also be big features that bring people
to some cloud providers. Not all SaaS requires you to trust your
data/availability to some random vendor. Of course those first-party SaaS
aren't typically suitable for lift-and-shift by their very nature, and they
can still have some rough edges, but IMO you can expect them to be almost as
reliable as IaaS

~~~
grantlmiller
First-party SaaS meaning things like RDS, DBaaS, queues, LBs etc? Most of that
I would sort of put into a IaaS controlled PaaS, rather than true IaaS SaaS.
Yes, these are generally higher on the trust spectrum as they don't involve
additional vendors accessing/managing/storing data.

~~~
opportune
A major one I'm thinking of is BigQuery, also of course all the various
db/queue solutions outside of your typical S3 clone as you mentioned. That
would make sense viewing them as platforms though

------
dijit
I work for a large video games publisher, as you might expect we use a lot of
windows.

Windows server licenses on AWS and GCP are hundreds of times more expensive at
our scale. Incidentally we actually do have some cloud infra and we like it,
but the licensing cost is half the total price of the instance itself.

In fact, you might not know this but games are relatively low margin, and we
have accidentally risked the companies financial safety by moving into the
cloud.

~~~
whatsmyusername
TBF windows licensing in general is a shit show, to the point where just
handling that is a specialized ability potentially warranting a full time
position.

------
tr33house
I'm a solo founder who's bootstrapped a saas that's in one state. I'd started
out with the cloud then moved to a private cloud in a colocated data center.
Saved more than 80% in monthly costs. Got faster speeds, better reliability
and a ton of extra compute and network capacity. I just bought used servers
from eBay that are retired from big corps. Nothing significant has really
changed in the last five years on compute so I'll happily take their
depreciated assets :)

__Modern__ servers are really awesome and I totally recommend them. You can do
a ton remotely.

------
drej
Many of the stories here are from large companies, where the costs are quite a
different beast. I want to offer an opposite view - from a small company
(20-30 people), which is usually the kind of company best suited for the
cloud.

We ran a number of modelling jobs, basically CPU intensive tasks that would
run for minutes to hours. Investing in on-prem computers (mostly workstations,
some servers), we got very solid performance, very predictable costs and no
ops issues. Renting beefy machines in the cloud is very expensive and unless
you get crafty (spot and/or intelligent deployment), it will be prohibitive
for many. Looking at AMD's offering these days, you can get sustained on-prem
perf for a few dollars.

Three details of note: 1) We didn't need bursty perf (very infrequently) - had
this been a need, the cloud would make a lot more sense, at least in a hybrid
deployment. 2) we didn't do much networking (I'm in a different company now
and we work with a lot of storage on S3 and on-prem wouldn't be feasible for
us), 3) we didn't need to work remotely much, it was all at the office.

Obviously, it was a very specific scenario, but given how small the company
was, we couldn't afford people to manage the whole cloud
deployment/security/scaling etc. and beefy workstations was a much simpler and
more affordable endeavour.

------
snarfy
The idea of the cloud is to only pay for what you use. Your on-premise server
is idle 99% of the time so why are you paying for a full server?

If that's not true, it turns out it's quite expensive to run things in the
cloud. If your workload is crunching numbers 24/7 at 100% cpu, it's better to
buy the cpu than to rent it.

~~~
Polylactic_acid
Cloud servers tend to be more reliable as well if you don't run your own
datacenters. We have lost our internet connection or power 3 times in the last
year in the office. Its not the end of the world since we can go to 4g for our
own usage but if our servers were hosted locally this would be a huge issue.

~~~
dathinab
Don't forget that between cloud and servers in the company there are still VPS
and rented dedicated hardware in a data center.

So you:

1\. Don't manage hardware.

2\. But manage a server (OS+software stack).

3\. Have reliable internet, power and physical security from the data center
you are renting your hardware from (if you trust them fully!).

4\. Have fixed cost but also fixed resources. Tends to be cheaper for many
tasks. Especially CPU/GPU heavy ones.

~~~
Polylactic_acid
I consider VPSs to be cloud servers. Is this not common?

~~~
XCSme
I mentioned in another comment that I use VPS and not cloud services. I think
of cloud as the auto-scaling infrastructure with dynamic pricing. I think of
VPS as just sharing a dedicated machine with others, so each one gets a few
cores and shares other resources. The implementation of VPSs nowdays is
probably more similar to cloud services, where your own space might be moved
around to another physical machine without any downtime.

~~~
Polylactic_acid
So you consider cloud servers to be what most people call serverless
(S3/serverless functions/etc)?

~~~
XCSme
I do hate the term "serverless" as it makes no sense, but I think of cloud as
a system that automatically spins-up/down VPSs based on your current usage.
This means the infrastructure/software also allows for automatically load-
balancing between those VPSs. So I think of cloud as the VPS servers that are
used to host the actual data + the layer on top that does all the scaling,
provisioning, load-balancing, etc.

------
mattbeckman
We spend ~$50k/mo on serverless infrastructure on AWS.

It hurts sometimes, given we were fully colocated about 4 years back, and I
know how much hardware that could buy us every month.

However, with serverless infra we can pivot quickly.

Since we're still in the beta stage, with a few large, early access
partnerships, and an unfinished roadmap, we don't know where the bottlenecks
will be.

For example, we depended heavily on CloudSearch, until it sucked for our use
case, so we shifted to Elasticsearch, and ran both clusters simultaneously
until we were fully off of CS. If we were to do that on-prem, we'd have to
order a lot more hardware (or squeeze in new ES cluster VMs across heavy
utilization nodes).

With AWS, a few minutes to launch a new ES cluster, dev time to migrate the
data, followed by a few clicks to kill the CloudSearch cluster.

Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher
upfront, lower long term, but ceiling.

~~~
brickbrd
If "Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher
upfront, lower long term, but ceiling" is true, then how come the revenue of
cloud companies keeps going up?

That would mean the incoming rate of users who are just starting off and find
Cloud worthwhile is more exit rate of mature users who are finding on-prem
more worthwhile than cloud

~~~
wvenable
If you're spending more than 50k/month on AWS where is the money to move to
on-prem? When they got you, they got you.

------
walterbell
The (startup) Oxide podcast has good history/stories about on-prem servers,
from veterans of pioneering companies. They are fans of open-source firmware
and Rust, and are working to make OCP-based servers usable for on-prem. In one
podcast, they observed that cloud is initially cheaper, but can quickly become
expensive with growth. There is a time window where you can still switch from
cloud to on-prem, but if that window is missed, you're left with high
switching costs and high cloud fees.

[https://oxide.computer/podcast/](https://oxide.computer/podcast/)

~~~
mapgrep
Their co founder Bryan Cantrill gave a talk at Stanford on what they are
trying to do, essentially offer on prem servers comparable to what
“hyperscalers” like Google and Facebook put in their data centers — highly
efficient and customizable (in low level software) iirc.

[https://youtu.be/vvZA9n3e5pc](https://youtu.be/vvZA9n3e5pc)

------
PaulWaldman
Manufacturing. The cost to a factory if the internet is down is too great.
Each facility has its own highly redundant virtualization infrastructure
hosting 50 to 100 VMs.

~~~
eitally
I was in manufacturing IT before moving to big tech. Our big campuses in the
US & Europe had 40-80mbps internet circuits. The remote facilities in
developing countries often only had 10mbps MPLS connections to a regional hub.
To be 100% honest, we had 10x the outages caused by crappy local
infrastructure than anything having to do with a SaaS service or IaaS/PaaS
provider. Seriously, things like bad storms, a snake (cobra!) sneaking into
the server room and frying itself and a machine it was snuggling against,
utility workers accidentally severing cables, generators failing during power
outages, labor strikes, and so much more. Moving to the cloud -- or even just
hosting everything centrally -- was much more stable than maintaining a fleet
of distributed machines.

------
MaulingMonkey
I work in gamedev. Build servers, version control, etc. are almost always on-
premise, even if a lot of other stuff has been transitioned to the cloud.
There's a few reasons:

1) Bandwidth. I routinely saturate my plebian developer gigabit NIC links for
half an hour, an hour, longer - and the servers slurp down even worse. In an
AAA studio I am but one of hundreds of such workers. Getting a general
purpouse internet connection that handles that kind of bandwidth to your
heavily customized office is often just not really possible. If you're lucky
your office is at least in the same metro area as a relevant datacenter. If
you're really lucky you can maybe build a custom fiber or microwave link
without prohibative cost. But with those kinds of geographical limitations,
you're not so much relying on the general internet, so much as you're
expanding your LAN to include a specific datacenter / zone of "the cloud" at
that point.

2) Security. These servers are often completely disconnected from the
internet, on a completely separate network, to help isolate them and reducing
data exfiltration when some idiot installs malware-laden warez, despite clear
corporate policy threatening to fire you if you so much as even _think_ about
installing bootleg software. Exceptions - where the servers _do_ have internet
access - are often recent, regrettable, and being reconsidered - because of,
or perhaps despite, draconian whitelisting policies and other attempts at
implementing defense in depth.

3) Customizability. Gamedev means devkits with strict NDAs and physical
security requirements, and a motley assortment of phone hardware, that you
want accessible to your build servers for automatic unit/integration testing.
Oddball OS/driver/hardware may also be useful for such testing. Sure, if you
can track down the right parties, you might be able to have your lawyers
convince their lawyers to let you move said hardware into a datacenter, expand
the IP whitelists, etc... but at that point all you've really done is made it
harder to borrow a specific popular-but-discontinued phone model from the
build farm for local debugging when it's the only one reproducing a specific
crash when you want to debug and lack proper remote debug tooling.

...there are some inroads on the phone farms (AWS Device Farm, Xamarin Test
Cloud) but I'm unaware of farms varied desktop hardware or devkits. Maybe they
exist and just need better marketing?

I have some surplus "old" server hardware from one such gamedev job. Multiple
8gbit links on all of them. The "new" replacement hardware often still
noticably bottlenecked for many operations.

------
mrmrcoleman
There's a renewed interest in on-prem bare metal recently with a lot of
different offerings helping to make various parts of the stack easier to
manage.

Awesome bare metal is a new repo created by Alex Ellis that tracks a lot of
the projects: [https://github.com/alexellis/awesome-
baremetal](https://github.com/alexellis/awesome-baremetal)

Also we (Packet) just open sourced Tinkerbell, our bare metal provisioning
engine: [https://www.packet.com/blog/open-sourcing-
tinkerbell/](https://www.packet.com/blog/open-sourcing-tinkerbell/)

------
chime
1\. 500TB of storage for 3-6mo of CCTV footage.

2\. Bought a hanful of $700 24 core Xeons on eBay 2 years ago for 24/7 data
crunching. Equivalent cloud cost was over $3000/mo. On-Prem paid off within a
month!

3\. Nutanix is nice. Awesome performance for the price and almost no
maintenance. Got 300+ VDI desktops and 50+ VMs with 1ms latency.

~~~
junar
> $700 24 core Xeons on eBay 2 years ago

Can you clarify? I don't think such a product exists as a single chip at that
price point. The Threadripper 3960X costs $1400, and that released less than a
year ago.

Edit: Looking up Intel chips on Wikipedia, I think you might be using
12-core/24-thread chips...

[https://en.wikipedia.org/wiki/Skylake_(microarchitecture)#Xe...](https://en.wikipedia.org/wiki/Skylake_\(microarchitecture\)#Xeon_Bronze_and_Silver_\(dual_processor\))

~~~
fiveguys94
I picked up three Dell R900's for an average of $200 each, with 4x Xeon E7450
and 128GB of ECC RAM. No hyperthreading (2011!), it's 24 real cores.

They're noisy and use lots of power, but you can't argue with the value for
money.

~~~
icedchai
I have one similar to this. I don't think it is an R900, but it's an older 1U
rack mount. I forget if it's a 12 or 24 core xeon, but it was dirt cheap, came
with 72 gigs of RAM, and sounds like a jet engine turning it on. I recently
built a Ryzen box with 128 gigs of RAM and it's much quieter...

------
mattmireles
CEO of an AI startup & former AWS employee here.

The cloud sucks for training AI models. It's just insanely overpriced in a way
that no "Total Cost of Ownership" analysis is going make look good.

Every decent AI startup––including OpenAI––has made significant investments in
on-premise GPU clusters for training models. You can buy consumer-grade NVIDIA
hardware for a fraction of the price that AWS pays for data center-grade GPUs.

For us in particular, the payback on a $36k on-prem GPU cluster is about 3-4
months. Everything after that point saves us ~$10k / month. It's not even
close.

When I was AWS, I tried to point this fact out to the leadership––to no avail.
It simply seemed like a problem they didn't care about.

My only question is why isn't there a p2p virtualization layer that lets
people with this on-prem GPU hardware rent out their spare capacity?

~~~
blueblisters
Are TPUs too application specific to replace GPUs? It seems cloud TPUs could
be competitive with GPUs in terms of $ per number of target epochs, provided
you can do data parallelism for your workloads.

Also, IBM offers bare metal pricing which is _somewhat_ cheaper than
virtualized instances attached to GPUs (and faster too).

I think GPU virtualization is not quite there yet because Nvidia does not give
access to core GPU functionality needed for efficient virtualization - you're
stuck with using their closed-source libraries and drivers.

------
kasey_junk
Bandwidth is the big reason to stay on-prem. Good peering contracts can more
than make up for any cloud advantages for bandwidth intensive uses.

Now the hard part is turning those cost advantages into operational
improvements instead of deficiencies.

------
INTPenis
Becuse we don't trust a foreign cloud provider with our client's data. Why is
that so hard to understand?

All the best cloud providers are from the US and as a european company with
clients in european government and healthcare we are often not morally or
legally allowed to use a foreign provider.

The sad thing is that this is an ongoing battle between people on a municipal
level who believe they can save money in clouds, and morally wiser people who
are constantly having to put the brakes on those migration projects.

~~~
v4dok
What about the existing/upcoming technologies of let you use the cloud without
trusting it?

~~~
INTPenis
Doesn't matter because all US companies are subject to US laws and agencies.
Even if the data is in Ireland, they are obliged to cooperate and before you
know it all our patient records are leaked in the states.

And speaking of Ireland, I have a memory of a Microsoft EULA saying that even
if the data is stored in Ireland they can't guarantee that it won't be
transferred to the US.

------
shortlived
Small company here (30 total people, 8 in IT/software).

\- unwillingness to seed control of the critical parts of our software
infrastructure to a third party.

\- given our small team size and our technical debt load we are not currently
able to re-architect to make our software cloud-ready/resilient.

\- true cost estimates feel daunting to calculate, whereas on-prem costs are
fairly easy to calculate.

~~~
aspyct
I agree with the last 2 points. Estimates are hard to get right, and
rearchitecting an existing app is probably not worth it.

What about your first point though? Do you not trust a 3rd party to maintain
infrastructure properly? In what way?

------
janstice
I'm application architect at enterprise-type org. We have a few SaaS
applications, but all the big dogs, including custom dev, run in-house on a
dual data centre vmware environment. It's cheaper for us to spin up more VMs
in-house, so there's no real cloud driver for things that just live on VMs. On
the other hand, our ops team are still working on network segmentation and
self-service, but I regularly get a standard VM provisioned in less than 20
mins. If we had to buy new data centres it might be different.

But the real reason we're not deeper in the cloud is that our business-types
insist on turn of the century, server-based software from the usual big
vendors, and all the things that integrate with them need to use 20th century
integration patterns, so for us migrating to the cloud (in stages at least)
would have drawbacks from all options without the benefits. It's only where we
have cloud-native stuff that we can sneak in under the radar for stand-alone
greenfields projects, or convince the business types that they can replace the
Oracles and Peoplesofts with cloud-first alternatives will things really
change.

~~~
jasonv
Last company I was at was more or less as you describe.

Now.. at a company in a different industry, there's a 5+ year plan to move
100% to cloud. Nascent efforts are about 18 mos old already, no apps are live
yet.

Fortunately, they've been using a container approach for their on-prem stuff
for a while, so some stuff can move over pretty easily, a lot of things will
get a touch-up or more interesting upgrade along the path to the cloud
environment.

Not even talking about de-commissioning the DCs yet, but those will get
defunded as things go on.

------
olivierduval
Security... not against hacker but against provider's government.

I worked for some french or european companies, with IPs and sensitive
informations, and US business competitors. By the US law, US companies may
have to let the US gov spy on their customers (even non US, even on non US
location), so this may be a problem for strategic sectors, like defense for
example.

In that case, sensitive informations is required to be hosted in country by a
company of the country, under the country law.

Of course, it's not against "cloud" in general... only against US cloud
providers (and chinese, and...)

------
MattGaiser
I work on two projects, neither of which use cloud.

For my day job, it is privacy and legal constraints. I work for the government
and all manner of things need to be signed off on to move to cloud. We could
probably make it work, but in government, the hassle of doing so is so large
that it is not going to happen for a long time.

In my contract project, it is a massive competitive advantage. I won't go into
too many details, but customers in this particular area are very pleased that
we do not use a cloud provider and instead host it somewhat on-premise. I
don't see a large privacy advantage over using the cloud, but the people
buying the service do simply because they are paranoid about the data and
every single one of them could personally get in a lot of trouble for losing
the data.

Not my project, but intensive computing requirements can be much more cheaply
filled by on-premise equipment (especially if you don't pay for electricity),
so my university does most of its AI and crypto research on-premise.

------
axaxs
My company moved from all on prem, to all in AWS. Having used both, I'd much
rather go back to on prem. I did architecture, capacity planning, system
tuning, deployments, etc. I knew everything about all of them, and treated
them as sacred. The next generation came in, deciding not to learn anything
about systems and brought in the 'systems as cattle' attitude and everything
that comes with it.

I try to remain objective, there are some pros to AWS, but I still much prefer
my on prem setup. It was way cheaper, and deployments were way faster.

------
sqldba
\- Uptime

The number and frequency of outages in Azure are crazy. They happen non-stop
all year around. You get meaningless RCAs but it never seems to get better,
and if it did, you'd have no way of knowing.

Compare this with doing stuff internally - you can hire staff, or train staff,
and get better. In the long run outsourcing and trusting other companies to
invest in "getting better" doesn't end very well. Just because they moved
their overall metrics overall from 99.9 to 99.91 may not help your use case.

\- Reliability

Their UIs change every day, there's no end to end documentation on how things
work. There's no way to keep up.

\- Support

Azure's own support staff are atrocious. You have to repeatedly bang your head
against the wall for days to get anyone who even knows the basic stuff from
their own documentation.

But it's also difficult to find your own people to do the setup too. Sure,
lots of people can do it, but because it's new they have little experience and
end up not knowing much, unable to answer questions, and building garbage on
the cloud platform. Because there's no cloud seniority, it hasn't been around
for long enough.

\- Security

Cloud providers have or can get access and sometimes use it.

\- Management

I've seen too many last minute "we made a change and now everything will be
broken unless you immediately do something" notifications to be happy about.

\- Cost

It's ridiculously expensive above a certain scale, and that scale is not very
big. I don't know if it's because people overbuild, or because you're being
nickel-and-dimed, or if you're just paying so many times above normal for
enterprise hardware and redundancy. It's still expensive.

Yes, owning (and licensing) your own is expensive too.

For smaller projects and tiny companies, totally fine! It's even great!

\- Maturity

People can't manage cloud tools properly. This doesn't help with costs above.

PS: I don't think any other cloud service is better.

------
TedLePoireau
Not exactly on-premise but we rent 2 big dedicated server (ovh) + install
VMWare ESXi on them. Going to the cloud would cost more, the price would be
unpredictable, only to solve a scale problem we won't have. And customers love
to know their data are hosted in France by a French company, not by Google or
Amazon :)

------
ROFISH
Network storage. MacBooks don't have a lot of space and artists like to make
HUGE psd files.

Bonus points for small stuff like RADIUS for wifi and stuff. Groups charging
$5/user for junk like that is absolutely awful with a high number of staff.

With a staff of 100, a single box with a bunch of hard drives is two months
worth of cloud and SaaS.

TCO needs to come down by like at least 100x before I consider going server-
less.

------
throwaway7281
We own infra because we need to own and control it and also because it's just
vastly cheaper at the scale we use.

Besides, we do have things like our own S3, k8s and other cloud-ish utilities
running so we do not miss out that much, I guess.

------
nemacol
This conversation sounds a lot like the debate around globalization and
outsourcing manufacturing to me. It might be a stretch but there is something
here.

There is room for both Cloud and On-Prem to exist. This endless drive by
industry to push everyone to cloud infrastructure and SaaS, in my humble
opinion, will look exactly like the whole supply chain coming from the east
during a pandemic.

The economics of it look great in a lot of use cases, but putting our whole
company at the mercy of a few providers sounds terrible to me. Even more so
when I see posts on HN about folks getting locked out of their accounts with
little notice.

It does not take much to bring our modern cloud to a grinding halt. For
example, a mistake by an mostly unheard of ISP lead to a massive outage not
less than a year ago(1).

It was amazing to see the interconnections turn to cascading issues. 1 ISP
goofs. 1-2 major providers have issues and the trickle down effect was such
that even services that thought they were immune from cloud issues were
realizing that they rely on a 3rd party that relies on a different 3th party
that uses cloudflare or AWS.

So, even though I think the cloud is (usually) secure, stable, resilient,
etc... I still advocate for its use in moderation and for 2 main use cases.

1 - elastic demands. Those non-critical systems that add some value or make
work easier. Things we could do without for several days and not hurt the
business much.

2 - DR / Backup / redundancy. We have a robust 2 data center / DR fail over
system. Adding cloud components to that seems reasonable to me.

(1)[https://slate.com/technology/2019/06/verizon-dqe-outage-
inte...](https://slate.com/technology/2019/06/verizon-dqe-outage-internet-
cloudflare-reddit-aws.html)

Edit: Spelling and clarity

Edit2: New reasons to stay on prem are happening all the time.
[https://www.bleepingcomputer.com/news/security/microsofts-
gi...](https://www.bleepingcomputer.com/news/security/microsofts-github-
account-allegedly-hacked-500gb-stolen/)

------
throwaway028374
I worked for a famous large tech company that makes both hardware and
software. On stuff that runs on customer datacenters.

There are plenty of companies that run their infrastructure to keep their data
secure and accessible.

It's not the type of companies that blog about their infra or are popular on
HN. Banks and financial institutions, telcos, airlines, energy production,
civil infrastructure.

Critical infrastructure need to survive events larger than a datacenter
outage. FAANGs don't protect customers from legal threats, large political
changes, terrorist attacks, war.

------
sudhirj
The way I think about it this: not using the cloud is like building your own
code editor or IDE after assembling your own laptops and desktops. It may make
you happier and it’s a great hobby, but if you’re trying to run a business you
need to do a cost benefit analysis.

We currently have double digit petabytes of data stored in our own data
centres, but we’re moving it to S3 because we have far better things to do
with our engineers that replace drives all day, and engineering plus hardware
is more expensive than S3 Deep Archive - but it wasn’t until Deep Archive came
out.

We put out hundreds of petabytes of bandwidth, and AWS is horribly expensive
at first glance, but if you’re at that scale you negotiate private pricing
that brings it to spitting distance of using a COLO or Linode/Hetzner/OVH -
the distance is small enough that the advantages of AWS outweigh it, and it
allows us to run our business at known and predictable margins.

Besides variability (most of our servers are shut nights, weekends and when
not required), op ex vs cap ex, spikes in load (100X to 1000X baseline when
tickets open), there’s also the advantage of not needing ops engineers and
being to handle infrastructure with code. If you have a lot of ops people and
don’t need any of the advantages, and you have lots of money lying around that
you can use on cap ex, and you have a predictable load pattern, and you’ve
done a clear cost benefit analysis to determine that building your own is
cheaper, you should totally do that. Doesn’t matter what others are doing.

~~~
nihil75
You're not building everything from scratch on-prem. There are excellent tools
for deploying and managing infrastructure like Terraform, Ansible, Puppet that
funny enough are used to deploy to cloud as well. Add a self-hosted Kubernetes
cluster to that and your on-prem is not that different than a cloud.

As for ops people - you might not need an engineer to replace failed hard-
drives, but you'll need a DevOps person to manage Cloud Formation templates
and such, and they cost more.

------
skiril
One of the reason is a legacy systems. Some companies are too tied up to old
custom made systems built on old software and hardware virtually not
convertible to cloud. You will be surprised but there are big corporation
still using AS400 and not planning to switch anytime soon. If you heard in
recent news US unemployment system was still built on COBOL... In 2020...
Another reason is a cost. I love AWS! Its fantastic to be able to create and
launch servers or the whole farm of servers in the matter of minutes! And
ability to convert physical servers to virtual and upload them to the cloud is
breathtaking! But my monthly bill started at $300 and grew to $18K per month
in less than 3 years. And that was for just a few virtual servers with Windows
OS and SQL. My company realized that we can have a state of the art datacenter
with WmWare and SAN on premises for the fraction of that price. Put second one
on the other coast (one of our other offices) and you have your own cloud with
better ping and six digits figure saving a year. For the last I would name
vendor lock. With vSphere its very easy to move your virtual servers between
AWS, Azure and Google (assuming you can afford all 3 and licensing cost of
WmWare) but have you ever tried to "download" your server back to premise?
It's virtually impossible or made so hard by cloud players trying to keep you
up they're in the clouds. With all said I read that Netflix (I believe its
Netflix) saving hundreds of millions dollars per year by using Amazon instead
of its own servers. I also read somewhere that Dropbox moved away from AWS...

------
bluedino
A job or two ago:

Everything on-site for a couple reasons (50 servers). Mainly because as a
manufacturing company, machines on the shop floor need to talk to the servers.
This brings up issues of security (do you really wan to put a 15 year old CNC
machine 'on the internet'?). Also, if our internet connection has issues, we
still need to build parts.

The other big part of it is the mindset of management and the existing system,
which was built to run locally, does Amazon offer cloud hosted Access and
Visual Basic workers?

------
nitwit005
We're looking at moving from AWS to having some machine space rented in two
data centers. The reason is purely cost.

There are still some computers on site due to equipment being tied to it,
telephony stuff, etc.

My last company was looking at "moving to the cloud", with the idea that its
data centers were too expensive, but found out that the cloud solutions would
be even more expensive, despite possible discounts due to the size. They still
invested in it due to some Australia customers wanting data to be located
there.

------
tcbyrd
I haven't personally done detailed cost analysis lately, but if you have
systems that regularly operate at 80+% of capacity, I can't see how the
operating costs of any cloud operator can be cheaper than operating it
yourself. Their whole pricing model is based on being able to over-provision
compute and move workloads around within a datacenter. As much as people talk
about failing hard drives and other components at scale, failure rates are
still low enough that you could operate dozens of systems at full capacity for
3+ years with maybe a handful of legit hardware failures. To rent that same
compute from any cloud provider would cost significantly more. The cheapest
"Dedicated Host" on AWS will cost you almost $12k over 3 years if you pay for
it on-demand, and it's equivalent in specs to something you can buy for ~$2k.

> am I missing something?

I'd want more background behind what you mean by "at least for business". What
kind of business? Obviously IaaS providers like Digital Ocean and Linode are
are type of business that would not use other clouds. Dropbox and Backblaze as
well would probably never use something like S3. And there are legitimate use
cases outside of tech that have needs in specific teams for low latency
compute, or its otherwise cost and time prohibitive to shuttle terabytes of
data to the cloud and back (3D rendering, TV news rooms, etc). If you're
talking about general business systems that can be represented by a website or
app with a CRUD API, then most of that probably doesn't require on-prem. But
that's not the only reason businesses buy servers.

------
marvinblum
As most have mentioned already: cost

We started out with Emvi [1] on Kubernetes at Google Cloud as it was the
"fancy thing to use". I like Kubernetes, but we paid about 250€/month just to
run some web servers and two REST APIs. Which is way too much considering that
we're still working on the product and pivoting right now, so we don't have a
lot of traffic.

We then moved on to use a different cloud provider (Hetzner) and host
Kubernetes on VMs. Our costs went down to about 50€ just because of that. And
after I got tired managing Kubernetes and all the complexity that comes along
with it, we now just use a docker-compose on a single (more powerful) VM,
which reduced our cost even futher to about 20€/month and _increased_
performance, as we have less networking overhead.

My recommendation is to start out as simple as possible. Probably just a
single server, but keep scaling in mind while developing the system. We can
still easily scale Emvi on different hardware and move it around as we like.
We still use Google Cloud for backups (together with Hetzners own backup
system).

[1] [https://emvi.com/](https://emvi.com/)

------
hdmoore
Anything with massive storage and massive compute that doesn't need low
latency is a great fit. I still host ~300TiB and ~250 cores at home because
the cloud cost would be astronomical. Edit: This is for personal stuff related
to internet-wide scan data and domain tracking. See
[https://github.com/hdm/inetdata](https://github.com/hdm/inetdata)

------
desc
1\. Our customers run our software on their own machines for security and
data-control reasons. As soon as something's running on someone else's
hardware, the data is out of your control. Unless you're going to accept the
(often massive) cost of homomorphic encryption, AND have a workload amenable
to that, it's a simple fact.

2\. Everything we do in house is small enough that the costs of running it on
our own machines is far less than the costs of working out how to manage it on
a cloud service AND deal with the possibility of that cloud service being
unavailable. Simply running a program on a hosted or local server is far far
simpler than anything I've seen in the cloud domain, and can easily achieve
three nines with next to no effort.

Most things which 'really need' cloud hosting seem to be irrelevant bullshit
like Facebook (who run their own infrastructure) or vendor-run workflows
layered over distributed systems which don't really need a vendor to function
(like GitHub/Git or GMail/email).

I'm trying to think of a counterexample which I'd actually miss if it were to
collapse, but failing.

------
lettergram
I actually am moving my startups servers from AWS to a home server.

Reasoning:

* We know how much compute is need.

* We know how much the new servers can compute.

* We have the ability to load balance to AWS or Digital Ocean or another service as needed.

* This move provides a 10x speed improvement to our services AND reduces costs by 70%.

For reference, had to call the ISP (AT&T) and they agreed to let me host my
current service. It’s relatively low bandwidth, but has high compute
requirements.

------
ggm
We operate an X509 PKI with a Trust anchor and CA. Its not impossible to run
the Hardware Security Module (HSM) in the cloud but its off the main path. Its
more plausible to run it in a D.C. but it invites security threats you don't
have, if you run it inside your own perimiter. Of course you also then invite
other risks, but its a balancing act and it has to be formally declared in
your Certification Practice Statement (CPS)

We also run some registry data which we consider mission critical as a
repository. We could run the live state off-prem, but we'd always have to be
multi-site to ensure data integrity. We're not a bank, but like a bank or a
land and titles administration office, registry implies stewardship in trust.
That imposes constraints on "where" and "why".

Take central registry and the HSM/related out of the equation, if I was
building from scratch I'd build to pub/sub, event-sourcing, async and in-the-
cloud for everything I could.

private cloud. If you don't control your own data and logic, why are you in
the loop?

------
SoylentOrange
We are a research group at a company with 2000 or so employees. We have a few
GPU machines to train models, which are utilized nearly around the clock.

AWS and co’s GPU-enabled servers are exceedingly expensive. Most of the GPU
models on those machines are also very old. We pay maybe 1/3 or less to
maintain these machines and train models in-house vs paying AWS.

Mind you, we use AWS for plenty of stuff...

------
Blackadderz
I work in R&D for a telecomms/optics company.

All servers are on premises. Not allowed to have a laptop. No access to
emails/data outside of the office. No USB drives, printing documents, etc.

Reason? Protect IP. From who? Mostly Huawei.

Good and bad: When I walk out the door... I switch off. The bad is that
working from home isn't realy an option. Although they have accommodated
somewhat for this pandemic.

------
mrweasel
We sell hosting to large number of different customers who for whatever
reason, mostly legal, are required to keep data within the borders of Denmark.
There is no Google, Amazon, Azure or DigitalOcean data centers in Denmark, so
cloud isn't an option for them.

Regarding cost, well it depends. We try to help customers to move to cloud
hosting if it's cheaper for them. It almost always will be if they take
advantage of the features provided by the cloud providers. If you just view
for instance AWS as a VMware in the cloud, then we can normally host the
virtual machines for you cheaper and provide better service.

You have to realize that many companies aren't developing software that's
ready for cloud deployment. You can move it to EC2 instance, but that's not
taking advantage of the feature set Amazon provides, and it will be expensive
and support may not be what you expect. You can't just call up Amazon and
demand that they fix your specific issue.

------
pmlnr
> The more I learn, the more I believe cloud is the only competitive solution
> today, even for sensitive industries like banking or medical.

Then learn A LOT more and start with mainframes and their reliability.

------
gameswithgo
Our business has had greatly increased load due to COVID-19. It would have
been very nice to buy a 128 core EPYC bare metal server to run our SQL Server
on at this time, to buy us time to rearchitect to handle the load. Instead we
are stuck with 96vCPUs because that is the most Amazon can do.

Its also very very expensive to have a 96vcpu VM on amazon!

------
apetersonBFI
I'm one of two IT persons at a food processor, in a small town. Despite living
in an area where a majority of IT & Programmers work at the hospital or an
insurance company, my boss has run our own networks and servers since the days
of Novell, and we continue to run Windows servers on-premise, instead of the
cloud. It does lead to interesting situations, like finding out a Dell server
shrieking next to you is running max fan speed because the idrac is not
connected.

Neither of us have any experience with the cloud, whereas we have a lot of
Microsoft experience. We still rely on OEM licenses of Office, because Office
365 would be 3x or more expensive. We have a range of Office 2019, 2016, 2013
OEM, and we get audited by them nearly every year.

We use LastPass, Dropbox and Github, but only the basic features, and LastPass
was an addition last year after someone got into our network through a weak
username/password.

In our main location, we have three ESX boxes, running several virtual
servers, and then we have a physical server for our domain controller, file
sharing and DHCP, DNS in other locations. We also switched to a physical
server for our new ERP application server, which hasn't yet been rolled out.

Projects like upgrading our ERP version can take months, but we have a local
consulting team, with a specialist in our particular ERP solution, as well as
a Server and Network specialist, and we also have a very close relationship
with our ISP, who provides WAN troubleshooting.

Our IT budget is small relative to our company revenue, so most cloud
proposals would raise our costs manyfold. We continue to use more services
like Github and Lastpass, and we both have multiple hats.

I'm a developer, in-house app support, Email support, HR systems support, ERP
support, PC setup, and I run our Data Synchronization operation and my boss
runs EDI. I do a lot of PowerShell and Task Scheduler, but I've got familiar
with bash through git bash.

------
mikorym
Here is a stupid example: Excel vlookups work on a network drive, but not on a
cloud service like Dropbox or OneDrive. The absolute path can't resolve if
it's used across multiple Excel users. If the users store the file locally,
each will have a different path on their computers. Excel stores actual paths.
[1]

There is one way around it: Mounting the cloud server as a network drive (some
providers do this by default, but OneDrive is not one of them, neither is
Dropbox).

I don't know of a way of mounting OneDrive as a virtual drive; I would be
interested to know.

It sounds stupid, but the above was a real life scenario.

[1] Only if the files are closed. Excel can change the path if you have the
file open, but it can't change it to multiple option across different PCs. But
as I have mentioned before, Excel doesn't seem to document all of their more
subtle features.

------
jolmg
Unreliable internet.

A retail company may decide that the best place to put up a new branch is
coincidentally (though there might be a correlation) at the edge of what the
available ISPs currently cover. They might have to make a deal to get an ISP
to extend their area to where the store is going to be. However, because of
lack of competing ISP options on the part of the retailer, and the lack of
clients in the retailer's area on the part of the ISP, that service is
probably not going to be all that reliable.

Also, that retail company may experience a big rise in sales after a natural
disaster occurs, when communications (phone/cell/internet) are mostly down for
the area. One tends not to think about stuff like that until it happens at
least once.

It's very important for the ERP/POS systems to be as operational as possible
even when the internet is down.

------
joshuaellinger
On the smaller end of the scale, I have a $12K/mo spend with Azure. I decided
to go back to Coloc.

For under $50K, I have 4 machines with an aggregate 1TB RAM, 48 cores, 1 pricy
GPU, 16TB of fast SSD, 40 TB HHD, and infiniband @ 56GB/sec. Rent on the
cabinet is less than $1K/mo. It's going to cost me about $20K in labor to
migrate.

So the nominal break-even point is six months but the real kicker is that this
is effectively x10-30 the raw power of what I was getting on the cloud. I can
offer a quantitatively different set of calculations.

It also simplifies a bunch of stuff: 1\. No APIs to read blob data -- just
good old files on an ZFS share. 2\. No need to optimize memory consumption.
3\. No need for docker/k8s/etc to spin up under load. Just have a cluster
sitting there.

There are downsides but Coloc beats the cloud for certain problems.

------
busterarm
I'm going to buck the trend and say cloud is great. We do cloud, on-prem and
colo (16 racks in two different DCs).

Procurement is a nightmare especially when your vendor is having problems with
yields (thanks Intel!) and the ability to scale up and scale down without
going through hardware procurement process saves us millions of dollars a
year.

We avoid the lock-in by running on basic services on multiple cloud providers
and building on top of those agnostically.

Spend is in the millions per month between the cloud providers, but the
discounts are steep. We're essentially had to build our own global CDN and the
costs are better than paying the CDN services and better than running our own
hardware & staffing all those locales.

It's a no brainer. We'll continue to operate mixed infrastructure for quite
some time as certain things make sense in certain places.

------
the_svd_doctor
PhD student/researcher. Most of the compute I do are HPC/scientific computing
style and run on University or US National Lab machines. We thought about
cloud, but the interconnect (MPI-style code) is very important for us and it's
not very clear to me what's available there in the cloud.

------
frellus
Self-driving company here. We're doing on-prem because we have a clear
projection on the amount of data we'll need to injest, store and process using
GPUs.

The advantages of running things in a cloud are clear -- and as an
infrastructure team we have challenges around managing physical assets at
scale, however it's clear with the cost of cloud providers that eventually we
would have to pull data into a datacenter to survive at some point.

Co-location costs are fixed, and it's actually easy to make a phenomenal deal
now-a-days given the pressure these companies are under.

The real trick of it all is that regardless of running on-prem or in the
cloud, we need to run as if everything is cloud native. We run Kubernetes,
Docker, and as much as possible automate things to the point that running one
of something is the same as running a million of it.

------
jaboutboul
There is not one clear cut answer on this. It depends on what your company
values, i.e. cost vs. agility. If you are using the cloud for what it was
meant for availability, scalability, elasticity--and those are the things that
your org values--its the right fit for you. If on the other hand you value
cost then it clearly isn't the right fit.

One other point I'll make, the true value of cloud isn't in IaaS, renting VMs
from anyone is relatively expensive compared to the costs of buying a server
and maintaining it yourself for a number of years. The true value of the cloud
is when can architect your solution to utilize the various services the cloud
providers offer, RDS/DynamoDB, CDN, Lambda, API Gateway, etc. so that you can
scale quickly when you need to.

------
jcrawfordor
For hobby projects, I own a moderately outdated 1U "pizzabox" installed in a
downmarket colocation facility in a nearby major city. Considering the monthly
colocation rate and the hardware cost amortized over two years (I will
probably not replace it that soon but it's what I've used for planning), this
works out to appreciably less than it would cost to run a similar workload on
a cloud provider. It costs about the same or possibly a bit less than running
the same workload on a downmarket dedi or VPS provider, but it feels more
reliable (at least downtime is usually my fault and so under my control) and
the specs on the machine are higher than it's easy to get from downmarket VPS
operations.

Because my application involves live video transcoding I'm fairly demanding on
CPU time, which is something that's hard to get (reliably) from a downmarket
VPS operation (even DO or what have you) and costly from a cloud provider. On
the other hand, dual 8 core Xeons don't cost very much when they're almost a
decade old and they more than handle the job.

There are a few fairly reputable vendors for used servers out there, e.g. Unix
Surplus, and they're probably cheaper than you think. I wouldn't trust used
equipment with a business-critical workload but honestly it's more reliable
than an EC2 instance in terms of lifetime-before-unscheduled-termination, and
since I spend my workday doing "cloud-scale" or whatever I have minimal
interest in doing it in my off-time, where I prefer to stick to an "old
fashioned" approach of keeping my pets fed and groomed.

And, honestly, new equipment is probably cheaper than you think. Dealing with
a Dell account rep is a monumental pain but the prices actually aren't that
crazy. Last time I purchased over $100k in equipment (in a professional
context, my hobbies haven't gotten that far yet) I was able to get a lot for
it - and that's well less than burdened cost for one engineer.

------
_bxg1
One of my favorite things (at least for personal projects) about using the
cloud is so-called "platform as a service" systems like Heroku, where I don't
have to get down in the weeds, I just push code and the process starts (or
restarts).

Is there something like that I could use on my own hardware? I just want to do
a fresh Linux install, install this one package, and start pushing code from
elsewhere, no other configuration or setup necessary. If it can accept
multiple repos, one server process each, all the better. I know things like
Docker and Kubernetes exist but what I want is absolute minimal setup and
maintenance.

Does such a thing exist?

~~~
mappu
You're looking for Dokku

Same git push deploys, heroku-compatible buildpacks or Dockerfiles, all on
your own hardware, MIT license.

~~~
_bxg1
This looks perfect, thank you! I knew there was no way I could've been the
first person to think of this

------
frogbox12
After a big cloud-first initiative, several managers left; leaving
implantation to Linux sysadmins now in charge of cloud. Treated cloud as some
colo facility, dump all apps in one big project/account, cloud costs spun
quickly out-of-control and lots of problems with apps not being segregated
from each other. Cloud declared 'too expensive' and 'too insecure', things
migrated back on-prem, team now actively seeks to build and staff colo
facilities with less than 10ms latency somewhere outside coastal California
(Reno, Vegas, PHX) which just isn't gonna happen because physics.

------
annoyingnoob
International Traffic in Arms Regulations (ITAR) Compliance - much easier to
keep it on site, off-site compliance is costly. Also, cost over time. Better
control of performance requirements for certain applications.

------
otabdeveloper4
We're in the process of moving a greenfield project from AWS to a more
traditionally hosted solution.

AWS turned out to be 5-10 times more expensive; what's worse, our developers
are spending more then half their time working around braindead AWS design
decisions and bugs.

A disaster any way you look at it.

There are good reasons to chose AWS, but they're never technical. (Maybe you
don't want to deal with cross-departmental communications, or you can't hire
people into a sysadmin role for some reason, maybe you want to hide hosting in
operational expenses instead of capital, etc.)

------
Nextgrid
Bandwidth costs.

Most dedicated servers come with unmetered bandwidth so not only is it cheap
to serve large files but your bandwidth costs won't suddenly explode because
of heavy usage or a DDoS attack.

------
TuringNYC
Here is how we went about w/ CSPs (AWS, Azure, GC, Oracle). Thoughts welcome

Getting Started --> Definitely go w/ CSPs. No need to worry about infra.

Pre Product Market Fit + Steady Growth --> On Premise, because CSPs might be
expensive until you find a consistently profitable business.

Pre Product Market Fit + HyperGrowth --> CSPs since you wont be able to keep
up [we never got to this stage]

Product Market Fit w/ Sustainable Good Margins --> CSPs, pay to remove the
headache [we never got to this stage]

Side Note: w/ GPUs, CSPs rarely make sense

------
acwan93
I agree with you OP.

Our company provides on-premise ERP systems to small (we’re talking at most 20
person companies) wholesale distributors.

Pre-COVID, I was pushing for a cloud solution to our product and pivoting our
company towards that model. We’re at a hybrid approach when COVID hit.

What ends up happening with an on-premise/hybrid cloud model is we end up
doing a lot of the sysadmin/IT support work for our customers just to get our
ERP working. This includes getting ahold of static IP addresses (and absolving
responsibility), configuring servers/OSes, and several other things along the
same vein that’s wholly irrelevant to the actual ERP like inventory management
and accounting.

Long story short, these customers of ours end up expecting us to maintain
their on-premise server without actually paying for help or being knowledgable
about how it all works. We keep pitching them the cloud but they’re not
willing to pay us a recurring fee even though it actually saves the headaches
of answering the question “who’s responsibility is it to make sure this server
keeps running?"

I think a lot of these answers here are dealing with large-scale products and
services where the amount of data and capital costs is so massive it makes
sense to start hiring your own admins solely to maintain servers. For these
small mom-and-pop shops who are looking for automation, the cloud is still the
way to go.

~~~
jolmg
Deja vu. I think you totally hit the nail on the head with that last
paragraph. On-premise ERP systems probably only make real sense for (non-
small) companies that wish to avoid relying on the internet (because e.g.
their business strategy requires that freedom) and can hire long-term
sysadmins/programmers that can provide support to those systems.

~~~
acwan93
Have you had experience selling on-premise systems? I’m really curious how
other companies handle the sysadmin and IT support issue.

------
allenrb
Finance space, under 100 people. Most servers are either latency-sensitive or
24/7 fully-loaded HPC. Neither case fits the cloud model. We do use cloud for
build/test, though.

------
Jugurtha
Many organizations do have private clouds.

If by cloud you mean a public cloud like Google, Amazon, or Microsoft, then
forget about it; not with these companies piping data directly to U.S
intelligence.

~~~
p1necone
What's the difference between private cloud and on prem?

Does it just mean you let everyone spin up VMs etc rather than requiring them
to go through IT?

~~~
toomuchtodo
Mostly. Cloud is just bin packing compute and data storage. There are many use
cases where it’s cheaper for you to host the hardware you cloud on instead of
a public cloud provider.

------
krageon
You've been managing servers for quite some time but you've never considered
the security implications of hosting all of your sensitive data on someone
else's computer? You say you're not trolling but I genuinely don't see how
those two facts are compatible, except if you work in an industry where the
data just doesn't matter. If that were the case though, you shouldn't feel
compelled to judge what banking or medical institutions are doing.

~~~
aspyct
We use a public cloud, and believe me: we do consider the security
implications. We veeery much do.

------
nikisweeting
Quebec power and internet pricing is really competitive. For residential
services I pay $0.06/kw + $80/mo for 1Gbps fiber with 1ms ping to 8.8.8.8
(USD).

As a result, I run a power-hungry Dell r610 with 24 cores and 48GB of ram with
20+ services on it for many different aspects of my company. All the critical
stuff runs on DigitalOcean / Vultr, but the 20+ non-critical services like
demo apps, CI/CD, cron workers, archiving, etc. run for <$200/yr in my closet.

~~~
blaser-waffle
This is also why there are a lot of data centers in the Greater Montreal area,
FWIW.

------
ex3ndr
We are small team startup and i am personally annoyed about pricing of a
clouds.

I have a 3 smallish VM for build server + managed SQL. It cost 500$/mo. It
doesn't make sense. Having my own VMs on ESXi makes everything very different
- most of the time this VMs do nothing, but you want to make them performant
from time to time, so there are a plenty of resources because all other vms
are too mostly IDLE.

In cloud they are billed as if they are 100% loaded all the time.

I am not really satisfied with latencies and insane price for egress traffic.
I just can't do backups daily since it could cost whooping 500$/mo just for
the traffic. This is just insane, i can't see how it could scale anywhere for
B2C market. For B2B it might work really well though since revenue per
customer is much higher.

We are not moving to our own DC, but just keep realtime stuff in the cloud,
but anything that is not essential is being moved somewhere else. Bonus is
that you need off-site backups anyway in the case if cloud vendor will just
ban you and delete all your data.

Startups might move fast and iterate, but if you don't have your own servers
you always reduce your usage because it could grow fast effectively reducing
your delivery capacity.

------
leroy_masochist
Background: I consult extensively in the SaaS space and ask people this
question all the time in the course of strategy reassessments, transactional
diligence, etc.

1\. Physical control over data is still a premium for many professional
investors. As a hedge fund CIO told me recently when I asked her why she was
so anti-cloud migration, "I want our data to live on our servers in our
building behind our security guard and I want our lawyer, not AWS's, to read
the subpoena if the SEC ever comes for us."

2\. There are a lot of niche ERP- and CRM-adjacent platforms out there --
e.g., medical imaging software -- where the best providers are still on-prem
focused, so customers in that space are waiting for the software to catch up
before they switch.

3\. A lot of people still fundamentally don't trust the security of the cloud.
And I'd say this distrust isn't of the tinfoil hat, "I don't believe SSL
really works" variety that existed a decade ago. Instead it's, "we'd have to
transition to a completely different SysAdmin environment and we'd probably
fuck up an integration and inadvertently cause some kind of horrendous
breach".

------
Scaevus
For our case, the need is very specific as we are working with mobile apps.

Building iOS apps require macOS and even though there are some well-known "Mac
hosting" services, none of them are actual cloud services similar
DigitalOcean, Azure, AWS, etc.

So it is much less expensive and actually easier to scale and configure to
host the Macs onprem.

(Off the record: If it is for internal use only, you can even stick in a few
hackintoshes for high performance.)

------
arghwhat
Quite frankly, "cloud" is a convenience and elasticity service at a steep
premium, with downsides.

Contrary to popular belief, it does not in the slightest save you a sysadmin
(most just end up unknowingly giving the task to their developers). And
contrary to popular belief, the perf/price ratio is _atrocious_ compared to
just buying servers.

For some of the loads I had been doing the math for, I could rent a colo _and_
buy a new beefy server every year with money to spare for the yearly cost of
something _approximating_ the performance in AWS...

------
ai_ja_nai
It's the people cost, not the hardware.

Hardware is super cheap:

-A 40 slots rack, with gigabit fiber, dual power and a handful of public IP addresses, costs on average 10000€/y.

-A reconditioned server on eBay with 16 cores and 96GB of RAM costs 500€ (never seen them break in 3 years).

-A brand new Dell Poweredge with AMD EPYC 32 core and 64GB of RAM will cost 3000€.

-Storage is super cheap: 500GB of SSD costs 80€ (consumer stuff is super fine as long as you wisely plan between redundancy and careful load) and rotational disks are even cheaper. Never seen a rotational disk break.

Once bought, all of this is yours forever, not for a single month. You can
pack very remarkable densities in a rack and have MUCH more infrastructure
estate at disposal than you would ever afford on AWS.

The flip side of the coin is that you need operation expertise. If it's always
you, then ok (although you won't be doing much more than babysitting the
datacenter). Otherwise, if you need to hire a dedicated person, people is the
most expensive resource yet and that should definitely be added to the cost of
operations.

------
dathinab
A company which:

1\. Does "security" critical stuff (like affecting the security of people not
data if breached)

2\. Which besides certain kinds of breaches has lowish requirements to
performance and reliability (short outages of a view minutes are not a big
problems, even outages of half a day or so can be coped with)

3\. Slightly paranoid founders, with a good amount of mistrust into any cloud
company.

4\. Founders and tech-lead which have experience in some areas but toughly
underestimate the (time)cost of managing servers them self, and that _kinda_
hard to do secure by yourself (wrt. long term DDOS and similar).

So was it a good reason? Probably not. But we still went with it.

As a site note while we did not use the could _we didn't physically manage
server either_. Instead we had some dedicated hardware in some compute center
in Germany which they did trust. So no "physical" managements securing etc.
needed and some DDOS and network protection by default. Still we probably
could have it easier without losing anything.

On the other side if you have a dedicated server hardware in some trusted
"localish" compute center it's not _that_ bad either to manage.

------
prirun
I did a startup with a co-founder in 1998, before cloud was a thing. We hosted
at above.net first, then he.net following above.net's bankruptcy. Both were
very good and we never had colo-related problems with either, though he.net
was significantly cheaper.

We started with 2 white-box PC's as servers, 2 mirrored RAID1 drives in each.
We added a 3rd PC we built ourselves: total nightmare. The motherboard had a
bug where, when using both IDE channels, it overwrote the wrong drive. We
nearly lost our entire business. Putting both drives on the same IDE channel
fixed it, but that's dangerous for RAID1.

A few years in, we needed an upgrade and bought 5 identical SuperMicro 2U
servers with hardware RAID1 for around $10K. Those things were beasts: rock
solid, fast, and gave us plenty of capacity. We split our services across
machines with DNS and the 5 machines were in a local LAN to talk to each other
for access to the database server. The machines' serial ports were wired
together in a daisy-chain so we could get direct console access to any machine
that failed, and we had watchdog cards installed on each so that if one ever
locked up, it automagically rebooted itself. When I left in 2005, we were
handling 100's of request/s, every page dynamically generated from a search
engine or database.

Of course it took effort to set all this up. But the nice thing is, you
control and understand _everything_. Some big company doesn't just do things
to you, you have no idea what is happening, and they're not talking. And if
things do go south, you can very quickly figure out why, because you built it.

The biggest mistake we made was in the first few years, where we used those
crappy white-box PCs. Sure, we saved a couple thousand dollars, but we had the
money and it was a terrible trade. Night and day difference between those and
real servers.

------
quanto
> The more I learn, the more I believe cloud is the only competitive solution
> today, even for sensitive industries like banking or medical. I honestly
> fail to see any good reason not to use the cloud anymore, at least for
> business. Cost-wise, _security-wise_ , whatever-wise. [emphasis mine]

Most people here seem to point out cost and utilization. I would like to offer
another perspective: security.

I worked in both of these industries: finance ("banking", not crypto or PG)
and medical (within a major hospital network). The security requirement, both
from practical and legal perspectives, cannot be understated. In many
situations, the data cannot leave an on-prem air-gapped server network, let
alone use a cloud service.

It costed us more to have on-prem servers as we need a dedicated real estate
and an engineering team to maintain. Moreover, the initial capital expenditure
is high -- designing and implementing a proper server room/data center with
climate control, power wiring, and compliant fire extinguishing system are not
trivial.

------
gorgoiler
For business logic at a fairly large school the free and open tools that make
the cloud so productive get used here a lot. We get to leverage commodity
hardware and network infra very effectively for on-premises[1].

You have to have a good recovery plan for when equipment X’s power supply
fails but when deploy is all automated, it’s very easy to overcome swapping
bare metal, and easy to drill (practice) during off hours.

This makes it _much_ easier to meet regulatory compliance: either statutory or
regulations your org has created internally (e.g. financial controls in-org,
working with vulnerable people or children, working with controlled
substances, working with sensitive intellectual property.)

Simply being able to say you can pull the plug on something and do forensic
analysis of the storage on a device is an important thing to say to
stakeholders (carers, carers families, pupil parents.)

I’m so grateful to be living in the modern age when “cloud” software
exists[2], but I don’t have to be in the cloud to use it.

The downside: you need trained staff and it’s completely inappropriate if you
need any kind of bandwidth, power consumption, or to support round the clock
business (which we do not because, out here on the long tail, we work in
single city so still have things like evenings weekends and holidays for
maintenance!)

— [1] Premise vs premises is one of those “oh isn’t the English language
awful” distinctions. Premise is the logical foundation for some other theory
(“the premises for my laziness is that because the sky is grey it will
probably rain so I’m not going to paint the house”) where as the premise_s_
means physical real estate property (“this is my freshly painted house: I
welcome you onto the premises”.)

[2] Ansible, Ubiquiti, arm SBCs like raspberry pi, Docker, LXC, IPv6 (making
global routing for more tractable, IPv4 for the public and as an endpoint to
get on the VPN.)

------
hourislate
If you interested, here is an article about how Bank of America chose to build
out its own cloud to save about 2 billion a year.

[https://www.businessinsider.com/bank-of-
americas-350-million...](https://www.businessinsider.com/bank-of-
americas-350-million-internal-cloud-bet-striking-payoff-2019-10?op=1)

------
ps
Cost.

We recently moved one rack to the different DC in the same city and used
Digitalocean droplets to avoid downtime. Services running on Linux were
migrated without high-availability (e.g. no pgsql replication, no redis
cluster, single elasticsearch node...) and we just turned off Windows VMs
completely due to licensing issues and no need to have them running at night.

The price of this setup was almost 4x higher than what we pay for colo. Our
servers are usualy <5 years old Supermicro, we run on Openstack and Ceph (S3,
rbd) and provide VPNaaS to our clients.

AWS/GCP/Azure was out of question due to cost. We considered moving Windows
servers to Azure with the same result - the cost of running Windows Server
(AD, Terminal, Tomcat) + MS SQL was many times higher than the price of colo
per month. It is bizarre that you can buy the server running those VMs
approximately every 3 months for the Azure expenses (Xeon Platinum, 512GB
RAM).

------
majkinetor
There is no cloud solution that lets you ship and forget a system.

Come in 15 years ? It still works. Is that possible with cloud even in short
periods, like 2 years ? No.

Will it ever be possible? No.

Thats primary reason for me. I can use cloud only for stuff that are nice but
not mandatory to have for service to work, like status page.

Plus, work is more enjoyable then using somebodies else stuff.

------
whatsmyusername
Cloud falls over if you have a lot of data egress. I don't work on those types
of workloads (mainly in PCI and a little bit of HIPAA) so I stick to cloud for
the sheer convenience factor (it's easier to fix a security group than having
to drive to the office and plug in somewhere like I had to earlier today).
Dealing with hardware has become a very specific skill set. I have it, but I
don't enjoy it, so I don't look for it.

I still have to build physical networks occasionally (ex: we are building a
small manufacturing facility in a very specific niche that's required to be
onsite for compliance reasons) but the scale is so small that I can get away
with a lot of open source components (pfsense netgates are great) and not have
to use things that are obnoxious to deal with (if I never have to deploy cisco
anything ever again I won't be upset).

------
darrelld
I have a friend who is an IT manager for a large chain hotel in the Caribbean.

I keep asking him about why they still use on premises equipment and it boils
down to:

* Cost for training / transitioning + sunk cost fallacy * Perceived security risk (right or wrong) * IT is mostly invisible and currently "works" with the current arrangement, why change?

------
avifreedman
We're (Kentik) a SaaS company and one key to having a great public margin is
buying and hosting servers. In our case, we use Equinix in the US and EU to
host, with Juniper gear for routing/switching, and the customary transit,
peering at the edge.

One secondary factor is that we've only monotonically increased, and it's way
cheaper to keep 10%-15% overprovisioned than to be on burst price with 50%+
constant load.

But the simplest math is - we have > 100 storage servers that are 2u 26x2tb
flash, 256gb RAM, 36 cores. They cost $18k once, which we finance at pretty
low interest over 36 months (and really last longer than that). Factor in
$200-400/mo to host each depending (I think it's more like $200, but it
doesn't matter for the cloud math).

That same server would be many $thousands/month on any cloud we've seen.
Probably $4-6k/mo, depending on the type of EBS-ish attached. Or with the
dedicated server 'alternate' they are moving to offer (and Oracle sorta
launched with).

It'd be cheaper but still > 2x as expensive on Packet, IBM dedicated, OVH,
Hetzner, Leaseweb (OVH and Hetzner the cheapest probably).

Three other factors for us:

1) Bandwidth would be outrageous on cloud but probably not as outrageously
high as just the servers, given that our outbound is just our SaaS portal/API
usage

2) We'd still need a cabinet with router/switch infra to peer with a couple
dozen customers that run networks (other SaaS/digital natives and SPs that
want to send infrastructure telemetry via direct network interconnect).

3) We've had 5-6 ops folks for 3 of the 6 years, 3-4 for the couple years
before that. As we go forward, as we double we'll probably +1. It is my belief
that we'd need more people in ops, or at least eng+ops mix, if we used public
cloud. But in any case, the amount of time we spend adding and debugging our
infra is really really really small, and the benefit of knowing how servers
and switching stuff fails is huge to debugging (or, not having to debug).

All that said - we do run private SaaS clusters, and 100% of them are on bare
metal, even though we _could_ run on cloud. Once we do the TCO, no one yet has
wanted to go cloud for an always-on data-intensive footprint like ours.

Good luck with your journey, whichever way you go!

And happy to discuss more, email in profile

------
kokey
I think when the economy is hit hard a lot of companies are going to have to
look at what they can do to make sure they remain profitable since investor
appetite has changed. This means some companies will have to look at what is
being spent on cloud providers. Renting RAM by the hour makes sense if you are
optimistic about future revenue growth, but if the market changes and you have
to worry about how you can sustain profits while just keeping your current
customer base then this makes a lot less sense. The cloud vs on prem argument
also really includes all the things in between, e.g. colo, managed servers,
VPSes and also better tooling to manage your own VM and container clusters on
these, which I think will now get increased attention and competition when
people are considering alternatives to the big cloud providers in order to
bring down costs.

------
MorganGallant
I picked up a few refurbished dell servers off Ebay for super cheap a while
back - and usually use these with Cloudflare Argo tunnels to host various
servers. However, since these are just sitting on the floor next to my desk,
usually rely on cloud for any applications with high uptime requirements.

Recently though, I've been working on some distributed systems type projects
which would allow these servers to be put in different physical locations (and
power grids), and still continue to operate as a cohesive whole. This type of
technology definitely increases my confidence in them being able to reliably
host servers. I wouldn't want to be reliant on the cloud for large scale
services though, from my understanding you can get some crazy cost savings by
colocating some physical servers (especially for large data storage
requirements).

------
pavelevst
AWS is one of most expensive options, and far not perfect, I can’t understand
why people consider it as default choice... To compare - dedicated server on
hetzner (core i7, 32gb ram, 20tb network) is cheaper than medium VM on AWS. If
the product is growing, cloud cost can quickly become the biggest expense for
company. Than it will make sense to spend some time make things run in more
cost effective way

I think if you choose cloud hosting that costs about same as renting dedicated
server plus settings virtualization by yourself - than it’s a fair choice (can
check on [https://www.vpsbenchmarks.com/](https://www.vpsbenchmarks.com/) or
similar)

Another sweet configuration is dedicated servers with kubernetes: good user
experience for developers, easy to setup and maintain, easy to scale up/down

------
bpyne
My employer is a mid-sized university. Cost is the main issue.

Our environment is a mixture of in-house developed apps and COTS. Until
recently, our major COTS vendors didn't have cloud solutions. Now they have
cloud solutions but they're far too costly for us to afford. So we need to
keep them in-house and continue to employ the staff to support it.

Our in-house apps integrate the COTS systems. Our newer apps are mostly in the
cloud. But the older ones are in technologies that need to stay where the
database server is, which is in our server room for the reason stated in the
last paragraph. Rewriting the apps isn't on our radar due to new work coming
in.

Historically, outsource vs. in-source seems to ebb and flow. The clear path is
usually muddied when new technologies come out to reduce cost on one side or
the other.

------
gen220
My SO’s brother works in the studio video recording industry, and is a very
IT-savvy guy. We had a long discussion last holiday season about the state of
cloud adoption in that industry. He told me (this is obviously secondhand)
that most of the movie industry is not only off the cloud, but exclusively
working in the realm of colocating _humans_ and data (footage).

This is for many reasons. The one that comes back to me now is that the file
sizes are HUGE, because resolution is very high, so bandwidth is a major
concern. Editors and colorists need rapid feedback on their work, which
demands beefy workstations connected directly with high bandwidth connections
to the source files. Doing something like this over a long distance network
(even if the storage was free) would be prohibitively expensive, and sometimes
literally impossible.

So the write loads are basically the antithesis of what cloud optimized for:
“random sequential reads of typical short length, big append only writes”. The
big production houses (lucasarts famously) are also incredibly secretive about
their source material, and like to use physical access as a proxy for digital
access.

It leads to some seemingly strange (to me as a cloud SWE guy) decisions. He
pretty much exclusively purchases top of the line equipment ( _hard drives
/ssds_), and keeps minimal if any backups for most projects because there
simply isn’t any room. It’s a recipe for disastrous data loss, and apparently
it’s something that happens quite often to this day. It’s just extremely
prohibitively expensive to do version control for movie development.

I don’t know to what extent cloud technologies can solve for this domain. I
asked him if Netflix was innovating in this area, since they’re so famously
invested in AWS, but he said that they mostly contracted out the production
stuff, and only managed the distribution, which makes sense. The contractors
don’t touch the cloud at all, for the most part.

Again most of this is secondhand, I’d be curious to hear more details or
reports from other people in the movie industry.

------
kgc
We moved all AWS servers to a colo. Saving 80% of the cost.

------
ocdtrekkie
There are very few niches where the cloud makes sense: Namely, where you are
either too small to benefit from a single server and a single entry-level IT
guy (think three or four person companies with low need for technical
competency), or where you are expecting rapid growth and can't really
rationally size out your own hardware for the job (in this case, the cloud is
useful initially, leave it later once your scale is more stable).

In every other case, you are paying for the same hardware you could buy
yourself, plus the cloud provider's IT staff, plus your own IT staff which you
likely need anyways to figure out how to deal with the cloud provider, _and_
then the cloud provider's profit margin, which is sizeable.

------
amq
Surprised no one mentioned a third option: cheap vps providers like
digitalocean or vultr. They've become real contenders to big clouds recently,
providing managed databases, storage and k8s. And their bandwidth costs are
close to what you'd get in colo.

------
cpascal
My company cannot run our infrastructure in the cloud because we do
performance/availability monitoring from ~700 PoP’s around the world.

Not running our infrastructure in the cloud is part of our value proposition.

Our customers depend on us to detect and alert them when their services go
down. We _have_ to be operational when the cloud providers are not, otherwise
we aren’t providing our customer with a valuable service.

Another reason we don’t run in the cloud is because we store a substantial
amount of data that is ever increasing. It’s cheaper to run our own SAN in a
data center than to store and query it in the cloud.

The final reason is our workloads aren’t elastic. Our CPU’s are never idle. In
that type of use case, it’s cheaper to own the hardware.

------
znpy
Many of the customers of my previous employer had hardware on premises,
including the customer that I was handling.

It had both compute and storage (netapp). It had two twin sites in two
different datacenters. The infra in each site consisted basically in six
compute servers (24c/48t, 128gb ram) and netapp storage (two netapp heads per
site + disk shelves).

Such hardware has basically paid itself across its seven or eight years of
life, and having one of the sites basically in the building meant basically
negligible latency.

The workload was mostly fixed, and the user base was relatively small (~1000
concurrent users, using the services almost 24/7).

It really checks all of the boxes, does all it is supposed to do and in a
cheap manner.

------
alkonaut
What do you mean by "servers"? Anything that isn't a client machine, or just
customer-facing infra?

We have a pool of 15 build servers for our CI. They run basically 100% cpu
during office hours and tranffer terabytes of data every day. They have no
real requirements for backup, reliability etc, but they need to be fast. If I
run a pricing calculator for hosting those in the cloud it's ridiculous. We
are moving source and CI to cloud, but we'll probably keep all the build
machines on-prem for the foreseeable future.

For customer facing servers the calculation is completely different. More
traffic means more business. Reliability, Scalability and backup is important
and so on.

------
urza
I would like to add, that even for small teams/projects the cost is the
reason. We have small business project, with only one server, few hundred
customers, with varying traffic (database sync between thick clients, web
portal,..). We were considering cloud, but with the features we needed (few
different databases, few APIs,..) it would be cca $1000/month (with reasonable
response times - could be cheaper but terribly slow). Having our own on
premise server, the price is back just after few months and then just the
minimal cost of connectivity, energy and occasional maintenance.. it just
didn't make any sense for us to choose cloud.

------
adev_
I worked in Switzerland and a reason to use on premise here is _Security_

Many detail banks, asset management company or high security company refuse to
use any public Cloud.

They want to have a strict and traceable list of people who have physical
access to their hardware.

This in order to control any risk of dataleak [1].

In practice they use generally on-premise installation. They ren space in a
computer center and own there a private cage monitored with multiple cameras.
Meaning they know exactly anyone touching their hardware and enforce security
clearance for them.

[1]:
[https://en.m.wikipedia.org/wiki/Swiss_Leaks](https://en.m.wikipedia.org/wiki/Swiss_Leaks)

------
jordanbeiber
My last three employments have had me and my team build three different
platforms with three different “providers”. Many lessons learned!

Chronological order:

1\. E-commerce, low volume (1000-5000RPM), very high value conversions, highly
localized trade.

We built an on-prem stack using hasicorp here. This place had on-prem stuff
already in place, the usual vendor driven crap - expensive hypervisor,
expensive spoffy SAN, unreliable network. Anyway, my platform team (4-5 guys)
built a silo on commodity hardware to run the new version of this site. This
is a few years back, but the power you get from cheap hardware these days is
astounding. With 6 basic servers, in two DCs, stuffed with off the shelf SSDs
we could run the site and dev teams no problem. Much less downtime compared to
the expensive hyperconverged blade crap we started on at basically no cost.
There’s a simplicity that wins out using actual network cables and 1u boxes...
LXC is awesome btw! Using “legacy” vmware, emc, hp etc for non-essential on-
prem? Cloud is tempting!

2\. Very high volume (Billions of requests per day), global network. AWS. Team
tasked with improving on-demand scalability. We implemented kubernetes on AWS
and it really showed what it’s about! After 6-7 months of struggle with k8s <
1.12 things turned around when it hit 1.12-1.13-ish we got it to act how we
wanted. Sort of, at least. Cloud just a no brainer for this type of round-the-
clock, elastic workload. You’d need many millions up-front to even begin
building something matching “the cloud” here. Lot of work spent tweaking cost
though. At this scale cloud cost management is what you do.

3\. Upstart dev-shop. No rpm outside dev (yet). Azure. About 30 devs building
cool stuff. Azure sucks as IaaS, they want you to PaaS that’s for sure. Cloud
decision had been made already when I joined. Do you need cloud for this? No.
Are there benefits? Some. Do they outweigh the cost? Hardly. In the end it
will depend on how and where your product drives revenue. We pay for a small
local dev datacenter quarterly, which i find annoying.

Just some quick thoughts off the top of my head (on the phone so excuse
everything).

Happy to discuss further!

------
Cthulhu_
Our software runs in core (mobile) network systems, but that's at our
customers. We ourselves have a rack in our office that runs things like Git
repos, project management, virtual machines for development / testing, build
servers, and instances for trainings.

We're concerned about corporate espionage and infiltration, so we can't trust
our servers being out of our sight. Most people don't have the code on their
physical machines either; I'm a cocky breath of fresh air in that regard in
that I prefer my stuff to run locally instead of the (slow, underpowered) VMs,
I trust Apple's encryption a lot.

------
rjgonza
I work for a US Stock Exchange, and some of the technologies that we rely on
are not permitted in the cloud. The performance metrics we need are usually
only achievable on highly tuned bare metal deployments as well so cloud is
usually not an option. I guess it really depends on your workload, but I think
there is a very healthy amount of production being deployed and worked on
businesses own datacenter/private clouds.

------
sokoloff
We have mostly switched to, or are in the midst of switching to, the cloud.

Services that we will continue to run on-premises (as an exception to that
rule) are some machine learning _training_ clusters (where we need a constant,
high-level amount of GPU and cloud provider pricing for GPU machines is very
far off the mark of what you can build/run yourself) and some file shares for
our production facilities where very large raster files are created,
manipulated, sent to production equipment, and then shortly afterwards
deleted.

Most everything else is going to the cloud (including most of our ML _deployed
model_ use cases).

------
DyslexicAtheist
this assumption doesn't consider threat models where the vendor could be part
of your problem. E.g. if you're based in country A and work on super secret
new Tech for an emerging industry, then hosting in country B may not be an
option.

Imagine a company in Europe that decides to host it's files on Alibaba Cloud
in the US.

Imagine the US Department of State hosting it's files with Google.

Imagine an energy company working on new reactor Tech, ...

Imagine a Certificate Authority which has an offline store of root
certificates which need to come online to sync twice a day.

Imagine cases where you need a hardware HSM.

Then there is also Cost as others have pointed out. AWS cost structure is so
complex that whole business models[1] have sprung up to help you optimize the
price tags or reduce the risks of huge bills. that's right: you need to have a
commercial agreement with another partner that has nothing to do with your
cloud just to work around aggressive pricing. The guy who started this ~2
years ago has grown to 40+ people (organically), is based in Switzerland and
is still hiring even in this recession. It should give you an idea of how
broken the cloud is.

Lastly there is also the lock-in. All the hours that you have to sit down and
learn how the AWS IAM works is wasted once you decide to move to another
cloud. The cost for learning how to use the 3rd party API is incurred by you
not the cloud vendor. For people who think lock-in isn't much of a problem
remember your whole hiring strategy will be aligned to whatever cloud vendor
you're using (look at job description that already filter out based on AWS or
GCP experience). Lock-in is so bad that for a business it is close to the rule
of real-estate (location, location, location), only that it's to the advantage
of the cloud vendor not you as the customer.

[1] optimyze.cloud

[2] _" I have just tried to pull the official EC2 price list via JSON API, and
the JSON file is 1.3gb"_
[https://twitter.com/halvarflake/status/1258161778770542594](https://twitter.com/halvarflake/status/1258161778770542594)

------
blackflame7000
If you have a lot of data but not a lot of users, it's prohibitively expensive
to pay monthly hosting and network egress fees when you can buy cheap hard
drives, use ZFS, and whatever server protocol you desire.

------
mcv
I notice all the banks I'm working for are moving to the cloud. A few years
ago they all had their own data centers, sometimes really big, well-designed
custom data centers. But they're all moving to the cloud now.

I've personally been wondering whether that's wise, because financial data and
the handling of many banking processes are a bank's core business. It makes
sense that a bank should be in control of that. And it needs to obey tons of
strict banking data regulations. But apparently modern cloud services are able
to provide all of that.

------
Xelbair
Cloud costs way way more compared to on-prem solution for my company.

We need random access to about 50TB of files, and quite a decent number of
VMs.

For storage on-perm vs cloud: buy was cheaper to have after 3(!) months.

For VMs(some of them could be containerized though): 1 year

It was cheaper to buy second-hand decent server, slap SSDs and just install a
decent hypervisor. Those costs also include: server room, power usage, admins
etc.

We do use cloud backups for the most important stuff.

Cloud is cheaper if your business is a something that is user-based - as in
you might need to scale it, hard.

If you aren't doing anything like that it is absurdly expensive.

------
benbro
Can you recommend dedicated hosting provider that: 1\. Has US, EU and Asia
regions. 2\. Let you rent servers per month. 3\. Has decent pricing. Not
premium, doesn't have to be low cost. I expect excellent egress pricing and
1/2-1/4 cost for CPU compared to the cloud. 4\. Reliable network. GCP premium
network is very good. How does dedicated providers and VPS providers (Linode
and DO) compare? 5\. Easy to use management and dashboard. Experienced really
bad dashboards and hard to use Java tools to install and manage dedicated
servers.

------
lrpublic
\- cost, as well evidenced in other comments here. The hyperscalers are orders
of magnitude more expensive than dedicated hosting or using collocation
providers.

\- lock in, all the hyper scalers want to sell you value add services that
make it hard or impossible to move away.

\- concentration risk, hyper scale providers are a well understood target for
malign actors. It’s true they are better protected than most.

\- complexity, if you think about how little time the hyperscalers have been
operating in comparison with corporate IT they have created huge technical
debt in the race to match features.

------
sgt
We do a hybrid approach which I think makes sense for a lot of companies. Our
mission critical stuff runs in the cloud, but anything that has to do with
staging environments and development we do on-premises. It's pretty easy to
host yourself, if you have a couple of decent engineers looking after it
(depending on scope, of course!).

Redundant power, redundant internet connections, and a few racks of Dell
servers and gigabit switches. Why did I mention Dell? They just don't seem to
die. We used HP for a few years but had a few bad experiences.

------
2rsf
Security (we're a bank)

~~~
aspyct
How is the cloud less secure than your on-prem servers? I would argue that
it's easier to keep track of all the threats with the tools available from big
cloud providers.

~~~
badpun
The big question is - how much do you trust these providers.

~~~
quicklime
I would guess that approximately zero banks own data centers that are operated
by their own employees. There might be a few exceptions to this, but the
reality is that most banks don't view technology as part of their core
business. So this largely gets outsourced to IT consulting firms like Infosys,
IBM, Wipro, etc.

The big question is - how much do you trust _these_ providers, and do you
think they are more competent at security than Amazon/Google/Microsoft?

~~~
dathinab
Or formulate it differently:

1\. Trust a provider which whole existence relies on trust and which you can
audit or at last cross-check the audit and security processes (as a Bank you
are normally not a small customer).

2\. Trust a provider which might 1st be a potential competitions in some
business fields. 2nd is so big that it can easily afford losing you. 3nd for
the same reason doesn't allow you any insights into there internal processes.
etc.

Plus many of the banks having their own hardware also have their own IT team.
So it's often about trusting your own people. I mean either you keep your it
or you outsource _and_ go into the cloud. Keeping local servers but
outsourcing IT at the same time seems kind not very clever tbh.

------
samcrawford
Cost is the sole reason for us as well. We have ~600-700 dedicated servers
around the globe, and generate a lot of egress traffic (~20PB/mo). We last ran
the figures a year or so ago, and it'd cost us around 13-15x in network costs
alone.

A common thread of a lot of the replies to this post is network traffic costs.
If one of the cloud providers can figure out a way to dramatically (and I mean
at least 10x) reduce their network transfer pricing, then I think we'll see a
second wave of companies adopting their services.

------
irrational
It is so so so much cheaper. We moved to AWS and tried setting up our servers
with specs that were comparable to what our physical servers had been. We just
about died after the first month. Our bill was higher for one month than for
multiple years running our physical servers. We had to way dial them back to
get the run rate down to a reasonable number. Now we have serious buyers
remorse. Performance is terrible. The cost is still more per month than we
ever had with our physical servers by a large amount.

------
fpierfed
We do not use the cloud. We operate (24/7) facilities in remote locations
where we do not have super reliable internet connection (we do have redundant
links including three different fibres on distinct paths plus radio links but
still). For this reason alone nothing critical can be in the cloud. In our
experience, however, cloud offerings are not that cheap compared to purchasing
and operating your own machines. Besides, one still needs sysadmins even when
operating infrastructure in the cloud.

------
kjgkjhfkjf
On-prem can make sense where your computing needs are constant and
predictable. You can arrange to have exactly what you need, and the total cost
of buying and operating it may be less than it would be to get comparable
resources in the cloud.

If your computing needs vary over time, then provisioning on-prem for peak
load will mean that some of your resources will be idle at non-peak times. It
may be cheaper to use cloud resources in cases like these, since you only need
to pay for the extra capacity when you need it.

------
blodkorv
Am the CTO of a small company making payroll software. We don't have on
premise servers but we currently are moving away from azure to renting VPSs
from a local provider.

The cost benefits are huge and since our app is mostly a normal web app we
dont need that many fancy cloud things. And i dont see us needing it in the
future.

I really dont understand why a company doing similar things would wanna go the
could route. Its so damn expensive and its not always easy to use and setup.

------
iseletsk
We are software development company. Most of our compute/storage needs are for
build/test cycles. We recently bought ~100K worth of additional hardware to
migrate some of that work off AWS. The storage / virtualization is done using
Ceph & OpenNebula. Including colocation/electricity/networking costs, the
investment will pay for itself in ~9 months. If I would include deployment
costs & work to migrate the jobs off the AWS -- it will pay for itself in 11
months.

------
starpilot
PII and draconian security policies. We are not a tech company, so we can't
fine-tune or have nuanced policies, we just have to build a wall around
everything . In our web password recovery process, we can't tell people if
their login was correct or not, because that might help a brute force attacker
infer they got that right. Even though we have that rate limited anyway. I
don't know why we can't just tell people the login was found or not found like
banks etc.

~~~
vikramkr
What industry? It's good practice to not share information like that in any
context, since attackers that have a bank if email addresses that are trying
to figure out which if a few reused passwords might be used for a given
website would have a harder time if they dont even know if the email has an
account with a given website, or if an alternate email is used instead etc.

------
kkielhofner
Not a company but a personal project:

[https://github.com/krisk84/retinanet_for_redaction_with_deep...](https://github.com/krisk84/retinanet_for_redaction_with_deepstream/wiki)

I haven't analyzed the TCO yet but the bandwidth costs alone from hosting my
curated model of 100GB in the cloud (Azure blob) have greatly exceeded my
total operation spend from downloading the entire dataset and running
training. By an order of magnitude.

------
jll29
The smartest approach is to be able to run anywhere, which is increasingly
practical due to VMs, Docker etc.

(At least funded) startups should start with the cloud as speed to completion
is key, but can later optimize for cost.

Elasticity of the cloud is also great, dealing with peak demands dynamically
without having to purchase hardware.

I'd suggest larger companies to use at leas two cloud vendors to add
resilience (when MS Teams went down, so did Slack - I was told they both use
MSFT's cloud).

------
tpae
Because Elon doesn't believe in the cloud. We were one of the few teams that
got AWS access, and we were told not to rely on it too much because it's
temporary...

------
thecolorblue
Just talking about my side project here: a local server + ngrok is easier to
use and cheaper than anything in the cloud.

In general, I would say any noncritical system I would host on-prem.

------
edoceo
Cost. We're a small company, five person team and we need a development
environment. All our stuff is built around VM and Docker. So scores of little
test nodes that get run, or beta environment in the cloud was costly (>$300/mo
base, sometimes 3x). For $1000 we put a box in the office we all VPN to that
runs all the necessary VM and Docker. The variable, sometimes expensive cost
for Test/Beta in cloud was replaced with low fixed cost.

------
nurettin
Sure, for tiny intel servers it makes sense to rent vms. It won't start to
hurt until 1.5 years later at which point the project needs to become
profitable anyways.

I run a couple of on-premise xeon gold machines with 96 gb ram and 40+ cores
on each. Their total purchase cost was the monthly cost of renting them on the
cloud. Also, you will never get the full benefit of the servers you use unless
they are dedicated instances with no virtualization layer.

------
pachico
It depends on what you have to do. If your stack includes a series of
microservices/monoliths connected to the typical OLTP DB then you might very
well sit entirely on cloud. Things change when you need heavy lifting like
having big ElasticSearch or ClickHouse clusters, or any other operation that
requires heavy CPU and high RAM capacity. In that case using providers like
Hetzner can cost you 1/10 of the bill compared to AWS.

------
AdrianB1
Manufacturing plants across the world controlling production lines: no way to
go in the cloud for that. We put the reporting data in the cloud, no problem
there.

------
Yizahi
"Server" is such a broad term. In our case, aside from cost as others already
mentioned, the distance and latency is very important. The servers must be
located as close to client devices as possible and reasonable, and they are
synced with clients using PTP to microseconds (depends on the actual distance
and topology). Cloud is a no go, and we a using bare metal K8S for graceful
handling of failures and updates.

------
iso1631
Most of my equipment has physical interfaces, video and audio in and out

Some equipment is very latency sensitive -- I'm talking microseconds, not
milliseconds.

More generic tasks need easy access to that specialist equipment (much of
which doesn't quite grasp the concept of security)

Given that we therefore have to run a secure network across hundreds of sites
on multiple continents, adding a couple of machines running xen adds very
little to the overhead.

------
thorwasdfasdf
Well, for one thing: jobs. It takes a lot of IT man-power to manage an on-
premise solution, especially when you run everything on-premise. Just imagine
if the CTO were to switch the company to a cloud based solutions, it would
save the company millions of dollars but also it would mean cutting a lot of
jobs. Gov departments that use on-premise do so for security reasons and to
maintain existing jobs.

------
timbre1234
Amazon/Azure/GCP - they're _businesses_. They charge you 50-70 points of
margin in order to run your computers. If you're R&D-limited, that's not
important to you, but if you're a more mature company that's cost-limited then
it matters a lot. If it's a core part of your business, it'll never be cheaper
to pay another company to profit from you. Period.

------
StreamBright
Yes, some of the leadership thinks that they can build a better cloud than MS
or AWS. It is pretty hilarious to watch how spectacularly they fail.

[https://forrestbrazeal.com/2020/01/05/code-wise-cloud-
foolis...](https://forrestbrazeal.com/2020/01/05/code-wise-cloud-foolish-
avoiding-bad-technology-choices/)

------
mathattack
We are going to the cloud, but you have to be careful. With on-prem the limit
of the cost is the server. Someone writes inefficient code and it just doesn’t
work. In the cloud there are 1000 ways to overspend and the vendors
purposefully don’t make it easy to track or keep things under control.

It’s kind of like outsourcing. If you don’t know what you are doing, cost goes
up and quality goes down.

------
Mave83
Cloud is only good if you don't care about costs and plan to scale without
looking back.

For example, building a Ceph based Software Defined Storage with croit.io for
S3 comes for 1/10 to 1/5 of the AWS price in TCO. Same goes for any other
product in the cloud.

If you only need it a short time up to 6 months, go to the cloud. If you plan
to have the resources longer than 6 months go to Colocation.

------
gbasin
Cloud will continue to dominate, even if it's more expensive. Why? Because the
best companies are able to focus on what makes them special, and outsource the
rest.

Cost and security are important, but they may not be most important. In a
business, the scarcest commodity is FOCUS. By outsourcing anything that isn't
core to your product, you can excel at what differentiates you.

------
Darkstryder
On top of what others have said, when outside the US the Cloud Act has been a
big one for most previous companies I worked for.

Using AWS / Azure / Google Cloud (even using datacenters from your own
country) implies that the US government can access your data at will.

As soon as you treat sensitive information, especially related to non-US
governments, this becomes a blocking factor.

------
Areading314
In some industries, cloud is not an option. For example, certain privacy laws
like HIPAA preclude uploading data to third parties, in which case, you need
things to be on-prem. There are also a lot of places in the world where
internet access is limited. Sometimes you need to solve problems beyond the
simple "web saas in cloud" use case

------
reverseengineer
Own servers cheaper than cloud. We calculated two years ago, it was up to 4x
for HPC. Best cost-wise option is used servers.

------
sradman
Legacy seems to be missing from the comments. Before the advent of Cloud IaaS
(Infrastructure as a Service) a very large ecosystem of On-Premise hardware
and software flourished. The question needs to be considered in the context of
greenfield vs brownfield systems as the trade-offs involved differ
drastically.

------
Spooky23
It all depends. Outside of SaaS, If you have a mature data center operating
model and truly understand your costs, there won’t be a strong cost savings
story for many types of workflows.

If you suck and don’t understand costs, or don’t automate, or spend a lot of
time eating steak with your Cisco team, you’ll save money... at first.

------
mister_hn
There's a fake believe in my company that all the data must be on premise
because of privacy concerns.

We've never used cloud services and we do not want to use it.

Some are saying it's a matter of costs, but you know? For a dual node server
(hot standby) we were asked 120K € + 50K € only for configuration fees.

------
32gbsd
Cloud servers tend to hide their limitations behind payment tiers which makes
it hard to really know how far you can push things. Also there are various
turns, conditions, cache rules, change management strategies that are hidden
when dealing with someone else's constantly changing box of magic.

------
lvturner
We install in to remote locations - you can't access the cloud if your
connectivity is down so local resources is a hard must.

Though we have adopted something close to an "Edge Computing" solution... I
guess it comes down to "Why not both?" :)

I think it also depends on your definition of "server"

------
sergiotapia
Just want to say I love threads like these. I hope one day where I work we can
have two or three obscenely beefy servers and be done with it. I'm planning
for something similar probably Q2 2021 as our expenses grow too large on a
manged hosting platform like Heroku/Render/Aptible.

------
dahfizz
There are some things you just can't do in a VM.

The company I work for actually develops and hosts an AWS clone for the Linux
Foundation, but with very specific requirements. They have special needs that
requires baremetal machines and "real" networking between them across 6+ NICs
per server.

------
withinrafael
Not many services are available in gov cloud regions, so we're stuck with on-
prem nearly everything.

------
jarym
my 2cents: the 'big' clouds (AWS, GCP, Azure) and the big-brand clouds
(Oracle, IBM, etc.) are attractive for BigCustomerCo because:

1\. Replace capital expenditure of in-house infrastructure + staff with OpEx
that can be dialled down

2\. Get to benefit from the economies of scale that the cloud vendors get
(those Intel CPUs get a lot cheaper when purchased in the 1000s)

3\. Get to leverage big shiny new tech like Big Data and AI that's
'advertised' as 'out-of-the-box'

My only concern really is that the big cloud players are all fighting for
dominance and market share. What happens in the next 5-10 years time when they
start raising prices? Different kind of lock-in - customers won't have the
expertise in-house to migrate stuff back.

------
trelliscoded
Zseries are a huge deal to move to the cloud; it’s just not worth the risk for
most organizations.

------
icelancer
Yes. Cost of GPUs if you want them on regularly and reliably, and you don't
need 2000 of them. We run small/mid-sized operations on-demand and the latency
of spinning up instances is not competitive and the cost is outrageous to have
them on standby.

------
collyw
There is extra complexity with managing cloud based solutions. Logging in,
setting up ssh keys. Ok it's all automatable but if I want a basic server set
up for doing a simple task quickly it's often a lot easier to run it on an
internal server.

------
dryst
If your product doesn't run connected to the internet, it is difficult to make
the case for cloud development. You need developers who understand hardware,
and abstraction layer cloud provides is a handicap instead of an enabler.

------
julienfr112
I think it depends on what is the alternative and what hardware you are
buying. If you go for top of the line HPE or dell or Cisco hyperconverged
stuff that allow you to be, sort of, your own cloud, you will end up with the
same large bill.

------
skunkiferous
I work for a German company. They run their own DC (unfortunately, I'm not
privy to precise numbers I could share, but we must be in the 1000s of
hardware servers).

Why? Because our (German) clients don't _thrust US cloud providers_.

------
exabrial
Hybrid cloud is the moneymaker solution, but there are no out of box solutions
for it.

------
maltelandwehr
Gigantic Elastic Search cluster - according to Elastic the largest in Europe
(as of 2 years ago) - used in production. Broke again and again on AWS. We
needed more shards than AWS supports. Moved to bare metal again.

------
iamgopal
When cloud computing started, alternative software deployment was complicated,
after a decade, much has been improved. So ease of management is also one of
the factor. Not all need state of the art data and AI.

------
kevlar1818
We use on-prem GPU nodes for training deep learning models. Our group
estimated the cost vs. cloud and it was significantly cheaper to go on-prem. I
can't speak to security-wise and whatever-wise though :)

------
caseyf
We colocate and as a small tech business, I can't imagine doing anything else.
We don't spend more on payroll due to colocating and AWS/etc would easily
double our annual non-payroll expenses.

------
cm2187
One good reason for your list: diversification. You don't want all the banking
systems of a country all running on AWS. It's an unacceptable single point of
failure risk for a country.

------
joaodlf
Like many said, cost.

But also - legalities. Most cloud providers have very unclear rules and what
exactly happen should you be in breach. For this reason, our business prefers
to have most of the control.

------
z3t4
There are several levels, not just on-prem vs cloud. You can for example co-
locate, rent dedicated servers, rent a single VPS, or put your website up on a
web hosting provider...

------
benbro
How do you manage installation and upgrades of dedicated servers? What do you
use for block storage and object storage? Kubernetes and Ceph seems to be hard
to setup and maintain.

------
altmind
to control the costs. after the original purchase, the MRC is a fixed cost for
hosting and for the network access. also, for the total control - the network,
io, cpu performance does not change randomly. with better predictability our
IT team can give more precise SLA.

we're not 100% on-prem, but aws, gcloud and azure are the worst examples of
3rd party hosting - unpredictable and with complicated billing. we're
considering the alternatives to big 3 for the "cloud hosting"

------
manishsharan
I work in a very large financial institution in small tech team. I would give
an arm and a leg to not deal with soul sucking bureaucracy that is our IT
department.

------
PudgePacket
This thread has been illuminating, a lot more non-cloud people than I thought!
Drinking the cloud kool-aid had me thinking cloud was the only realistic way
to go.

------
JoeAltmaier
My client's customers are geolocated (continental US) and their personal data
is sensitive. So their server is in their own firewalled server closet.

------
jerzyt
Client data confidentiality. I know it's a weak argument, but if the contract
requires that we store data in-house, there's no choice.

------
kortilla
It’s not secure if you have air gap requirements or issues with employees from
another company technically having access to all of your data.

------
FpUser
I normally host on prem and also rent dedicated servers elsewhere as standby.
Way cheaper than the cloud and full control the way I want.

------
GaryNumanVevo
We run one of the largest Hadoop clusters on the planet. On-prem is very cost
efficient if you're running jobs flat out 24/7

------
natmaka
Confidentiality. Protecting sensible data (avoiding letting some hostile
obtain or even modify them) seems impossible on the cloud.

------
hans_castorp
We are staying on-premise for security and privacy reasons.

We can't store our data outside of the company (or even worse: outside of the
EU)

------
moonbug
TCO.

~~~
aspyct
Which, all things considered, seems lower on the cloud. Could you give more
details on this answer?

~~~
lazylizard
A 2u asus with 2 xeon silver, 64gb ram and 4 x 2080ti is maybe us$15k?

We'll use it for as long as its producing useful output. Lets say 5 years?
Probably a little longer?

A 60bay 4u western digital with 14tb drives is under us$50k?

definitely got a dell md1280+1u server with 70x10tb 2yrs ago for under us$50k.
Fully populated the following year..

A 2u dell with maybe 20 cores n 128gb ram each should cost less than us$10k.

And we just got 4 or 5 dell switches with 48x10g ports for 50-70k? I'm not
sure.

What's the equivalent in the cloud?

~~~
aspyct
Are you keeping tabs on the cost of the building and associated security?
Video cameras, locks... Also the manpower needed to install and maintain that
equipment.

Similarly, your 70*10tb tells nothing. What's the redundancy on that, how much
of that space did you lose to it, where do you store your backups and at what
cost?

As for networking, having those switches is nice, but you still need your
internet connection if you're serving anything online from this.

~~~
lazylizard
The manpower to run that stuff is already doing desktop support.

Storage is usually a stripe of 10 disks in raidz2 + 2 spares. Then we do a
nightly zfs send..

The internet connection for the hosting is 100mbps. The users share 1gbps of
internet.

We pay only for hardware. Rent and utilities. And the internet connections.
Everything else we can do it ourselves.

I mean. Seriously. Whats the cost of running 100 x 4 x 2080ti in the cloud for
a year? Or storing 500tb(1 instance only,no redundancy) ?

------
leothecool
If your customers still want the system to run air-gapped from the internet,
cloud is basically off the table.

------
mlang23
Cost and data security, while cost actually weights more. It is simply not
true that the cloud is cheaper.

------
totorovirus
I work in a data science company with 30+ engineers. We've spent 80k dollars
on GKE last month only..

------
yread
Yes just moved from 4 servers in AWS costing 700 usd/month to a single
dedicated one that costs 40.

~~~
gizmodo59
Just out of curiosity, what downtime is in acceptable range for that single
server?

~~~
yread
At night noone cares

------
fulafel
Cloud is even more muddled as a term in this space than usual (Hybrid/private
added to the mix).

------
som33
>I honestly fail to see any good reason not to use the cloud anymore, at least
for business. Cost-wise, security-wise, whatever-wise.

Problem is point of failure, many businesses need to be independent and having
data stored in the cloud is a bad idea overall. Because it produces point of
failure issues. Consider if we ever got a real nasty solar wind and the
electric grid goes down, the more we rely on the internet and centralize
infrastructure into electric devices, the more it becomes a costly point of
failure.

While many see redundancy as "waste" in terms of dollars, notice that our
bodies have many billions of redundant cells and that's what makes us
resilient as a species, we can take a licking and keep on ticking.

Trusting your data to out-side sources generally is a bad idea any day of the
week. You always want to have backups and data available in case of diaster,
mishap, etc.

Like no one has learned from this epidemic yet. Notice that our economic
philosophy didn't plan for viral infections and has forced our capitalist
society to make serious adjustments. Helping people is an anathema to liberals
and conservatives / republicans / democrats, so for void to come along and
actually force co-operation was a bit tragically humorous.

As a general rule you need redundancy if you want to survive, behaving as if
the cloud is almighty is a bad idea, I'm not sold on "software as a service"
or any of that nonsense. It's just there to lull you into a false sense of
security.

You always need to plan for the worst case scenario for surviveability
reasons.

------
funny948
At least a few years ago, tons of company were afraid of the 'cloud'.

It does change right now.

------
blaser-waffle
Big public clouds have a genuine purpose but there is a shit-ton of marketing
and FUD being thrown around about them -- I'd bet my hat that's what this post
is, given the phrasing up top.

I'm not a fan. In short:

Cost. CapEx and depreciation vs. OpEx. The numbers look amazing for ~3 years
until the credits and discounts wear off. Then it's just high OpEx costs
forever. Meanwhile I can depreciate my $10k server over time and get some cash
back in taxes; plus it's paid for after a couple years -- $0 OpEx outside of
licenses, and CentOS has no license cost.

Once you have significant presence in someone's cloud, they're not going to
just lower costs either -- they've got you now. What in American Capitalism
circa 2020 makes you think they won't find a way to nickle and dime you to
death?

It's not going to reduce headcount, either. Instead of 14 devops/sysadmins,
now I have 14 cloud admins, sitting pretty with their Azure or GCP certs.
Automation is what's going to reduce those headcounts and costs, and
Ansible+Jenkins+Kubernetes works fine just with VMware, Docker, and Cisco on-
prem.

Trust. The Google Cloud just had a 12-hour outage -- I first read about it
here on HN. AWS and Azure have had plenty of outages too... usually they're
just not as open as Google is about it. You also have to trust that they won't
get back-doored like what happened to NordVPN's providers, and that they're
not secretly MITM'ing everything or dup-ing your data. We (and some of our
clients) compete with some of the cloud providers companies and their
subsidiaries, and we know for a fact that they will investigate and siphon any
data that could give them an advantage.

Purpose. _We just don 't need hyper-scalable architecture._ We've got a
(mostly) fixed number of users in a fixed number of locations, with needs that
are fairly easy to estimate / build for. Outside of a handful of sales &
financial processing purposes, we will never scale up or down in any dramatic
fashion. And for the one-off cases, we can either make it work with VMware, or
outsource it to the software provider's SaaS cloud offering.

If we were doing e-commerce -- absolutely. Some sort of android app? Sure, AWS
or Azure would be great. But it's a lot of risk and cost with no benefit for
the Enterprise orgs than can afford their own stuff.

------
hendry
Bandwidth is the problem. You can't run a Youtube on any cloud provider.

------
alpenbazi
Security. And Independency.

------
aripickar
I work for AWS, so (in the most pedantic way possible) technically yes

------
whorleater
Yes, but we work in financial information so very weird requirements.

------
gramakri
We use servers on the cloud (IaaS) but still self-host all our apps.

------
_wldu
Local caching recursive DNS servers work best close to the clients.

------
Abukamel
Yes online.net and the like companies save costs

------
itworker7
In physics, a gallon of water weights 8.34 lbs. (For this analogy, a gallon of
water is a unit of work.) And the gallon of water weighs 8.34 lbs irregardless
if it is sitting on my desk in a physical building, or on your desk, in the
cloud. Same weight, same unit of work. same effort. For a brand new,
greenfield application, the cloud is a no brainer. I agree 100%. But for
legacy applications, and there are so, soooo many, the cloud is just some one
else's computer. Yes, the cloud is more scaleable, yes, the cloud is more
manageable, and yes, you can control the cpu/storage/memory/network in much
finer amounts. But legacy applications are very complicated. They have long
tails, interconnections to other applications that cannot immediately be
migrated to the cloud. I have migrated clients off of the cloud, back to on
premise or to (co-lo) local hosting, because without rewriting the legacy
application, the cloud costs are simply too great.

The essence of IT is to apply technology to solve a business problem.
Otherwise, why would the business spend the money? The IT solution might be
crazy/stupid/complex but if it works, many business simply adopt it and move
on. Now, move that crazy/stupid/complex process to the cloud and surprise, it
is very, very expensive. So, yes, the cloud is better, but only for some
things. And until legacy applications are rewritten on-premise will exist.

One final insight. The cloud costs more. It has been engineered to be so, both
from a profitability standpoint(Amazon _is_ a for profit company) but also
because the cloud has decomposed the infrastructure of IT into functional
subcomponents, each of which cost money. When I was younger, the challenge for
IT was explaining to management, the ROI of new servers, expanded networking,
additional technology. We never quite got it right and often had it completely
wrong. That was because we lacked the ability to account for/track and manage
the actual costs of an on-premise operation. Accounting had one view,
operations had another view and management had no idea really, why they were
spending millions a year and could not get their business goals accomplished.
The cloud changed all of that. You can do almost anything in the cloud, for a
price. And I will humbly submit, that the cost of the cloud - minus the
aforementioned profitability, is what on-premise organizations should have
been spending all along. Anyone reading this and who has spent time in a
legacy environment, knows that it is basically a futile exercise of keeping
the plates spinning. On-premise failed because it could not get management to
understand the value on in-house IT.

As I said, the costs are the same. A gallon of water weighs what it weighs
regardless of location. It will be interesting to see, I predict the pendulum
will swing back.

------
shakkhar
Because we don't trust either amazon or google.

------
loeg
Cost-wise is still a pretty compelling argument.

------
Stierlitz
Annual costs, backups, security and latency.

------
markc
on-premises not on-premise

------
sarasasa28
we are not zoomers

