
Ask HN: If I Close My Data Centers, What About the People/Jobs Lost? - throwawayacct4q
(throwaway account)<p>I have a chance to basically migrate 90% of my F50&#x27;s data centers to commercial providers. The cost savings are awesome, but what about the people currently doing legacy stuff? theyre doing stuff like manual config, and future state is everythign automated as much as possible in the cloud and shutter the data centers.<p>I think the harsh truth is that many wont have jobs post move, but maybe I&#x27;m being cynical. maybe theres a chance to retrain the employees, but even doing that automation generally reduces the workforce.<p>whats your take? what do I tell current employees? how many can I realistically be able to save from being jobless?
======
jedberg
When Netflix moved from the datacenter to the cloud, there were a whole bunch
of really good people with skills we didn't need anymore -- sysadmins, DBAs,
datacenter techs, etc.

Netflix went to them and said, "do you want to learn some new skills that
apply to our new cloud based infra?" Some said no, and were given big
severance checks and glowing recommendations. Others said yes, and were
retrained.

And some were retrained but it didn't work out, it turned out they weren't
interested in the new work, or just weren't needed, and so they too were given
big severance checks and glowing recommendations, and went to look for new
jobs using both their old and new skills.

It's nice that you're worried about them, but the best thing you can do for
them is offer them the chance to learn new skills, either through new jobs or
paying for them to learn new skills.

~~~
brianwawok
This seems best - offer them the choice.

Don't go "learn to code cloud formation in 30 days or lose your job with no
severance".

------
boulos
Disclosure: I work for Google Cloud (and I assume/hope someone at Google is
competing for your business).

Some of those folks happily do updated things, but others (like people that
rack servers) won’t have an obvious new role in a cloud focused world.

My (now) colleague at Nic Harteau gave a talk at our Leaders’ Circle event
talking about precisely this in Spotify’s move to Cloud: how could he keep the
infrastructure team as they moved to cloud? As it turned out, most of them
happily adjusted what they were doing because the mission was the same: serve
the product teams / groups that need infrastructure from them. Sure, now you
don’t configure a physical firewall, but there are still firewalls in cloud.
In a sense, they’re even more important!

I _believe_ Nic said only one person from the group left. The engineer was
focused on building their own SDN stack on-prem. That job, and other “things
the provider completely handles” do go away. But that person likely would have
been massively useful in helping guide how Spotify should use Google Cloud’s
own SDN setup. There are lots of ways to architect how services interact in
clouds, and a lot of it comes down to funny networking.

So I’d say more than you might think, but please don’t mislead them.

------
tucaz
I will assume that at the current state you are able to pay both the
datacenter costs and people costs since it seems that is what you are
currently doing.

With that assumption in mind, if you do migrate and realize that you now don’t
need 80% of the people you needed before, it could mean one of two things:

\- you are now able to pocket whatever savings you were able get from the
migration and either save it or invest in something to potentially keep that
money in the economy, just in a different way

\- you now have 80% of your people free to do whatever other things you didn’t
do because they had to maintain datacenters. That is huge in terms of
efficiency gains and allows you to redirect that money to increase the value
provided by your company either by repurposing the current workforce or
getting rid of them and bringing people you could previously not bring on
board due to existing costs

Either way I believe that everybody is better off After the changes.

Efficiency gains such as the one you are describing is what makes it possible
for economies to grow and keep moving forward.

Can you imagine if you had to handle cows, pigs and farms to have a BLT? It’s
very good that such things are outsourced because that allows for improvement
that would otherwise not be possible.

~~~
subway
Odds are high that the anticipated cost savings sits largely on the folks they
intend to sack, rather than on any power/real estate/bandwidth/hardware
savings.

~~~
throwawayacct4q
Actually the cost savings hinge on the massive cost savings from moving to a
commercial cloud, laying off workforce would save more, but would prefer to
save a bunch by moving to cloud and keep the workforce for other things.

~~~
mmt
That's extremely surprising.

I see two possible explanations (both of which may apply):

(1) You're getting the cloud servers for a remarkable discount, which, as
other commenters mentioned, may not last if you don't put in the engineering
effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost
as otherwise implied).

(2) You're grossly over-paying for your current environment. Given what I know
about the purchasing habits over merely large (nowhere near the size of F50,
mind you) companies and, more importantly, the pricing of "enterprise"
hardware and _support_ , this seems likely.

If the second is true, then I expect you could gain the same savings, if not
more, by implementing your own "private cloud" by emulating the purchasing and
ops habits of larger-scale startups or established tech companies (but not yet
so huge as to be too custom) who run their own hardware.

You'd still have to retrain and put an end to manual configuration, but I
expect that would ultimately be welcomed. Some might be able to keep their
jobs almost as-is.

What likely would _not_ be welcomed, and therefore might make it impossible to
do, is the political upheaval, since you'd have to put the kibosh on buying
expensive, brand-name gear, all but a few bare-bones service contracts, and
the level of enterprise sales perks that the decision makers have come to
expect. You'd probably have to hire someone who knows how to be frugal like
that from an engineering/ops standpoint for architecture/purchasing and is
willing to work for such a large company and isn't already employed by FAAMG.

~~~
throwawayacct4q
Maybe so, I haven't made this move before, which is part of the reason I'm
asking here, as unlikely as it may be that "someone who is in this position
would be asking on HN." Sorry for being vague on all of this. Regardless:

1.) No specific discount, this is standard, public pricing on major commercial
clouds, with some tricks applied like reserved instances, which is part of the
reason I am not worried about low prices first and then ballooning pricing
later,

2.) Our operations and maintenance and sustaining are in fact more expensive
that anyone would reasonably expect, I believe partially due to vendors
getting their hooks in decades ago and expanding scope and increasing prices,
I've walked the data center floors and coded with the ops teams.

It is #2 that is really hitting us. I have personally moved several apps as a
PoC and you wouldn't believe the savings, or maybe you would, which is even
more when not ported 1:1 but used in conjunction with smaller VMs and
autoscaling.

I would like to retrain to to end all manual configuration and have the
current workforce take over all of this, which is a big deal for me
personally. Thank you for the response.

~~~
exikyut
I'd been wondering about the possibility of #2 when I made my other original
reply a few hours ago. Thanks for clarifying.

The first thing I'm reminded of is [https://www.backblaze.com/blog/petabytes-
on-a-budget-how-to-...](https://www.backblaze.com/blog/petabytes-on-a-budget-
how-to-build-cheap-cloud-storage/). It's from 2009 but this image jumped out
at me: [https://www.backblaze.com/blog/wp-
content/uploads/2009/08/co...](https://www.backblaze.com/blog/wp-
content/uploads/2009/08/cost-of-a-petabyte-chart.jpg)

Let me guess... you're using kit from the vendor at the bottom of that list?
:)

No need to Y/N; I'll assume Y because the sentiment's probably
representatively/loosely correct.

The commentator your/this reply is to points out an interesting aspect:

> _What likely would not be welcomed, and therefore might make it impossible
> to do, is the political upheaval, since you 'd have to put the kibosh on
> buying expensive, brand-name gear, all but a few bare-bones service
> contracts, and the level of enterprise sales perks that the decision makers
> have come to expect._

This sort of jumped out at me a bit, it's a very reasonable assumption to
make, particularly as you say you're an old-school operation, which suggests
to me that the tech components are heavily politicized due to their management
by non-technical types.

Well... the landscape is indeed changing, as clarified by the 9-year old
article above.

So major shifting-around of vendor selection is going to ruffle (and maybe
squash) a few feathers, but these "old world" tech enterprises cannot
reasonably expect everything to keep going indefinitely because commodity-
hardware based solutions have become so accessible that it's harder than ever
to sugarcoat proprietary systems to even mildly-technical management,
particularly with compute and storage, so the writing is on the wall, and you
can use that in your favor.

(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)

And the thing is, for a $B operation (for really any value of $B..$BBB), the
cost-benefit absolutely goes in your favor if you build everything out and run
it yourself. Plus, if you really are a $BBB operation, _you can build your own
servers and network kit_. As in, you get to start answering questions like
what features you want on the motherboards (no Ethernet since you're supplying
your own 10G kit? sure thing... oh you don't want any onboard SATA either?
okay), how you want the systems to boot (build your own secure boot
implementation!), what BIOS/EFI features you want (you want the half-hour RAM
test disableable for your staging boxes? fine), how long you'd like your
enclosures to be (36" is really long but your racks can handle the length
you'll be able to fit a remarkable number disks in 4U), what firmware
optimization suggestions you have for your HDDs (a writeable vendor area in
SMART config for how many seconds you want each disk to pause for before
spinning up so systems come up slowly and don't kill their breakers? done)...
and all of these things will directly translate to cost savings.

If there's one thing that differentiates "old-world" datacenters from shots of
FAANG, Backblaze, etc, it's that all the big players' servers look really,
realy _boring_ , particularly the enclosures. (Well, okay, the blue LEDs make
up for it. :P) And this is because there's almost nothing in those machines
that isn't needed. Can you get away with removing VGA? (You almost certainly
can with virtualization hosts, even if you're just doing KVM or Xen on top of
Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower
power consumption.) All of these things serve to save money. How much exactly
probably depends on what sorts of overheads you're currently dealing with;
just switching to barebones systems that do have a few bells and whistles but
are still very minimalist may get you an appreciable saving that all the
obscure things I've listed here may not be necessary (ie may have
negative/negligible/not-worth-it cost-benefit analyses).

I guess my question is at this point, is the decision scoped down to "what
vendor are we selecting" or is building your own DC still on the table? I so
hope it is, because IMO that's where the cost savings really is.

Hm, maybe you could go "I have one more idea I want to try before we commit to
this, it'll require $$$,$$$" and then go and replace your most expensive
storage array or something :P that could be pretty convincing...

> _I would like to retrain to to end all manual configuration and have the
> current workforce take over all of this, which is a big deal for me
> personally._

I think it's cool you're really getting behind this and making your job
description part of your identity :P hopefully this is respected and you don't
get too heavily stepped on.

There are some really interesting comments in this thread, and a fair bit of
reiteration that clouds are generally expensive solutions. I hold this
sentiment myself, FWIW, despite admittedly having a LOT less experience with
the practical side of things than everyone else here.

This was what I was getting at with the "the offer will be what the buying
team wants to hear" \- at this point it's almost like the vendor sales are
actors in a real-time movie that's exactly what the buying team wants to
watch. On top of that one vendor is really going to be as good as another
(I've been mentioning GCE and AWS, I forgot Azure.)

Cannot keep writing as I must go, may continue later

~~~
mmt
> Plus, if you really are a $BBB operation, you can build your own servers and
> network kit. As in, you get to start answering questions like what features
> you want on the motherboards

I suspect it's this kind of over-reach that ends up scaring decision-makers
off from _any_ in-sourcing (for lack of a better term) solution. To me, the
key is in sticking with commodity solutions.

Even putting aside that the computer tech part of the business isn't their
core competency, custom hardware is _hard_. In the worst case, you create
interdepedent vendor lock-in and indefinite delays. Is there an example of any
large company, other than Google, who has mastered this?

I found it interesting that you would mention Backblaze since, even back then,
I questioned (I think even here on HN) their decision to go with a custom
chassis and the non-standard, low-performance technology of SATA port
multipliers. Well over a year before that blog post, there was enough
competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro,
SuperMicro) that I was using them at a cash-strapped startup instead of used
Dell MD1000s.

What we _don 't_ know, however, is if those backplanes were actually a
_commodity_ yet, at their scale. Maybe Backblaze couldn't actually get enough
cheap SuperMicro chasses soon enough. I do recall some resellers charging what
seemed like comically high markups, but maybe they were banking on a scarcity
I was blissfully unaware of, at my low volume. Still, the fact that the option
is conspiciously absent from their bar chart tends to make me suspect their
summaries and conclusions, even if I always appreciate their raw data (e.g.
disk stats).

To tie this all back to the original topic, sticking with commodity solutions,
be they hardware or cloud, and re-training your staff on them will help them
avoid the potential career dead-end of over-specialization.

~~~
exikyut
> _Even putting aside that the computer tech part of the business isn 't their
> core competency, custom hardware is_ hard.

Hmmm. Good point. (Maybe I scared OP away, heh.)

> _Is there an example of any large company, other than Google, who has
> mastered this?_

Well, I certainly don't know. I've heard Cloudflare gets their systems made,
for some definition of "from scratch", by Quanta. Presumably there's a bit of
a decision-making process in there, and presumably Quanta handles everything
related to the "why is the devkit sample broken again #51383" side of things
and ships a more or less finished product (maybe even with finished
enclosures). My reference point for this knowledge is
[https://www.youtube.com/watch?v=LA-
gNoxSLCE&t=19m31s](https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s)
(2015).

Regarding SAS, maybe past a certain ordering quantity (or anything similar
that would reveal a massive operation) the market would expect certain
behavior like requesting value-adds, ordering with only one supplier, or other
things that would generate lock-in. I of course have absolutely no idea [if
this sort of thinking is reasonable].

Your points about commodity at scale sound quite reasonable. (You clearly have
more experience than I do, which is kind of zero, heh.)

As for suspicion about summaries and conclusions, well, (tangential example)
the radial graph on [https://about.gitlab.com/](https://about.gitlab.com/)
(the one from Forrester) doesn't show GitHub :D

Yeah, building own hardware is sadly still within the realm of over-
specialization at this point. I guess this is because 4+-layer PCB printing is
still eye-watering so the amount of cheap iteration that can be done is
restricted. There are probably similar cost-benefit disparities as well.

I guess, for the sake of discussion, then... what's the best vector into the
middle of "commodity solutions" that scale massively (which this operation
sounds like it needs a bit of)? I mean in terms of rough vendor selection etc,
what to look for, etc. Admittedly this question is based on my own complete
naivete and OP probably has at least some info on this sort of thing available
or accessible.

~~~
mmt
> I've heard Cloudflare gets their systems made, for some definition of "from
> scratch", by Quanta

I looked just glanced at that video, it looked familiar. What I'm reasonably
confident of is that no meaningful definition of "from scratch" applies.
AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just
slightly tweaked (if at all).

Now, it's certainly possible that Quanta offers customization that SM doesn't,
like deleting extra on-board parts, but I'd be surprised if they allow even
something like a choice in BMC/IPMI platform. Certainly nothing as extreme as
exotic power options.

My point being that, if Quanta suddenly decided to jack up the price,
CloudFlare wouldn't be over a barrel due to a custom solution. Worst case,
they'd just have to pay (presumably higher) SuperMicro prices. Best case, they
find a different white-label vendor, because what they're looking for is a
commodity, nothing special, readily available.

> Regarding SAS, maybe past a certain ordering quantity (or anything similar
> that would reveal a massive operation) the market would expect certain
> behavior

That seemed true 2 years prior (2006) when high-port-count SAS expander chips
were mainly available super-expensive standalone devices, presumably targeted
at the "SAN" market. It took a while for manufacturers and then customers
(other than me) to twig to the fact that one could attach a cheap SATA disk to
each SAS port on an expander, thereby making it relevant to the lower-
end/frugal market.

> I guess, for the sake of discussion, then... what's the best vector into the
> middle of "commodity solutions" that scale massively

Before I answer, I'll pick a nit.. I'm not actually convinced that this
business needs truly _massive_ scale.

Especially given the OP's mentioning somewhere in the thread that the greatest
savings in a POC cloud migration were from a _smaller_ VM with auto-scaling.
That suggest that even the current environment is over-sized, and that can
easily get worse as computers get more powerful for the same size/cost.
Remember, it's a traditional F50 business, not a web-scale tech company.

> (which this operation sounds like it needs a bit of)? I mean in terms of
> rough vendor selection etc, what to look for, etc

That said, the job of vendor selection is pretty easy. Since it's commodity,
it's pretty much about price.

At times, that has meant that, with a steep enough discount, you could
consider Dell (and I've heard HP, but I never had such a discount) for
servers, just not for storage. Supermicro seems to have been low-markup all
along, and there are always white-labels, though that may not make sense at
low scale.

Networking gear isn't _quite_ commoditized, but we're close, with open-
source/SDN type platforms.

One thing I've learned is to be open to different vendors for different
commodity parts. Sometimes this means considering actually doing some of ones
own assembly work, such as mounting disks in arrays. The modern knee-jerk
reaction is to reject anything involving that much of the customer's labor [1]
on the grounds that it's not scalable or somehow categorically too expensive,
but that just leads to an insidious form of lock-in (which is, at least,
temporary and limited to the transaction at hand). The integrator may be
giving you a great deal on the server but soaking you on the disks (or maybe
they're getting soaked on the disks and merely passing it along).

[1] Which can be contracted out, though that's fraught with its own problems,
not the least of which are cost inflation and, perhaps more importantly, less
"invemstment" in the quality of the final product. Really, though, I think the
amount of labor is grossly over-estimated, especially working in anything
approaching assembly line fashion.

------
southern_cross
Since you're thinking towards the future, think also to the time when you will
probably have to move an awful lot of your stuff back in-house, which might be
much sooner than you'd expect. Plenty of companies which outsourced in a big
way some years back have already been through a similar cycle.

A former Fortune 500 employer of mine went through the outsourcing boomerang.
Now I understand that they have big plans to move everything to the cloud
within the next two years. Knowing their systems like I do, and knowing that
the CEO who is driving this is fairly noobish (note that I said CEO rather
than CIO), I don't think this plan is going to work out too well for them. But
given that since I left they've had to file bankruptcy, and in doing so
involuntarily dropped to about half of what they were before (employees,
assets, etc.), I guess this is just par for the course for them now.

------
gjkood
You obviously have a good heart for asking the question!

I have a feeling that whatever decisions you finally take would take into
consideration the wellbeing of your employees.

I would assume that the folks who are doing the legacy stuff have the
necessary aptitude to pick up the skills that are needed to automate for the
future. Can their existing data center skills be translated into learning
about AWS/Azure, VMs, Containerization and the necessary scripting and DSLs
around those toolsets?

Obviously there will be reductions in headcount but give them the option to
retool for the future. Even if they are not all retained but if you retrain
them then they will still have options elsewhere with relevant skillsets.

You are finally running a business after all where revenue and profit are the
drivers.

Good luck and I would say that your employees are luckier than those who
consider them as mere resources and not people with lives and loved ones to
take care of.

------
supahfly_remix
Are you sure the savings will be real? I've read that real savings results
from using the cloud for bursts while keeping the base load on-prem. (Never
done this personally so can't speak from experience.)

Once you let those people to you will be losing their knowledge in this area.
Not sure if it matters.

------
friedman23
I really think the worst thing you could do is continue to run the business in
an inefficient manner. Improving technology and as a result efficiency is the
reason we have advanced so much as a species.

However, you don't need to be callous in your pursuit of efficiency. You
should try to give these people an opportunity to update their skills and get
a new position. You might not be in a position to do that however.

------
joewee
There are a lot of benefits to migrating to the cloud, in my experience, cost
isn’t one of them. Unless of course you are mucking around with accounting
tricks by cutting employees and moving that cost to services or cost of goods
sold, I’m not a expert in this, but it works and will save you money because
you can control these expenses more easily and because of how payroll vs
services are accounted for.

You should aim to keep 50% of your workforce. 25% won’t want to change 10-25%
won’t be able to. I would invest in training on automation and multicloud /
hybrid cloud management. I recently worked on a large scale migration and you
will definitely discover that the cloud wont work for a lot of your high
performance computing, or it will be cost prohibitive. You also need to take
into consideration intellectual property, does the company really want all of
its IP on another conglomerates infrastructure? But most importantly you need
to avoid lock-in to one provider at all cost, training your trusted resources
who understand the business requirements on hybrid cloud or multi cloud
implementation will really enable cost savings because you will have leverage
during price negotiations. If you get locked into one provider you are going
to get royally screwed on long term cost, seen it happen.

------
eksemplar
We’re in the process of moving our infrastructure to Azure. Being the public
sector we take things relatively slow, but we’ve already fully embraced Azure
ad as well as office 365, one drive for business, planner and all those other
things.

The next step is our servers, they are already virtual and hosted in our own
cloud, but moving our database cluster to a commercial cloud is still a big
step politically.

Anyway, you retrain your employees. You may need fewer in the long run, but
finding people who are actually professional at stuff like setting up on-
premise security inside Azure hasn’t been easy for us. We’ve had 3 very highly
regarded consultant agencies to help us, and they quite honestly didn’t know
anything but tech-book Azure. More than once we’ve had to send consultants
home because it became obvious they knew less about this stuff than our
people.

Azure has also opened up for a range of automation through services, but
that’s not really automatic either. We’ve had to retrain our people in
powershell, and now we’re running a lot of stuff through the orchestra
service. So right now, we’ve actually not needed fewer people, just better
trained ones.

I image you could replace Azure with Aws or any such service and it be the
same story.

~~~
int_19h
If you don't mind me asking - why Azure specifically?

~~~
eksemplar
We’re the public sector, we operate more than 500 it systems, some of which
won’t be leaving the windows platform.

We run office 365, Skype for business and handle our Security through af and
adfs for 10.000 employees.

So we’re already heavily invested in Microsoft as far as what our IT staff is
trained in, and they prefer Azure to say AWS often for non-technical reasons,
but that sort of motivation is still important from a management point of
view, even if you can run windows in AWS.

That’s one reason. The other factors is that we’re in the EU and Microsoft
were the first to implement our laws, without fighting them. Unlike AWS I can
physically visit our Azure servers, though this requires you to throw a lot of
money at Microsoft. Last but not least, Microsoft is a trusted partner that
we’ve worked with for 25 years, Amazon is not.

There weren’t any other options than AWS and Microsoft, because we’re in the
eu and in the public sector, and AWS wasn’t under the privacy laws at the time
we started.

A lot of our neighbors are choosing AWS though, so it’s certainly viable
today.

------
codingdave
If you are in a position to make decisions for that many people, is it a safe
assumption that you have a fairly robust knowledge of how to enact
organizational change? Because based on my limited knowledge, major change
like this has processes to be worked through which will help answers some key
questions -- what is the current and future state, who will gain/lose, what
will your communication plan and transition plan be, etc.

You have much more knowledge to give those answers than we do. What to tell
employees, and who can/not be saved from losing their job depends on those
answers.

I will say that open communication, with complete answers of what people
should expect, will help the change to happen without losing the trust of
those who do get to stick around. You will have some harsh truths to share. If
I worked for you, I would want to hear them as soon as there are enough
answers to not leave me in limbo on how it affects me, personally. Not sooner,
not later.

~~~
cjalmeida
I would second this comment and add that proper change management is _needed_
to ensure a smooth transition. As soon as the word is out, if you don’t have a
plan and/or don’t communicate in the right way, your people will start sending
out resumes and that can wreck havoc in your business.

As you might expect, replacing top notch techies is quite hard. So managing
the transition and making sure those that won’t be kept are better off is not
only a good heartened approach but important to the success of the project.

Management consultants usually get a bed rep but that’s something they’re
pretty good at. If you’re an F50, you have the budget to hire a top one.

------
lsc
so, uh, I'm not weighing in on the job stuff... but I suggest you check your
numbers on how hosting in the public cloud is cheaper than your own data
centers.

I mean, sure, automating things is great, and running your own hardware cloud-
style is great for a lot of workloads... but that doesn't have a lot to do
with the decision to rent rather than own your servers.

Cloud providers charge a _lot_ compared to owning your own hardware, unless
your load is extremely bursty.

The bandwidth prices are particularly eregrious. They are like what you would
pay for bandwidth 10+ years ago.

I personally ran a small VPS company and managed to turn a (small) profit
while selling compute for rather less than amazon, even though I had to rent
space and power from a fancy pants datacenter by the rack. I was essentially
paying retail, and I only had 20-30Kw of capacity, max, and even at that
scale, owning was cheaper than renting.

I can only imagine that at F50 scale, the economics tilt even further towards
own, as you can much better amortize the labor than I could

(and it's not like the datacenter loses money, either. Renting datacenter
space is like renting commercial realestate. When your lease is up, the
renewal rate goes up to whatever competing space costs + what it costs you to
move. Owning is so much better in that respect.)

~~~
empath75
Amazon etc negotiate prices for large accounts so the price you pay will not
be the same price that a Netflix pays.

~~~
lsc
Sure, and it's possible OP really _is_ saving money by renting their hardware
instead of owning. I don't know the details of their needs or of their costs
or what cloud provider they are moving to or what deal they cut with that
provider.

I'm just saying, there's currently a huge push to rent your essential services
from service providers such as amazon, and from what I've seen, it's pretty
easy to "move to the cloud" and then find that your total cost goes up pretty
dramatically vs. owning hardware.

I mean, yea, "cloud" does make sense sometimes. But... it is sold as being
cheaper, and quite often it is not.

Even if the rates look good up front, you've got the same problem that you
have when leasing commercial realestate from professionals; they are
professionals, which means that when it comes time to renew the lease, the
rate will go up to match what renting something similar would cost from a
competitor, plus slightly less than whatever they think it will cost you to
move. If you can't switch cloud providers and your cloud provider knows it,
you can count on your negotiated rates going up next contract renewal.

~~~
mmt
> it's possible OP really is saving money by renting their hardware instead of
> owning.

Is it, though? Other than the special case of Netflix, it strains credibility
that any cloud provider would be willing to give enough of a discount to
match, let alone beat, ones own hardware. How would they make money?

This can be especially true if one "unfairly" compares a cloud provider's
relatively expensive hardware choices to ones own cheaper choices. For
example, AWS with dual 10GE motherboards and CPUs that weren't the best
performance-per-dollar when they bought them, if all you need is single
gigabit (and don't need particularly high single-core CPU speed).

> the rate will go up to match what renting something similar would cost from
> a competitor, plus slightly less than whatever they think it will cost you
> to move.

Fortunately, there's an upper bound, which is the public price, which has, at
least historically, been coming down. Also, AWS has the spot instance market,
which can provide downward price pressure from even that ceiling.

~~~
lsc
>Is it, though? Other than the special case of Netflix, it strains credibility
that any cloud provider would be willing to give enough of a discount to
match, let alone beat, ones own hardware. How would they make money?

Right now, as I understand it, Uber is losing money driving _me_ to work every
day, and I'm not really sure how they will ever make a profit off of that. if
they double their prices? I'm gonna buy a car, drive my own self to work, and
use uber once a week when I go to the bar. I don't really see how buying fake
market share (rather,market share that only exists when you are dumping) helps
them when it comes time to not dump anymore.

Point being, there are plenty of people who are willing to sell two dollar
bills for a dollar when the economy is hot. (I'm not suggesting amazon does
this... but they certainly could. It's possible. )

Even in more rational markets, loss leaders are a big thing, and in leasing
situations, where it's hard to move once you are in? loss leaders are a really
common sales tactic. How often have you gotten "first month free" rent deals
on one year leases, that work out to below market for the first year (because
of the free first month) but above market after that, even if they don't
actually raise the rent?

Datacenter leases often have a similar 'ramp up' period, where you pay less at
the beginning of a contract than towards the end, even not counting the fact
that they jack up your bill by however much they think it will cost you to
leave.

I mean, these all can be really good deals if you know that it's easy for you
to move later,and it's certainly possible (though maybe not easy) to setup
your infrastructure so you can switch cloud providers with little cost.

>Fortunately, there's an upper bound, which is the public price

This is true...but that's a pretty high upper bound, when you compare to
owning hardware/leasing pipe and power at all but the smallest scale.

>which has, at least historically, been coming down.

Historically, the cost of compute has come down, though slower than the cost
of hardware (at least until lately) the cost of bandwidth has come down...
much more slowly.

>Also, AWS has the spot instance market, which can provide downward price
pressure from even that ceiling.

I think spot instances are a difference in kind (and actually a really great
deal, if you have a workload that can be interrupted like that) - but most
workloads I have worked with, especially in corp, don't really work that way.

~~~
mmt
I think most of us here understand the concept of loss leaders and of
sacrificing profits (even operating at a loss) for market share.

My point isn't that it doesn't happen, nor that it _couldn 't_ happen in this
space. Rather, my point is, why would they bother (be willing)?

> there are plenty of people who are willing to sell two dollar bills for a
> dollar when the economy is hot. (I'm not suggesting amazon does this... but
> they certainly could. It's possible. )

In fact, we pretty much know, because Amazon has revealed it, that AWS is
making money hand-over-fist, and they do this while having sizeable (existing)
market share.

Would the other providers compete on (non-public) price just to gain (new)
market share? It's _possible_ , but I maintain it's an extraordinary
proposition, considering Microsoft's overall software ecosystem integration
advantage with Azure (much better lock-in) and Google's various attempts at
features, including AI hardware.

Of course, this is all fairly academic, because..

> This is true...but that's a pretty high upper bound, when you compare to
> owning hardware/leasing pipe and power at all but the smallest scale.

The OP admitted that the cost savings is based on public pricing, so the
decision makers don't actually care about the hugely inflated cost of cloud
infrastructure! (Or, as in my personal experience, they refuse to believe it,
which makes job hunting particularly surreal). That further supports my belief
that the providers have no motivation to negotiate price.

> Historically, the cost of compute has come down, though slower than the cost
> of hardware (at least until lately)

(Did you mean the parenthetical to apply to this or to the statement about
bandwidth?)

Is there anywhere that tracks this? I've only ever done that-point-in-time
pricing analysis, never historical tracking. I tend to wonder how much of the
difference is due to Amazon's specific choice of hardware for each generation.
That is, would the trend look different if you limited the comparison to the
cost of hardware with only the same specs as theirs (which are, of course, not
necessarily, or even ever, the best bang-for-the-buck)?

> I think spot instances are a difference in kind (and actually a really great
> deal, if you have a workload that can be interrupted like that) - but most
> workloads I have worked with, especially in corp, don't really work that
> way.

Agreed, and they require additional engineering work (and forethought) to take
advantage of properly. I just like to mention them to avoid any cries of "but
cloud doesn't _have_ to cost full price" when venturing into the territory of
how expensive it is. (Also, with AWS, previous generation, and reserved
instances, though knowled of the latter is implied with the spot market).

~~~
lsc
>Would the other providers compete on (non-public) price just to gain (new)
market share? It's possible, but I maintain it's an extraordinary proposition,
considering Microsoft's overall software ecosystem integration advantage with
Azure (much better lock-in) and Google's various attempts at features,
including AI hardware.

The better their lock-in, the more sense it makes for them to lose money at
first, if that's what it takes to get you into bondage. (assuming they have a
way of raising the prices later, or of dropping their costs later) I'm just
saying that it would totally make sense to do this if it got you more
business. Maybe they only do it for especially juicy customers? I mean, yeah,
I'm reaching for some way that it might conceivably be cheaper to rent servers
when you have already done the work of setting up your own datacenter, and
it's hard.

(now, I do understand that most in house datacenters have ridiculous policies
that managers would be happy to pay 5x to get around. But if you are at f50
scale, you can setup your own "internal cloud" with amazon like policy
affordances.)

In the aughts, the standard thing to do when starting a VPS provider was to
come in as cheap as you possibly could, get as many customers as you could,
and then just leave your prices the same as the hardware (and thus power)
costs fell out from under you. I mean, I don't know anyone who lost money on
their marginal customer, but plenty started out with margins that wouldn't
have been any good long-term.

That said, I don't know of any cloud infrastructure providers who seem to be
losing money on their marginal customer, and as you point out, there's very
good evidence that AWS makes a killing. My impression is that it's an industry
with pretty good margins, once you scale enough to amortize your labor costs,
which is harder than it looks. I mean, in those days, hardware(and thus power)
costs were dropping dramatically, labor costs were not, and even if you have a
scalable system where one person can handle a lot of servers, you need a
minimum size team, which gets expensive fast.

>(Did you mean the parenthetical to apply to this or to the statement about
bandwidth?)

the recent change is that hardware has gotten... more expensive in the last
little bit. not just ddr4 (but mostly ddr4... and that's most of your cost, a
lot of the time, as that's the thing you least want to oversubscribe) To me?
this is shocking. Honestly, I haven't really been keeping up with bandwidth
prices since I sold my thing, so I wouldn't know as much about that side of
things.

>Is there anywhere that tracks this? I've only ever done that-point-in-time
pricing analysis, never historical tracking.

pcpartpicker has nice historical graphs for hardware pricing that are publicly
accessible (but that don't go back so far) One of my super early projects was
a web price scraper for books... I wish I had kept up with that project and
expanded it to computer parts, because there's a lot of data that I don't know
how to get anymore.

[https://pcpartpicker.com/trends/price/memory/](https://pcpartpicker.com/trends/price/memory/)

>I tend to wonder how much of the difference is due to Amazon's specific
choice of hardware for each generation. That is, would the trend look
different if you limited the comparison to the cost of hardware with only the
same specs as theirs (which are, of course, not necessarily, or even ever, the
best bang-for-the-buck)?

From what I've seen, the x86 server market is pretty efficient. Yes, yes, the
professionals will take you if you let them, but it's pretty easy to get to
within 10% of the tray price even on small purchases, and it's hard to get
much below tray prices, even on large purchases - (dramxchange is a pretty
good resource if you want to know about 'tray price' for ram) I mean, x86
hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses
the same chipsets, and most of the cost is the cpu/ram, so there's really not
a lot of difference in who you buy from, as long as you get quotes from more
than one vendor.

The times I've seen people get really ripped off, they got sold different
types of hardware; like one place I worked was getting sold blade servers
using FBDIMMs. Blade servers for "efficency" \- of course, there was only
enough power for one blade chassis per rack. And FBDIMMs used like 3x the
power of low voltage reg. ecc ddr2. I pointed out the existence of the low
power xeons that took reg. ecc ddr2 rather than FBDIMMs, and ran enough
benchmarks to show that they were about as good as the regular xeons of
similar speed... and used a lot less power. I'm not aware of any similar traps
in the market right now, but I am a little out of the loop.

My personal theory as to the lack of cost concern in "cloud" is that times are
hot. When times are hot? you jump. make hay while the sun is shining. If
spinning up cloud servers is faster than developing your datacenter
infrastructure? who cares if it costs 5x as much. Setting up openstack is a
huge pain in the ass, and you're going to have growing pains for the next six
months, if your team is really good.

My personal theory is that this trend will reverse with the next downturn.
Meanwhile, I'm working for other people, which is pretty nice in a lot of
ways, even though it does feel sad that all the time I spent in managing x86
hardware is something people just don't care about anymore.

~~~
mmt
> nice historical graphs for hardware pricing that are publicly accessible

Even then, that's only half the comparison, though. Do you know of anyone
keeping historical AWS prices? That would seem more likely, actually, than a
reliable pc-parts database.

> I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it
> mostly even uses the same chipsets, and most of the cost is the cpu/ram, so
> there's really not a lot of difference in who you buy from, as long as you
> get quotes from more than one vendor.

Noooo.. no you too! :)

I find that this is true, but only if the workload is primarily CPU- _and_
RAM-bound. My sense is that somewhere between the dot-com boom and now (500x
increase in speed?) CPUs got fast enough, and software got parallel enough,
that the challenge in scaling shifted CPU capacity to I/O. (More RAM always
seems to help, but I agree there's not much differentiation there).

As such, at any given point in time, each vendor's offering at a particular
(otherwise competitive) price point might not make sense. The mobo might not
have enough PCIe slots with enough lanes (or too many on too few slots). The
onboard 10GE might use a chipset that performs poorly under OpenBSD. The
chassis might hold comically few disks (ahem, Dell) or have no SAS expander
backplane option.

The other differentiator, is at the high end, for >2S systems. It's arguably
no longer "commodity" hardware, but the parts are more similar than different,
and there's no lock-in if one can wait at least the full release cycle. Still,
it's rare that anyone is brave enough to buck the trend and pay the 2x-4x
price premium to scale "up" before embarking on the engineering nightmare that
is scaling "out" (which I'm not convinced has any less of a hardware price
premium, just that it can grow without bound).

> Blade servers for "efficency"

I've never understood that mass delusion, either, though at least not there do
exist high-power-density datacenters that can house them. I assume someone
managed to spread the myth that it's _space_ that's expensive at a datacenter,
rather than power and cooling.

> I'm not aware of any similar traps in the market right now

I'm not, either, at this very moment, but every time I've looked, out of
curiosity, there's something like that. All the vendors will have at least one
product that seems objectively worse in every way to the older products,
including being more proprietary, that I wonder if it isn't some custom design
that flopped that they're trying to dump.

>My personal theory as to the lack of cost concern in "cloud" is that times
are hot.

Agreed. Easy money, be it from VCs, low interest rates, and/or a booming
economy (perhaps fueled by the first two) helps explain it.

> Setting up openstack is a huge pain in the ass,

Is that really the go-to option? I'd think there are much lighter-weight
alternatives. I'd have thought provisioning the base infrastructure were much
less a PITA than writing the "infrastructure as code" which one has to do even
for public cloud.

> growing pains for the next six months,

Is it really that long? Or is that just the perception?

> if your team is really good.

Maybe that's the problem. Good help is hard to find?

Still, what about 1-2 years down the line of making hay at those 5x costs?
There are plenty of companies like that and, at some point, it has to at least
be slightly tempting to go through the effort, including hiring. Maybe the
money really is too easy.

> My personal theory is that this trend will reverse with the next downturn.

I've been thinking the same thing, but I'm wondering what will happen with all
that excess capacity that cloud providers are suddenly left holding.

> Meanwhile, I'm working for other people, which is pretty nice in a lot of
> ways, even though it does feel sad that all the time I spent in managing x86
> hardware is something people just don't care about anymore.

I've always worked for other people (which also makes me worry about too
strong a downturn), and the hardware management has only been a _part_ of what
I do.

I'm not so much sad at the apathy toward one of my skills. It's more that I'm
so limited in how much performance I can squeeze out of the infrastructure or
how low I can keep the cost. It makes for weird tension.

------
segmondy
Just do it. Most old school sysadmins want to write their Perl and sh scripts
not terraform and cloud formation scripts.

I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and
SRE before most of the admins. Most of them left to go safe companies where
they can stick to their old ways.

~~~
gaius
_I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops
and SRE before most of the admins._

Yes, devs are often enthusiastic because they think they can ditch those
boring old operations people who always stop them having fun.

Next thing they know they're on call 24/7 and don't get to write application
code anymore, just operations stuff...

The truth is the need for ops people doesn't evaporate when you go cloud, be
wary of anyone who says it does. This is the same dynamic that played out with
MongoDB.

~~~
dev_dull
The new reality is that the barriers are gone. Ops must dev, dev must ops.

Who can get away with a job that just requires them to open a Java IDE? Those
jobs are just as extinct as a pure ops role.

~~~
gaius
With MongoDB the devs said "we don't need a DBA anymore". But what they failed
to understand was that the DBA was _not_ the person who knew all the database
command syntax, but rather the person who was responsible for the data - its
availability, security, and so on. And getting rid of that person didn't get
rid of the reason the job existed. I dare say that came as a bit of a shock.
Well, serves them right for stabbing a fellow Worker in the back! And it is
playing out exactly the same way in Cloud. All the responsibilities still
exist, apart from the most basic ones such as "sweep the floor of the
datacentre". Now who's going to take them?

------
zer00eyz
Cloud/Comerical <> automation.

I have been burned in the past by "commercial", thankfully I wasn't the guy
who had to go to the board hat in hand and say "this isn't in our control"...

Every comercial provider is going to do everything in their power to "lock you
in" they are going to put up features that are "better enough" that you
migrate to them and then they own you.

Automate, reduce your workforce as needed (it will be less than your plan) and
then look at the commercial provider as a second step, look at them in
relation to a real DR plan. You might not like what you see for day to day
operations but having them on stand by in the even of an emergency could make
you a hero and present a massive cost savings already.

------
georgebarnett
I have some experience in this area. My thoughts:

Retraining can be done as part of the project, but you need to figure out what
the project and sub projects looks like first. Then you an know roughly how
you’ll move people around at various phases. It takes time to build new
systems and time to train. Line those up and you can take advantage of them.

Be careful with costs. They will blow out if you aren’t mindful.

It’s going to be messy for a LONG time, which causes a huge amount of pain in
the org. Be ready for this.

Moving to cloud brings a whole new set of problems you haven’t thought of yet.
For example, cleanup becomes more critical (old resources cost money, but
can’t be “seen” because they’re not physical and just a hidden line item
somewhere).

There’s more. My email is in my profile if you wish to reach out.

~~~
nasmorn
Cleanup is even a problem in smallish startups with some dozens off devs. I
cannot imagine the amount of security groups a F50 must have after some years

------
synesso
This happened to my friend in his 50s. He was a storage specialist working for
a bank. Since they outsourced to IBM a few years ago he's been stacking
shelves in a supermarket.

~~~
markbnj
If you let yourself get that specialized then you're always going to be one or
two innovations away from a layoff. I kind of feel that anyone who inhabits a
niche like that for a long time is going to be in danger of becoming a victim
of their own comfort seeking. Our industry is not kind to those who don't
continually challenge themselves and learn new things. As a counter example to
your friend's situation, I'll be turning 58 this november, and I'm an
SRE/devops engineer working on kubernetes-based infrastructure in google
cloud. Five or six years ago I was a back end .NET guy, a few years before
that I was doing ASP web apps. Before that C++ and COM/DCOM stuff, and before
that Orbix/CORBA, and before that... you get the picture. The old saw is that
nothing stays the same, so if you're not moving with the world you can be sure
it's moving past you.

------
kriss9
You're taking the marketing approach as opposed to coming up with real
solutions -- think critically about the business need, investigate the cost
savings and make a transition plan for those who actually add value to the
organization.

1) Consider if your organization has a future strategic value for having the
org capability in house (is this datacenter simply a necessary evil or does
your organization deploy new technologies to new locations for any part of its
business)?

If so, you may be outsourcing a core part of your organization's capability
(consider it). If not (e.g. these are all simply HR and other general business
systems [ or services which do not depend on anything physical], you should
proceed to evaluating costs

2) By the thought that you would outsource the majority of jobs means that you
probably shouldn't have had the employees in the first place (in the Fortune
50 I know, we mostly used contractors with a quarter dozen employees). Hence I
would evaluate if there are other business units which might benefit from
their skill (power related, analyst related et al) and offer training per the
previous notes.

A good decision to be a part of and my hope is that your answers here inspire
the right questions (to allow for the right transition plan for your
organization and company as a whole).

------
ezequiel-garzon
Out of curiosity, what does F50 refer to here? Google doesn’t help, nor does
[https://en.m.wikipedia.org/wiki/F50](https://en.m.wikipedia.org/wiki/F50) .

~~~
dankohn1
It refers to the 50 biggest companies in the
[https://en.wikipedia.org/wiki/Fortune_500](https://en.wikipedia.org/wiki/Fortune_500)

------
downrightmike
Good luck being at someone else's mercy when the problems that are being held
together by duct tape come to a head.

~~~
omeid2
At the same time, I rather negotiate with a service provider than employees
who have implemented their position into the system.

------
kylec
Since it sounds like you're going to save a bunch of money, I suggest letting
them go and using that money to provide very generous severance packages
(months of salary at least). Then they can choose to retrain themselves, take
some time off, whatever. It works for Netflix.

------
ultrasaurus
Kudos for keeping your sights on helping your team! I'm a big fan of holding
on to a group of people that understands your business (I'd rather retrain a
technology than instill passion and domain experience).

Not 100% relevant (and not my better writing) but here are some of the things
I've seen companies do in terms of moving people around from NOCs that might
give you some ideas [https://www.pagerduty.com/blog/future-of-
noc/](https://www.pagerduty.com/blog/future-of-noc/)

If you'd like a soundingboard to brainstorm with feel free to ping me at
dave@euri.ca

------
noobermin
Hate to do this, but be mindful people bearish on the benefits of moving in
the first place will be downvoted crazily, potentially by people who have an
economic incentive to convince you this is a good idea.

------
suff
They need cloud certifications. They should see this as a fabulous opportunity
to get paid to do cloud training, hands-on work. If you become a cloud
architect or a cloud security expert, whether it is at your company or
somewhere else, there will always be great opportunity. If they refuse to
learn to serve a changing market, that is where they will run into problems.
You can only ride a single skill for so long. If you stayed in the horse-shoe
business after about 1904, life was not great.

~~~
dankohn1
I'd recommend CNCF's training for Kubernetes:
[https://www.cncf.io/certification/training/](https://www.cncf.io/certification/training/)

We offer a free edX course and then a paid one that prepares you for the
Certified Kubernetes Administrator exam.

Disclosure: I am executive director of the CNCF and helped create these
courses.

P.S. No job will be around forever. You are not doing you or them any favors
by trying to maintain them in inefficient roles. But you can invest the
resources to bring them with you to the new world.

------
jroseattle
Your employees need skills that are useful and in demand. They need those
going forward, whether you close your data centers or not.

I'd suggest you be upfront with them. Use this as a pro-active way to show you
want them around, but it's not business-as-usual. The good ones will see this
as an opportunity for the long term, while others may view this as a time to
move on.

To reinforce plans, show them what skills you need after the move. This will
give a north star to motivate those who want career mobility.

------
j45
The implementation of hybrid cloud support is the key thing to consider
pursuing both from a strategic and technical perspective. I have provided
complex hosting for large corporations for more years than I like to remember
and it is the only thing of lasting and permanent value for a few reasons:

Having a completely migratable cloud setup, between any cloud provider, or
back to your own environment (OpenStack, etc), is the single most important
long term investment to consider for your stack.

Your situation sounds very advantageous in having the expertise ready to
containerize machines, and dockerize applications that may not be. This too,
is a very marketable skill for people who may be leaving.

Historically, connectivity has driven location of data. Popularity has
oscillated between on-premise vs client-server setups (cloud). Currently, the
cloud is in vogue, for many great reasons. The pendulum though seems to have
historically swung both ways.

While network speed will continue to increase, and computing power will
continue to increase, other factors may play into why a corporation wishes to
keep it's data in-house, be it political, security, industry practices, or a
sudden wish to change directions due to a failure too many of a cloud
provider.

The cloud providers do a very good job of making it easy to get sucked into
their world.

It would be interesting if you followed up here with the decision you made and
why. Not enough thought seems to be out in the open about these types of
decisions, let alone at your decisions.

------
true_tuna
This sounds like a very exciting time, the work is changing, the very
economics of the business are changing. Remember to lead throughout this
transition. Communicate well, frame the transition in a positive way and take
care of people. Do not allow your exuberance at the potential cost savings
blind you to the pitfalls in a project of this magnitude. The transition will
be harder than you think, take longer and cost more. That being said, it’s
probably worth doing. Along the way you must remember that your employees are
a super important part of your business so treat them accordingly.

Offer training. The automation work is more gratifying and valuable anyway.
You’ll be offering them a step forward. Some will take it, some will move on.
Even those who move on may find that the change catapults them forward as
well.

I trained a colleague in cloud automation in two months. Admittedly he is very
capable and committed, but you’ll find that in some of your people too. He’s
now worth more than anyone I could have hired. You have the opportunity to
elevate your existing employees significantly. Embrace that opportunity.

PM me if you’d like to see the training materials I’ve put together.

~~~
throwawayacct4q
I would love to see those training materials! I don't see any contact info,
but maybe you could shoot me an email to the email in my profile?

------
jhowell
If your business has competitors, and they move to the cloud, conceivably they
will be able to lower prices, and/or provide more service/features because of
cloud scale. In most instances, the less your competitors charge, the more
customers flock to your competitors. It is not personal, but economic in my
opinion.

~~~
throwawayacct4q
Our business is an old, traditional business that happens to have lots of data
centers for legacy reasons. Even though we spend $BBB on data centers, they're
not the core of the business and a very small part of overall revenue.
Realistically, even if we didn't move from data centers we would be good as
long as the core non-tech business functioned.

~~~
jhowell
As an economic principal what you suggest is not the best use of inventor
assets and compounds when you factor in employee efficiency/return per
employee. If you invested your hard earned money into this business you would
want cost efficiencies and high productivity. Company executives have a
fiduciary responsibility to do so in the US. If you're running a business and
are not seeking cost efficiency but instead are more interested in employee
satisfaction, likely you should be in HR and let the CEO handle day to day
operations and the bottom line, so you can contribute to the company by
driving employee retention, satisfaction or some metric that reflects
positively on the bottom line. It's OK to the a CEO. It's OK to be in HR. It's
not OK to be a CEO in HR. It's not OK to be a HR as CEO.

~~~
throwawayacct4q
Point well taken. But as many have mentioned in this thread, it is very hard
to hire good people and the people I have already have deep knowledge of the
systems and business and are good at what they do. Just from that efficiency
perspective it makes sense to retrain per fiduciary responsibility. The fact
that I can keep lots and lots of people working instead of terminating is an
awesome bonus that I feel great about.

------
apapli
I imagine it is going to take you a LONG time to do this. Some apps will
migrate easily, others not so much. If you want to look after your people
build a program of work in place that helps you realise immediate benefit
while giving you the change to reskill people as you migrate to newer tech.

------
rshlo
Hire for attitude, Train for skills - If you've good dedicated people that you
can trust, I'm sure they can be trained for other positions. Good people are
always hard to come by, and they will be even more loyal if they will see that
the company doesn't ditch them.

------
sdfin
Your competency is eventually going to do the same and save money and have
advantages over you if you don't take advantage of the awesome cost savings.
Then you may cease being competitive and would have to close the data centers
anyway.

As others said, you could retrain them.

------
tyingq
Retrain them in AWS, devops, or something that has similar need, versus hiring
from outside.

~~~
throwawayacct4q
Best way to retrain without having to pay AWS a huge fee for all the certs?

~~~
alonmower
Negotiate a discount for the certs as part of the large contract you’ll be
signing with AWS

------
jonex
While automation is removing some jobs, we are far from a fully automated
society. I obviously don't know about the specific job markets where your data
centers are, but in general, the ones having the hardest time finding a job
aren't the ones with years of work experience in running data centers.

They might have to retrain and if they live in a small town they might have to
move/commute to find something new however. This can be hard enough for some
people that they won't be happy about it.

In the end, I think it's hard to argue that we should keep being inefficient
just to save jobs. History haven't been nice to that kind of companies.

------
mathattack
First path is to retrain them. Getting more cost efficient can coincide with
overall corporate growth.

Some people prefer to remain in their specialty or struggle to make the leap.
In that case if they have tech skills, someone will take them. It may require
a move, but they will land. It's one thing if you're firing people in the dark
days of a deep recession. (Folks let go in 2001 had to leave California) It's
another thing if you're letting people go into a market of 4% unemployment. If
they can't get jobs with similar pay, then you were overpaying them.

------
stretchwithme
Tech jobs exist because of automation and they eventually go away because
automation. But every dollar saved in salary is available to spend on hiring
people to do other jobs. It works out in the long run, but that doesn't make
it easy on those who have to transition.

I think it would be useful if employers eliminating jobs could have sort of a
reverse auction where other employers offer comparable jobs in exchange for a
portion of the savings. And the workers could either take their severance or
accept one of those offers with no interruption in their benefits or work.

------
sjg007
I would build your own cloud. You have customers. Don’t give them away.

------
jwildeboer
My rule of thumb: invest 30% of the cost savings into training your people on
skills you need now but also with a view to future plans. Invest in preparing
your people for new ways like DevOps, microservices. With their experience and
knowledge they can save you even more looking forward.

And even if you don’t need them all, by training them you give them a better
chance when finding new jobs. A small investment can go a long way. Also helps
your companies reputation when people don’t simply get removed.

------
Mc91
You should do the move. Sysadmin is now a niche discipline, people work in
devops now. The tech economy is good and their skills are getting more
obsolete.

Just start slowly migrating things, and train them in devops skills. Then,
even if you have a reduction in a year, they will have a year worth of modern
skills in EC2 devops and the like.

It's going to happen, so better it happens now than in the future.

------
gboudrias
I've never managed more people than myself, BUT don't underestimate how
difficult it is to find people in IT. It's really, really hard, and even
harder to find people willing to stay longer than 2 years.

Loyal tech workers are invaluable right now. But then again I'm not the one
who has to make that choice.

------
panic
Maybe this is unrealistic somehow, but what about paying the difference in
cost directly to the affected employees?

------
srah763
You can reach out privately to other companies and leaders you know if they
may have a position for them. HR can also help find positions for people,
that’s part of their job.

What you can’t do make a decision that’s good for a very small number of
people at the expense of everyone else in the company if you’re a leader.

------
notatoad
Your people shouldn't be doing manual config regardless of if you're on-prem
or cloud. If they are, you're going to have a tough transition to cloud, and
you're still going to need a lot more of those people than you think you will.

------
xupybd
Can you last long term if you don’t move? If there are huge savings to be made
others in the same space might jump on the opportunity.

It’s a horrible thing to have to let people go but the alternative may be far
worse.

------
exikyut
Saying first what I want you to see the most: please add a throwaway email to
your bio profile (and maybe add a new comment with it so more see it). I
guarantee people will want to say hi privately - not [just] vendors but random
people who have on-the-ground advice.

\--

Hopefully you still see this even though it's been a few hours.

Glad I went through the thread: there was quite a bit of extra narrative and
detail in your comment replies.

> _Our business is an old, traditional business that happens to have lots of
> data centers for legacy reasons. Even though we spend $BBB on data centers,
> they 're not the core of the business and a very small part of overall
> revenue. Realistically, even if we didn't move from data centers we would be
> good as long as the core non-tech business functioned._

> _Thousands [of engineers], many close to retirement age. Think a company
> like UPS /Fedex that isn't at its core a tech company but with tech
> datacenters for logistics and billions in revenue._

> _[T]he cost savings hinge on the massive cost savings from moving to a
> commercial cloud, laying off workforce would save more, but would prefer to
> save a bunch by moving to cloud and keep the workforce for other things._

\--

Ok, here's my take.

I wonder what your [job] position is, heh. Maybe someone with a CTO's ear, or
maybe you're an unusually empathetic CTO.

In any case you're clearly at the point where you've had the internal
discussions, the trigger has been assembled and is ready to pull, and you
and/or the team (committee?) has probably already been groomed by all the
gigantic providers' sales teams, which of course pull out all the stops in
your case. If I understand these things correctly, if you haven't yet had an
amazing lunch or two on one or more vendors' dimes, you will :) (or at least
the high-level people will, would be nice if you get included).

Sales is important. But always remember, particularly for large potential
accounts like yours, the sales numbers will always be what the high-level
execs want them to be, and the sales teams will be spinning the story the
execs want to hear. The on-the-ground experience is always going to be rough.
You've probably experienced similar transitions, and that's what's informing
your skepticism some of your team won't have jobs. Nice you're giving your
people some thought.

On the one hand, it doesn't really matter who you sign up with, as vendor
ceases to matter somewhat short of the $B mark.

On the other hand, try to mitigate the risk of discovering that a vendor can't
provide a given feature at some eleventh hour, or you may end up in a confused
tangle of AWS and GCE deployments (because the other major vendor will of
course have the missing feature, and the fragmentation will begin).

How I'd achieve this: every time you go "ok, so this is our implementation
blueprint", start the rounds of internal technical enumeration over again.
Spend time with the engineers to understand pain points. Talk to as many
people on the ground as possible, especially the forgotten people in the
corners. One-on-ones, or small groups of people who are known to gel well with
each other (no impedance mismatches, work well together), could be good for
extricating annoying gotchas. This could look like devs aimlessly paging
through source code and saying what they think (this could catch tiny things
well), or it might be more structured with general architectural discussion.

Do NOT have the cloud provider vendors do this work (of talking to your
existing team); I wouldn't be surprised if they offer to do it, but if they
do, all their feedback ultimately goes to the sales guys (via an internal
engineer digestion process), and not directly to you.

Well, the analysis team will probably generate a high-level report that
contains all the information they've discovered (this is probably legally
mandated for full disclosure about what they now know), but all the on-the-
ground details about inefficiencies, vendor-specific pain points and so forth
will only be derived by vendor-internal engineering debriefs, and you'll
naturally never get any of this particularly valuable information -
unfortunately this approach is within the vendors' business focus, as
inefficiency means more compute resources used ($), and pain points mean more
vendor-infrastructure maintenance ($). You could respin this statement to say
that the vendor who makes the most attractive offer is the one who knows their
system is the worst fit for your system, but I don't know :)

Instead, if you don't have the internal resources (or freshness/objectivity)
to put together a bird's-eye view, get an outside consultancy to help,
directing them to investigate your current architecture top-to-bottom-and-
back-to-top-again, and advise on ways the system can be rearchitected for
efficiency and optimization within modern cloud infrastructure. (They'd then
be the ones doing the one-on-ones and so forth I mentioned above.)

Another thing that may be extremely useful would be to take the "we're moving
to the cloud traction" and carefully smudge out (some of) the dividing line
between "we're overhauling everything to migrate it and make it work most
efficiently within $vendor" and "we're rearchitecting major components so they
work more efficiently overall" so the line between these two sentiments is
blurred, and both things wind up happening in lockstep. Obviously the two
ideas are intrinsically distinct and this can only be taken so far - "rewrite
everything from scratch to get it running in the cloud" wouldn't pass _any_
accounting costing or time-boxing :) - but you can probably push this far
enough to extract some usefulness from it (eg, you probably have an idea how
long upper level will be willing to wait for this transition to finish - 2
years? 1 year? 6 months? - and you might be able to squeeze some interesting
and useful tidy-up operations into that window).

Blurring the lines between these two things will play to your existing devs'
strengths, as rewriting/restructuring is easier on the brain than the "replace
engine in flight" of picking up all the existing state, punting it into to the
cloud ad-hoc, and running after all the resulting fires that get created and
scrambling to get it all working perfectly.

It won't be a perfect transition if you're really a $BBB operation, and
restructuring will be the easiest way to cohesively get the existing devs into
a flexible frame of mind and ready to deal with whatever curveballs get thrown
their way.

This is one way you could efficiently achieve keeping a lot of your existing
team.

Plus, restructuring means everything can slowly be built up onto the new
system; picking everything up at once and expecting it to land on its feet
probably won't go very well.

Depending on how things go, if you bring in new techs to assist with
understanding modern best practice/hitting the ground running, I cannot
recommend enough pitting them against your current team when doing
onboarding/interviewing, as they'd be an EXCELLENT catalyst for judging
candidates' character.

These are a couple of lateral ways the current workforce could be creatively
employed during this time. I reckon careful management would see everyone kept
on until retirement.

The current team knows all the systems inside out. Letting them go would make
everything a thousand times worse.

Also, another reason I thought it would be cool to put an email is because
your post makes for a perhaps-unwitting but nonetheless effective and
interesting recruitment drive; I'm aware of very few (okay, zero) high-level
people at equally high-level companies who really care about their employees.
In my case this is because I don't have any connections, but the same probably
applies to many others.

I don't know why
[https://news.ycombinator.com/item?id=16283252](https://news.ycombinator.com/item?id=16283252)
stood out to me a few months ago, but your post resonates similarly. Probably
because it incorporates a similar human response too. (Note that I have no
association with the potential companies listed and have never talked to any
of the users in that thread, I just thought it was interesting)

~~~
throwawayacct4q
These are really helpful insights. Thank you. Position wise yes, high level
newly-minted CTO-ish-type office sounds about right, sorry to be vague.
Decision is made, implementation and vendor selection is the hard part. I have
already moved many applications as proof of concept and the savings seen is
one driver of this change, and me being the one that moved them is one reason
I get to be involved/doing/advising/engineering this. We are not FAANG and not
located in cool locales so harder to attract great talent.

The cloud vendors are in fact already offering to take on a lot of this work
of transitioning, which I am avoiding to avoid lock in at all costs.

And for others reading it's not UPS/Fedex just to throw that out there, I saw
another thread on UPS yesterday trying to move huge organization's ops to 21st
century and was trying to think of examples of what I assumed were large
legacy companies that are not core tech and those popped to mind. That thread
also served as a catalyst as I'm also wrestling with moving a huge orgs ops to
the 21st century.

I have taken your advice on email: throwawayacct4q@gmail.com

~~~
mmt
> Decision is made, implementation and vendor selection is the hard part. >The
> cloud vendors are in fact already offering to take on a lot of this work of
> transitioning, which I am avoiding to avoid lock in at all costs.

This gives you a distinct advantage, in that you could, subsequently, move
_back_ and be your own "private cloud" vendor for even further cost savings.

> We are not FAANG and not located in cool locales so harder to attract great
> talent.

You may find that becomes less true as housing prices in cool locales continue
to rise. Alternatively, you could open offices in those locales, since, if
your infrastructure is going to be cloud-based (public, private, or hybrid),
it's less important that the staff be co-located with it.

There's also the notion of a remote-first engineering culture that I'm sure
you can find plenty of evangelism for, just by searching.

------
Havoc
If you don't do the move someone else will eventually given the monetary
incentive. This isn't something that can be stopped.

Better that it happens under the watch of someone that clearly cares

------
empath75
You’re not going to be able to just flip a switch and do that migration
instantly. You have time to retrain people and some people will adapt better
than others.

------
gsich
Why not automate your stuff at your current datacenter?

------
lmeyerov
Embracing automation and moving to cloud systems means layoffs for a
dying/stagnating company and an opportunity engine for an innovating one. Our
startup does GPU visual graph analytics for boosting visibility, and we see a
world of a difference in velocity etc when we engage with a company that has
cloud data lakes etc than one that doesn't. And, that includes F50s and
federal agencies: super normal!

Technology employees should embrace new tools; that's part of the job that
ensures compounding effects over time.

------
gus_massa
How many are them? (10? 100?)

Are some of them close to retirement age?

~~~
throwawayacct4q
Thousands, many close to retirement age. Think a company like UPS/Fedex that
isn't at its core a tech company but with tech datacenters for logistics and
billions in revenue.

~~~
russtrpkovski
Thousands? I really find that hard to believe.

~~~
russtrpkovski
Im not sure why I was down-voted. Even if you close data centers and move to
the cloud, you still need personnel to manage the infrastructure.

------
late2part
What type of commercial providers are you saving money by moving to?

------
mesozoic
Progress

------
buahahaha
Just retrain them.

They're going to have tangential skills to the new platform.

Get your network team to learn all about VPCs and networks and security groups
and peering and VPNs, train them to be your in-house experts to deal with
changes and expansions and to help trouble-shoot issues.

Chances are you're still going to have systems, your systems engineers can be
repurposed to better tie in your monitoring tools, your security tools, etc.
You're going to need authentication and identity management, and probably have
internal people handling it.

If they're motivated to keep their jobs, all it should cost you is a
$5-20K/head in training.

That's peanuts compared to the turmoil of everyone thinking they're going to
lose their jobs, or the costs of hiring new people w/ pre-existing cloud
skills. It is cheaper than the stop-gap of paying outside consultants till you
can bring in the interior staff. Also, these people _know_ your existing
systems. They know your business logic, they know what needs to be up and they
know what can't break on Thanksgiving weekend. That is valuable knowledge that
is a hidden cost in rehiring the work force.

You're going to find new challenges in the new system. If your employees are
willing to transition, then transition them. It is your best bet.

If you have a subset of employees that can't/refuse to transition, put them on
performance improvement plans. No rumor mill - if you're on a PiP you need to
shape up or get out; if you're not on a PiP keep doing well, your job is safe.

~~~
ma2rten
But is OP really going to have the same cost savings if they retain everyone?
I am assuming labour is a huge part in that figure.

~~~
michaelt
The ideal situation is to be in a growing company, and to move employees from
shrinking areas into growing areas.

For example, if the company needs 8 fewer people to install and set up
servers, but 10 more people to work on robust automated code deployments to
support business growth, and both jobs need Unix skills.

Hence, the setting-up-servers department budget shows cost savings, the
overall technology budget grows by 2 employees instead of by 10, and business
growth means overall profit increases.

------
arminiusreturns
> theyre doing stuff like manual config, and future state is everythign
> automated as much as possible in the cloud and shutter the data centers

Ok, who let the C'level out of his cage? Quick, convene a meeting about
meetings and distract him with new synergistic products that mesh the work
output of the teams into a new daily reporting product with shiny graphs for
him to make informed decisions about, preferably by proposing an "cloud only"
data scientist.

But seriously. This comment alone shows this person doesn't have a clue wtf
they are talking about. If your guys were so focused on manual config that was
a management problem, and isn't an inherent fact about the current state of
data centers. So you decided to move data centers, you just moved the jobs
somewhere else, to the new provider (Let me guess, amazon, microsoft, or
google...)

Say it with me: "There is no such thing as "the cloud"."; Say it again: "There
is no such thing as "the cloud"."

Your shit still runs on servers, that at the end of the day someone manages
the systems of. So they manage the systems at a lower level that is
transparent to you as the modern datacenter is able to hotload blocks,
devices, vm's, containers, etc across a swath of machines, clusters, etc.

At the end of the day, you made a business decision that decided managing your
own on prem wasn't for you. For some companies, that can be the right
strategic move, but as the salty ol bofh who has seen it happen, it's not all
puppy dogs and rainbows and there are many pros and many cons to such a
move... with "the boss" usually completely underestimating the cons.

Next up: the boss asks how to ease the transition of the low level programmers
because he's hiring a new wave of nodejs/meanstack devs because "there's so
many of them he can pay them less than the greybeards" and "they are young and
know all the cool new devops agile cloud cycle methodologies" and "c++ is
dying".

Whatever f50 this is I'd longshort the fuck out of it. I bet it's in a
downslide for other reasons and IT is the easy dept to squeeze politically
while ignoring the root causal issues that are probably management related.

tldr; Data centers aren't going anywhere, your shit still runs on servers,
people still design build and rack boxen, design, fine tune, and diagnose
networks, munge and wrangle data, etc, etc, and the skills they have are still
in high demand the world over. Disagree? Tell me how many data centers as a
percentage that have fully automated robotized racking?

~~~
throwawayacct4q
You really have this completely wrong. I am a systems/network engineer and am
newer to the business side. Asking here to get some upsight into the people
side. Thank you for pointing out that things run on servers, that was news to
me.

~~~
mmt
Sarcasm is rarely well-received here. Also, I think anyone reading any of the
rest of the thread would see that you do have an understand of the issues, so
no need to defend yourself.

Conversely, the parent commenter uses hyperbole to the point of appearing
_not_ to have that understanding.

For example, even without "fully robiticized" datacenters, there are
remarkable economies of scale before that, including ordering servers by the
rack (or maybe even row at a time), pre-assembled by a vendor.

Things really _can_ be different, even though they _can_ be the same.. even
both at the same time!

------
poster123
Jobs in the private sector do not come with lifetime guarantees. If you have
new jobs you could train them to do, great. Otherwise, give the people you
will lay off notice and severance, and let them figure out what to do.

~~~
s73v3r_
To any companies wondering why their employees jump ship at the drop of a hat,
or why there's just no loyalty anymore, this attitude is it.

~~~
Spivak
Sure, but it's a cycle that is easy to fall into and hard to break now that
we're here. Distrust begets distrust.

~~~
s73v3r_
Then those with the power, the employers, can work to earn that trust.

~~~
mmt
This can't be over-emphasized, that, especially with no union, the power
dynamic is _immensely_ unbalanced. Even just the information assymetry is
huge, since, despite recent innovations like Glassdoor, workers know very
little about even each other's circumstances.

The parent's comment use of the word "cycle" implies, by association with
phrases like "cycle of violence", that there might be some equality or that
both parties may be victims to some degree.

In the face of such assymmetry, such a claim would be extraordinary, requiring
a strong argument, more than just an implication.

