
The cloud skills shortage and the unemployed army of the certified - rajeshmr
https://itnext.io/the-cloud-skills-shortage-and-the-unemployed-army-of-the-certified-bd405784cef1
======
coleca
I did some freelancing cloud work recently and was shocked at what I saw out
there. From small one person operations to startups to large multinational
Fortune 500 corporations I saw the same pattern repeated over and over. People
using the cloud to spin up infrastructure that have no experience building out
infrastructure and making really bad and dangerous mistakes.

Amazon, Google, Microsoft and others make it simple to get going but the devil
is in the details when it comes to building it right, safe and ready for
production. Just a few examples:

• Putting RDS SQL Servers on a public IP with no protection

• No templating of servers so if it disappears you have no record of how to
get a new one running again

• Servers with SSH password authentication turned on with passwords of
“Password”, because SSH key auth was “too hard for our devs”

• No backups because “it’s in the cloud, isn’t the cloud backing it up for me,
like iCloud?”

I have an AWS Solution Architect Professional certification and agree you
could get a cert and still not be qualified to run or design much on AWS but
it does put you ahead of much of what I’ve seen out there.

~~~
drieddust
I work for a large IT outsourcing company and I have seen the opposite of it
with our clients. Most of our clients wants to be on cloud but without any
change to their habits so we end up building complete traditional data centers
in the Azure or AWS.

Most applications are simply cloned from existing On-Premises Infrastructure
because every project have a 2 weeks deadline and every single stakeholder
just wants to stick it in. Patterns like Auto-Scaling etc is just a far
fetched dream.

Supporting such Infrastructure where every server is a pet trying to survive
in a Slaughter House is a deeply painful.

~~~
raincom
In big companies, 95% of apps are still old school: firewall -- load balancer
-- 5 front ends -- 3 back ends -- two database servers. And the people who
developed most of these apps left these companies long time, what these
companies have bunch of managers and some developers who maintain that code.

People of this kind are extremely conservative, because they are not in a
position to fix major issues if something goes wrong--esp new architecture.
So, they want to lift as is to aws.

A decade ago, I was involved in physically moving three solaris servers that
were running un-supported BEA weblogic instances. When I powered on these
three servers on a different rack, weblogic cluster failed. No IP change, just
moved machines to free up racks in one row. And the people who developed the
app, and the dev architect behind it were so mad at us. In the end, they gave
up on that app.

That's how it goes.

~~~
user5994461
It's not old school, it's what every web company is. A few load balancers, a
few apps and a few databases.

~~~
drieddust
In principal yes, they look like any modern web application but reality is
different.

For example stateless web component is a primary requirement to allow servers
to fail as well as auto scale. Most enterprise applications I have seen falls
flat on face.

I have seen enough cases in which customer endlessly debates why his snowflake
of a web server went down and expects a RCA with god knows what details.

------
angarg12
The question: "Why it’s so hard to find roles in cloud technology, while jobs
go unfilled."

The answer: "The key thing here is that businesses are not hiring fresh-out-
of-certification AWS experts to take over the infrastructure of their
established operations teams. They want people with solid experience, a risk-
free bet that the new hire can execute the tech flawlessly. This often means
insane job requirements [...] and an existing track-record of success."

So jobs openings stay empty because companies have unrealistic expectations.
Is that something new and surprising? This is the pattern in tech recruiting
as far back as I can remember.

~~~
atemerev
The idea of “you can find a job immediately after getting a degree or
certification” ended a few decades ago. Today, you need some experience,
especially in cloud computing.

Thankfully, in this field there are few barriers of entry, save for having
some natural curiosity. All clouds have free tiers, excellent documentation,
sample projects, etc. Kubernetes is free and open source. Technologies are
interesting. Play and learn, and you’ll get a job.

~~~
coldtea
Only personal tinkering doesn't count as "experience" in those companies
books...

~~~
ldoughty
I tend to hire a lot more people that personally tinker without
certifications, than those with certifications that don't have a pet project.

Certification proves memorization. Tinkering proves interest. I don't want a
candidate that is not interested personally.

------
theredbox
I am sick of this non sense. It's because nobody knows what they want and
those that should be the ones knowing it all also do not know what they know
and what they need to learn.

We built immensely complex systems that allow us to scale. The thing is we
have increased the complexity so much that we no longer can justify hiring
incompetent.

Simplicity is beautiful and we will have a renaissance of simpler systems.
Something along the lines of what Go brought to the game.

~~~
sascha_sl
>Something along the lines of what Go brought to the game.

Where the biggest adopter, Kubernetes, is ironically a way too complex system
of automation routines that still regularly fail in production environments.
And oh dear, if they fail, it's gonna be hard to figure out what exactly went
wrong. We uncovered a bug in the job controller just yesterday that was
probably not biting anyone else due to how a race condition usually plays out,
but this kind of bug is way too common.

~~~
nova22033
>Where the biggest adopter, Kubernetes, is ironically a way too complex system
of automation routines that still regularly fail in production environments.

regularly fail? Do you have any data to back that up? No, a google search for
one kubernetes failure doesn't count as proof of "regular failure"

~~~
sascha_sl
I'm in an SRE-like position for an otherwise unmanaged Kubernetes cluster. We
have code merged in the repo as well as a few PRs still open.

I should note that by regularly I mean edge cases and quirkiness, not that
your run of the mill cluster mostly using deployments is going to randomly
explode. But our teams still find ways to _regularly_ break it in unexpected
ways.

So I guess you could call that anecdotal if you want.

------
fma
"The jobs opening up include a requirement to know all these skills in
addition to everything else."

I've been casually looking at job postings and sometimes wonder what drugs the
job poster is on when writing job requirements.

~~~
GoToRO
I you really want to have fun, just look at the job posting for your own role
in the organization. Then, you can really compare stuff.

~~~
yjftsjthsd-h
Funny story: Once upon a time, I got promoted, and we needed to backfill my
(old) position. They proceeded to post a job description that 1. I might not
have qualified for, and 2. I never would have applied for.

------
andmarios
Every year cloud providers announce tens of new services, poorly tested and
far from production ready. Non-ops eng teams and management are always happy
to use the latest buzzword out of AWS, then ask their ops team to make it
somehow work.

Certifications are moot. I don't care if you are certified, or if you have
used a cloud (or kubernetes) following a tutorial. I do care if you have seen
enough issues throughout your career, so you can debug every time (not when,
nor if) mud hits the fan. I don't care if you can type a port in a yaml file
or in your browser. I do care if you know how many ports are there, what's the
ephemeral port range and so on. Systems are 75% experience, 25% basic
knowledge, you can't get away just with certifications which matter only after
you get experience and knowledge.

If you are experienced in ops, I know you can learn and work out any cloud. If
you claim you are experienced with a specific cloud though, then better have
to share a ton of war stories or it will just be a huge red flag on your CV.

Let's call it for what it is, clouds, like data-science are hot fields right
now. It's only natural that many people will go after a good-paying job
claiming expertise in the field.

~~~
scarface74
If you have experience in ops and don’t know the specifics of how to do
operations in the cloud and take that same mentality to A cloud provider, you
are going to end up spending more and not get any benefits from being with
AWS/GCP/Azure.

------
henrik_w
"The Creeping IT Apoclypse" is on the same theme:

[https://news.ycombinator.com/item?id=18930781](https://news.ycombinator.com/item?id=18930781)

------
HocusLocus
"Hi! I'm your new cloud outsourcing specialist. I'm here to take your old IT
structure and those servers in the closet devils-you-know with something so
intangible and 'elsewhere', when it stops working some day you cannot possibly
send an army to the closet to fix the problem. You can all just wallow in your
helplessness and go home and wait for a text message. When do I start??"

~~~
nostrebored
Yeah, except that team in the closet is now one of the most experienced teams
in the world responsible for thousands of companies and managing a global
backbone infrastructure that a smaller company couldn't possibly pay to
maintain.

~~~
setquk
Unless it’s Azure as they make small company mistakes like forgetting to renew
certs...

YMMV for anything you can’t see or touch regardless of the size or experience
or reputation so far.

------
paganel
Just partially related to the article, but mentioned in it nonetheless, I just
wanted to ask if “serverless” is now the new hype-word instead of
containers/Docker/K8? I’m curious because I’ve started reading about it in
different contexts more and more lately but afaik it doesn’t come with any
clear advantages over the “classical” way of doing stuff.

~~~
dsr_
Everything is a semi-rational reaction to something else.

Companies used to own and run their own machines on-premises. But doing it
properly (HVAC, power, raised floors, standardized racks, management,
redundancy, preparedness...) is expensive and not actually most company's core
competency. So they moved to colocated datacenters, where a competent company
would take care of the infrastructure for a fee which would hopefully reflect
a discount based on savings from scale.

But managing generations of servers at colo datacenters takes manpower for
hardware replacement, upgrades, cabling, and generally doing things right; the
customer companies mostly don't have that as a business feature but see it as
a cost. So managed infrastructure services came up, where the datacenter
company leases you the hardware (standardized so they can have spares) and
racks it for you and puts in network switches and remote services and gets it
to the point where all you have to do is log in to a console server and
install your OS and start deployment.

But sysadmins who can keep track of security updates and package dependencies
and keep an OS properly organized are relatively expensive, and what used to
be

"well it works on my laptop"

becomes

"well it works in my container"

and the container itself gets shipped out. It's cheaper not to do security
work, you know.

So VM and container accepting services come up, where the datacenter company
now runs a hypervisor and managed storage systems for you, in exchange for the
salary of those sysadmins. The start-up costs can be much lower, to the point
where it really doesn't make fiscal sense for any tiny company to do anything
themselves except ship their containers/VMs out and puzzle over IP allocation
schemes and load-balancing services.

It's a trap, but a delicious one.

So now your company is hooked on the ease and speed and cheapness of just
spinning up another container or VM any time someone expresses a desire, or
even automatically, when you get the bill at the end of the month and you
managed to spend how much? That's ridiculous. Why are we getting billed for
containers that we don't even need to run all the time?

Along comes "serverless", which is a logical successor of inetd. Yes, inetd,
the very old "internet super-server" which would read a table of ports and
programs, open all the ports, and when a connection came in would run your
program and connect stdin/out to the socket, isn't that great?

Serverless is just a management system that spins up your very restricted
single-function program on demand -- now it answers an HTTPS API instead of a
raw socket -- and does the accounting work to make it profitable.

Complexity is simultaneously the enemy of correctness and the source of
profit.

~~~
adityapurwa
This is so well written and beautiful. Thanks for putting it on this
perspective.

------
mancerayder
_Typically we have shortening 5-year cycles in tech that cause clamoring for
new skills. First it was the internet (“HTML! Perl! PHP!”) then mobile apps
(“Android! Objective C! React Native! Flutter!”) and now cloud (“AWS
Certification! DevOps! SecOps!”). IoT and machine learning are probably next.
The technology itself doesn’t really matter — companies just want an army of
tactical workers to build this stuff tirelessly._

It's Kubernetes and microservices! Even though Kubernetes is what, four years
old, where I am, numerous employers are chomping at the bit and demanding that
you be an expert at these. Now, I'm a crusty old Linux sysadmin/programmer who
puts "SRE" and "DevOps" on his resume, and I'm noticeably getting spurned
because I don't have "5 years experience" with "containers".

At first I was worried that my skills were obsoleted, having worked in large
enterprise environments doing infrastructure engineering for over 15 years. I
worried that I was too "systemsy" and not enough "cloud-y". Oh, and even
though I spent years using TeamCity and Jenkins, I was also spurned because I
didn't use the right lingo within "CI/CD pipelines" and demonstrate a Docker
deployment system _exactly like they used._

Then I realized something.

The job descriptions are asking for _five years experience total_ , number
one, and the second red flag: the salaries and contract rates are _lower_ than
the rates for large enterprise-y type jobs (minus management).

Conclusion: hipster tech coincides with youth and coincides with companies
following trends and buzzwords over wisdom and experience.

Back to the point. Someone with a strong basis in the core disciplines of
systems design, infrastructure automation, perimeter security and so forth are
going to be able to adapt to AWS knowledge, as I have (for the record I have
two AWS Associate certs and didn't find them hard). And as the author points
out, there are these 5 year cycles where all we look for is experience in
certain tech. Is it any wonder that there are horror stories about fragile and
insecure cloud environments?

It goes back to the incompetence of the hiring process which we see again and
again discussed on HN.

------
huffmsa
_> "10 years serverless" as a hard requirement..._

So what we have here is a failure in the hiring process. The people writing
the jobs don't actually know the space, but are looking for candidates with
literally impossible amounts of experience.

~~~
dankohn1
We refuse to hire any Kubernetes developers without at least 6 years
experience.

[https://landscape.cncf.io/selected=kubernetes](https://landscape.cncf.io/selected=kubernetes)

------
sailfast
That S-Curve on cloud seems off.

Most Federal agencies are on or moving to Cloud services right now. They are
squarely toward the top right of the S.

I don’t think you can say only “large trailblazers” are there. We can argue
about how many services have moved and how it’s being used but the whole “let
someone else manage it” argument is long done and normalized.

~~~
laurentl
A lot of statements in the article seem off. The position on the S-curve, the
claim that TDD replaced QA teams, even the fact that “serverless is coding for
hardware engineer” or however it was formulated (the author probably meant
“infrastructure as code”). With so many approximations in the argument, it’s
hard to accept the conclusions of the article at face value.

------
RickJWagner
29 year IT veteran here. I don't agree with a lot of this article.

I remember panic-talk when outsourcing began. When offshoring arose. Etc. None
of these extinguished the need for IT workers.

I view the cloud as the next generation of the mainframe. By that, I mean your
experience and understanding of the environment (nobody will master all of it)
will be a career differentiator. Knowledge of Kubernetes, Istio, etc. will
help you to navigate the next career paths. System admin skills (and
associated extras, like networking) will likewise move you up the ladder.

Bill Gates once said there is an infinite number of problems yet to be solved,
an infinite amount of work to be done. I believe him.

------
heyjudy
Disclaimer: ex preferred partner early AWS client-facing Fortune 200
consultant.

The senior sysadmin at the DMDC around mid California coast: only one
certification, VMware. A French offensive exploits broker, solid researcher
and reeigne: no certs. I only have a few certs because work paid tens of
thousands for offsite training. A few certs are okay because corporate HR
wants staff to demonstrate continual learning and achievement, lots of low-
quality ones (A+) are a negative signal. It's more important to keep up on
news, tools and techniques on the front lines.

My opinion is managing \ _aaS needs solid rack &stack ops experience at scale.
For IT folks: either start huge if you can or start small and move towards
huge. The biggest challenges are customer communication, managing
expectations, anticipating and planning where possible and showing empathy...
it's a 5% technical skill and 95% getting along productively game. Communicate
the business impact, don't overwhelm customers with _unnecessary* detail.

------
halbritt
I really appreciate this part:

    
    
      Some of these firms realize they need to renovate their IT 
      but they attempt to do this through hiring “Digital 
      Transformation” roles which do little but frustrate the 
      revolving door of hires because they have no executive buy- 
      in and budget.
    

This kind of thing makes it easier for me to hire. My org swallowed the risk
and moved our production infrastructure entirely to k8s. I've no expectation
that I'm going to be able to hire engineers with years of experience with
that, but my team has already cultivated those skills. Aside from all the
benefits and pains of being on k8s has, it makes it easy to attract great
engineers with lots of related experience looking to leave a company where
they've heard they need to adopt something and fail to put the resources
behind it.

------
tabtab
This is an age-old problem in IT. Some new technology and/or fad comes along,
and bunches of organizations suddenly want an _expert_ in it.

It's impossible to find enough experienced in the new field because it's so
new and/or growing too fast. But the HR department or technically-
inexperienced managers don't understand this and complain to politicians et.
al. of a "shortage". (Those in know have seen the pattern multiple times.)

The answer is simple: find a similar field and _accept_ a learning curve for
the employee to transition. If you need something quick, hire a consultant to
both move the project along and to mentor the new employee.

------
empath75
The real reason is that coding is hard and coders are hard to find and
certifications are just an acknowledgment that you have _domain knowledge_.

I don’t care if you know what an auto scaling group is. I care that you can
write code that automates builds and deployments. The AWS part is easy and you
can learn what you need to know on the job. In my experience, coding is
something you either can do or you can’t, and if you come to me with a cert
and no programming experience it’s very hard for me to know what you’re
capable of.

i spend like 5% of an interview asking about AWS stuff and like 50% talking
about code and automation.

~~~
theprotocol
>you can learn what you need to know on the job

Maybe I've had bad luck, but I've encountered very few prospective employers
who were OK with this. The ones I encountered expected to hire someone who
would fit in like a jigsaw puzzle piece and they wouldn't settle for anything
less. My confidence in my ability to pick up almost any tech quickly did not
impress them, and they latched on to my lack of that particular bullet point
and kept hammering on things like "oh, you don't know what a [platform-
specific term for auto-scaling groups] is?..."

~~~
Pokepokalypse
Same. And I spent 10 years at a company where my job, literally, was new small
projects, every 6-12 months, often radically different platforms and
technologies, where I had to "come up to speed". (has it's plusses and minuses
- you learn a lot of breadth, not a lot of depth). But interviewing for me,
has been a very frustrating experience. When I did land a spot, they were
desperate, as it turns out. And in interviewing and making hiring decisions
for my team; I've had to make these same types of judgment calls. I look for
candidates who have enough experience that I know they're not bullshitting.
And I look for a work history that shows a progression of continually taking
on new skills and roles. Because that's what this job really is. Anyone who's
been doing this in the real world for at least 10 years knows this. But it's
really puzzling to me how so many hiring managers at so many places just don't
seem to understand this.

~~~
mancerayder
The same thing happened to me. Recently, one company put me through three
Python challenges. One on the phone, and then later two in person. One was on
the whiteboard, with no "hello" or questions about who I was, starting
immediately as I entered the room. I completed his assignment (and in an
object-oriented way). Later he said, "We're looking for someone who can come
up to speed quickly." And later HR said, "Feedback was very mixed, but
unfortunately you don't come up to speed quickly."

I answered all the questions, have a proven track record, am self-taught for
15 years with increasingly high salary / rates, and yet for this one guy I was
incapable of learning quickly. This was determined in a 30 minute whiteboard
interview I passed.

Some of the interviewers have extremely low sociability on the psychological
scale, even if they're tech geniuses (which isn't even evident). Everyone
talks about the skillsets, S-curves and so on, but no one talks about the
mediocre evaluation process by those who aren't good at it.

------
hestefisk
“server admin is very hard, and mostly gone with virtualization.“ I would say
this is simply not true. For many companies, getting good admins and
infrastructure experts this is extremely hard, at least in big corporate.

~~~
ldoughty
Completely agree. Server admins are still super important. Not everything fits
in a nice server-less box. Developers also tend to have issues with
containers, and firewalls, and monitoring/troubleshooting (depending on their
skills and interest). People whom enjoy the server admin side can still find
rewarding work in a more serverless environment

------
netheril96
I'm guessing that another reason is that some of the existing IT personnel are
resisting the change, and one of their methods is to give HR unrealistic
requirements so no one will ever qualify.

------
lifeisstillgood
I read this as "develop and sell a niche tool that's useful in cloud ops or
transformations, and market it like hell"

------
ilaksh
Really interesting article. This line was funny though: "can remember the
differences between a stack and a heap".

------
taneq
> QA has been gutted by TDD

Wow. Just wow.

~~~
rcaught
It is probably fair to say that CI/CD have gutted QA though.

~~~
pjmorris
It is probably fair to say that managers have gutted QA.

------
JVerstry
Not so sure Docker is the only way forward when it comes to cloud scaling or
deployment. VM templates are a very good alternative. They are more stable,
more flexible/customizable and integrate more smoothly with CI.

~~~
talltimtom
Docker, cloud hosted VM from templates, vagrant, Azure, AWS, it doesn’t really
matter much to me. The important thing to me is that I no longer have any need
for anyone else doing company wide infrastructure. We have several department
heads engaged in a battle about who’s going to “own” our “cloud
infrastructure” they seem Oblivious to the fact that the only thing we need
from them is to negotiate an Azure or AWS subscription and after that they
will loose any utility.

A lot of talented people are going to find them self in a problematic
situation because the area in which their talents lie is only going to be
handled by low wage jobs at google, Azure and Amazon. Even big companies won’t
bother setting up their own hardware because the people are costly even if
they have slight savings on private hardware.

~~~
ownagefool
> they seem Oblivious to the fact that the only thing we need from them is to
> negotiate an Azure or AWS subscription and after that they will loose any
> utility.

I've seen similar and they tried the following:-

\- We'll provision VMs for you. Raise a ticket.

\- We're doing "Hub & Spoke". You're not allowed to route any internet traffic
except through our inspection proxies.

\- We've disabled the API. You can only use the Console.

Basically, a couple of old school guys will do anything they can to disable
automation, as otherwise they'll be accepting they can't really contribute
anymore.

~~~
laurentl
The old-school guys also think (rightly in some cases) that they have an added
value. 10 years ago I was building a cloud platform and explaining to the
security team that they would no longer receive tickets to manually configure
routes on firewalls, the customers would do it from a console. I thought
they’d be happy to be relieved of a menial, boring task but their reaction was
“when we receive a ticket requesting to open all ports from any IP address, we
can explain to the customer that it’s a dangerous idea. If they can configure
it themselves, who will tell them?”

~~~
technion
I lived through a "empower the developers with Devops", "prevent you doing
menial tasks" project a while back and ended up with:

\- A mail server which was an open relay, promptly shutdown for abuse

\- Every single internal server on an external IP address with an allow
any/any ACL

\- Brand new environments built with PHP 5.0 in 2018 to run new development
projects (EOL over ten years ago)

\- Managers patting themselves on the back about the power of Devops

