
Why isn’t there a glut of good software engineers? - jseliger
http://blogs.law.harvard.edu/philg/2013/05/01/why-isnt-there-a-glut-of-good-software-engineers/
======
tomkarlo
Because there’s a longstanding view that since programmers love what they do,
they shouldn’t be compensated as highly as other fields with demanding
requirements. Employees at large tech firms are as profitable as bankers, yet
they get paid a fraction as much compensation. Not to mention that it
generally takes years and years of _unpaid_ homework / hobby work to learn how
to be a good software developer - just taking four years of college CS with no
outside hobby work will not prepare you in any way. Nobody expects that from
doctors, lawyers or bankers.

So the pool of great programmers is generally limited to those who love the
job _so much_ that they’re willing to forgo income to do it. Someone with a
“125 IQ” and strong math abilities could get a job on Wall Street, or as an
actuary, etc and make more money for less frustration.

At Amazon, they target 75th percentile comp – but I really doubt they also
target hiring 75th percentile developers on average. So they’re basically
trying to hire 90th percentile employees for 75th percentile comp. I’d argue
the entire industry does that in comparison to non-engineering industries.

The next time a CEO complains they can’t hire enough engineers, ask them how
much over market they’re offering.

~~~
jskulski
I see your point but I think "forgo" is a strong word. I have made and will
continue to make a great salary with great benefits and be able to choose
companies based on cultural fit.

That's something that I would never give up for an extra 50k a year.

~~~
tomkarlo
Yes, but you're also hanging around on HN. :) I'm thinking more of the other
guy, who was just as smart as you but perhaps not in love with tech, and had
to decide if he wants to give up 50K to be an engineer. (I think you're
referring to choosing between engineering jobs - I meant choosing between
career paths.)

Also: it's sad that beyond the comp issue, they're often sacrificing prestige,
at least in the US. It's just as hard to become a qualified engineer at top
firm, but the general public just doesn't find that as impressive as being a
lawyer, doctor or banker. I know lots of lawyers and bankers for whom a big
reason they're in that job was the prestige and ego-boost associated with it.

~~~
rayiner
That's why you need to move to Silicon Valley, where being a programmer is
considered prestigious by people in barbershops.

~~~
tomkarlo
I did.

------
sehugg
_by now most of the world’s programming would be happening in low-to-medium-
wage countries and that customers wouldn’t have to pay a lot for high quality
software. But it hasn’t happened (e.g., see my posting about California’s
state government spending $327 million on a straightforward ecommerce site_

Uh, the reasons government IT projects are typically huge wasteful boondoggles
are _not_ because programmer salaries are too high or programmer talent is
mediocre.

~~~
solistice
I was puzzeled over that number too, as putting up a site like that, even if
you are buying the servers from scratch and hire a first rate web dev,
shouldn't cost more than 10-20k$. I went through to the linked artice, and
appeareantly, most of it went for consultants. I mean there are consultants
out there that are worth their money, and I bet the site is kai-zen optimized
very nicely, even though I can't tell, but 327 million $ is quite a hump for
that website. The main problem seems to be that government contracting is not
a regular market, else joe's jquerry shack would have ourrun and outpaced
accenture in both cost and quality.

~~~
jshen
"if you are buying the servers from scratch and hire a first rate web dev,
shouldn't cost more than 10-20k$"

this number is just as absurd as the $327 million.

~~~
tjr
At $125/hour and $5000 for servers, $10-20k could be spent up in a couple
weeks or so. That seems pretty reasonable, and certainly not as absurd as $327
million, which would amount to about 1200 years of FTE effort.

~~~
jshen
If you were the government or a big company with a reputation would you call
up 1 guy and say "you've got a few weeks and 120/hr, make a site and well run
it in our data enter"?

Think of all the things that can go wrong! Security, PII, etc.

~~~
tjr
Ah, I thought your remark was suggesting that it should be even cheaper than
$10-20k.

------
ben336
The article starts with the primary assumption that programming is a commodity
skill, then is astounded when the world doesn't back up his theory. Not
exactly going to learn a lot like that.

~~~
tempi35
Why programming isn't a commodity skill? Or will become such commodity soon?
Programming is like a house builder. Yes there are better and more experienced
builders, but most of them can build a house top to bottom with no problem.

~~~
paulbaumgart
Programming is not like building houses. The computers on which the software
runs are the "house builders"-- the parts of the system doing the repetitive
work. Programmers are the architects and civil engineers who are tasked with
inventing new types of houses every time.

~~~
siavosh
Not that I disagree with you, but have you looked at a typical architect's
salary and job prospects?

~~~
paulbaumgart
It seems there's a lot more demand for different types of software than there
is for different types of houses.

~~~
BerislavLopac
Especially since most of the house design is now done by software.

------
rayiner
1) I'd imagine programming ability is fairly correlated with IQ, which puts a
limit on the percentage of the population that has the requisite analytical
skills to be effective programmers.

2) Programming is as much about communication as it is about writing code.
People have this idea that you can reduce a program to a precise set of
specifications that can be formulated by non-programmers and shipped off to
China to be coded up. I find the idea ridiculous. People don't come with to
lawyers with the precise contours of the case they want to build and simply
ask for the lawyer to write up the briefs. They don't come up to doctors with
a precise treatment plan and simply ask them to administer the medications.
Programmers are problem-solvers, and that's a two-way street, and that makes
shipping critical programming work over to China and India untenable because
of communications barriers.

~~~
tomkarlo
"People have this idea that you can reduce a program to a precise set of
specifications that can be formulated by non-programmers and shipped off to
China to be coded up."

Yup... If this was actually possible, we could just write software to do it
and not need programmers at all.

------
VLM
"But that doesn’t explain why people in the Philippines or China aren’t
training themselves en masse to be able to soak up the $30-100/hour jobs that
would be readily available to them"

Rentacoder is readily available, its just that word on the street was most
jobs paid about $5/hr.

Amazon's Mechanical Turk has some extremely fuzzy stats and not much more than
anecdotes but earning around $7/hr has been reported. Higher pay, more fun,
less effort?

Waiter/bartender at a tourist restaurant is probably the most fun and could
pay even more.

Why is the market so fuzzy and illiquid? Why aren't there ten places I can go
to find "write perl to import this old fashioned CSV file into that database"
and see exactly what thats worth? Fix that and then wonder where the people
are.(edited to rephrase there's no market, so no one plays the market)

~~~
raverbashing
Being a waiter at a tourist restaurant is pretty much an 'on/off' occupation.

You need a small amount of specific knowledge to start working (at a specific
place), and leaving the job takes almost no knowledge away.

Software development is a _very slow_ start, depending on the project.

Just try and see how long until a developer has the development environment.
Yes, it should be a 'turn key' solution but that's often not the case.

~~~
redblacktree
Check out vagrant[1]. I can have a new developer set up in about 30-60
minutes.

[1] <http://www.vagrantup.com/>

~~~
raverbashing
Well, you're assuming that

1 - All projects can be run in a VM 2 - Setting the development environment is
all a new developer needs

Let me try to instil some expanded vision into things:

\- I can't have a VM for our specific embedded hardware with its specific
devices

\- I haven't seen a VM for AIX on Power (maybe except running on Power HW)

But most importantly, Vagrant can't put on the developer's head everything
specific (s)he needs to know to do the job. Like how to use the proprietary
utils that we need to integrate in a project. Or what the project is all
about.

~~~
redblacktree
I didn't mean to imply that Vagrant is appropriate for all situations.
However, it has certainly been a boon to my web project.

I hope you'll forgive me for misunderstanding "Just try and see how long until
a developer has the development environment." -- The language is ambiguous,
but "until a developer has the development environment" means software to me.
That is, that the developer has the tools he needs to make changes to a code
base. You're right that I may have taken too narrow of a view, but my
interpretation also seems reasonable.

------
arunabha
Its easy to run, all you need is a pair of shoes(not even that in the extreme
case) and a bit of open space. I wonder why we don't have olympic runners in
every neighbourhood ?

More seriously, I suspect its a consequence of the 10000 hour rule applied to
a discipline which builds some of the most complicated structures known to
man. It takes a lot of training, formal or informal and a lot of good feedback
to make a good software engineer. Plus, CS is not the easiest of disciplines
to study.

That, along with the fact that good practitioners, in almost any discipline
are by definition the top few percentage.

The bigger question would be, as the profession matures and builds up an
institutional body of knowledge like civil engineering, would the number of
'good enough' engineers rise.

------
willholloway
The author of this would have his answer if he understood the IQ floor
necessary to be a good engineer, access to computers at a young age and
willingness to troubleshoot obscure problems is relatively rare in the
population.

There are far less high IQ individuals in the world than the world needs right
now.

We can teach coding to young people, but as of now we can't reliably increase
IQ.

We have studies that prove agents such as piracetam administered to pregnant
rats increase the intelligence of their offspring.

If the Tech titans that want cheaper labor really want to solve the problem of
lack of quality engineers they could think long term and throw massive amounts
of funding at neuroscience and nootropic research.

In the information age we all live and die by our wits. Policy and R&D money
should be directed at raising the mean intelligence of the population through
novel scientific breakthroughs.

Far easier though to just lobby for government policies that make it easier to
import the rare kind of people Google, Microsoft and Facebook need.

So we have a situation where no one is focused on solving the root of the
problem (a need to increase the cognitive abilities of humans) and no one is
lobbying to protect the financial interests of working coders in the United
States.

~~~
solistice
Isn't Silicon Valled allready hooked on a constant supply of aderall and
modafil? Now add Hydergine, Piracetam and Deprenyl into school lunches, and
you've got your super children.

Or hack whatever is broken with the educational systems around the world. I'm
still in school (technical highschool, programming focus), and none of my
friends that is worth his salt in programming is good in school, and that
puzzles me because they keep telling us that they're desperate for good
coders. D is one of the best js coders I personally know, and he raves about
it all the time. He dropped out of the part of the school that I go to (which
would lead up to college), and likely won't stay in cs due to that. B is one
of the best php coders I know. He's also a D student. A's good too, but had a
2.0 GPA as well.

There's a fair share of good potential programmers around (even though my
observation might be somewhat biased), but many of them won't ever get into
college. And how do you face HR without a degree? Go to hell kid, we want
credentials.

I'm pretty sure you don't need more leads, you need better conversion, or else
we're all going into marketing.

~~~
ahlatimer
Speaking as the only person in my office who was a degree in CS (one has a
geology degree, the other dropped out half-way through his graphic design
degree, both make more than I do), no, you don't need a degree to get a job as
a software engineer. The job I had previously wasn't altogether different,
either. I think less than half of us had degrees in CS.

~~~
solistice
At least that leaves me with a spark of hope, but it's still quite
discouraging that you could be the best programmer in your entire school, and
still not be admitted for a CS degree. It kinda defeats the whole point of
school, degree, job.

~~~
ahlatimer
If schools judged CS applicants by their ability to program, I'd agree, but
they don't. They judge them based upon their likelihood to succeed in the
program (amongst other things, like building a diverse student body or
whatever). That is, they judge them by their ability to be a good student. If
all of the evidence goes to show that a prospect is a terrible student, why
would a University accept them knowing that they are likely to fail?

Now, your friends may be more than capable at succeeding in university, and if
a school only took the time to get to know them, they might see that, but, on
paper, they look the same as someone who likely couldn't succeed in school
(because they haven't, really, up to that point).

That all sounds all horrible about universities, disregarding these capable
students because of an inability to really get to know prospects, but that's
not on them. If you want to get into a good school, try harder.

As far as it defeating the point of getting an education, I'd wholly agree
that college is not a worthwhile pursuit for a future-SE if your only reason
for going is to get a job. You'll spend a lot of time in college learning
skills that are not directly applicable to your future employment. You could
arguably get to a point where you're good enough to hire more efficiently by
skipping school. Whether or not it's worth passing on the things you really
won't experience outside of school in exchange for a more efficient route to
money is really up to you.

------
coldcode
Why isn't there a glut of baseball players who hit .408 in the major leagues?
Because it's a rare person who can hit a ball thrown at nearly 100 MPH 40% of
the time. Software engineers are common but people who deliver quality product
after product is a rare bird. Programming at a high level is not as easy as
this author seems to think.

~~~
jpatokal
Philip Greenspun, aka "this person", has a pretty solid resume when it comes
to programming. <http://philip.greenspun.com/personal/resume>

~~~
sixbrx
And like many brilliant and talented people, they're completely clueless that
in the real world, many other people just don't have the natural abilities
that they themselves have, thus they are repeatedly perplexed that others just
can't do the "easy" things they do. Often I've seen instead that they will
instead ascribe others' failures as moral failures, which bites doubly against
those that are trying hard but can't measure up.

~~~
beat
I struggle with that myself. My attitude toward people who aren't
accomplishing much with their lives is "Well, work harder!" But while much of
my own success is due to hard, focused work, much of it is also due to raw
intelligence and talent. Not everyone has that.

------
ap22213
I think there may be a glut of potentially good software engineers. In fact, I
know several high-IQ programmers who are skilled, but sadly, probably
unemployable.

I think the problem seems to be (at least) four-fold. 1. They don't have the
prerequisite cultural background. 2. They are self-motivated to pursue their
own creativity but less interested in others'. 3. They don't have a fondness
for excess money. 4. They have a hard time being obedient and subordinate.

Maybe controversial, but the way I see it.

~~~
paulbaumgart
There is absolutely a market for really good creative people who aren't
obedient or subordinate. You have to be able to back up your disobedience with
results, though.

------
jbcurtin2
It just irks me when people come to me and offer me an idea, then say they'll
give me 1% of a company that I build all the software to it. Either I don't
understand why that happens or it's just plan greedy. I'd rather work in a
company that pays me well now while playing with my own ideas in parallel.

~~~
solistice
And likely, they'll want you to take over the financing and distribution, as
well as the marketing and accounting as well, because those are all hard
things they don't want to deal with. They just want to have the idea. And they
stay blissfully unaware that with just the idea (and maybe a patent pending),
it's be 3-10%....of wholesale price for them.

------
cateye
By good, it is often meant to be in the 5% of the standard normal
distribution. So "good" is an infinite shifting condition. When there are more
developers, it becomes just harder to be good enough. This scares therefore
often novice off. They implicitly think that they have to be "really" smart to
retain a life long job.

I think that there are enough developers. But everybody is searching for the
geniuses that can create a competitive advantage.

------
diego
For the same reason there isn't a glut of NBA players. Anyone can play, few
people can become good enough. Next question.

Edit: disagree? Downvote if you want, but please counterpoint.

~~~
jdminhbg
I didn't downvote, but programming is nothing like playing in the NBA. The NBA
is zero-sum: There are by definition (15 players per team * 30 teams) NBA
players in the world. There exists no such hard limit on "good programmers";
if there were more good programmers people would pay them to solve problems
that currently go unsolved.

------
beeffective
Why doesn't the pay reflect the supply and demand? I consider myself a decent
developer with 9 years of industry experience in several domains and my salary
is only $88k. It sounds like based on the demand, I should be pulling in a lot
more.

~~~
paulbaumgart
Where are you located? Geography is still a big factor in hiring decisions, so
demand is not evenly distributed. In Silicon Valley, $88k is what good-but-
not-great fresh college graduates get paid.

~~~
MatthewPhillips
The point still stands, unless you work for the handful (or less) of companies
that pay FU money to their engineers, you're still paid equivalent of what a
Dentist makes, and much less than what a good salesperson makes in a much much
smaller city.

I think the answer is that the demand isn't actually very high. Programming is
still considered a cost center for most companies, the quality of the product
is determined by how much they are willing to spend on salary, not vice versa.

~~~
paulbaumgart
Judging from how much attention all the good software engineers I know get
from recruiters and from conversations I've had with people actively hiring,
the demand in SF/Silicon Valley is incredibly high. I have personally taken
advantage of this demand through lucrative contracting gigs.

Sure, crappy companies in dying industries might not be able to compete, or
might not even realize that they're failing to compete, but why care about
them? They don't make the market for people who are actually good at software.

------
marcbarros
I"m not a software engineer, but I wish I had learned it in college instead of
studying business. Now as a startup founder I'm at a disadvantage because I
can't just start building products I see, I need a team just to start.

As the CEO of Contour it wasn't about pay for great engineers. It was about
having an interesting problem to work on and a company culture they wanted to
be a part of. Yes you compete on pay, but for a start-up, everything else you
offer is as important.

Growing up I didn't even know what you could do with engineering. I loved
math, but had no idea what you could do by being an engineer. I think a lot of
kids don't realize the creativity you can harness with engineering and how you
can "build" real things. It was always presented to me in terms of math and
being nerdy. Just like writing, I hated it because it was about "grammar" and
not about expressing yourself. Now I love to write.

For kids I think it is framed up wrong and until more engineers have kids and
teach their kids just what you can do, we will be short software engineers.

I'm now trying to teach myself with codeacademy and it's hard.

------
zampano
_Why wouldn’t the millions of unemployed Americans train themselves to code?_

I'd say this is exactly what my friends and I have done. We had been nerds
growing up but the majority of us went into liberal arts majors in college.
After graduating and finding out that our slips of paper were meaningless, a
few of us taught ourselves how to program and a few of us built upon college
IT jobs to go into system administration.

When we were younger, I think we generally had some pretty bad misconceptions
about the software development as a whole, especially regarding relative
difficulty compared to other careers. While the money and job security were
the things that drove (most of) us in that direction originally, we've all
become completely obsessed with a tech world that we generally misunderstood
when we were younger. I've spent a lot of time trying to convince other out-
of-work nerds to do what we did but I think most either don't believe you can
get work without a CS degree (by assuming that we just got lucky) or think
that it is well beyond their mental capacities. I'm not surprised a lot of
unemployed americans are scared of trying something intimidating, but I think
you'll see more and more young un(der)employed americans start learning code,
especially as the learning resources become more and more available and more
and more accessible.

~~~
numo16
I don't know about how other schools approached programming careers, but I
know my school tried to make it out like you had to be some crazy math genius
in order to do basic programming. Compound that with the fact that my school
didn't offer any programming classes for people to even see if it was
something they would be interested in, and we're left with me being the lone
software engineer in from my school for the past several years. They scared
everyone else away.

------
vampirechicken
Because businesses want to cut corners and ignore the good practices and
disciplines of software. Then they complain that it's the software engineers'
fault.

------
taylodl
_I am left to wonder how everyone could have been so wrong in predicting that
the world would be glutted with good programmers and sysadmins by now_

I stopped reading the article at this point. Seriously. I've been programming
for nearly 30 years and in that time no one ever thought there was a risk of
our having a glut of good programmers. Ever.

Hell, finding good talent back then was _much more difficult_ than it is now.
So stop your whining.

------
bigdubs
There is another force in the market that the author is missing; there are a
lot of quality engineers doing their own thing or starting their own company.

------
bradleyland
There is a tremendous difference between being able to "program" (the ability
to type out syntax that will execute) and being able to design large, complex
systems. Stop and think for a moment about the complexity in interaction
between components in a modestly complex software product. I know plenty of
people who can write code competently, but cannot design software very well.

------
ExpiredLink
> _Why isn’t there a glut of good software engineers?_

Because a career in computer programming (still) sucks:
<http://www.halfsigma.com/2007/03/why_a_career_in.html>

> _Programming is easier to learn than a foreign language_

Maybe 'coding' but not programming and definitely not software engineering.

~~~
waps
The advantage older coders bring to the table is full-stack understanding, at
least at the moment. Sure, maybe not as useful to bring your site off the
ground, but MUCH more useful for any kind of optimization.

------
scrame
Because most software engineers aren't very good.

~~~
CmonNoReg
...and they do not want to be better.

------
bengrunfeld
Because people great and small are afraid of programming. It is the big hidden
terror inside of a circus box that grown-ups use to terrify children and
village idiots.

Serious applications seem so complex with their millions of lines of code that
people are simply scared off.

I know that the industry is now trying to entice youth and professionals to
begin learning code with things like <http://code.org>, but they need to focus
more on the rewards (i.e. like how much money you can make, and job stability
even during economic crises).

Learning to code is hard, and you need to commit your brain to 3 years of
serious edumacation, and then 5 years of industry experience if you want it to
succeed.

That's a tough bill for anyone to swallow.

------
CmonNoReg
"As Google hires all of the world’s good software engineers" - probably an
exaggeration, but definitely not true. At least in UK salaries there are
lagging, only good for experience and to have it on your CV. Any company is
only hiring the best people they could identify of those who applied.

