
Why programming is difficult (2014) - deepaksurti
https://joearms.github.io/#2014-02-07%20Why%20programming%20is%20difficult
======
0x445442
I think there are a number of major problems with the industry.

Technology jumping based on anecdotal evidence of the efficacy of doing so.
Jumping around so often precludes a lot of practitioners from gaining a deep
experience and understanding with the tools they're working with. This may be
the only industry I know where pertinent knowledge and experience matters
little.

The notion of "Full Stack Developers" gives the impression that one can be
proficient top to bottom. What I find interesting about the label is full
seems to usually be bounded on the lower end at a place much higher than a
place where I would use the term full. This is one of the many areas in our
industry where hubris is in play. Specialization is usually preferable for
quality and scaling.

Real world business continually drives the fast and cheap legs of the stool to
be chosen by management. However, I think this grates against most
practitioners who not only care about quality but would admit, the aspect of
quality is one of the major reasons they are even in the field to begin with.
There is a large natural tension at play here. I think it's this tension that
Joe was articulating.

It's seems to me the most satisfied folks in this field are the ones fortunate
enough to be compensated for working within a bounded context where over time
they can gain more and more experience and knowledge. This naturally leads to
a growing confidence and a sense of control over the context with which they
work.

~~~
adventured
I've spent most of my adult life as as "full stack" developer, out of
necessity: I build the whole banana on my own and typically work on projects
on my own (about 3/4 of the last 20 years). I'm better at some aspects than
others and I enjoy some aspects more than others, of course (and I
offload/automate anywhere I can).

From the late 1990s until five or six years ago, it was fairly easy to keep up
with the latest interesting stuff in terms of full stack. Adjustments were
modest year to year. At this point I've entirely given up trying to keep up
with the very rapid fad switching that has been going on. Real full stack is
now only possible if you freeze pretty strictly across the stack and don't try
to keep up as the next fad rolls in for a given part of the stack. If you try
to keep up with all the fads now, it'll burn you out (and most likely make you
hate what you do as you realize how pointless the constant switching is).

I've found - as you'd expect - that abandoning the stack rat race hasn't had
any negative impact. Most of the latest fads are entirely unnecessary and
bring nothing to the table that prior numerous solutions didn't already
deliver toward building, launching and maintaining product.

~~~
mtberatwork
This is the other side of the coin. A lot of folks in the industry, especially
on the younger side, tend to believe they are operating at FAANG levels of
data and complexities. What happens is that you get convoluted, over-
engineered architectures and applications which really do nothing other than
pad resumes. The next time you are listening to a talk or reading a blog entry
that drones on about how complex their stack is, stop and think for a moment:
"Do they really have traffic/MAU levels that merit this approach? Are they
really operating on data that is PB scale or just a few GBs?" More often than
not, the answer is an emphatic No ... or "Can this application be distilled
down to a vanilla MVC app with a few cronjobs running?" Yes.

~~~
Mirioron
> _What happens is that you get convoluted, over-engineered architectures and
> applications which really do nothing other than pad resumes._

But padding resumes is one of those seemingly fundamental problems of any
industry (or bureaucracy). Many poor decisions by (middle) management come
down to padding resumes - when the consequences of bad decisions eventually
happen said managers are gone. Hell, you could even argue that a lot of higher
education is about padding your resume rather than learning skills.
Unfortunately, it seems that we have to live with the added baggage of
"padding resumes".

------
lmilcin
I think the reason programming is difficult is that really we are dealing with
incomplete information while the device that runs your program does not care
you have incomplete information and wants to screw you the first time you do
something not right in the slightest.

Programming is not just understanding how the device, language, libraries work
(ie. getting more complete information) but also limiting the ways the
adversary can screw you and limiting the fallout.

Programming is difficult because developers don't appreciate how important the
part of managing this incomplete information is. "I wrote this and it works!
Yeay!" And then it goes in prod and fails instantly because you did not take
this or that into account.

Some bright developers have the ability to manage this. But the problem is
that they can't work really well with other developers (at a typical corp) who
don't appreciate the problem the same way. This puts them in this untenable
positions where they can either do something by themselves, very efficiently,
or rely on extremely inefficient process if they want to work with other
developers.

~~~
hhas01
“I think the reason programming is difficult is that really we are dealing
with incomplete information”

Dealing with uncertainty and incompleteness is a core part of the job.

The problem is that far too many programmers are gloryhounds and cowboys who
think all that is beneath them; cramping their style when they’re trying to be
the coding heroes.

This is compounded by many programmers being lazy incurious bums who have zero
interest in doing anything that isn’t writing code. They don’t want to learn
the user’s problem domain, so they don’t. Instead they demand users describe
exactly how the software should work down to the very last detail; but if
users could do that for themselves then they wouldn’t need programmers to
write it for them!

Starting from ignorance, instead of writing software that encapsulates the
users’ expert requirements, they write clever code that solves their own
problems: entertaining themselves, and getting paid.

Writing code is not hard. Any monkey can do that. The challenge is knowing
_what code_ to write; and that’s made needlessly hard by programmers who can’t
be arsed to understand their users, what they do and why they do it, learning
their needs and problems in order to synthesize more effective solutions.

Instead of raising themselves up to users’ level of knowledge, they drag the
problem down to their level of ignorance and whale on it with their crappy
tools to give the illusion of “productivity”. Instead of being problem
solvers, they’re problem _multipliers_.

Slap on several more layers of equally useless management, marketeers,
beancounters, etc, and factor in that while deadweight accummulates expertise
walks, and it’s no surprise most software projects are rolling Katamari
disasters that makes _everyone_ miserable until finally put out their misery;
only for the whole process to start over again, with zero lessons learned.

\--

TL;DR: Programming is hard because programmers _make_ it hard. Because they
don’t want to do the job right. Because they’re children.

~~~
commandlinefan
> Programming is hard because programmers make it hard

This has been the MBA's wish/dream/hope for about a half-century now: “if only
programming were easy and programmers didn’t take so long,” they think, “I
could be making millions off of this business.” Since they want this to be
true, they reason, it must _be_ true. They then insist, in spite of mountains
of evidence to the contrary, that programming is easy, and the entire industry
of programmers - every single one, including the ones who were educated
recently, including the ones from foreign countries - are conspiring against
them to make it seem like it’s harder than it really is.

~~~
lmilcin
Also, MBAs assume then can hire anyone and then they complain the performance
suffers.

The truth is, development is a high skill profession. It should take decades
to master if we were really realistic about it.

------
imtringued
There is a reason why people make a distinction between "Programming" and
"Software Engineering". Programming is just the act of writing code. All you
have to do is learn a programming language and you are set.

"Software Engineering" encompasses everything else on top of programming. You
need knowledge of these things: build systems (Make/Gradle), filesystems,
networking (IP/TCP/HTTP), Data formats (JSON/XML), Databases(SQL/KV), software
architecture, layout/styling, documentation (both reading and writing). This
list is just scratching the surface but we can agree that all of these things
are critical to build applications.

These topics clearly aren't exclusive to "Programming" and knowing them
doesn't mean you are a "Programmer". You could also be a sysadmin that is
managing a large cluster for multiple applications. But if you are a Software
Engineer and not just a mere "Programmer" you know that your job is much much
more difficult than merely the "Programmer" part.

~~~
jaxx75
Is software engineering a real engineering discipline, though? I have a hard
time saying yes when "software engineers" make mistakes that are well
documented, no certification is necessary, and it isn't a legal definition.

~~~
kabdib
Man, the people who push certification systems on the software engineering
community really burn my biscuits.

I haven't seen a certification system that wasn't either a money-grab
(product-specific certifications, like Windows or Novell), a power-grab (by
someone like the IEEE, where the tests were half electrical engineering and
the software bits were garbage practices from the 70s and 80s), or a way to
artificially restrict the number of available practitioners (arguably another
money-grab).

No certification system is going to magically make large systems easier to
write or whisk away bugs. _Writing software is hard_. It's that simple. People
who have spent their entire careers writing software, from industry luminaries
that you read about every day to the really, really good engineers you've
never heard of, who are absolute wizards, these people are still writing
howlingly bad bugs because _writing software is hard_.

You can have your certifications and legal requirements for training and so
on, just don't pretend they will improve the state of the art. The best you
can do is hold people's feet to the fire after they make mistakes, and if the
best people in the industry can't be perfect, why would we expect threats of
punishment to fix anything?

Engineering is about designing and building systems, usually with cost, time
and reliability constraints. It's totally separate from whatever liability
framework is being enforced this week.

~~~
notabee
It's not about being perfect, it's about being fault tolerant, just like
regular engineering. I think especially for safety-critical situations (
_cough_ Boeing _cough_ ) there needs to be stringent, formal standards for
quality, review, and training. It should not be possible to cheap out or ship
a pile of duct-taped garbage in some situations, and that requires external
review by subject matter experts. (Not necessarily government bodies,
especially with the regulatory capture and brain drain going on right now.)

~~~
lliamander
I think kabdib's point is that the incentives of certification authorities
aren't aligned with the purported goal of the certification process. We
already have all sorts of certifications (in specific technologies, "agile"
practices, etc.) but I don't think most engineers see those certifications as
providing any value.

The only thing I'm aware of that seems to actually work is to hold engineers
accountable for results and _give them the autonomy to set their own process_.
I would argue that even very rigorous methodologies like NASA's Systems
Engineering[0] process match that description.

[0] [https://www.nasa.gov/connect/ebooks/nasa-systems-
engineering...](https://www.nasa.gov/connect/ebooks/nasa-systems-engineering-
handbook)

------
ptah
"Before I learned the art, a punch was just a punch, and a kick, just a kick.
After I learned the art, a punch was no longer a punch, a kick, no longer a
kick. Now that I understand the art, a punch is just a punch and a kick is
just a kick." \- Bruce Lee

------
TacticalCoder
Re- the Emacs config suddenly breaking... I do something getting more and more
common: I put my entire Emacs config in Git. It's a bit more work when
upgrading packages (because you have to commit them too) but then there's
hardly ever any breakage that's not one Git checkout away (I'd say anyway that
Emacs doesn't that often break at all: it's more the usual package
dependencies SNAFU of our trade that bites Emacs users once in a while when
they upgrade this or that package).

On a funny side note I had a very bad Emacs breakage recently and at first
couldn't understand why checking out the latest known correct Git version
wasn't working on my main machine. It was working on another computer (btw
copying my entire Emacs config is, well, also one git command away, so is
backuping). Turns out: for whatever reason the file containing the list of
files recently opened by Emacs got corrupted in a very strange way and that's
one of the rare Emacs file I'm not committing into Git. Took me a few minutes
to sort out.

~~~
PerryCox
I do the same with my .vim folder. You can place your .vimrc file in there as
.vim/vimrc and now everything is backed up in git. I've been working this way
for a few years now and can get setup on a new machine in a matter of minutes.

------
40acres
For me, programming is difficult because once you reach a certain level of
complexity it's very difficult to keep a robust enough model of the program in
your head.

TDD has really helped me in this regard because it forces me to start building
the model from ground up as opposed to in media res.

------
dragosmocrii
I find this article beautifully written and it resonated with me. Writing
maintainable code is damn hard for larger projects.

Also, I read a couple comments above, and I feel that people who wrote them
have no idea who Joe was. To me Joe is an inspiration, and I think his words
are usually not there to be debated ;), after all he cocreated Erlang, and
Erlang and the OTP is a masterpiece imo.

------
jkingsbery
"After a week of struggling with a broken LDAP sever written in C and badly
documented I had a lapse of memory and forget that my boss said and
accidentally implemented a server from scratch in Erlang during my lunch
break."

Joe was awesome, on so many levels. You have to love the healthy rebellious
streak and the talent to back it up.

------
commandlinefan
It's kind of the worst of both worlds: it looks like it should be easy, but
it's confoundingly difficult in ways that are impossible to see unless you're
actually trying to practice it.

~~~
wool_gather
Is this not true of most difficult things?

~~~
commandlinefan
When I look at the high regard most people hold, say, lawyers and doctors in,
who also do hard things (that are accepted as hard), I’d have to say no.

------
lbj
I agree that programming is difficult, but Im not sure he understands why he's
finding it so difficult.

> "I Google a bit and after a while find a posting where some poor unfortunate
> soul has encountered exactly the same problem that I have. My heart leaps
> for joy. My trembling fingers enter the magic spell that will remove the
> curse, and ... nothing. The problem remains."

and

> "I'm lazy, I'm a good for nothing slacker. But when I want to put a diagram
> into LaTeX I don't want to have to read a 391 page manual first. Now I know
> you will accuse me of laziness and of being of unsound moral character,"

I wouldnt go so far as questioning his morality. But if you're not constantly
focused on learning, even the minor details of the stacks you're working with,
you'll increase the difficulty enormously. He sums up the danger of that
attitude perfectly in his closing remark

> "Fortunately eleven minutes with the Google casino worked. The second
> suggestion of how to fix my problem worked - and I still don't know why
> emacs could not find aspell - and life is too short to find out why."

He's fixed it now. He has no idea what was wrong or how to arrive at a
solution. Next time he faces, even the same problem, he'll be back on Google
trying his luck.

~~~
fanboyism
>Next time he faces, even the same problem, he'll be back on Google trying his
luck.

There isn't a next time because Joe died last year.

I think you misunderstand the point. Time is truly limited thing you have in
life, and you are using much of that time understanding minor tool that are
not the major point of what you are trying to do, that doesn't seem productive
and is indeed infuriating.

About the final mark, those eleven minutes should even be needed, things break
unfortuntatly, and if who has deep understanding of every single tool you put
your hands on, you will not have the time to do whatever you need to do.

A few years ago, I had the "pleasure" of have to use docker on windows, on a
complex stack, the docker windows client was always breaking with the solution
always being revert to previous version, wait issue fix, update. Ofc, before
that we needed to waste some time to check if it was I who broke something or
if it was an update. The lesson was, only update sporadically, which also has
its downsides. Or should I instead have a deep understand on how docker for
windows client worker and submitting the fix myself? Do you know if I had the
time for it?

Too many things in our field are held by duct tape that is always tearing. I'm
also culprit of that. But saying that because tools not being newbie friendly,
or some tools breaking it's the fault of the user is not really a good a
approach. Being a programmer shouldn't be about fixing your editor every time
it breaks, or spend hours learning arcane tools that are not about
programming.

~~~
32gbsd
Webdev seems to be a duct tape paradise full of beginner tutorials that go
nowhere. And when the hard problems popup the suggestion is to google search
instead of KISS. I dont know somethings will always be difficult.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=7197255](https://news.ycombinator.com/item?id=7197255)

2016:
[https://news.ycombinator.com/item?id=11002731](https://news.ycombinator.com/item?id=11002731)

2019:
[https://news.ycombinator.com/item?id=19717877](https://news.ycombinator.com/item?id=19717877)

------
jblakey
Love this article! I don't think the guy is playing a 'persona' at all in this
piece. These are the kinds of things he thinks about on a daily basis. Heck,
these are the kinds of things I think about on a daily basis too, and I'm sure
most devs who have been writing code for a few years do.

In my daily work, I don't have time to understand everything. There, i've said
it. I'm not lazy. I'm not stupid. I just don't have time when I'm trying to
get a task/job/chore done. I appreciate good abstraction/encapsulation. A good
tool does this. A bad tool fails to do this. Same with documentation, really.

Don't get me started!:)

~~~
mdonahoe
FYI Joe died last year and is best known for inventing Erlang.

[https://en.m.wikipedia.org/wiki/Joe_Armstrong_(programmer)](https://en.m.wikipedia.org/wiki/Joe_Armstrong_\(programmer\))

------
contravariant
I feel like the constraints he added are a bit too restrictive, they would
force a program to be _perfect_ whereas for most circumstances it would be
enough for a program to be bug-free.

The real problem seems to be writing bug-free programs, which seems to be
difficult mostly because it is hard to specify what a program should do
formally (i.e. in code). Note that bugs are never about the program not doing
what it is told, but are always a mismatch between the program and the
programmers mental model of the program.

~~~
mint2
Not to mention, the program can be completely bug free but still terrible when
put in real life because any one of the 10,000 things about its intended use
was unknown or not prioritized.

------
yters
My experience has been we have a big pile of complexity with some way of doing
things. New shiny tech promises to eliminate this complexity. Perhaps it does,
but also introduces new level of complexity. Old complexity hasn't fully
vanished either, but now its problems have to be solved through the new level
of complexity, resulting in a further emergent level of complexity.

Fortunately, there is a new shiny technology that will make all this
complexity go away.

------
pts_
Oh but many people will say it's child's play and management is where it's at.
And then they you will have 10 people doing the work of 1.

------
gitgud
_\- The output of the program is beautiful._

 _\- The input to the program is beautiful._

 _\- The program is beautiful._

I think it would make more sense if the word _" beautiful"_ was replaced with
the word _" correct"_...

It would also help strengthen his argument for why programming is so
difficult!

~~~
dnautics
I suspect ie didn't specify correct because correctness is impossible; with
the following constraint (i'm surprised he didn't have as the last bullet
point): _and it must run on top of an unreliable subsystem_.

~~~
AnimalMuppet
On top of _several_ unreliable subsystems.

I think that's a big part of why it's so hard. You've got leaky, imperfectly
understood, unreliable subsystems, and you're trying to build the best system
you can on top of it. And what you produce inevitably is a leaky, imperfectly
understood, unreliable system... which becomes somebody else's subsystem. And
so it goes.

You say: We need to create perfect subsystems! Formal verification (or robust
type systems, or better requirements, or mathematical reasoning, or some other
approach) will save us! But it's not that simple. It starts clear down at the
hardware level. How much can you formally prove about your CPU? _After_ taking
into account it's errata sheet? The union of the errata sheets of _all_ the
revs of the CPU? If you're running on PCs, the errata sheets of _every CPU
that anybody could conceivably be running on?_

If you can't formally prove much about the CPU, how are you going to formally
prove the OS and/or compiler? And if you can't formally prove those, how can
you formally prove anything else?

It's leaky, unreliable, incompletely understood subsystems all the way down.

~~~
dnautics
All the way down to quantum mechanics, unfortunately.

------
quinnydinny
I think I'll start using the phrase "Google Casino" now.

