
Why Programming is Difficult (2014) - merlinsbrain
http://joearms.github.io/2014/02/07/why-programming-is-difficult.html
======
mbrock
In my experience, the overwhelmingly most difficult thing about programming is
writing code that makes sense, even after it has gone through a couple rounds
of requirements changes and bug fixes.

Any concrete coding task can be dealt with straightforwardly enough, but
projects as a whole tend to deteriorate by arbitrary fixes, maintenance
patches, and failure to uphold a conceptual schema.

The idea of composability is a good approximation to what it means for a
program to make sense. If any part of the program can be understood as a
sensible composition of parts, then the program probably makes sense.

But so many parts of real world code bases seem to end up incoherent. You look
at a function and instead of seeing code that obviously, say, maps the render
function over the list of widgets, you see three obscure conditionals, an
apologetic comment, a bug tracker issue number, two boolean parameters named
"force" and "dontRenderLast" (for some reason), and so on.

This is why I think the ideas from "domain-driven design" are so important,
including the notion of a ubiquitous domain vocabulary. Also the FP idea of
denotational semantics. And that well-known benefit of unit test driven
design.

There's not enough recognition and clear understanding of this problem, in my
opinion. We all know the problem, but somehow we don't talk about it enough,
and don't acknowledge it as an enormous problem for development speed,
programmer happiness, agility, and so on.

~~~
turaw
Do you think that sort of a problem (incremental changes eroding an initially
sound structure) could be fixed with better refactoring tools? It seems like
the issue you're describing is that logic is leaking from some parts of the
codebase to others, which might be fixable with sufficiently powerful
refactorings.

~~~
mbrock
Yeah, I think the sheer manual friction of moving things around is a big
impediment to change.

But also, I think there has to be some conceptual basis for us to be able to
think about factoring in a clear way, and that kind of intellectual labor is
hard on another level.

Eric Evans' book on domain-driven design has some examples of programming
tasks that generate confusing and unclear code, until the coder together with
the "domain expert" reaches some kind of epiphany—like "ohh, if we think of
these three things as being parts of the same kind of thing, then this part of
the program makes more sense."

A program is a kind of philosophical theory about a certain topic, and if the
theory isn't very clear, then you'll end up with a lot of special cases and
obscureness even if you have unit tests and refactoring tools. They say naming
is the hardest thing in programming and that becomes more true when you start
to introduce any level of abstraction.

"Design patterns" were an attempt to create more general small theoretical
pieces, above the syntactic level. In the FP world, a similar vision is that
algebraic and categorical abstractions can provide some of those pieces. We
still haven't seen much of what algebraic abstractions can do for ordinary
programming...

But these abstract pieces, I think, are pretty crucial if we want programs to
be understandable as anything but arbitrary collections of procedures and
types. So yes to refactoring tools, and yes to more discussion about design
patterns—it didn't end with the Gang of Four! read Christopher Alexander and
Richard P. Gabriel!—and yes to more inspiration from mathematics and
engineering and philosophy and all kinds of stuff, to provide inspiration for
factoring—yes to study of logic and language and ... I'm getting carried away.

~~~
amatic
Very smart comment. I've had a long brake from programming and all this
functional and lisp stuff is new, and it is not always clear what is the
benefit. But is great to see a diversity and experimentation.

~~~
johnmaguire2013
Lisp is actually the second oldest programming language (behind Fortran.)

1\.
[https://en.wikipedia.org/wiki/Lisp_%28programming_language%2...](https://en.wikipedia.org/wiki/Lisp_%28programming_language%29)

------
ksk
I don't think programming - in the sense of what 99% of programmers do - is as
difficult as programmers love to blog about. Most professional programmers
write stuff that is not all that hard, and most of it has been done by others
countless number of times. All the "hard" cutting edge stuff is done by a tiny
tiny fraction of programmers. Heck even the things people think are cutting
edge now, just taking a random example - VR, have been explored decades ago.
Seriously, go read some SigGraph papers.

However, programmers more than other professionals love blogging about their
work, which is great - because it helps spread ideas (more so than any other
field I've seen), but also creates this illusion that programming is this
super hard thing. I mean people complain about going to interviews and being
asked about algorithms and stuff that you could "just google" when, if you
were to ask an average scientist, the amount of active working knowledge they
have at any given time is simply astounding.

~~~
mbrock
There is another kind of "hard" though, which involves managing growing code
bases, doing programming within a business setting, and so on. We mostly let
the scientists work on well-defined problems in isolation, so they can really
focus on those deeply intellectually challenging things, but even if those
scientists are very clever I don't think they are necessarily good at larger-
scale system development.

There are lots of jobs that are harder than mine in some sense, but if you
look at the objective of building high-quality complex software in the minimum
amount of time, keeping it running correctly, and then being able to change
that software as effectively as possible, then that in itself is pretty much
an infinite problem. Maybe there is lots of scientific work to be done to help
us struggling coders, and maybe we need to get our shit together more as a
profession... It's not all easy.

~~~
rpgmaker
> but if you look at the objective of building high-quality complex software
> in the minimum amount of time, keeping it running correctly, and then being
> able to change that software as effectively as possible, then that in itself
> is pretty much an infinite problem.

The same can be said of almost any "high-quality complex" project in any other
profession.

~~~
mbrock
Agreed. It's interesting to compare software with other "intellectual labor"
like law, government, writing, architecture, philosophy, accounting, and so
on. Many of the same problems seem to come up. Programming as a profession is
very young, though.

------
xdc55
It is much easier to prove a particular program function correct when
complexity is properly handled, this is the true art of programming, handling
complexity in a way that makes sense.

Many complex phenomena in nature is explained by very simple mathematical
equations, likewise great engineering design is achieved when there is nothing
else to remove, and in order to achieve those results, a great deal of effort
and thinking has to happen first.

The thing is, as the author put out, the environment is just too hostile in
order to design beautiful programs, from the insane expectations of what can
be done, to incompetent peers, politics, distractions and what not.

If you are skilled, you can define a problem in a precise way, be able to
dissect it in smaller problems, solve them and prove the solution correct.

What is ultimately hard is how to handle the hostile environment you're
usually stuck with.

------
ScottBurson
_Programming is difficult because it is easy_.

Here is what I mean. Implementing new functionality where nothing existed
before _is_ easy: just figure out what you want the machine to do, and tell it
to do that. It isn't trivial, but on a relative scale, it's easy.

Precisely _because_ it's easy to create functionality, we do a lot of it. Our
programs grow. As they grow, they get complicated. As they get complicated, it
gets to be more and more difficult to understand what they're already doing so
that we can make them do something different. So, the less tongue-in-cheek
version of my thesis here is that _programming gets to be difficult because it
starts out being easy_.

This is why I always say that the essence of software engineering is _the
management of complexity_. And this is the part that people who have not
actually worked on large programs will probably never understand. (As someone
once said to me, "what you need to get is that management views programming as
primarily a _clerical_ function".) Viewed one-by-one, the tasks seem simple.
And yet, somehow, when you put them all together, they're not simple anymore.

~~~
engi_nerd
> This is why I always say that the essence of software engineering is the
> management of complexity.

Your point is valid, and you can delete the word "software" and make a more
general point that is also true.

------
sjclemmy
> Sleep. A lot of programming problems get solved while you sleep.

I love it when I go to bed mulling over a problem, and when I get up I have
the solution. The brain is truly a wondrous machine!

------
csydas
I think the author nailed it on the head with the Workplace parts about why
programming is difficult. The physical/socio-political environment that
someone is in compounds the normal issues in the programming process and I've
seen it drive quite a few programmers to straight up quit when the mountain of
issues became insurmountable. Without the freedom to tinker and explore a bit,
as the author bemoans, I think that it's when programming becomes a chore and

At my last job, I think the only time that I had ever seen our programming
team actually happy to work on a project was when by chance and via a minor
mutiny, we were able to break them away from the minutia of constant
maintenance and repair and let them actually just work on a project with no
expected outcome. It was an alternative path for our identity management
solution and the entire project was a challenge to the managers for the
Enterprise team. Our programmers were down-right chipper at the prospect of
just being able to flex their creativity and try something just to see if they
could do it.

~~~
Roboprog
I am thankful that at my current job, we do get some time to experiment here
and there.

Begin tangent:

I work for a university doing grant projects for a few state and federal
agencies, though. Some might like it, some might consider it hell for an
entrepreneur. It works for me :-)

We are in the process of re-writing our "flagship" app from 10+ year old java
libraries. We were able to dodge the bullet of "upgrading" (???) from Struts
to JSF. The new app is going to be a services based (REST + JSON) Java
(mostly) back end, with an Angular based front end. We just did a demo of a
subset of the new app at a trade show where a 3rd party app imports a partial
record (via web service) into our app, then launches a nested browser window
to run the rest of the relevant data entry. (the rest of the workflow will be
handled by the legacy app until the complete new version is finished)

No, I couldn't persuade my coworkers and supervisor to discard Java entirely
and go with Node.js :-) I'm not entirely sure I would be comfortable with
that, either. We are doing some server side Javascript lately, though, by
using the Java scripting engine interface.

We will likely soon be redoing another (30 or so year old) app from a sister
university that the state uses, as well. Interesting times!

For almost 5 years before this, I worked at a financial services company. I
got paid a bit more (bonuses), but I spent all my time, really, keeping 2
legacy applications going without ever getting to spend much time REALLY
updating some sore spots.

------
FreedomToCreate
The most important line from the author is "it takes time to be a good
programmer". Those nights of debugging and reimplementing will decrease as you
better understand how to develop code.

Also I find open concept offices terrible to work in. I'm not the type of
person who can put on a pair of headphones, blast music and pump out code. I
like solitude as I think though the problem and code. Unfortunately for me,
the open concept is here to stay it seems.

~~~
merlinsbrain
I love that line as well. I just told a friend of mine that if I spend the
next 10years writing 3 languages exclusively (one per domain of systems,
web/mobile application and high concurrency distributed systems) I would
hopefully probably be a "good programmer" as I envision it. That might be a
tall statement, but that's just _my_ interpretation of how it takes time to
become a good programmer, YMMV.

------
meesles
I see these posts every so often: "X is hard". Aside from just illuminating
the X, I don't really see the point of the posts. Most higher-order jobs are
difficult, that's why people train and educate themselves to work in the
field. Coding has it's own set of challenges, but I don't think anyone is
saying coding something slightly complex is easy. Just like how designing a
building is hard, or swimming an olympic time is hard, or solving a complex
science equation is hard. If I'm being honest, these posts come off a little
whiny.

------
WWKong
The bigger question is, why are we still programming the way we did 20yrs ago?
Higher level languages were a great improvement over machine language. But
what comes next? Why don't we have it yet? A decade ago I was setting up
database, writing queries, designing forms, writing the code to wire it all
up. I still have to do all these steps with more amount of effort today.

~~~
merlinsbrain
I think Backend-as-a-Service providers are a good step towards a place we want
to be, although I still think we're not super close to using such services to
deploy complex business logic on.

Services like parse cloud code and AWS lambda are another step closer.

Context: We (me + couple of collaborators) are writing a similar platform for
ourselves, although the purpose is dogfood more than $.

~~~
davidjnelson
Aws lambda does look like a particularly large innovation regarding complexity
management. Anyone have stories on using it to share?

~~~
room271
Nothing amazingly innovative, but I've used it a lot at work. For simple
jobs/event processing it is amazing as it removes a load of complexity
(logging, monitoring, scaling, deploying).

Lambdas tend to be < 100 lines of Javascript which means maintenance and
rewriting is trivial. And because the jobs are so simple people don't need to
be Javascript experts or aware of the JS ecosystem to write them.

Lambda is also very cheap if you have bursty or just very low-volume
workloads.

------
njharman
>code that produced it should be cleaned up and documented before the next
project was started.

A major why waiting to the end to test document and clean up code is a
failure.

Write beautiful, tested code from the start. Even throwaway code, because half
time prototypes, etc end up production.

~~~
gozur88
At most places I've worked there's not much point in documenting everything
meticulously at the start. The users don't know what they want yet, and they
won't think through the details until they get the first prototype.

What makes sense in testing and documentation is really project dependent.
What's right for flight control software is very different from what's right
for Yet Another Twitter Clone.

~~~
njharman
I mean programmer documentation, how to build this thing, why was this choice
made, what ideas were attempted and abandoned and why, what gotchas, what
shortcuts were taken/tech debt incured. A lot of this should be in commit
messages and/or "tickets" refed in commit messages. Rest goes into README,
code comments.

User docs are completely outside my realm as a software developer.

My point (and strongly held belief) is NO, when you do testing and
documentation is not really project dependent. All projects need to do it from
the start. Some, flight control software, need more rigorous process.
Everything needs to be viewed through ROI. My contention is too many devs
underestimate the return of "doing it right" from the start. and overestimate
the effort and effectiveness of doing it later/at the end.

------
aryehof
I think for those programming complex systems in business and industry, the
biggest problem remains how to represent concepts in code.

Even the simplest problem, like a Purchaser placing a Purchase Order for a
Product from a Supplier can't be easily modeled and programmed by the majority
of software developers - with any sort of predictability based on prior art or
their own experience.

At best, we approach a problem with a combination of modular procedural code
(modularity via subroutines)(from the 1950's) and structured programming with
functional decomposition (from the 1960's).

There is no shortage of technical prowess, but a warehouse system is not based
on design patterns, libraries, frameworks, or technologies. Instead, it is
based on people, places, things and transactions interacting together to solve
a problem.

How to represent real-world concepts and systems (real or abstract) into code,
remains the challenge it always has been. We just don't seem to learn any
lessons from the past.

------
namnam88
All good points, but this is not just specific to programming IMO. I work as a
sales engineer for a networking company and I face the same issues as you when
I work with customers who use our products. Documentation is never good,
you're always solving customer problems so you never really have time to dig
deep (read 391 page manuals are out of the question), new features are not
described correctly and when I try to lab things up (demo or otherwise) there
are about 100 things that can go wrong (I made up that number but you get the
idea).

Great article though- I think it applies to most technical roles.

------
shockzzz
_Now I find no particular joy in implementing archaic and perverse protocols
but often the quickest way to progress is to re-implement them from scratch._

whaaaa

------
jcoffland
The author and a number of commentors seem to be saying that incremental
changes eventually lead to tangled and unmaintainable code. My litmus test for
the quality of a code change is the amount of code it allows me to remove. If
a change results in many other parts of the code collapsing or getting simpler
then I know I've made a good decision.

------
njharman
I believe author means things like code, build system, etc when they say
"inputs are beautiful". Cause In my experience one of the major inhibitors of
beautiful code, what takes many tests, and what makes dev hard is how fucked
up input is. Esp user, as in interactive human, input.

------
dclowd9901
If David Sedaris was a software developer...

------
yasuo
It is not difficult, trust me. I know C, C++ , Java and Phyton. It is all
about logics and the only thing that differs is the syntax. Get in touch I
might help you

~~~
njharman
Too many guys like this are what makes software dev hard. Too many yahoos who
think its just about code. Code is only an implementation detail of how most
dev is done today. Even today, code isn't about the code. It's about
explaining your program, your process, wtf you where thing when designed this
thing to the next guy/gal.

Sadly, what too much code tells us is "Design? That's what gui people do,
right? I just kept typing stuff until it worked enough for non-technical
person to think it was OK or we ran out of money/time."

~~~
dclowd9901
So much this. My cohorts and I aren't spending hours talking about why we
should use es2015 or typescript or whatever else. We're spending hours nearly
coming to blows over how to properly implement a Flux architecture.

