
Things I've learned about writing software after 12 years - landongn
https://medium.com/@landongn/12-years-later-what-i-ve-learned-about-being-a-software-engineer-d6e334d6e8a3
======
lordnacho
There's really one issue here. Budget.

You want two pairs of eyeballs on everything as you write it? Okay. Pay two
guys.

You want complete test coverage? Okay. Can take as long as writing the code
itself. Maybe more.

You want full documentation? Okay. Guy can't spend as much time on coding.

You want new APIs to be included? Okay. Guy's got to read docs and try
samples. Less time.

What I've seen is you always end up with someone noticing that you could just
spend all your time coding, and you would still produce an output that was ok.
At least for small projects, or the start of large ones. It's as if there's a
true productivity that's much lower than the imagined, and whoever is setting
priorities doesn't notice it. He only sees code that's been written. He
doesn't see technical debt. He doesn't see when a test catches a budding
error. He doesn't see what that second pair of eyes is for. He doesn't know
why you're always reading about some new thing.

~~~
chii
which is why a software project requires someone with technical knowledge
doing the management, and not an MBA or "project manager" who cares only about
on budget and on time at the cost of technical quality.

~~~
Too
If you know the management is not technically competent, the responsibility
also lies on the developer to communicate things on their level. If they ask
how long will X take, you don't answer 1 day, and then another day to test it.
You answer 2 days. If that's outside their budget then forget about that
feature and let sales find a new feature they can sell that can be developed
within the budget.

This also boils down to the ego thing, many junior developers are proud to
give short estimates to show off how quick and good they are at their work.
But those short estimates are always always always just the estimated time to
get a quick prototype working, where things are configured as you go in the
debugger, not something that is robust and will work together with the rest of
the application in a customer deployed environment. Estimation is really where
you can see the difference in an experienced engineer and a non experienced
one.

------
Kenji
We get the software we deserve. Apparently, people aren't ready to spend more
money on performance, reliablility and security. When they buy software, they
pay the price of a giant ball of mud (if they aren't expecting it to be free!)
and they get a giant ball of mud. Simple. You call it insanity, I call it
market, supply and demand. Of course, the perfectionists are unhappy about
that (being one myself I can certainly empathise) but you have to realize that
this kind of perfection in software costs huge amounts of money and requires
the most skilled workers in the field.

~~~
Animats
_" We get the software we deserve."_

Painfully true. There are techniques for writing better software. They work.
They take longer and cost more. They are not widely used outside aerospace.

~~~
dgemm
Is there a high level overview somewhere I can read? Don't even know what to
google here.

~~~
throwawayf2o
[http://spinroot.com/gerard/pdf/P10.pdf](http://spinroot.com/gerard/pdf/P10.pdf)
(some are more/less appropriate for non-embedded-systems)

In general, the answer typically involves formal specification and formal
methods that check the code against these specifications, combined with
testing and coding standards that result in analysable code.

More references:

[https://www.cs.umd.edu/~mvz/cmsc630/clarke96formal.pdf](https://www.cs.umd.edu/~mvz/cmsc630/clarke96formal.pdf)

[http://research.microsoft.com/en-
us/um/people/lamport/tla/fo...](http://research.microsoft.com/en-
us/um/people/lamport/tla/formal-methods-amazon.pdf)

------
stinos
_Unless you’ve written a buggy program, you don’t realize that you’re
addressing our intellect. This is why I think that every engineer on the
planet looks at a bug report and feels a twinge of pain as they read whatever
detail that was left to serve as a figurative shame sticker on the report card
of their creation. It really sucks when you’re just flat out wrong.

Being wrong — rather, being incorrect — is an extremely humbling experience.
The catastrophically incorrect, which is when software crashes, money is lost,
or the absolute worst, data is stolen, is the kind of thing that makes you
question your career choice. It makes you want to curl up into a ball and weep
at how completely fucking stupid you were when you’ve found the problem._

Just had to quote this - I urge people who stopped reading or down't want to
read the article to read this anyway. I've read a lot on developping but I
think it's the first time I saw someone putting it like this. And oh boy, is
he right. At least for me. Every bug report (well, the ones which points to
something I obviously fucked up) hurts. What hurts even more is the dreaded
_reopened_ because I fucked up again. Especially because sometimes that means
the whole set of classes surrounding the bug are just textbook examples of all
code smells in the universe. And the _only_ thing that can be done about it is
the nearly impossible _write good code 100% of the time_.

~~~
greggman
I have a slightly different take. That used to be my initial reaction and it
probably still is but taking a cue from other engineers better than me they
seemed to internalize that there will be bugs, period. So might was well get
over it and just fix them as they're found rather than get upset over them.

Of course they try to write good code, follow good practices, write tests etc
but there's just going to be bugs, period. So don't beat yourself up. Just fix
the bug, learn whatever you can from it so hopefully you won't do it again and
or if it calls for it adjust your build infrastructure or testing
infrastructure so you're more likely to catch them in the future.

~~~
landongn
"Just fix the bug, learn whatever you can from it so hopefully you won't do it
again"

The baggage associated with bugs is so prevalent that we can't talk about them
without a real sense of shame. "Hopefully you won't do it again".

Bugs are just part of the cycle. I completely agree with your assessment that
people who can realize that are happier and more productive because they can
plan for the probable case.

------
jquast
From Edsger Dijkstra:
[https://youtu.be/RCCigccBzIU?t=13m54s](https://youtu.be/RCCigccBzIU?t=13m54s)

> ...just after the first successful moon landing [...] I knew that each
> Apollo flight required some 40,000 new lines of code. [...] I was duly
> impressed that they got so many lines of code correct. So when I met Joel, I
> said, "how did you do that? [...] Getting that software right." [...] He
> said that in one of the calculations of the orbit of the lunar module the
> moon had been defined as repelling instead of attracting. They had
> discovered that error by accident. Imagine, by accident five days before the
> shot.

Sound familiar to anyone?

Some 16 years experience myself, the clarity of "the problem of time" has
become the foremost factor of bad software that I knowingly write. I imagine
NASA has the same problems that we do.

I've been through about 5 cycles that all occur in the same form: 1) We put
the under-performing developers on maintaining "legacy" systems and the
seniors begin on the next one. 2) Time is purposely estimated shorter than
viable, corners get cut in the software, prototypes are put into production
once again, and a new legacy system is born. 3) The good programmers leave,
some juniors become seniors and others are hired, and the cycle repeats.

There are many additional mismanagement mistakes that accumulate, but limited
time always seems to be at the center of the problem for me. I whole-heartily
believe that really great software can be authored by just two or three high
quality engineers over the span of a few to many years with a clear vision.

~~~
sopooneo
I agree, but I think "with a clear vision" is the rub here. That is possibly
the real impediment in all these cases.

------
krick
Many people here already acknowledged that this text misses some details, but
I'll put it even harsher: even though it's interesting to read, in fact it's
just completely wrong and somewhat misleading.

Author starts by mentioning huge, complicated and ambitious projects and
pointing out how seemingly insignificant mistake makes all difference between
"it works!" and "it doesn't work!", which sure will sound familiar to what we
experience every day, but on different scale. With this impression he moves on
to talking about modern software in the industry and at home, and how we fail
to write _correct code_ and why.

But in reality, nobody (except the programmer himself, maybe, if he fails to
see the bigger picture) gives a fuck about _correct code_. Glitches and errors
are ok, if they don't make software unusable and prevent us from achieving
some global business-goal. It's ok for landing-site to have a CSS bug, that
makes some button look weird in certain situation, unless it really annoys the
customer and we lose sale — the chances are that nobody even notices, and it
may be fixed next week or even never. It's ok to put your business-critical
back-office software on production and find out that it contains some really
nasty bug, which you'll fix in a hurry in the next 20 minutes. It's ok for
Ubisoft to spend millions of dollars on production and then ship game with
bugs, which will be fixed with the next patch-release. It's ok to have stupid
over-complicated networking protocol, that might have been 5 times more
efficient, but still is usable and allows people to do something they couldn't
have done before.

What is _not ok_ is to spend millions of dollars on development and then never
ship the product. It is not ok to let your client to find a solution which
works for him, but is cheaper. In a word, whatever actually hurts your
business is not ok. And the truth is that cheap solution that is "somewhat ok"
is usually better, than the expensive "good solution".

------
makmanalp
There's a sentiment in the article I see a lot that keeps coming up these
days: "NASA sent a spaceship to the moon with something 100x slower than my
computer, so why does it take so long to load a page"?

I think it's entirely unjustified. What makes modern software engineering
difficult is that it's one of the few engineering disciplines where not only
are you making a Boeing 747 fly, and all the parts were made by different
people in different locations, but you're also replacing the engine _in
flight_!

Also the specs on the parts are kind of spotty. They weren't agreed upon by a
single group of people, and they certainly weren't decided by you. Some of the
parts are decades old. There are parts in there that are older than most of
us!

Often, the parts decide to change how they work. They don't ask you whether
they should. Sometimes, also, the parts decide to disappear. It's up to you to
figure that out.

In fact, there aren't really specs on how the parts go together, come to think
of it. None of the people who are building this are talking to each other. You
could say that most of it is kind of emergent. There are no top-down quality
controls.

There are layers upon layers of abstraction, and no one knows all of it. No
one even knows most of it. In fact, most people don't care about any of it
other than a cubic centimeter of their own few layers.

And it works - by golly it works. And a person with no training can
absentmindedly navigate all this by flicking their finger about while walking
down the street and drinking their coffee. IMHO, this is a miracle in
proportions that are indescribable. The moon landing really was nothing
compared to this. So yes, there's some overhead involved in getting that to
work. :-)

~~~
jared314
> The moon landing really was nothing compared to this.

I used to believe that, but I have slowly moved to the other side.

Things are complicated. There is no doubt about that. But, landing on the moon
was a fight against nature (gravity, air, distance, etc). Current programming
is a fight against stuff someone else dreamed up and no one ever fixed,
because "LOL, that's old school" or "You're just doin' it wrong".

Most of the points about parts not being specified correctly, not working as
expected, or disappearing happens all the time in other industries. (That's
one reason hardware kickstarters fail so often.)

~~~
XorNot
Except it's an applies-to-oranges comparison to start with. The mathematics of
space-travel is _very_ simple - it's just calculus. You need to be accurate,
but it's easy to quantify and well controlled.

Which is why you can go to the moon with a decent graphics calculator, but you
definitely can't have a self-driving car with much less then a modern
supercomputer.

~~~
jared314
> The mathematics of space-travel is very simple - it's just calculus. You
> need to be accurate, but it's easy to quantify and well controlled.

We're not talking about calculating our way to the moon. We're talking about
going to the moon.

The space program had much more than a graphing calculator. They had buildings
full of people, systems, and realtime communication with the craft.

------
robmccoll
The main thing that resonated with me is keep your software simple. Don't try
to be cute or clever. Don't whip out your favorite OO design pattern when a
simple function will do. Don't try to optimize the function you spend 1% of
your time in. Follow Unix design principles. Simple is easy to maintain, easy
to debug, easy to learn, easy to understand. Impress your peers by writing
good code that works well rather than code that only you understand (and even
then you will have to relearn when you come back to it in six months).

~~~
dba7dba
If I may dare to add:

... writing good code (that others, including yourself 1 year from now, can
maintain easily) that works well ...

------
bch
It's a shame the article is so egotistical, hyperbolic, and poorly edited.
Also, not crediting obvious sources[1] for your punchlines smells like
plagiarism.

Otherwise, I _really_ _do_ like reading articles like this as touchstones for
engineering and development.

[1] [http://blog.codinghorror.com/coding-for-violent-
psychopaths/](http://blog.codinghorror.com/coding-for-violent-psychopaths/)

~~~
landongn
Credit where credit is due, that was lifted nearly wholesale from
codinghorror. Wonder how you do liner notes with Medium. I'm a terrible
writer, so the rest of the items I've stolen or otherwise isn't to stand on
the shoulders of giants like Atwood, it's to reinforce a point the best way I
knew how.

I lifted all of the comments from stackoverflow, too:
[http://stackoverflow.com/questions/184618/what-is-the-
best-c...](http://stackoverflow.com/questions/184618/what-is-the-best-comment-
in-source-code-you-have-ever-encountered)

I'm curious where the egotistical portions were, though. I wanted to at least
prefix some of the more "thou shalt" sections toward the end with a disclaimer
that they're personal opinions on how I operate.

Glad you liked it, though!

------
jordanpg
Excellent piece of writing. Very funny at times, witty like a satirical
novelist. Particularly enjoyed the anecdotes from NASA and the author's
conveying of the powerful emotional context of high stakes programming.

I _pine_ for a golden, future age of programming where documentation is as he
describes.

> This image brought to you by preventable catastrophe, budget overruns,
> political wrangling, infighting, incalculable complexity, and against it’s
> best effort, humanity.

Beautiful caption. The dismal history of NASA in one sentence.

~~~
gibrown
"dismal" \- that's an absurd word to apply to an organization that has
successfully enabled humanity to explore the solar system over the course of a
short 50 years.

There is plenty to Monday-morning quarterback, but "dismal" is not the word I
would ever use.

~~~
jordanpg
You're reading me wrong. Dismal means sad, and the caption is meant to be wry
or ironic.

------
ninjakeyboard
Budget as an excuse for poor software seems like a bad argument. There are as
many people talking about how much more expensive it is to fix bugs late in
the game as there are people pressuring teams to deliver shit code on tight
timelines. It costs less to write better software. Invest upfront and rejoice
that you did later or rush it out the door now and pay to maintain it forever.

------
jdiez17
The ideas behind this article are good. The author gives some guidelines that
have helped them write better software. However, it makes me think about how
we can make "making software" better. It seems like the best current way of
going about it is to implement these zen-like self-discipline techniques that
have helped the OP.

What's so inherently wrong about software? Why is it so damn difficult to
write software that doesn't explode if you don't hold your tongue at the right
angle?

~~~
JustSomeNobody
"Why is it so damn difficult to write software that doesn't explode if you
don't hold your tongue at the right angle?"

Hubris mostly.

~~~
dasil003
I don't know, it takes a certain amount of hubris to write any software at
all.

------
ams6110
The Big Ball of Mud is not unique to software. Ever been in a building that's
had a lot of renovations and additions? There are weird half-flights of stairs
here and there, convoluted paths from one side to the other, different styles
of fixtures, etc. Eventually they get to the point where it's easier and
cheaper to just build a new building somewhere else, or tear the old one down
and start from scratch. Just like we do with software.

~~~
landongn
Absolutely true. Shantytowns are a common parallel to big balls of mud, and I
think that would hold true for anything that gets constructed.

------
McP
It was a good rant until he encouraged writing a test after writing the code
to be tested :( As a developer also with 12 years experience, I implore you to
write your tests _first_. Otherwise your organisation will almost inevitably
end up with brittle tests & it will often cost more to patch up the tests when
they break than the cost of the possible increase in genuine bugs. I don't
necessarily espouse slavishly adhering to TDD (it works for some devs, not
everyone) but experience has taught my heart to sink every time I hear someone
say "I've implemented the feature, now I just need to write tests."

------
mod
Leave it to me to count the zeroes:

> It contained a defect of effectively point zero zero zero zero one inches.
> that’s 0.000001.

------
CyberDildonics
Sean Parent would bring up the fact that given enough scale all systems end up
being a network problem. If you look at the internet, you can think of it as
one big system. Huge and hideously complex, its pieces are connected but not
dependent.

If you can boil your software down to parts that can afford latency (even if
they need throughput), you can pass messages between them. That is possibly
and probably less efficient, but your macro pieces that pass well defined
messages between each other can exist without adding to the complexity of the
other pieces.

------
flatfilefan
To the point of automation, my credo is to automate all hairy stuff because
you spend your focus and frustration budget on that otherwise. The rule of
thumb is more like "if you can't make this your daily priority but need to do
it repeatedly take a day and make it a nobrainer"

------
SubuSS
Great article. Perhaps because I suck at grammar, I didn't even notice the
language issues. Interesting that HN is getting bogged down analyzing grammar
rather than the actual content. Were the mistakes so bad that they detracted
you away from the message? FWIW I was able to comprehend it just fine.

One interesting issue I see is with specific documentation rather than 'self-
documenting code': Considering docs don't compile, I have seen (in my 13 years
;) that docs tend to drift apart from the actual code and become more of a
PITA than the code itself. At least once I have caught myself down a wrong
path because I trusted the docs.

I don't have anything against docs though - write a bunch myself: I am very
circumspect against trusting it completely though.

~~~
andrewflnr
Yes, they really do detract from the message. They're a jarring distraction,
like a skip or buzz in a piece of music. And sometimes even when you think
you've comprehended it fine, you and author are thinking about different
things because you silently filled in a gap with the wrong idea (can't say for
sure if there were cases like that in TFA)

------
pronoiac
"It’s like creating Frankenstein and then realizing that you didn’t bolt the
legs on. After you’ve hit it with lightning and given it life. While it’s
attempting to walk."

------
damian2000
Inexperienced developers would benefit most from reading this, but its also
harder for them to put any of it into practice ... in your first job you're
usually just scrambling around day to day trying to get a handle on the work,
and also to fit into the workplace culture.

------
nerdy
Wow. It feels like I wrote that article I agree with it so much. There are a
few things I disagree with:

1\. I'm not a TDD purist but I do think a red/green TDD cycle does strongly
encourage all of the following things to happen in a more natural, flowing
way: "Tell yourself what you’re going to do, and then implement that sentence.
When you’re finished, take that sentence and write a test, then just move on."
Coding before testing isn't the issue as much as the potential for needing to
change code just to be more testable (an efficiency issue).

2\. No sale on the idea of documenting every piece of code; I truly feel like
doc comments muddy the contents of the file, plus resulting documentation is
less likely to be read than code and more likely to fall out of sync with the
contents of the code. Completely separate documentation is hard to keep
updated. It's often something to maintain with often no tangible benefits
unless you are churning developers like crazy. Certainly it might be
reasonable to document high-level interactions for complete view of the system
but documentation at lower levels seems an increasingly bad value proposition
to me. In a perfect world, sure... but in real life there are deadlines and
tests+code almost always serve as enough of a developer guide to understand
clearly. If not, I'd rather spend time refactoring instead of writing docs.
For the few new developers brought on, reviewing the system in a paired
programming manner is a good way to get to know them and ramp them up at the
same time. Parts of the system that are difficult for them to comprehend might
warrant refactoring attention.

3\. "If something is performed more than three times in a 4 hour span" It's
difficult to make a wholesale suggestion on things like that. If said task is
1 second clicking the "get mail" button and automation is auto-checking every
n minutes, automation can be a net loss in terms of productivity. I think the
XKCD[1] chart is probably a better measure ;) Anyhow it's definitely better to
not waste loads of time doing things you do often, and the more often you do
something the more likely it is that automating the task would pay off; just
don't do it with an utter lack of discretion.

Some typos in the post: \- Stray period before "I just call it by what it was:
A big ball of mud." \- "Simple isn't sexy." should maybe be "Complex isn't
sexy" or "Simple is sexy" \- "internet" (proper noun) \- "explaining just how
the interact with" s/the/they \- "english" (proper noun)

Very thought-provoking and fun read (lots of laughs)- I really look forward to
future posts!

1\. [http://xkcd.com/1205/](http://xkcd.com/1205/)

------
dmkram
12 years? Sorry to break it to you, but that's a short time.

~~~
bprater
12 years is the middle years for a career programmer. Enough to know what
patterns aren't serving you, and enough to know that there is still a bunch to
learn.

~~~
marssaxman
Middle years!? How long are these "middle years" supposed to last? I'm 22
years into my career and there's no way I've hit the mid point yet. It still
feels like a joke to have a job title with the word "senior" in it.

~~~
will_work4tears
How long have you been programming full time though? I find too many people on
HN are 25 and count "15 years experience" because they started programming at
10. It's just not the same thing, especially when most people only count
professional experience (and full time at that). 15 years of programming
experience < 10 years full time experience in an enterprise or startup
environment.

~~~
marssaxman
I'm counting from when I dropped out of college and started working full-time.
Maybe only 21 years ago, it's a little hazy now. I'd been writing code for
seven or eight years at that point.

~~~
will_work4tears
It's pretty impressive, then, and nonetheless, I'm just a lot less impressed
when somebody says they have 10 years experience and it equates to 2 years of
full time, enterprise or startup experience, and 8 years of "I'm going to
school full time and program 3 hours a night." I did program and experiment
when I was in my teen years too (late 80s, early 90s) and still cringe when
people count that as actual experience. Not that I'm constantly learning and
growing at my full time job, but meetings, documentation, paperwork, etc are a
large part of the job and count more than not having to bother with it.

------
the_cat_kittles
being wrong is ok! try not to hate it! try not to let the fear of it guide you
in any way!

------
radius
Not sure what all the hate is about. He presented some interesting anecdotes
about the quirks of software engineering and then tied it all together with
some concrete lessons.

Maybe 5K words without any sort of lists or clickbait titles is just too much
for the average reader.

~~~
walshemj
Or writing at higher level than a 12 or 13 year old :-)

~~~
courtf
Beyond the grammatical errors, misplaced punctuation and misspellings, and
despite the relative novelty of the anecdotes, I thought the writing itself
was dull and simplistic. Many paragraphs were just a string of blunt
statements without any independent clauses.

According to a copy+paste of the text into
[http://sarahktyler.com/code/sample.php](http://sarahktyler.com/code/sample.php),
the article as a whole reaches Flesch-Kindaid grade level 7.

~~~
JustSomeNobody
So, it was written such that a 7th grader could read it. How is that a bad
thing?

Edit: I pasted a sample of John D MacDonald in there and it showed a level of
6.61. Poor John, no wonder he was never a success. /S

~~~
courtf
Your comment isn't really addressing mine from the context in which it was
given. My comment was in reply to someone implying criticism of the piece
stems from its reading level being above that of a 12 or 13 year old, which
isn't the case.

I'm simply proposing that the criticism is founded on actual flaws with the
piece, rather than it being too complex to be understood by its detractors.

~~~
walshemj
It was what your comment implied to me - that is some one with a low level of
English reading comprehension who needs short simple sentence structure.

------
acjohnson55
I got halfway through this article and then gave up without any idea of what
the author learned. Wish the author would have edited this post better.

~~~
mrits
I stopped at about 5 words in with the oddly placed comma and capitalization.

~~~
RussianCow
You stop reading things as soon as you encounter an "oddly placed comma"?

~~~
mturmon
Like gcc, one misplaced comma and the whole thing might as well be rubbish.

~~~
koja86
-Wall -Wextra -Werror -pedantic

------
CmonDev
Don't forget to thank nazis, who taught you how to build rockets and soviets
who showed you that space travel is possible.

