
Organizing complexity is the most important skill in software development - speg
http://www.johndcook.com/blog/2015/06/18/most-important-skill-in-software/
======
msandford
This is incredibly true. I once turned 60kLoC of classic ASP in VBScript into
about 20kLoC of python/django including templates. And added a bunch of
features that would have been impossible on the old code-base.

It turned the job from hellish (features were impossible to add) to very
nearly boring (there wasn't much to do anymore). So with this newfound freedom
I built some machines to automate the data entry and once that got rolling the
job got even more boring. Because it was a small company with a very long
learning curve the owner didn't let people go, but instead kept them on so
that he didn't have to hire and train new people as growth accelerated.

But with all the automation some slack found its way into the system and
problems that had normally required me to stop working on my normal job and
help put out fires now got handled by people who weren't stretched a little
too thin.

Sadly there's (seemingly) no way to interview people for this ability so we're
stuck with the standard "write some algorithm on a whiteboard" type problems
that are in no way indicative of real world capabilities.

~~~
moron4hire
I got fired from a job because I had done this and they couldn't figure out
how to keep me billable after that. Well, I got fired because I would "show up
late" and "leave early" because there was nothing else for me to do (even
though I was actually pulling a full 40 hours, they were expecting 60 out of
people).

They could have given me more work, put me on new projects, etc., like I kept
asking to be done, but instead they just wanted me to figure out some way to
bill the client for 60 hours a week, and not call it "bug fixes" or "new
features", because the client wasn't paying for it. That left data entry,
which I had automated from a whole day to 10 minutes. They wanted me to go
back to entering data by hand.

I would have quit then, but I didn't want to give them the pleasure. A flood
had recently destroyed all of my stuff and I used the insurance check to get
out of debt rather than replace the lost stuff, so I suddenly didn't need
money for a LONG time. It's funny how easily debt can make you a compliant
little slave towards people who want you to perform bullshit tasks.

I've been freelancing ever since, making twice what they paid me for only half
a week's worth of work. And the flexibility of the freelancing enabled me to
pursue a long-distance relationship that eventually lead to marriage. So I say
getting fired was the greatest thing that ever happened to me.

~~~
xacaxulu
Great takeaway. Debt tends to make 'good little workers'. Stay free my
friends!

~~~
moron4hire
I think P.T. Barnum wrote the prototypical "Self-help:
finance/entrepreneurship" book in "The Art of Money Getting: Or, Golden Rules
for Making Money" [0]. A lot of it sounds like the stuff you hear getting
hammered to death on motivational TED talks. It's a pretty short book and he
doesn't go into too much exposition on a lot of the issues he addresses, with
the exception of debt. He probably spends half of the already short book on
talking about the evils of debt, especially debt put into things that aren't
assets. It's not the debt that is the problem, it's the mindset you have to be
in to take the debt, the mindset that the debt puts you in, and the mindset of
the type of person that continues to use debt, especially destructively. There
are some closely related issues with spending money, which tie in nicely.

It's an interesting read. This book was written in 1880, and if not for the
dated writing style, would fit right in to the modern day self-help genre. It
really helps underscore the idea that there's "nothing new under the sun."

[0]
[http://www.gutenberg.org/files/8581/8581-h/8581-h.htm](http://www.gutenberg.org/files/8581/8581-h/8581-h.htm)

------
knodi123
Mirror since the site is currently down:

The most important skill in software development

Posted on 18 June 2015 by John Here’s an insightful paragraph from James
Hague’s blog post Organization skills beat algorithmic wizardry:

When it comes to writing code, the number one most important skill is how to
keep a tangle of features from collapsing under the weight of its own
complexity. I’ve worked on large telecommunications systems, console games,
blogging software, a bunch of personal tools, and very rarely is there some
tricky data structure or algorithm that casts a looming shadow over everything
else. But there’s always lots of state to keep track of, rearranging of
values, handling special cases, and carefully working out how all the pieces
of a system interact. To a great extent the act of coding is one of
organization. Refactoring. Simplifying. Figuring out how to remove extraneous
manipulations here and there.

Algorithmic wizardry is easier to teach and easier to blog about than
organizational skill, so we teach and blog about it instead. A one-hour class,
or a blog post, can showcase a clever algorithm. But how do you present a
clever bit of organization? If you jump to the solution, it’s unimpressive.
“Here’s something simple I came up with. It may not look like much, but trust
me, it was really hard to realize this was all I needed to do.” Or worse,
“Here’s a moderately complicated pile of code, but you should have seen how
much more complicated it was before. At least now someone stands a shot of
understanding it.” Ho hum. I guess you had to be there.

You can’t appreciate a feat of organization until you experience the
disorganization. But it’s hard to have the patience to wrap your head around a
disorganized mess that you don’t care about. Only if the disorganized mess is
your responsibility, something that means more to you than a case study, can
you wrap your head around it and appreciate improvements. This means that
while you can learn algorithmic wizardry through homework assignments, you’re
unlikely to learn organization skills unless you work on a large project you
care about, most likely because you’re paid to care about it.

~~~
jaytaylor
Here's an archive.org cache:
[https://web.archive.org/web/20150622134205/http://www.johndc...](https://web.archive.org/web/20150622134205/http://www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/)

------
bbotond
The site seems to be down.

Cached version:
[http://webcache.googleusercontent.com/search?q=cache:Mf9074z...](http://webcache.googleusercontent.com/search?q=cache:Mf9074z6m7sJ:www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/+&cd=1&hl=en&ct=clnk&gl=us)

~~~
Djehngo
mirror (text for the sake of people trying to find this with ctrl+f)

------
edw519
_the number one most important skill is how to keep a tangle of features from
collapsing under the weight of its own complexity_

Agreed.

 _very rarely is there some tricky data structure or algorithm that casts a
looming shadow over everything else_

Agreed, BUT...

In order to organize, sooner or later, you will have to get clever (with
tricky data structures or algorithms).

How I have always built something big and/or complex:

    
    
      1. Add lines of code. 
      2. Add lines of code.
      3. Add lines of code.
      4. Holy shit! What have I done?
      5. Refactor.
      6. Combine similar sections.
      7. Genericize with parameter-driven modules.
      8. Still too many lines of code! Optimize Step #7 with something clever.
      9. Go to Step 1.
    

2 years later: What the hell was this clever parameter-driven multi-nested
process for? Why didn't I just code it straight up?

For me, organizing complex code has always been a delicate balance between
readibility and cleverness.

Don't remove enough lines of code and it's too much to navigate. Remove too
many and it's too much to comprehend.

Organization + Cleverness + Balance = Long Term Maintainability

~~~
onion2k
_Still too many lines of code!_

Being organised has nothing to do with fewer lines of code. A 10 line script
can be disorganised and a million LoC app can be well organised. What makes
code organised is things not being tangled up together - having well
encapsulated logical modules that do one thing (or a set of related things),
with a well designed API to get each bit to work with the other bits.

I've found that the more I write testable code the better organised my code
gets. Automated testing forces you to design maintainable, organised code -
because your test suite really won't cope with disorganised code.

~~~
ghthor
> I've found that the more I write testable code the better organised my code
> gets. Automated testing forces you to design maintainable, organised code -
> because your test suite really won't cope with disorganised code.

It can, but it will get out of hand really quickly. Worst case scenario the
test suite becomes so tightly bound to the implementation that is becomes a
part of it. This forces the test suite to be rewritten in the implementation
changes, this is the worst case scenario and must be avoided at all costs.

------
dsr_
The most important skill in operations: systematic debugging. If the
developers did well at organizing, this is much easier.

The most important skill in low-level technical support: diplomacy.

The most important skill in high-level technical support: figuring out what
people are actually complaining about.

Note that many low-level technical support problems look like high-level
problems, and vice versa.

~~~
donatj
I find diplomacy a highly important skill as a developer as well. Strangely I
feel I've become worse at it as the years go on.

~~~
mattmanser
One of the problems I hit was that I would get really grumpy when particular
things were asked for.

Once I learnt to think about why I was getting grumpy, and vocalize the actual
problem it was easier.

e.g. A client said to me recently could clicking the blog link open a new
window? When I started programming I'd have said "Yes! I can do that!". Then
after a few years I'd have got grumpy about it and said something passive
aggressive "I guess so. That's not normal though...". Now I think about it,
realized I hate it when links on the web behave unexpectedly and that's what's
making me grumpy. Then explain to the client that that's you shouldn't take
control of other people's browsers on public websites, and the better way is
to have a clear link to get back to the shop on the blog.

~~~
loopbit
Related to what you are saying, I read somewhere, many years ago, that the
best thing to do in cases where you feel like being grumpy or dismissing
something that was asked for was to, instead, ask 'why'.

The reasoning is that sometimes what the client is asking is his best guess as
meeting a need he has. By asking why, chances are you'll get to the actual
need and, oftentimes, find a better solution where both the client and you are
happy.

Over the years as a grumpy developer, this piece of advice has served me
wonderfully well.

------
userbinator
I'd say it's not really about "organizing complexity" \- because that tends to
just push it around somewhere else - but _reducing_ complexity which is most
important.

In my experience it has been that designs which look "locally simple" because
they have such a high level of abstraction are actually the most complex
overall. Such simplicity is deceptive. I think it's this deceptive simplicity
which causes people to write a dozen classes with 2-3 methods of 2-3 lines
each to do something that should only take a dozen lines of code (this is not
that extreme - I've seen and rewritten such things before.)

Perhaps we should be focusing on teaching the techniques for reducing
complexity more than hiding it, and that abstraction is more of a necessary
evil than something to be applied liberally. From the beginning, programmers
should be exposed to simple solutions so they can develop a good estimate of
how much code it _really_ takes to solve a problem, as seeing massively
overcomplex solutions tends to distort their perspective on this; at the
least, if more of them would be asking things like "why do I need to write all
this code just to print 'Hello world', and the binary require over a million
bytes of memory to run? Isn't that a bit too much?", that would be a good
start.

~~~
necrodawg
This can be hard sometimes. If the business is a mess then your code will be a
mess seeing as your code is just a representation of your business.

I work for such a business right now and it's killing me. No amount of high
quality code will fix this.

------
wskinner
“Here’s something simple I came up with. It may not look like much, but trust
me, it was really hard to realize this was all I needed to do.”

This reminds me of something Scott Shenker, my computer networking professor
at Berkeley, drilled into us every chance he got: Don't manage complexity.
Extract simplicity.

Finding complex solutions to complex problems is comparatively easy. Finding
simple solutions to complex problems is hard.

~~~
thirdtruck
> Don't manage complexity. Extract simplicity.

I can't second this enough. This is why it's so important to _throw away_ your
prototype code, sometimes, instead of trying to fix it. It's also the big
value-add of TDD: Well written tests make that extracted simplicity explicit.

------
rsuelzer
This times 1000! The really sad thing is that 9.9 out of 10 technical
interviews are all about how many Algo's you know. Even if your job will never
require actually implementing a single one.

These interviewers do not seem to care at all about the actual process of
writing and refactoring code, or what your finished products actually look
like.

I know plenty of programmers who can write highly optimized code, but do so in
a way where it is completely impossible to maintain.

It's especially heightened when you are given a problem such as "validate if
this uses valid brackets, and you have 10 minutes." When under time
constraints to solve what is basically an algorithm 99% of programmers are not
going write their best code or write code in the way they would normally write
code.

If you are using lots of algo's on your programming interviews, I suggest you
take a step back and determine if those skills are actually what you want to
be testing for in this job. Odds are that it is NOT algo's. Give your
interviewer some really sloppy code to refactor into something beautiful. Give
them a few hours to work on it, watch how they put the pieces together.

If your position isn't going to require someone to write advanced algorithms
on daily basis, testing for them only cuts out a huge swath of potential
talent. I also think it probably leads to less diversity in your work place,
which is a bad thing.

A Web Developer will never need to solve the towers of Hanoi problem, but they
will need to write clean code that can be maintained by others.

</rant>

~~~
TrophyWipes
Interviewee*

And as someone who is about to graduate computer science, your rant makes me
hopeful :)

I am much better at refactoring and finding patterns than memorizing
algorithms and other things (mainly because they are relatively easy to look
up). Do you suggest that I try to strike a specific balance between practicing
dealing with complexity, optimizing and making code readable, versus actually
trying to memorize and implement certain algorithms for learning's sake?

~~~
rsuelzer
I'm not a computer science grad, and it puts me at a huge disadvantage simply
because I never had any formal training about algorithms. And my day job is to
write testable, well-designed production quality code. I don't get to spend a
lot of time rewriting algorithms, which as you note, can be easily looked up.
The fact is that as a full-stack developer I can count with zero fingers the
amount of times I've had to use an self-made algorithm in a production system.

That being said, I do force myself to practice algorithms thru various coding
challenges only because I know that this is the specific skill that will land
you a job. Which is a shame.

That is not to say that there are not positions that really do require this
specific skill, but the vast majority of development positions don't and yet
nearly every employer tests their candidates as if their success depends upon
their ability to solve an algo in under 5 minutes, instead of their ability to
write a good library.

It's a disservice to those of us who do not have CS degrees and have learned
to code by spending a great deal of our time writing code.

------
jondubois
I agree that it is an important skill in engineering to recognize when the
complexity got too high - At that point you need to take a step back (or
several steps back) and find a different path to the solution - Sometimes that
means throwing away large amounts of code. It's about acknowledging (and
correcting) small mistakes to make sure that they don't pile up into a giant,
disastrous one.

Another thing I learned is that dumb, explicit code is highly desirable - It's
better to have 10 dumb components to handle 10 different use cases than 1
clever component that can handle all 10 cases.

I think the most important skill is being able to break down problems into
their essential parts and then addressing each part individually but without
losing track of the big picture.

~~~
Gracana
> Another thing I learned is that dumb, explicit code is highly desirable

I knew for a long time that "dumb" code was easier to reason about, but it
wasn't until I started looking at compiler output and profiling my code that I
realized being "clever" in the source text _really_ didn't matter. Now I'm
quite happy to write code in the most straightforward fashion, rather than
foolishly "compress" things. I imagine the same experience could be
enlightening for a lot of newbie and intermediate programmers.

~~~
TillE
There are definitely exceptions to this (generally correct) rule, in
situations where performance does matter. RAM is still incredibly slow
compared to the CPU caches, so painstaking cache-optimization can make a huge
difference when that's the bottleneck.

Of course, if you're just writing some I/O bound program anyway, it hardly
matters.

------
ak39
Organisational skill is actually an entrepreneurial skill. I'm learning the
hard way that there are two diametrically opposing skills you need to master
for both running a business and programming effectively:

1\. Skills in Creating 2\. Skills in Organising those creations

The thing is, the Creating part is always exciting but it's disruptive in
nature. The Organising part is boring because it's about taming or tempering
the creation in such a way that it can be referenced later on (just like
filing your invoices, or timesheets - yuck - but necessary).

Unless you've got a systematic method to organise your creations, you will
always be alone with your ideas, find it hard to resume creative chains of
efforts and ultimately flounder without profit.

Both in business and in programming.

Damn right it's the most important software development skill.

~~~
Coincoin
This is why most of my personal toy projects are coded like a 12 years old
with giant functions, god objects everywhere and horrible, horrible, one
liners. It's liberating.

This is also why I like game programming that much. A big chunk of the
codebase is throwaway code that won't be transported to the next project. When
the final deadline comes, you don't feel bad making hundreds of hacks
everywhere, and it's fun.

------
jimbokun
I disagree with the premise that organizing code is not a recognized or
appreciated skill among developers.

At least not since this was published:

[http://www.amazon.com/Refactoring-Improving-Design-
Existing-...](http://www.amazon.com/Refactoring-Improving-Design-Existing-
Code/dp/0201485672)

Martin Fowler really struck a chord all the developers trying to do the right
thing by cleaning up badly structured code, by giving the practice a name and
explaining why it's important. Refactoring is definitely a widely acknowledged
and accepted practice today, although probably more so in some communities
than others.

------
mathattack
This has been my experience too. I've been involved cleaning up a half dozen
or so projects that got out of control. In each case, technical complexity was
blamed as the reason. After digging in, I found that following commonalities:

\- Incomplete, conflicting and misunderstood requirements.

\- Lots of "We never thought we would need to do X".

\- Poor team communication.

\- Mistrust, frequently well earned.

\- Harmony valued over truth.

Once these were winnowed away, the problems rarely overwhelmed the technical
teams. This isn't to diminish the importance of technical skills. Rather -
when everything else is f*cked up, you can't blame the technology or expect a
10xer to pull you out of it.

~~~
crpatino
The advantage of the 10x-er is that he will be reporting directly to a Jr.
Executive and therefore cutting ahead of the middle management echo-chamber.

If you genuinely can write code 10x as fast as the average developer, and get
feed requirements through a 10x faster channel, that would be the life! I
suspect that's where the mythical 100x developer comes from.

~~~
mathattack
I agree. There are actually two 10x things that can happen - sometimes in one
person. One is choosing the right things to do the first time around. The
other is doing them very efficiently. When you get a developer who intuitively
understands the domain, and programs very quickly, the sky is the limit.

I certainly don't intend to diminish the impact of these 10x developers, just
that projects generally get messed up independent of developer quality.

------
humbleMouse
I like this little article a lot. Personally, I try to write code in a way
that reads like a book. Lots of comments, explicit function names, explicit
variable names, object names, class names, ect. Talking about languages higher
level than C/assembly here obviously.

I am amazed at all the code I see that has terrible/too generic names for
functions and variables and objects. Some people get so obsessed over short
function names, one character variable names, and complicated looking one
liner evaluations.

~~~
pinkunicorn
Totally agree on variable/function names, but for some reason I love
complicated one liners. I know its wrong, but its my way of measuring how
smart I really am(even though its not).

~~~
humbleMouse
Much respect to you for acknowledging that it gets your brain off mentally!
Need a 12 step program for folks like you. Complicated one-liners anonymous.

------
linkregister
Google cache, for those experiencing connectivity issues.

[http://webcache.googleusercontent.com/search?q=cache:Mf9074z...](http://webcache.googleusercontent.com/search?q=cache:Mf9074z6m7sJ:www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/&hl=en&gl=us&strip=1&vwsrc=0)

------
matt_s
There is just a nice magical moment though when you grok what an application
does and how it is organized, especially when you didn't write it, which is
likely the most often case.

I think a large issue is when an application has new people working on it or
entirely new teams that maintain it. That is when the original authors
methodology and organization of the application falls to the immediate need of
the day.

The functionality of the software should speak for itself. Commenting your
code with why you are doing something is important to help other maintainers
later on, including yourself, understand what you were thinking when it was
written.

~~~
donatj
This. Organization is in the eye of the beholder. I've seen my code bases
taken over and completely reversed, and frankly I have done the same. If only
there was a better way to communicate exactly what you were trying to do.

~~~
matt_s
I don't know if this is solvable, it could just be human nature.

Some folks like the analogy of software and building construction. I like the
comparison to cooking... in this case the closest thing would be how a chef
sets up their work space and how it is organized for what they are cooking.

Some like it one way, some like it another - is one way better or worse?

------
agounaris
I would add to the conversation the fact that most projects fail on the set of
the initial requirements. In my experience so far I have seen that constantly
changing what you want the app to do creates a huge mess in the codebase even
if you use latest tool and methodologies.

Looks like there is a great value to organise your app in way to be able to
throw away large chunks of code and start over in case there is a big design
change.

------
joshburgess
"The art of programming is the art of organizing complexity, of mastering
multitude and avoiding its bastard chaos as effectively as possible."
\--Dijkstra

:)

------
aspirin
Organization is the hardest part for me personally in getting better as a
developer. How to build a structure that is easy to change and extend. Any
tips where to find good books or online sources?

~~~
Toine
Same here. I only have 2 years of experience, but right now my opinion is that
this is not something you learn in a book (for the moment).

You can learn how to create clean, readable methods and classes with a book
(1).

You can learn how to refactor old methods and classes with a book (2).

You can learn how to organize a small team to allow fast iterations with many
books.

But building a project lasting more than a few months with constant changes in
the requirements, new developers every month, new SDKs and frameworks every 3
days, without the code rotting to death and everything going out of control is
a different story, at least for me.

I guess you just learn by watching old guys do what they do after decades of
experience...unless someone has a magic book for me?

(1) [http://www.amazon.com/Clean-Code-Handbook-Software-
Craftsman...](http://www.amazon.com/Clean-Code-Handbook-Software-
Craftsmanship-ebook/dp/B001GSTOAM)

(2) [http://www.amazon.com/Refactoring-Improving-Existing-
Addison...](http://www.amazon.com/Refactoring-Improving-Existing-Addison-
Wesley-Technology-ebook/dp/B007WTFWJ6)

~~~
sterlingw
I just finished a book called "Practical Object Oriented Design in Ruby." It
teaches you how to build OO applications that are pleasant to maintain. I
highly recommend it.

[http://www.amazon.com/Practical-Object-Oriented-Design-
Ruby-...](http://www.amazon.com/Practical-Object-Oriented-Design-Ruby-Addison-
Wesley/dp/0321721330/)

------
Beltiras
I disagree. Communication is the most important. It's the number one cause of
failed software project. Miscommunicated features, capabilities, scope,
failure to name a few. My favourite is the last one. Not standing up and
recognizing that an approach is not working due to fear has to stand out as a
big one.

------
sgt101
Which is why jupyter/ipython is so great -> you can do some fantastic
documentation of code with working examples and visualisation, you can do it
while you write it !

------
petejansson
Relevant: [http://www.safetyresearch.net/blog/articles/toyota-
unintende...](http://www.safetyresearch.net/blog/articles/toyota-unintended-
acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code)

Summary: Toyota settled an unintended acceleration lawsuit connected with
analysis of the source code for a 2005 Toyota Camry showed it was defective
"spaghetti code."

There'a a lot of poorly-organized code in the world, and a typical excuse for
not cleaning it up is that "it works" so there would be no return on fixing
it. In the Toyota case, the code may have contributed to unintended
acceleration, and did result in a legal exposure for which Toyota felt it was
necessary to settle a lawsuit.

------
anton_gogolev
This can be reduced to:

    
    
        Do Not Program Yourself into a Corner

------
jaequery
Nice post, organization definitely is one of the most overlooked aspects of
programming. It takes a lot of experience and thinking to be able to organize
properly. It's really what separates the beginner programmers and the
experienced ones.

~~~
jaequery
I think what it boils down to is coding from top down, rather than bottom up.
As the OP points out, you need to see the entire landscape.

------
mkramlich
I agree that complexity is far up there. But also risk. Also long term
thinking. And net cost or net profit. The more years I have under my belt, I
think more and more not only about complexity, but also risk, cost, profit.
Code and hardware is just a means to an end. Not the end itself.

But yes, seek the minimum amount of complexity to materialize the inherent,
necessary complexity. But don't allow a drop of complexity more than that.
Architecture astronauts, pattern fashionistas, I'm looking at you. KISS. Spend
your complexity dollars where it gives you something you truly need or want.
Don't do things Just Because.

------
blazespin
Design patterns by the Gang of Four was great for this.

~~~
ebiester
No. This is an orthogonal concern. This is the equivalent of knowing to use
each pattern.

~~~
qznc
The promise of Design Patterns (tm) is that you can systematically learn to
organize your code. The article argues that only experience does.

~~~
tempodox
If it were that easy, you could write a compiler compiler that knows all the
“design patterns” and generates perfect solutions for all problems.

Remember the 239th Rule of Acquisition, “Never be afraid to mislabel a
product.” That goes for methodologies, too. If they could get away with it,
they would probably say it cures baldness, too.

~~~
bottled_poe
I believe that would actually be possible if we had a standardized problem
specification format.

~~~
tempodox
So it would boil down to the quest for the “sufficiently smart problem
specification format”. OK, aren't programming languages standard problem
specification formats? Why do you suppose there are different PLs? Right,
because all problems _can't_ be specified advantageously in the same format.
Which brings you full circle to where you started. That's by no means a
discouragement. If you invent a PL that's better than what we have now, it's
still progress. But, to use a famous quote, “there is no silver bullet”.

------
varchar
Interesting. Most comments seem focused on code complexity. In real life
situations the complexity is blend of human interactions (attitude,
consistency,team,leads, peers, family, emotions) business, market,
competition, budget, time, attrition, unexpected events, and more.

Life is complex. Business and workplace dynamics can be complex. People are
complex with their own strengths, quirks and situations. Having a broad
outlook, developing patience and skills to deal with life and work is part of
becoming mature.

------
jblow
It is a reasonable point that organization is important but I have to disagree
about "most important".

The most important skill in software development, by far, is managing your own
psychology.

------
abc_lisper
This is always a good idea. Here are some of the things I do

\- Establish conventions early; Conventions in managing projects, conventions
in code. And stick to those conventions. Be predictable, in the code. Use
simple names.

\- Protect your interfaces. By this I mean, use a central place like a wiki to
document your interfaces, so all involved parties may agree upon. Write unit-
tests for interfaces. Use libraries like mockito and hamcrest that make it a
breeze.(You would lock your home every time you go out, don't you?)

\- I mentioned this in the previous bullet, but write tests. Write lots of
them, write tests that document any tricky, magical behavior. Cover all the
cases(A boat with one hole is bad as one with two holes). Cover all the
invariants, any thing that you notice but didn't mention in the source code.
Write tests for the bugs you just fixed.

\- If you are developing in Java, please use an IDE. I use Intellij, but
Eclipse is good too. It makes refactoring code much easier. Rename fields,
pull classes up the hierarchy, create abstract classes, create getters and
setters automatically with refactoring tools. I am not against emacs or vi,
but it is hard to manage Java's sprawl with them.

One of the best programmers I know writes code like it has been generated with
a program. It is boring, dull and looks alike in every direction. Every field
and method is documented, it says what its purpose is, and why it is needed.
He is very fast(fast for a startup, not your average enterprise), accurate and
gets a lot of stuff done _without_ magic.

------
bikamonki
No part of a system is the most important part, from a car to a huge
organization, all parts are equally required to interact and hence make such
system 'work'.

Having a clear functional organization at the start (and respect it throughout
development) is very important, but after that is equally important to code
clean and efficient code, to test, to debug, etc. Then, going up and dow on
the solution stack is important to make the right decision on hardware, OS,
server, services, etc.

~~~
erikb
Okay, if you need to be so detailed you also have to argue keep in mind that
some parts are more important than others. Example with the car: If you leave
out the head-rest you might sell your car for a few bucks less or have a
harder time to get a customer (low importance), but if you have a car without
some kind of motor you don't even have a car (high importance).

While there are many things that are important they all have different
priorities, though.

All that said I'm also very sceptical that there is one thing you can learn
and then you are a the best programmer. You have to learn thousands of things.
And sometimes learning a currently low priority thing now might save you
hundreds of hours of pain further down the road. So it's really hard to say
what you have to learn.

~~~
bikamonki
Nope, a car is not a system that was built to be 'sold', it was built to
transport people/things safely. If you remove a part of a system and the
system still works as intended, it was not part of the system. The head-rest
can be removed and the car will still run but with less safety. Hence: all
parts of the system are equally required to interact and make the system do
what the system should do.

------
stinos
Not only is it the most important skill (unless for small home projects
maybe), it's also the one which takes the longest to learn and hence the one
you really improve on during the years and which distinguishes the experienced
ones from the lesser experienced ones. Coincidently, it's also the skill for
which you won't find a ready-made answer on stackoverflow or any other
site/book.

thinking of it, I've also seen this as a typical difference between fresh CS
graduates and those who have been programming for 10+ years. The latter would
sometime take way longer to come up with clever math-oriented algorythms than
the first, because the graduate has been trained for it and still has it fresh
in memory, but experienced programmer would make up for that by being able to
use the algorithms in all proper 'best practice' ways one can think of.
Whereas the graduate would just slam it in somewhere and call it a day even
though there are now x more dependencies and whatnot, you get the picture.

~~~
collyw
This is pretty much how I feel just now (after 12 years). I know Django pretty
well after 4 years, and know where and how to add things to avoid breaking
other parts of the code and make it work consistently with other parts.

Yet nearly every job I apply for these days wants a 90 minute online test. (I
just had to implement a sorting algorithm in psuedo code - I have never in 12
years had to implement a sorting algorithm as I choose an appropriate library
to do that for me).

~~~
bluedino
>> Yet nearly every job I apply for these days wants a 90 minute online test.
(I just had to implement a sorting algorithm in psuedo code - I have never in
12 years had to implement a sorting algorithm as I choose an appropriate
library to do that for me).

True, but if you can't implement a simple sorting algorithm how are you going
to implement _____ ?

~~~
oldmanjay
Since the process of implementing _____ is almost certainly completely
orthogonal to regurgitating a memorized algorithm one would never directly
write themselves (to a fantastically close approximation of never) I'm not
sure how that question has any validity at all.

------
bcheung
I can't load the page, but I would definitely agree with the title.

From my own experience programming here are some the most common and best ways
to better organize complexity:

1) Create DSLs. The Sapir-Whorf hypothesis is the theory that an individual's
thoughts and actions are determined by the language or languages that
individual speaks. By creating DSLs we are able to reason about a problem
domain with increased efficiency.

2) Reduce cognitive load by reducing state. By reducing the number of
variables in a given section of code we can more easily reason about it.
values.map (x) -> x * x is a lot more understandable than newArr = [] ; for
(i=0; i<values.length; i++) { newArr.push( values[i] * values[i] ); }

3) Build tools to build tools. The history of computing is one of building
tools on top of tools. From assembly language to C to high level languages.
What is is the next step? I suspect it is some kind of polyglot environment
that is a hodgepodge of languages all working together combined with automated
code creation from AI.

------
jackreichert
For those not able the view it.
[Cacheview]([http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/))

------
crimsonalucard
Since so many people prioritize getting then task done than writing
organized/beautiful code more often then not we get code that isn't organized
properly.

Thus, as a result: Interpreting complexity is by far the most important skill
in software development. More-so then organizing complexity.

------
zackangelo
Aside from personal discipline and experience, I’ve found that using strongly
typed and compiled languages combined with good tools are the best way to
accomplish this.

Being able to search for and manipulate symbols at the AST level goes a long
way towards eliminating any resistance to refactoring.

------
tome
This is why I love Haskell. Haskell seems to increase my ability to manage
complexity by one level.

Disclaimer: Haskell is not a silver bullet, not a panacea and I'm only
claiming a modest increase, not miracles, but it helps me deal with complexity
better than any other language I know.

------
grandalf
In a recent job I ended up rewriting some of the codebase with this kind of
stuff in mind... the results:

\- 10% of the LOC as previously

\- Code significantly more understandable

\- Jr. developer on the team suddenly became a rockstar b/c he could
understand what was going on.

~~~
notNow
Reducing LOC is not always necessarily a good thing. I'd always favor a more
readable codebase that takes some effort to explain its purpose than terse one
that is vague and obscure that makes your head spin trying to figure out its
raison d'etre

~~~
grandalf
I agree. In this case the code got significantly better organized and more
readable in the process.

------
o_nate
Or in other words: code should be as simple as possible, but no simpler. I
guess you could call it organization or readability or just good design. It
requires deeply understanding what you're trying to accomplish and structuring
your code to reflect that. I don't think there's any rote, step-by-step
procedure that will get you there. Often it is a flash of creative insight
that breaks the log-jam and reveals the hidden inner structure of the problem.
Once that is revealed the code writes itself. In other words, good code should
always make the problem that was solved look easy.

~~~
blakeyrat
Developers understand the word "simple" in different ways.

I personally think of the UI/API: what the code exposes to the outside world
should be "simple".

Some people might think about language features. "Simple" is being very
selective about what language features you use and what you avoid.

Others might think design patterns, and think the code is "simple" when you
can point to any class and immediately recognize what pattern that class
implements.

I've come to learn the word "simple" as in KISS (Keep It Simple, Stupid) isn't
very useful at all.

~~~
o_nate
I was thinking of more than the UI/API, although I agree that should be simple
as well. I think the internals of the code should be simple, in the sense of
being constructed out of simple parts with each part being simply connected to
the other parts. This means each method should have a clear purpose and it
should accomplish it cleanly without surprising side-effects, and that the
flow of method calls should be straightforward.

------
autotune
I take it from his site not loading that the most important skill is learning
how to scale your site and ensuring it remains accessible during high load
times, or using CloudFlare to at least ensure it gets cached.

------
kriro
"""But there’s always lots of state to keep track of, rearranging of values,
handling special cases, and carefully working out how all the pieces of a
system interact."""

I'm not a functional programming evangelist but that reads like a very good
reason to go for FP. I think a similar point was made in "Functional
JavaScript". I don't remember it exactly and it's on my shelf at home but
there was some passage about the biggest downside of typical OOP codebases
being the mental effort of keeping track of values and value changes.

------
AdieuToLogic
> Do the difficult things while they are easy and do the great things while
> they are small. A journey of a thousand miles must begin with a single step.
> (source:
> [http://www.brainyquote.com/quotes/quotes/l/laotzu398196.html...](http://www.brainyquote.com/quotes/quotes/l/laotzu398196.html#rGRvFjhwr4m4FXbJ.99))

This applies to systems as much as any thing else it possibly could.

------
BerislavLopac
This pretty much aligns with my take on that from four years ago:
[http://berislav.lopac.net/post/13061099545/the-most-
importan...](http://berislav.lopac.net/post/13061099545/the-most-important-
skill-for-software-developers)

As wise men said: All problems in software can be solved with more layers of
abstraction, except of the problem of too many layers of abstraction.

------
agumonkey
And Metalevel complexity. I may have been able to craft beautiful code, but I
sensed chaos in the way I operate and manages my own resources / tools. I've
seen people being organized at many, if not all, layers, solving problem and
solving how to help solving problems. Witnessing that makes me feel calm and
envious at the same time.

ps: it's also reminiscent of recursion, dogfooding etc.

------
markbnj
It's a very important skill for a programmer to have, especially in the modern
environment where distributed systems built from many integrated components
are the norm. That said, it's awfully difficult to disentangle the various
skills needed for programming and assign an importance to each one, mush less
to determine which of them is actually most important of all.

------
jtwebman
The website is down but here is a link from the archive.org site.
[http://web.archive.org/web/20150622134205/http://www.johndco...](http://web.archive.org/web/20150622134205/http://www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/)

------
brazzlemobile
Here's the Google cache if someone is looking for a mirror:
[http://webcache.googleusercontent.com/search?q=cache:www.joh...](http://webcache.googleusercontent.com/search?q=cache:www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/)

------
scuba7183
Text only mirror:

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/&strip=1&vwsrc=0)

------
thewarrior
Yeah this is so true.

I worked on a system where we had to support imperial and metric units. It was
done in a pretty bolted on fashion with if statements all over the place. And
sometimes it isn't even clear if it could be done in any other way.

Any HN'ers have suggestions on how to do it elegantly.

~~~
notNow
language?

------
ademarre
> _Organizing complexity is the most important skill in software development_

I agree with this profoundly. Unfortunately, complexity is in the eye of the
beholder. When comparing solutions to a problem, different developers will not
always agree on which is the least complex.

------
edpichler
Our humans are "comparing machines". For this reason we tend to valuate people
that solve problems more than the ones who never create them. This is really
bad.

Also, in business, if someone is really good administrator, it seems he never
does nothing.

------
Omnipresent
On the topic of organization and related to another post about good Software
Development books. What are some books that teach code organization as
discussed in this post. One I can think of is "Refactoring" by Martin Fowler.

What are some others?

~~~
tempodox
I would submit the main point is: You can't learn that from a book but only
from experience. The best balance isn't cast in stone but must be found for
each problem domain. As much as you can, do your own experiments and
experience the consequences of your decisions yourself. Remember that for
nontrivial problems, a usual part of the job is learning exactly what problem
it is that you have to solve.

------
ljk
I was thinking that recently. The way I try to stay organized is to
comment(with timestamp) every time something is changed so I can refer to it
later. Does anyone have more tips on how to stay more organized?

------
signa11
there is an _excellent_ book by jon lakos called 'large scale c++ design'
which treats organizational or physical design of large scale c++ projects (>
1giga loc). highly recommended.

~~~
bmonkey
I've considered this book a few times, but the following review always stopped
me cold:
[http://www.amazon.com/review/R1ICFUDTH8VXQ5/ref=cm_cr_dp_tit...](http://www.amazon.com/review/R1ICFUDTH8VXQ5/ref=cm_cr_dp_title?ie=UTF8&ASIN=0201633620&channel=detail-
glance&nodeID=283155&store=books)

Any thoughts on it?

~~~
blub
I would give "API design for C++" by Martin Reddy a shot. The only thing I
remember now from the large scale book is to pay attention to compile-time and
link-time dependencies, although when I read it I remember that it contained
some nice tidbits of info that I had not seen in other books

The big takeaway was that the way the project is split into libraries, files,
folders etc and how these map to classes, modules and subsystems matters.

------
justonepost
Arguably, this is what higher level languages like Java and C++ provide. Tight
organizational language metaphors that help implement design patterns in a
thoughtful, consistently structured manner.

~~~
jeremiep
The abstractions provided by Java and C++ aren't very good compared to those
provided by, say, Haskell and Clojure.

For the most part, design patterns exist merely to fix the lack of a simpler
lambda construct in the language. They more often than not add complexity
rather than remove it.

The biggest balls of spaghetti code I've ever seen we're _always_ OOP with
design patterns. I've seen far more clean C codebases than Java or C++ ones.

~~~
trb8
I would be hesitant of this so easily accepting this view due to selection
bias.

Java and C++ are often found in huge, legacy, or enterprise oriented code
bases.

The are plenty of ways to use the abstractions in Java and C++ to write nice
code. Just like it is possible to find a Scala or Clojure code base that went
crazy with the usage of "cutting edge" features and abstractions in those
languages.

And at least with Java, your IDE will always be able to navigate the code
effectively.

~~~
jeremiep
I completely agree it takes discipline and experience to write clean code in
any language.

What I'm saying is that it takes more discipline to cleanly use Java or C++
than it does to use Haskell or Clojure. For the simple reason that most of the
abstractions provided by the former languages add to the program's complexity
rather than remove it.

There's an excellent explanation by Rich Hickey in Simple Made Easy:
[http://www.infoq.com/presentations/Simple-Made-
Easy](http://www.infoq.com/presentations/Simple-Made-Easy)

------
brobdingnagian
Important according to what metric? Making the software developer feel good,
or making the company money? The former is almost certainly true, the latter
is almost certainly not.

~~~
davidjnelson
Avoiding rework can cut costs. Rapid Development by Steve McConnell explains
this well.

------
headShrinker
google cache record:
[http://webcache.googleusercontent.com/search?q=cache:Mf9074z...](http://webcache.googleusercontent.com/search?q=cache:Mf9074z6m7sJ:www.johndcook.com/blog/2015/06/18/most-
important-skill-in-software/&client=safari&hl=en&gl=us&strip=1&vwsrc=0)

The server seems to be getting hammered.

------
hamsterdog
Does anyone know of any book or website or other resource that has before and
after examples of good refactoring, or something similar?

------
snarfy
There is a term for it. It's called managing scope. I'm surprised the article
didn't mention this.

~~~
kyberias
No, that's a totally different thing. The scope (requirements) of the software
might be huge but the software could be designed and implemented cleanly and
with maintainability in mind. Yet the scope might be tiny and the
implementation a mess. This article discusses the skill of taking a mess and
turning it into something clean.

~~~
snarfy
I guess it depends your usage of scope. One is used in defining the problem,
the other, the solution. I'm thinking of it's usage when describing the
solution.

------
Stratoscope
Out of all the things I know how to do in programming, reducing complexity is
probably the one I'm best at. So how do I get a job doing this? Or a series of
lucrative consulting gigs? :-)

I'm pretty sure I'm not as smart as I used to be, and I'm definitely not as
smart or productive as some of the younger programmers I've worked with.
(Sorry for the ageist remark!)

This may be my secret advantage: I _have_ to keep my code simple enough that
even I can understand it.

Here's a fun example that I've seen more than a few times in various forms:
four-way navigation, either involving up/down/left/right or
north/south/east/west, or both.

In one (somewhat disguised) project it worked like this: the code had several
different modules to provide a keyboard interface for geographic navigation,
while keeping the geo code separated from the low level details of key codes
and events and such.

There was a keyboard manager that mapped keycodes to readable names that were
defined in an enum:

    
    
      switch( keyCode ) {
          case 37:
              return KEY_LEFT;
          case 38:
              return KEY_UP;
          case 39:
              return KEY_RIGHT;
          case 40:
              return KEY_DOWN;
      }
    

Then an event manager broadcast navigation messages based on the KEY_xxxx
codes:

    
    
      switch( keyEnum ) {
          case KEY_LEFT:
              BroadcastMessage( 'keyLeft' );
          case KEY_RIGHT:
              BroadcastMessage( 'keyRight' );
          case KEY_UP:
              BroadcastMessage( 'keyUp' );
          case KEY_DOWN:
              BroadcastMessage( 'keyDown' );
      }
    

A navigation manager received these messages and called individual navigation
functions:

    
    
      // Don't forget to reverse the directions here
      events.on( 'keyLeft', function() {
          moveRight();
      });
      events.on( 'keyRight', function() {
          moveLeft();
      });
      events.on( 'keyUp', function() {
          moveDown();
      });
      events.on( 'keyDown', function() {
          moveUp();
      });
    

These navigation functions panned a map in one compass direction or another:

    
    
      function moveUp() {
          map.pan( maps.DIRECTION_NORTH );
      }
      function moveDown() {
          map.pan( maps.DIRECTION_SOUTH );
      }
      function moveLeft() {
          map.pan( maps.DIRECTION_WEST );
      }
      function moveRight() {
          map.pan( maps.DIRECTION_EAST );
      }
    

Of course most of you reading this can see the problem at a glance: Besides
having so many layers of code, how many different names can we give to the
same concept? We've got KEY_LEFT, keyLeft, moveLeft, and DIRECTION_WEST that
all mean pretty much the same thing!

Imagine if math worked like this: You'd have to have two of every function,
one for the positive numbers and another one for negative numbers. And
probably four different functions if you are dealing with a complex number!

That of course suggests a solution: use numbers instead of names, +1 for up
and -1 for down, ditto for right and left. And pass these numbers on through
any of these layers of code so you only need half the functions. If you need
to flip directions along the way (like the left arrow key navigating right),
just multiply by -1 to reverse it instead of having to make special cases for
each direction name.

You might even decide to combine the two axes, so instead of vertical and
horizontal, you've got +1 and -1 there too (or 1 and 0, or _something_ that
lets you handle both axes with one piece of code). Now you could be down to a
quarter of the original code.

Unfortunately, I was brought in on this project near the end to help wrap up a
few other tricky problems, and all this navigation code was already set in
stone. (And to be fair, working and tested, and who wants to go back and
rewrite proven code, even if it is four times the code you need?)

But this would make a pretty good "how would you clean this code up" interview
question!

~~~
cactusface
It's actually reasonable. Step 1, translate key push. Step 2, broadcast key
message. What you're missing is that there's an alternate Step 1, translate
joystick hat push (or mouse drag, or whatever). So Step 2 should be broadcast
generic up/down/left/right input message. Step 3, translate message into
movement. But what you're missing is that dependent on context, there's going
to be other things besides movement that those keys do. Step 4, pan the map
with N/S/E/W. Again here, you have the flexibility to work with a rotated map.
Your code was probably overdesigned for the use case, but I've seen your
solution many times, in books and in real life.

However, I've also solved this problem by passing around polar coordinates,
it's elegant and very flexible but you have to munge some data at the
beginning. You can also pass around a simple vector [{-1, 0, 1}, {-1, 0, 1}],
which is basically what you describe.

------
DevPad
Yeah, the best developers I've ever seen are about simplifying things, avoid
overcomplication for too much "flexibility".

The zen of Python
[https://www.python.org/dev/peps/pep-0020/](https://www.python.org/dev/peps/pep-0020/)
say:

    
    
        Simple is better than complex.

------
jdimov9
I agree that most complexity in software systems comes from managing state. So
here is a simple solution - stop doing it. Stop managing state.

Use the right tools for the job. Most mainstream programming languages are
ridiculously inadequate for building any production software of any
significant complexity without introducing more problems than you are trying
to solve.

Use a mature functional programming language that works with immutable data
and is designed for building complex industrial systems.

Use a language that was designed from the beginning with the understanding
that your software WILL be full of bugs and errors, yet systems must always
continue to run.

Use Erlang.

~~~
philbarr
Here we go again with the "functional programming will solve all your
problems" thing. It won't. Every language has it's inadequacies.

~~~
NovelSpinGames
Some inadequacies are much bigger than others. The two big ones for me are
nulls AKA the billion dollar mistake and mutability by default. I agree that
functional programming won't solve all your problems, but it has a lot of very
useful features. Some features, like higher order functions, have made their
way to more imperative languages. I use higher order functions all the time in
my C# programming. And a lot more functional features (or at least F#
features), such as tuples, pattern matching, records, and immutable types, are
in the C# 7 worklist:
[https://github.com/dotnet/roslyn/issues/2136](https://github.com/dotnet/roslyn/issues/2136)

------
evandrix
page not found

~~~
shaneofalltrad
seems we overloaded the server

------
brightball
Great read. 100% agree.

------
yessql
Oh, I thought the most important skill in software development would be SCRUM.

~~~
ExpiredLink
You certainly are a manager, not a developer.

