
Why does software development take so long? - cheiVia0
http://blog.sesse.net/blog/tech/2016-10-26-09-29_why_does_software_development_take_so_long.html
======
jbb555
People STILL confuse the construction of software with the construction of
buildings. We can estimate fairly accurately how long it will take to build a
building once we have reasonable plans for it. I can pretty accurately say
that it will take about 4 minutes to build the software once I have the plans
to build it. The compiler pretty much automates the whole job.

Writing software is NOT construction. Much of it isn't even design. Most of it
is gathering detailed requirements and writing them down in unambiguous form
(code).

My asking how long it's going to take to write a software it's like saying to
a building contractor how long will it take to design every single detail of a
city block including gathering all the requirements.

Also the requirements for software are much more detailed than building.
100000 lines of code represents 100000 decisions. I bet not many buildings
have 100000 decisions. And 10000 is tiny for a software project.

~~~
snowwrestler
The reason designing a building is faster is that there are fewer decisions.
The reason there are fewer decisions is because buildings are way better
understood than software.

A lot of designing the structure of a building is just the implementation of a
few core concepts that have been perfected for thousands of years, like doors
and windows and beams and arches.

There is sufficient human knowledge about buildings that people expect every
single building to stand up, work properly, and be safe the very first time it
is built. There aren't many self-taught building engineers who just picked it
up in their spare time during high school.

And a lot of the design is simply choosing fixed options. Architecture and
construction firms don't design and engineer the ceiling light fixtures or the
faucets. They select a supplier + model, and install them with standard
hardware in standard ways. In many cases there is even government code that
tells them exactly how it must be done.

Compare to software. Every single decision is up for grabs on every new
project. Programmers have no standard certifications, and in fact are often
actively hostile to standardization and formal training. The state of the art
in making sure things work is just brute-force, constant testing. Of course
projects can run into problems.

Software, as an engineering discipline, still has a long way to go toward
being a mature, well-understood human endeavor. Which makes sense, since we've
been doing it about 1/100th as long as we've been making buildings.

~~~
cesarbs
> Programmers have no standard certifications, and in fact are often actively
> hostile to standardization and formal training.

I hold the unpopular opinion that we need those things. The current state of
things is that we're not much more than kids hacking away on computers with
minimal supervision.

I'd love to see most (if not all) software written with the rigor required to
write safety-critical systems.

~~~
majewsky
The problem, to add to the buildings metaphor, is that we don't have a few
thousand years of experience building software under our collective belt yet.
We quite literally don't know how to build good software. Or rather: we are
still figuring out how to build good software, and we don't know yet how far
along we are in this process. Therefore it looks like a questionable idea to
set the things that we currently believe to be true in stone.

~~~
sobri
We're also never building the same software (or at least very rarely). The
industry is such that we're always trying to invent something new, something
that hasn't been done before.

If you build the same (or near same) piece of software 100 times, you can know
almost exactly how long it'll take and you can do it quickly, same as if you
were building 100 buildings. But we don't do that, because you build software
once then just copy it 100 times.

You only build software if you're making something new that hasn't been made
before.

~~~
boggydepot
> You only build software if you're making something new that hasn't been made
> before.

Or has been proprietary. This is why we should choose copyleft licenses over
permissive ones.

This project I am working on is a bit boring, outsourced (or exploited) here
in my country. It's about ETL, transferring data from prod db to analytics
db.. I am of the opinion that all this is a solved problem and I am probably
repeating mistakes. However, due to the nature of Capitalism, exploitation
....

~~~
codingmyway
I've been there. I even got a fair way into discussing starting a 'data
migration' startup. Customers only care about getting their data migrated and
wouldn't mind you holding the rights to the software tools you write to do it
so you could get better and better as you build up your toolkit.

------
lordnacho
You can see the Pacific Ocean on one side, and you can see the Atlantic on
another.

You know of something called a shovel, machetes, labourers and engineers. And
you have a budget.

Of course, it must be possible to dig a trench connecting the two oceans.

When you start, you find your men get ill from the climate. And perhaps you
don't quite know how to build a reliable lock, but you figure the engineers
will find out somehow. And meanwhile there are constraints coming in from
government.

Things take long because it's easy to see the big picture, and getting more
detail (ie learning) normally means discovering that you need a bit more time
to do fix some issue that you didn't see standing on the hill.

~~~
jacquesm
Except that in software 1000's of groups have already dug 1000's of canals and
_yet_ the next group that will build a canal will insist on doing it their
way.

If our profession would only be able to simply collect lessons learned and to
pass those on to the next generation then we'd be moving faster on average and
we'd produce better software. we don't need 500 crappy, slightly different
ways of doing the same thing, we need maybe 5, battle tested and hardened
bullet proof solutions that are well engineered. Fragmentation is going to be
open source's death at some point if this continues, that's one area where
closed source actually has an edge: focus.

In part the problem is that software producers have managed to convince the
buyers of their goods and services that they are never ever liable for their
product (something no other industry has managed to do), in part it is because
it seems superficially easy to re-invent until you reach the same level of
complexity (which inevitably happens) as what those 'archaic' solutions were
already addressing.

I don't have any answers on how to solve this, unfortunately, but there are
software eco-systems that really get this right and others that seem to be
totally lost in ever expanding cycles of endless rewrites.

~~~
Aaargh20318
No, unlike e.g. civil engineering you're never doing the same thing twice and
you're always doing something you've never done before because if you have,
you could just use that code.

You can't copy-paste a bridge but you can re-use software.

~~~
jacquesm
You possibly don't now much about civil engineering, let me start out by
writing that _every_ project in civil engineering stands on it's own, it is
the knowledge and the practices that get re-cycled from one project to the
next and on a higher level from one generation of civil engineers to the next.

So it is extremely unlikely that civil engineering will be done twice for an
exactly identical project. But no civil engineer is going to make his own
table for bolt tensile or sheer strength just to scratch an itch.

Then we come to software, in software the same problems appear over and over
again and we have in the 100's of possible solutions for some problems that
are all almost (but not quite, of course due to sloppy design and lack of
standardization) interchangeable.

Think web frameworks. They are themselves an attempt to abstract away some
elements that are common, but in the meantime there are almost as many
webframeworks as there are things that they were trying to abstract away to
begin with (and all of them fail on one or more of the details). Or
programming languages, another area in which we have re-invented the wheel so
many times that the number of programming languages (1000+) will at some point
probably exceed the number of human languages (about 6500 in active use, but
about 2000 of those have < 1000 speakers).

[http://www.infoplease.com/askeds/many-spoken-
languages.html](http://www.infoplease.com/askeds/many-spoken-languages.html)

[http://www.99-bottles-of-beer.net/](http://www.99-bottles-of-beer.net/)

~~~
Aaargh20318
> You possibly don't now much about civil engineering

I don't, so maybe it's not the best example.

But take houses for example, it's not common to see a dozen or more identical
houses being built. After you've done 11, number 12 is not going to be much
different.

~~~
jacquesm
But the foundations might be slightly different to account for terrain
differences. And that will not cause the civil engineers involved to re-define
what a bulldozer or backhoe can or should do and how it should work, nor will
it change the process they will use to make that foundation.

In fact, the house sitting on top of the foundation will likely be the same
project if they really are identical, just like copies of a piece of software,
in other words, from an engineering perspective the job as far as the house is
concerned was done with #1 if you want more than one house.

It's all about the tools, processes, materials, soil knowledge and so on that
are being employed in order to solve the problem, _not_ about what is actually
built.

In the software equivalent there would be a new build process with associated
terminology and tool obsoletion for almost every engineering project. Imagine
workers coming to the job one morning to find that the tools they used the day
before have now en-masse been declared obsolete and all their technology is
slightly different, incompatible and un-tested. Then, a few weeks later at
another job site it would be the same story all over. In the meantime, the
engineers would be re-inventing engineering from first principles for every
5th job or so.

In other professions that would be called madness.

~~~
numlocked
I think that's the wrong perspective.

Software is a tool. What you do with it is an outcome.

A bulldozer is a tool. The house is the outcome.

So, we already invented a bulldozer (some piece of software) and it's trivial
to duplicate it and reuse it all over the world. But when we need a new type
of machine it's harder because, well, it's new.

In the software engineering world the goals is generally not to facsimile
something like e.g. a house. It's to create a legitimately new tool. And since
there are no real capital investments required (unlike building new tools in
the physical world), and folks often value their free time at a weirdly low
marginal rate, you end up with tool proliferation in a way not seen in the
physical world, which makes it easy to think that we're building houses when
in fact we're inventing the bulldozer.

~~~
jacquesm
> It's to create a legitimately new tool.

More likely: to create a minor variation on an existing tool. There have been
relatively few instances of legitimate new tool creation but many 100's (and
probably 1000's) of instances of re-creating slight variations on the same
tools.

For instance: build tools, programming languages, libraries (95% duplicate
functionality with some other library, and of course an inconsistent
interface) and so on.

> And since there are no real capital investments required (unlike building
> new tools in the physical world)

Software requires enormous amounts of capital, in part due to all this re-
invention. We even have a name for it: NIH, and we have a term for what
happens to a software project that is a few years old (technical debt).

Our tools and our processes are ill equipped to deal with the challenge and
what is supposed to be building houses more often than not ends up with people
re-inventing the bulldozer.

Now there are times when that is the right thing to do, but most of the times
it isn't.

~~~
Retric
Slashdot, Reddit, and HN are different tools. On the surface they seem
similar, but so does a Philips head screwdriver and a Frearson, yet they
really are different tools.

~~~
jacquesm
> Slashdot, Reddit, and HN are different tools.

They're different end products, but they are not different tools.

Conceivably they could have been built with the same toolchain, but instead
they were all built with totally different tools (Perl, Lisp/Python, Arc)
respectively.

~~~
SomeStupidPoint
But how is this not complaining that some buildings are concrete and some are
wood and some are _gasp_ a hybrid?

I mean, if those engineers were better at what they do, they'd use just one
material, right?

And they have hundred or thousands of kinds of screws! Why do they keep
reinventing them? They're just meant to hold things together -- one job!

~~~
jacquesm
Because in engineering there really are such requirements. In software we
_rarely_ have a good reason to start yet another 'from scratch' new hip thing
that is so much better than that old thing (where 'old' is likely less than 5
years old). Technology cycles are now so short that libraries and frameworks
don't even bother pretending to have a life-span longer than the stuff that
will be built upon them.

And those engineers with their 'hundreds or even thousands kinds of screws',
they very much favor standardization, lots of work goes into attempting to
create families of compatible tools and consumables and typically a change
like that will result in _decades_ of stability in the fastener industry, it
is quite rare for something revolutionary to happen, most of the changes are
incremental and logical.

If you attempted to come up with a completely new thread or screw head there
would have to be a reason better than 'I don't like the other screws' if you
expect to gain any acceptance of you shiny new tech.

I'm trying to imagine a world where every other week the whole of engineering
would be up-ended, everybody would have to totally re-train and we'd discard
everything we learned process wise over the life-time of the industry.

That would be the rough equivalent of what we do in software. Ok, maybe not
two weeks, but we might actually get there, life-cycles are getting ever
shorter.

~~~
SomeStupidPoint
So if you stepped back 5000-10000 years in engineering, you'd notice that
civil engineers... Basically did exactly that.

You're just being extremely unfair in comparing a field with ~10,000 years of
development with one that has like... 100.

Everything you've called out as unreasonable is pretty much what every field,
ever, has done when it first became a discipline, and your specific criticisms
border on absurd.

Programming languages, for instance, are like materials: of course there's
thousands, most of them are meant for research purposes, and the hundred or so
used ones each represent different tradeoffs in base construction strategy. No
different than the dozens of kinda of wood and concrete used by civil
engineers.

Similarly, engineering of civil projects has some notable massive overruns on
budget and complete design failures in even recent history. Their projects are
bespoke and often use novel ideas that don't work out in practice.

The main "difference" is you're comparing a high assurance subset of one side
to the general of the other, which is naturally quite unfair.

Want to compare the fields in full? I bet I can find a dozen bad handyman
repair jobs for every bad JS framework or library.

~~~
jacquesm
But we already _have_ engineering as an example of how to do something like
this right, there is no reason to go through another 5000 or 10000 years to
figure all this out once more.

And the handyman and the engineer have very little in common.

~~~
SomeStupidPoint
Civil engineering had thousands of years of development when Galloping Gertie
happened. Why would their methodology work in other, different engineering
fields when it fairly routinely doesn't work in its own? I mean, if you look
at things like project failure percentage (and cost!) over time spent
developing the field, you probably have software _winning_.

Software is developing in to an engineering profession at an astonishingly
fast pace. It's just currently at the point where it's differentiating between
tradesmen and engineers, and without that clear distinction, it's hard to
compare apples-to-apples between fields.

> And the handyman and the engineer have very little in common.

Why not? You're lampooning software development for the fact that a lot of
not-quite-professionals develop mixed results because they churn a lot of
product onto the market. The damage done to house integrity all over the world
by questionable repairmen estimating the engineering impact of a change they
make is comparable.

My experience with high-assurance software is that it's similarly well
constructed and engineered to high-assurance civil engineering, eg, major
bridges. Failures happen, but are sporadic rather than regular.

~~~
nickpsecurity
"My experience with high-assurance software is that it's similarly well
constructed and engineered to high-assurance civil engineering, eg, major
bridges. Failures happen, but are sporadic rather than regular."

Where did you experience high-assurance software and is there a report on how
it was made? There's very few here that even know what that means although
I've been working to change that over the past year or so. Always collecting
more from that tiny field for any lessons learned that could be shared.

~~~
SomeStupidPoint
I worked in control firmware and systems middleware/OSes for chemical
processing equipment (and related control systems). None of the super fancy,
huge plants; more single room sized processing pipelines for R&D uses. That
said, the bigger chambers were like 0.2m^3 and operated at 2500PSI @ 100C, so
you'd definitely know if one catastrophically failed.

We didn't necessarily develop a lot of process in-house, because our senior
engineers had backgrounds with Boeing and/or NASA, which both had extensive
ideas about how to design reliable systems.

If I were summarizing, there's only really two points that cover about 90% of
what you need to do for high assurance software:

1\. Realize almost all bugs are because of politics and economics, not
technological or engineering faults per se. That is, we make choices about how
we set up our culture and corporate system which incentivizes people to create
and hide bugs, while also failing to incentivize others to help fix that bug.
The first step in combating bugs must be to change the fundamental incentives
which create them. In particular, a focus on the success or failure of a team
as a whole. Development is a communal activity, and the entire team either
succeeds or fails as a unit. Someone else committed a bug (and it got merged
all the way to deployment!) that brought down the system? It's because _you_
didn't provide the necessary support, teamwork, and help with engineering your
peer needed to succeed. What can _you_ do to help them succeed next time?
After all, everyone is human and makes mistakes. What's important is that the
people are interlocked in layers, where one person can catch another's mistake
(without punishing the person who made a mistake, because that just
incentivizes them to hide them!) and help fix it before the code reaches
client machines. Successes may be individual, but failure is always the
system's fault, never an individuals.

2\. Almost all technical bugs, in any field, are because of leaky abstractions
and implicit assumptions. From abstract ones like mathematics to physical ones
like carpentry. Be explicit. About every possible detail. And if you think
you're being too explicit, you probably forgot 80% of the cases. Ever see the
average house blueprint? Puts software engineering design to shame, easily. If
you want to build something that runs as reliably as a highly tuned, expertly
designed engine, you can't start with anything less than as detailed of a
specification as they use. Be _explicit_.

Once you get to the point where you're working in genuine engineering teams
rather than as individual engineers on a team and you have explicit, detailed
specifications, the technologies to actually convert that reliably in to
software that runs stably are pretty straight-forward.

The reason we don't see this all the time is simply that it's expensive: the
politics require a lot of redundancy of time spent (eg, code has to be read
several times by several people); the explicitness requires a lot more upfront
planning and effort in documentation, which requires more time invested per
unit of actual coding; etc.

Of course, much like we have building codes for houses and larger structures,
I think it's perfectly fair to expect minimal standards from software
engineers. (Especially now that, eg, IoT botnets are DoSing things.)

~~~
nickpsecurity
Appreciate your write-up. It all sounds great. I'd say the tech to go from
detailed specs to reliable or secure systems isn't necessarily straight-
forward except in the easy cases. It can take some serious work by smart
people, esp if it's formal verification or spotting unknowns. We can easily
get 90% of the way there, though, without the hard stuff just by people giving
a shit and doing stuff like you said.

Of course, Cleanroom Software Engineering that does a lot of what you said
without as much formal stuff as Praxis's method. Thing you should remember in
these conversations is to point out to other party that what you recommend and
what these methods did knocks out tons of debugging and maintenance costs.
Since both phases cost labor, with maintenance fixes multiples of development
cost, there were many cases where Cleanroom projects actually cost _less_
since they knocked out huge issues early on. Can't bank on that being normal
with it often being 20-30% or so more. It was around the same or less cost in
many case studies, though, due to knocking out problems earlier in
development.

Btw, I found a nice write-up on Cleanroom without tons of math or formality
with case studies on college students if you're interested in references like
that:

[http://infohost.nmt.edu/~al/cseet-
paper.html](http://infohost.nmt.edu/~al/cseet-paper.html)

I like dropping them on people in regular development in these discussions
when they say you can't engineer software, it takes ridiculous skill, or it
would cost what NASA's team spends (rolls eyes). Eye opening experience for
some.

~~~
SomeStupidPoint
> I'd say the tech to go from detailed specs to reliable or secure systems
> isn't necessarily straight-forward except in the easy cases. It can take
> some serious work by smart people, esp if it's formal verification or
> spotting unknowns.

I was being a little facetious. Of course the technical work is highly complex
-- some of the brightest minds of our time work on DARPA related projects on
foundational mathematics related to automated theorem proving. (Looking at you
HoTT crew and related projects!) Why is DARPA funding automated theorem
proving? Because they want to create high assurance software for government
infrastructure to counter the threat of AI-based cyberwarfare, and our current
mathematical techniques aren't up to muster.

However, we've made tremendous progress on that problem in the span of around
100 years. By contrast, the problem of how to not incentivize your workers to
do a shitty job, causing you problems later on... has been with us, I think
it's safe to say, for thousands of years. (And it too, has attracted some of
the brightest minds over the years.)

So in a relative way, the technical aspects are "straight-forward" to address,
compared to the underlying political problem. And the more technical methods
only add some assurance that you've correctly implemented the spec as defined,
not that you're doing the right thing. It's certainly good to, eg, check that
you're using total functions or not misassigning memory, but it's not magic.
So while they catch a lot of dangerous problems, they can also lead to false
confidence about the existence of other classes of problems.

> they say you can't engineer software, it takes ridiculous skill, or it would
> cost what NASA's team spends

It mostly just requires that we consistently act with discipline and
professionalism, which are both tiring compared to not doing them, so by and
large, people just don't bother. I know I don't when I can get away with not
doing it (even if I know that's a bad habit).

------
forgottenacc57
My most recent project has taken more than 18 months.

I implemented 2 major subsystems that I later realized I could simplify the
system by throwing out.

My product is very technical so there was probably about four months trying to
make things work and learning what did and did not work.

I had to learn about 75% of the development tools and languages I was building
with.

I don't like the idea of releasing half baked, half working software that
represents half of my vision.

I had to fit the project in between my money paying job, family, friends,
life.

I had lots of great ideas and the fun bit about developing software is when
you get to implement those "wow it would be cool if...." functions.

I just want to fix bugs after the software is released, not just be starting
on the code.

I want the software to be so far along that it would cause a competitor to
think twice about copying.

I think software should be great, not just barely functional, so I added lots
of things to polish and make it awesome.

Near the end of the build I realized I could make a major leap forward so I
redesigned a key area of the UI.

I built as much of it as I could before release because after release its MUCH
harder to add new features.

There you go, that's why it took me so long.

~~~
jacquesm
Kudos on taking the risk to release a bit later and delivering something that
meets your higher standards, that's going against accepted business wisdom but
it is only because 'everybody else releases barely functional crap too' that
that became the norm for competition.

~~~
forgottenacc57
I get a vision in my head of what the software should be and I'm compelled as
though by force to make it match the vision. I don't seem to have much choice.

------
rogual
And this is just for a sole developer who knows what he's doing. When you have
a team of people at varying levels of ability, a meeting for every decision,
JIRA-wielding project managers with burndown charts to keep happy, standards &
practices, legal, translators, UI designers, all regularly taking out
exclusive locks on each other's time... it's a wonder anything gets done at
all.

~~~
hex13
It's correct. I've noticed that when I work in a team I work with maybe 10% of
my solo efficiency. Just because of this whole team work bullshit and because
of legacy code (it takes me often more time to understand how code written by
someone works than it would take to write such code from scratch).

------
Edmond
I think a big part of the problem can be attributed to tool makers and the
lack of true innovation in that arena. As a developer tool maker (HiveMind:
crudzilla.com) I am almost always disappointed when I come across a "new"
development tool and find that it is the same tired code editor tricks (key
bindings, syntax highlighting, code completion, multi-pane editing) with some
"flavor of the day" gimmick (git) thrown in. IDEs have been around for more
than 20 yrs and the most popular IDEs have not advanced the state of software
development much at all.

Compare IDEs with chip fabrication as an easy to compare example. Improvements
and advances in chips can be directly tied to advances in fabrication, the
same can't be said for software because there is almost no innovation
happening in those products despite appearances.

~~~
jbattle
I'd love to see some advances in IDEs, but the author is saying pretty clearly
that the bulk of the time taken was in high-level decision making, not in
struggling to get his ideas into the computer.

> "In a sense, programming is all about what your program should do in the
> first place."

~~~
evincarofautumn
Yup. I’ve been working on a project off and on for a few years now, where _if
I had known_ exactly what I was building in the first place, then it would
have taken me just a few weeks. But writing it (and rewriting it) was the only
way to actually _arrive_ at that knowledge.

------
CrLf
Because software development is actually research disguised as engineering.

(And that's why "engineering" when related to software is filled with mumbo-
jumbo and cargo-cultism.)

~~~
jacquesm
That's true in some cases, and in those cases time/budget overruns are
probably more acceptable than in CRUD app #34.

~~~
CrLf
It's also true for CRUD app #34, because the hard part is never the
implementation, it's dealing with shifting requirements, either due to the
nature of businesses, unsufficient understanding of the problem being solved
or sabotage from project stakeholders (conflicting agendas, politics, etc.).

When you plan to build a bridge between A and B, everybody can see that the
problem being solved is connecting A to B. You then have to match material
knowledge and construction experience from other bridges with the specifics of
the new location. Also, you define project completion when you can cross the
bridge.

When you start out building a "simple form" to solve some business need,
nobody agrees on what that need is or what the form should actually solve.
Navigating this is research. Also, nobody can tell when the project is indeed
finished (i.e. solves the problem).

~~~
jacquesm
That does not begin to explain the sometimes orders of magnitude difference
between estimates and final costs, elapsed time and required manpower.

I've sat in on meetings with people discussing a new project, it is absolutely
incredible how much of the future derailing of a project you can see happen in
real time during those preliminary meetings.

A probable clue you can find in this nice video of a product meeting between
'engineering' and 'customers':

[https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be](https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be)

------
MyNameIsFred
> _In a sense, programming is all about what your program should do in the
> first place. The “how” question is just the “what”, moved down the chain of
> abstractions until it ends up where a computer can understand it, and at
> that point, the three words “multichannel audio support” have become those
> 9,000 lines that describe in perfect detail what 's going on._

This closing paragraph is excellent.

While I do wonder if the author's progress along these wouldn't have been
_accelerated_ if more of this evolution took place before the first line of
code, the overall message here is very true and very well put.

When I made this realization for myself, it was a major turning point in the
quality, speed, maintainability, and usability of the code I produce,
especially when I can successfully define the final code structure to directly
reflect this progressive refinement from intent to implementation.

~~~
tremon
_I do wonder if the author 's progress along these wouldn't have been
accelerated if more of this evolution took place before the first line of
code_

Doesn't that require much more thorough interface specifications and
documentation than we have now (industry-wide)? I don't do that much software
development anymore, but I have always been put off by the amount of trial-
and-error required because of ill-defined interfaces.

~~~
MyNameIsFred
This is true.

I must admit that my more recent, positive experiences are only possible due
to my accumulated understanding of my dependency chain and toolkit. I work on
multiple things, but my largest, indefinitely-running one has only a single
dependency...but it's a truly beastly, poorly documented enterprise system. It
took me almost 2 years before I properly understood the system's intended
usage, interfaces, intents, bugs, and pitfalls.

The JavaDocs are awesome in their uselessness:

    
    
      /**
       * Sets the Wimbulator.
       *
       * @param wimbulator
       *     the Wimbulator to set
       * @param identifier
       *     the identifier to use
       * @param flags
       *     flags
       * @param legacy
       *     used to apply legacy
      */
      public void setWimbulator(Wimbulator wimbulator, String identifier, long flags, boolean legacy);

~~~
JoeAltmaier
Hey! That's every JavaDocs I've ever encountered. See the Apple CoreAudio docs
for instance. Auto-generated and never revisited.

Such doc systems are a crime - the Manager asks "Is the documentation
complete?" and then Engineer says "Well the auto-docs were created but..." and
the Manager says "Good, lets move on"

~~~
majewsky
On the other side of the spectrum, we have something like the Qt
documentation:
[https://doc.qt.io/qt-5/qwidget.html](https://doc.qt.io/qt-5/qwidget.html)

------
shados
If you read the comments so far, everyone has their favorite. The answer is
probably "all of the above".

That said, my personal favorites:

Most software engineers are bad and refuse to admit it to themselves (which
would let them get better). Too much ego. Given a large enough
department/team, only a small percentage is actually doing significant work.
Yes, even at Google/Facebook/Amazon/whatever.

Second, software engineers are extremely conservative. For all the flack fast
moving ecosystems like JavaScript get, the only reason we see so many
iterations is that the end goal is visible from far, but people reject it. So
we just make 100 intermediate steps to get there. Eg: a subset of functional
programming paradigms and type systems. Think TypeScript...which has to be
comfortable for the peanut gallery, while trying to support advanced features
many know need to be there in a modern language, and the struggle between
those 2 goals.

That ensures we're going slower than we need to, while basically ensuring
those techs will be obsolete sooner than later (because we need a stepping
stone to the "right" solution as to not alienate developers)

~~~
clifanatic
> Most software engineers are bad and refuse to admit it to themselves (which
> would let them get better)

Well… not to be confrontational, but in the context of this post, you seem to
be implying that if most software engineers (which would, statistically
speaking, include me) were “better”, they could put software together faster.
I have to say, the implication bothers me, as does my suspicion that it’s a
sentiment shared by most non-programmers out there. The problem is, there is a
lower limit on how long it can possibly take to complete a programming task -
and I think even the MBAs will concede that that lower limit is higher than
the time spent typing out the words that make up the program. So, if we accept
the premise that “good programmer” = “fast programmer” with no reasonable,
objective measure of how fast fast is, the people doing the judging will
always end up using the metric “how long I wanted it to take”, which is
usually about a day for any task.

~~~
shados
I did say the answer to this problem was "all of the above". We're talking
about human beings dealing with each other. That means there's an infinite
amount of problems. The ones I mentioned are just a few out of many. They're
just often overlooked by engineers for obvious reasons.

I hate to use the 10x developer meme...but I will. Personally, I've been
around a while...and I worked at a lot of places. A lot of obscure little
startups, some non-tech companies, many big and huge names... And 10x doesn't
even begin to describe the difference between what some companies consider a
"super star" and some consider "bottom of the barrel".

I've seen big name tech companies do with an army in a year, what some of the
engineers I worked with would have done in a few weeks, alone, -in the same
environment- (thats the important part).

Once you've seen that happen enough, you just can't deny it anymore. The
difference between engineers is so great, it matters a whole lot. At one
company, a PM will come with vague or shitty requirements and the engineers
will run away with it and produce garbage over several months... while another
dev would just call it out for the bullshit it is, point out what's wrong and
pump out a correct solution by the end of the day.

And all that doesn't even consider the time some will spend googling basic
shit while others can just infer the right thing by thinking about it for a
sec. The end result is a massive difference in $$$.

------
drinchev
Once I outsourced my side-project, since the pay per hour ratio worked better
for me, compared to what I earn, combined with the complexity of that project.
The "only" thing I had to do is write specs.

The specs took me 3 weeks to write ( finding a lot of logical "bugs" in my
project ), but the detail I went to ( SQL schema suggestions, framework
suggestions, front-end plugins suggestions, ready pixel perfect design with
different states, urls for the different pages, etc. ) was one of the biggest
benefit for the final price. The company that took the project estimated 4
months and delivered it in 3 and half. They said those were the best specs
they've ever seen.

Moral of the story : Do full specs before you start coding ( I know it's
boring ).

~~~
VladimirGolovin
I did the same for my outsourced side-project - I went full waterfall.

The spec turned out to be 250 pages long (without pics) and took almost a year
to fully develop from the conception stage. In addition to this, I hired a
designer to develop an additional design spec for the UI.

So far, this worked great. My total time spent on this project since the
moment I gave the spec to developers is about one workday in total. The
questions from the devs are mostly trivial, and solvable in an hour or less.

------
Futurebot
\- Requirements that change mid-stream, dozens, sometimes hundreds of times

\- Team has to do speculative development for ill-formed or understood problem
domain

\- Requirements that are improperly gathered

\- Requirements that are underspecified

\- Poorly understood problem in general

\- Fighting bugs in system libraries

\- Fighting bugs in third party libraries

\- Fighting issues with integration on deployment and/or third party anything

\- Complex underlying technology requiring its own kind of discovery (like in
the OP)

\- Poor/incorrect documentation

\- Project members that go off on their own, refuse to communicate, deviate
wildly from the project style, break things

\- Management that insists on doing things from scratch

\- Management that insists on using buggy, broken, etc. 3rd-party or even in-
house systems/libraries for political or faux-business reasons

\- Constant software developer interruptions

\- Long meetings that consist of circular discussions, politics, back-and-
forth arguments about personal preferences disguised as "important project
stuff"

\- Administrative minutiae that never stops adding up

\- People leave a project partway through, taking project knowledge with them

\- Broken/buggy tools/environments/systems

\- People getting pulled off a project to work on another project

\- Bad use of project management, source code management, other tools

\- Insistence on wheel reinvention for well-understood problems for career
advancement, resume building, or boredom-relief purposes

\- Low morale / poor focus due to bad management, co-workers, bad tools, bad
work environment

\- Low morale / poor focus due to reliance on: late night work (leading to
insufficient sleep), weekend work, little/no time off or breaks, reliance on
crunch/death marches

\- Dealing with company/project politics in general

\- Lack of focus due to team member resume-polishing/interviewing part way
through a bad project

\- Occasional project sabotage

~~~
majewsky
Hello, Mr. Brooks. Didn't see you coming in.

------
p0nce
Business-time goes seemingly faster than engineering-time. Engineering
requires luxurious calm and ample amounts of time to achieve anything of
value. Hence the disconnection, the business will ask "why do engineers take
so much time?" while the engineers ask themselves "why does the management
changes the plan so much?". But a key point to have successful engineering is
not panicking upon this perceived, unbearable, uncontrollable slowness and
still invest in quality: capital instead of debt.

------
hex13
It looks for me that despite what author wrote, he WAS indeed in exploratory
phase: `So now we have a sequencer device, how do we get events from it? Can
we do it in the main loop? Turns out it probably doesn't integrate too well
with Qt, (...)`

`My initial thought was making a grid of spinners,(...) but then I realized
that there isn't an easy way to make headlines in Qt's grid. (...) So after
some searching, I found out that it would be better to have a tree view(...)`

And many similar things. Exploration and technical spikes - nothing wrong with
that but author wrote: ` It's pretty common to do so if you're in an
exploratory phase, but in this case, I had a pretty good idea of what I wanted
to do right from the start, and that plan seemed to work. `

and then he contradicted himself writing about various difficulties and
explorations... (I don't see anything wrong with that, absolutely. I think
that exploratory phase is essential to good design).

~~~
icebraining
I think the author means exploratory as in writing throw-away prototypes to
test an idea. There's obviously always exploratory work during development,
unless you're reading everything from an extremely detailed spec.

------
koolba
Software development is 5% inspiration, 20% perspiration, and 75%
procrastination.

~~~
eludwig
>> and 75% procrastination.

I laughed at this and it's true. But I would like to champion a bit of
procrastination in creative pursuits. I know the way my mind works and the
simple truth is that sometimes it pays to wait until my head has worked
through what feels like an incessant cycle of building up possible approaches
and relentlessly tearing them down.

So lets say I'm given an assignment (or a stand-along idea occurs to me). I
immediately start to visualize possible outcomes and the approaches that I
would need to take to get there. I throw out 99% of these subconsciously. Some
of these ideas don't even make it into a full-blown thought, but they are
there lurking below the surface. All of this takes some time. Subjectively, it
just doesn't feel like the right time to start yet.

This may look like procrastination, but it isn't. I'm not arguing for always
waiting until every detail is mapped out in my mind to begin an effort, but
sometimes it pays to let your head work through stuff for a bit.

Of course, moderation in everything (including moderation).

~~~
MyNameIsFred
Yes!

Sometimes I will have a have a big problem that I will decide not to fix
immediately, even if it's more "important" than other things I'm working on.
Then, when I come back and rapidly write a simple solution, the question of
course is "Why didn't you do this a week ago if the solution was so simple?".

The answer is that the answer wasn't simple at first. Understandings of a
problem and solution often live and die many times over in a mental background
thread, before they ever reach a keyboard.

Sometimes, when under pressure, I am forced to ignore this fact and start
writing the moment I reached the "Yeah, I _think_ that would _probably_ work,
let's git 'er done!" phase. In almost every such case, it takes longer overall
as I iterate through incomplete solutions and rewrites.

------
Tistel
Its impossible to accurately estimate an unknown. Once you have experience, a
rough sketch of the solution will pop into your head right away, but as you go
along all these devilish little details will pop up. Also, meetings run by
people who can't write code who just want to talk to people for an hour
because it breaks up their dull day will eat a decent chunk of time. When
starting a new project and in the "give me estimates phase", I do my best,
but, I know they are essentially made up numbers. If I were in charge, I would
eliminate time estimates. Just have high level goals, keep breaking them down
(or up, in bottom up dev) and have lots of automated test cases. I guess like
that continuous dev (which I have never done, just read about) style.

~~~
douche
> Also, meetings run by people who can't write code who just want to talk to
> people for an hour because it breaks up their dull day will eat a decent
> chunk of time

Is there any other reason for weekly status meetings?

------
dqdo
Software development takes a long time because of the users. If you think
about every piece of software that has ever been written, they have been
created to solve a human problem or resolve a human concern. As we build more
impressive systems, our expectations also change. A state of the art website
just 5 years ago is unacceptable today. Think about all the work that we now
have to do in order to make things mobile responsive. Features such as chat
and video, which were revolutionary in 2004-2005 (when Youtube and Facebook
were found) are common requirements of projects today. In additional to an
increase in user expectations, we have to understand that when we undertake a
project it is to solve a new problem whose implications cannot fully be
understood from the beginning. In most software projects, the problems only
becomes clear after spending months building it. Once we have the software as
an artifact, we will find new ideas and new extensions that we could have
never imagined before. Additionally, a large part of our software has to
interact with other artifacts in the real world. These artifacts are built by
other people and continuously change. A large part of maintaining a project is
ensuring that is still compatible with all the programs that it depends on.

~~~
icebraining
_Features such as chat and video, which were revolutionary in 2004-2005_

They were revolutionary in 1968: [https://www.youtube.com/watch?v=yJDv-
zdhzMY](https://www.youtube.com/watch?v=yJDv-zdhzMY)

~~~
wolfgke
Chat and video was revolutionary in 2004-2005 _on the web._

~~~
jacquesm
The hell they were, I pioneered that in 1995.

------
scrrr
Because it's an art. It's like painting a picture. Good art takes time.

~~~
jacquesm
You're just asking for it:

[http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm](http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm)

~~~
rkeene2
Reading through the article, and having no way to comment there, I'll comment
here:

> It is true that both painters and programmers make things, just like a
> pastry chef makes a wedding cake, or a chicken makes an egg. But nothing
> about what they make, the purposes it serves, or how they go about doing it
> is in any way similar.

This misses the mark almost entirely, from my understanding. Hackers and
painters CREATE something -- it does not exist and then it does. The wording
Paul Graham uses is "makers", but it's not what I feel so maybe that is the
criticism being expressed in this part of the article ?

The purpose of artists who paint and hackers /IS/ the same -- to create. To
get what is inside of you out. To have the expression which has no form to
take form and be in the world as something that can be shared with another
soul. The methods are the same -- extract what is inside of you into concrete
form, mold it, remove the bits that are not right, add more bits, move the
bits around.

Certainly this is not true of computer programmers, which the article refers
to hackers as but this is a failure of the article not of the original
comparison. Nor is it true of all painters, since the word is too generic,
which is a failure of Paul Graham's choice of wording for this comparison
since based on the rest of the reading that is what he is referring to.

Computer programmers program computers, perhaps soullessly.

Painters paint things, perhaps soullessly.

Hackers hack because they need to.

Artists that paint paint because they need to.

As to the rest, I don't know if Paul Graham's intention was to attempt to
borrow "coolness" from artists, but for me it is not. It's an attempt to
explain why one must hack and why that is an identity and not an activity or a
profession.

------
mloranger2
Okay, so developing a "multi channel audio support" feature took three months.
Is that expensive? What would this type of feature cost to implement without
software? Answer: a LOT or even borderline impossible. Look, to a large degree
software like this replaces what used to have to be done by electronics (or,
more likely, not at all). The fact that it took three months is a borderline
miracle. Oh, you want to make this process faster? Okay -- time to get the AI
community involved because I don't really see how human beings are going to
perform any faster than they already are. Look, my point is that anyone who
claims that this could be done faster or is the result of "bad engineers" or
anything else is basically delusional IMHO. 'Nuff said.

------
stevefeinstein
It doesn't. It takes as long as it takes. It's a constantly evolving moving
target. Why does writing a book or making a movie take so long? It's a
creative thing. It's hard.

------
jimjimjim
Following on the analogies already presented here:

Coding is not construction (that's the compiler). Coding is the
architect/drafter.

If you want a pre-planned house, no problem, that's the same as buying shrink
wrap.

If you want to add a swimming pool under then that a whole bunch of
conversations, customization and checking.

If you want a completely bespoke house then then that is a conversation that
lasts for months and the requirements will always change.

Think about that Grand Designs tv show. that is software and the architect is
software development

------
Pica_soO
Because a lot of stuff gets written several times. Doesent matter wether its
agile or waterfall, and i dont care who started it - customers or software-
engineering, wether the rewrite happened before roll out, or while the thing
was already in the wild. Software development takes so long, because we tear
down what we build and redo it, alot of times over.

------
amelius
What is the best resource which explains an outsider why developing software
is hard and takes a lot of time in general? I'm thinking of a small accessible
book (perhaps even a cartoon) that starts with a bunch of analogies, and then
explains why those analogies are true in real life (this last part is
important).

~~~
vram22
This may not be the "best" (not sure if there is a single best), but is pretty
well known:

[https://en.m.wikipedia.org/wiki/The_Mythical_Man-
Month](https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month)

It also has a 20 or 25 year later anniversary edition.

I like his chapter / point about "No silver bullet" in the 2nd edition.

Edit: He (Fred Brooks, the author) was in charge of the OS/360 project (OS for
the IBM System 360 mainframe series). A pretty large project of the time -
years long.

The cover image is a good analogy - tar pits.

------
oneplane
Because software is a half-visible moving target, that's why.

~~~
prefect42
I like your answer the best, quite succinct.

Software is / has way too many intangibles, by its nature. If you try to show
the customer some tangible progress every three weeks, that's all very good,
but strongly highlights the "intangiblesness" (sorry made up word) of it all.
Once the customer starts seeing some "real" feature take shape, then the
feature requests _really_ start pouring in, and any previous understanding of
this feature before this point, becomes a dim ephemeral dream in everyone's
minds. Constant moving target.

------
ausjke
Maybe most programmers are amateurs? If you're really good at data-
structure/algorithms/etc and practice them like a surgeon frequently, maybe
the landscape will be very different.

"Anybody can code, even an idiot can learn to code in 24 hours"

------
pasta
The 'readable' button in Firefox was not working for me. Maybe others also
would like to paste this in the url bar:

    
    
      javascript:document.body.style.width='600px';document.body.style.margin='0 auto';void(0);

~~~
StavrosK
Firefox is wonky with its "readable view" button, huh? I visited the page, it
wasn't showing a button. Came back here, clicked again, and it did show a
button. Clicking it worked well.

I don't know why it's breaking so easily.

~~~
pasta
ha! you are right. A revisit made the button appear.

------
lazyjones
It doesn't really, unless you overgeneralize your solution (just like the
rhetorical question in the title) and make wrong toolset choices.

20 years ago we also had fewer choices, hence fewer bug-ridden abstraction
layers and moving targets. We mastered one programming environment and target
platform, both were much less complex than what we have today. We could also
tack something together quickly in an awkward way without putting the code on
a world-visible platform as part of our resume. Also, more precautions are
necessary today, like security, general code quality (no more tricks that work
only on one compiler).

------
njharman
Cause I (we?) spend too much time reading articles like this instead of
coding.

------
seanwilson
If you were delivering very similar projects several times (in which case you
should be reusing existing code anyway), estimates would be a lot easier and
more accurate. As most software projects are using a combination of tools,
people, requirements etc. that have never been combined before, accurate
estimates are always going to be difficult.

I routinely read that estimates get better with experience. The only way I see
this as true is in that you learn to broaden your estimates the more risk that
is involved, not that you eventually are able to give accurate and narrow
estimates.

------
0xmohit
I'd be tempted to say that a lot depends on being clear on what needs to be
achieved. I've personally seen deadlines slip _mostly_ for the lack of clarity
on _what_ needs to be done. An analogy might be to say that what was initially
described as an elephant turned out to be a giraffe.

Of course, there are instances when the actual approach, implementation or
choice of technologies tend to weigh. But the latter is much less frequent.

------
tsewlliw
Why does some software development take so little time?

My suggestion for the answer is that when the goal of the software project is
selected carefully in the context of the surrounding ecosystem you can connect
a few things and all the real magic was in picking the goal.

When instead you pick the goal and then figure out how to achieve it, who
would be surprised that there is often more work involved after picking the
goal?

------
dockd
I suspect it has something to do with a) The programmer not actually using the
software they create. They end up spending time trying to understand the
domain. b) The programmer uses the software they create and wants it to do
everything (see also Second System Effect, Zawinski's Law/Law of Software
Envelopment). c) The programmer expects components to work as documented.

------
imichael
To me it all boils down to this: hardware is all about making billions of
things, all identical. Software is about building innumerable things on top of
that hardware, all different.

What would you like it to do, what can be done, how to do it, how to organize
a group of people to do it, how to make sure it works and stays working, it's
all creative work and it takes time to do it well.

------
tyingq
_" Why does software development take so long?"_

I can see this as a valid question for a subset of problems. For example, the
basic database CRUD applications that get built over and over.

The closest solutions I've seen were DabbleDB (acquired by Twitter and
shuttered) and Quickbase (laudable, but the pricing model doesn't work for
many).

------
wickedlogic
Software development takes a long time because we write code still. Thats
essentially the crux of it. Humans then plan/optimize efforts on the wrong
layers, and underestimate the cost of updating code over time. Surprisingly so
at scale.

The systems that tend to produce the fastest (that also last) results are the
one's that give the developer the most control over the environment to get the
task done (LISPs). Or one's that generate code for you, and let you deal with
abstract flows as the control unit. We don't really use either of those things
in industry... but niche segments do to great success.

------
keithnz
my real takeaway from this article is the amazing feat of typing at 800
characters a minute!

according to [http://smallbusiness.chron.com/good-typing-speed-per-
minute-...](http://smallbusiness.chron.com/good-typing-speed-per-
minute-71789.html) a good typist does around 335 cpm

so this is way over twice as fast

is this a typo and supposed to be 80? that would be very slow.

~~~
Nadya
My guess is that is with "bursting" and is not very sustainable. Not to
mention, being able to type so quickly is very impractical when you need to
think about what to say (well....type) and with programming especially, typing
speed is irrelevant.

To give a comparison - I can type 700~ cpm steadily, which means I can
maintain it for large periods of time. Only practical when copying text when
OCR fails me or when doing data entry. With "bursts" (unsustainable but rapid
typing) I can type 160-170wpm with a negligible error rate (<1%) but I can
only keep it up for a few minutes. Realistically, I type closer to 110-120wpm,
because I take longer pauses between words while I consider what I am about to
say (again....type).

My hunch is the author actually meant 800cpm, but I do have my doubts to the
accuracy or susustainability of the figure. It isn't "impossible" but it is
"The top 1% of the top 1%" levels. :)

[http://i.imgur.com/NW2Kq4p.png](http://i.imgur.com/NW2Kq4p.png)

------
ilaksh
Why does it need another thread if ALSA uses poll()?

------
eagsalazar2
Buildings are totally different in that they are all, roughly, the same or
small variations on the same design. Consequently it is very easy to reuse
massive amounts of reusable components in extremely predictable ways and to
define standards for review that can be broadly applied across the majority of
projects.

This is not the case when builders are doing something that is actually
totally novel in that they don't have many existing examples and it is built
of entirely custom components. In cases like that it is actually exactly like
software in that there is a lot of figuring things out as you go along,
schedules are usually laughably off by orders of magnitude, and there are
frequently major screw ups (Big dig, Seattle big bertha fiasco, SF Bay Bridge,
... to name a very few)

Of course in software we aren't always reinventing everything as people
unfairly complain. We reuse way more than we reinvent - postgres, elixir,
phoenix, browsers, OSX, etc, etc, etc represent 1000s of lifetimes of work and
the majority of the totality of work behind an actual product. The time we do
spend, and consequently the variability in schedule, is due to the fact that
most software is actually quite different in its requirements from all
previous software and usually previous examples (aka competitors) have not
made source code available for reuse.

One huge difference between software development and "custom" building
development is that good software developers embrace this variability and
uncertainty - hence agile. Most other fields of engineering are still caught
in an archaic and delusional world of gantt charts and hard deadlines with
feature lists developed by architects. It never works that way insofar as the
work is _custom_ - in any field!

The thing I really hate about this conversation is how so many people are so
eager to just chalk all this up to software developers being half-assed hacks.
Well there are lots of teams at places like HP staffed by only "certified"
software engineers and guess what? They still are terrible at predicting
outcomes and they are actually far far worst in general at their jobs than
small, modern, agile teams who make the best of the chaos by addressing that
reality in how they work (again, agile).

This is software development people. It isn't about bad devs and lack of
certification, it is just hard. For comparison I did integrated circuit design
for 7 years right out of school up through being a lead on multiple larger
teams before switching to software. The rigor you perceive in other areas of
engineering is a total fantasy. Bridges and buildings are maybe the exception
but again this is only possible because it is such a canned art by comparison.
If you disagree please just enumerate some specific "building codes" or
"certification requirements" that you think would actually make any difference
on your average complex software project.

------
draw_down
Because nobody really knows what they are doing.

~~~
jacquesm
That's roughly the gist of it. There are exceptions but not many.

