
What we actually know about software development, and why we believe it’s true - ZeroGravitas
http://www.slideshare.net/gvwilson/bits-of-evidence-2338367
======
swombat
It looks like it was a nice talk, but unfortunately it failed to address _the
reason why_ there is so little reliable science when it comes to software
development.

The reason is, of course, that we have yet to come up with a reliable,
meaningful metric for programmer productivity, one that would apply to the
lone ruby on rails coder as well as to the enterprise java cog-in-the-great-
machine or the ThoughtWorks elite agile programmer in a 6-people iterative
team.

Until there is a meaningful way to compare the productivity of different
software methodologies, it is pointless to berate the lack of scientific
studies comparing the productivity of these methodologies.

As a final nail in that coffin, I don't think anyone can claim that Google is
lacking in the "let's measure everything" department, with their reputation
for intensive A/B testing of every single pixel change. And yet Google has yet
to resolve this problem... which would tend to indicate that this is a real,
non-trivial problem, not just a case of people not trying hard enough.

~~~
nopassrecover
I'm of the opinion that the reason there is no quantitative metric for
programmers is the same reason there is no quantitative metric for writers,
artists or musicians.

~~~
gaius
Yet there are quantitative metrics for engineers. Is your Ruby on Rails
application _really_ more complex than the software flying an airliner?

~~~
prospero
There are metrics for problems which are studied in-depth before engineers are
let loose to solve it. That's not true of most software projects.

~~~
gaius
Who do you think does the studying? Engineers...

~~~
derefr
You're agreeing with the parent. There are metrics to measure _solving_ well-
known problems, but no metrics for _researching_ problems to make them well-
known.

~~~
gaius
But the first person or team who realized that you could put a computer in an
aircraft weren't solving a known problem - however they were using engineering
methods, which is why it's very rare for avionics to (literally!) crash.

Here's my theory: we _do_ know how to build very high quality software in a
repeatable way. However very few organizations are willing to pay what it
costs to do that, and _that_ is why most software is so bad. Admittedly a lot
of people writing code these days are "cowboys" (they don't care and neither
do their clients, they just want speed) but that doesn't mean that the body of
knowledge doesn't exist.

~~~
pmichaud
I don't know why this comment was voted down, but I think he could be right --
given the correct resources (time mainly), and assuming the right talent
level, I think we could consistently build rock-solid software. The constraint
seems to be a (logical and necessary) trade off between total cost and total
utility.

As we all know, something that works 95% of the time at 5% the price is very
often better than a "perfect" solution. With critical systems like plane
navigation, the tradeoff doesn't make sense, so you pay full price for near-
perfection... your niche e-commerce site? 95% is sufficient.

~~~
Agent101
Plane navigation is in someways easier to e-commerce sites as well. The
environment that the plane is in, isn't actively malicious. You can collect
accurate statistics on weather, where as the attacks and usage (with
bots/scraping etc) consistently change.

~~~
Quarrelsome
Not at all man. The physical environment can be much more malicious than you
seem to think.

I've done some software for buses, it's toy in comparison (with planes) but
you get issues with noise, power supply, vibration, temperature, light (UV
killing a touch screen) and probably lots more i've never encountered. The
physical world can be every bit as unpredictable as the virtual one.

~~~
yters
The major difference is that environmental agents are generally deterministic
or stochastic, i.e. obey the law of physics. On the other hand, realms with
intelligent agents, such as humans, are inherently unpredictable due to their
free will (which is unpredictable by definition).

~~~
Quarrelsome
You also have the free will of humans messing with the system (in my case,
fraud prevention), also consider animals interacting the machinary while
someone is not watching. I remember hearing a story about a huntsman spider
regularily tripping a sensor for a program that measured the weight of
lorries.

The only consistent thing you can state is that all these "real world" issues
are to do with hardware whereas yours are more softwarey. However the
consideration of such issues is necessary across the board.

------
forensic
Most professions choose their best practices through a form of natural
selection and evolution. Rookies try all the different ways and 30 years later
they are Veterans. The Veteran then passes on the best ways he learned to the
Rookies, who copy it, and then over the course of their career slowly improve
on it as well.

Even the most "simple" task like scaffolding (for instance) has heaps of
"craft" in it. What research team decided that scaffolders should thread
scaffold clamps between their fingers when they carry them? There is a subtle
technique to this that has been passed down through generations.

This is how all professions learn the tricks of their trade. People try stuff,
figure out what works best for them personally, and then pass that knowledge
down to the rookies. Certainly with programming this happens at university -
I've had profs who give practical advice like, "Make sure your program is
always capable of compiling throughout the entire dev process." This is folk
knowledge yes, but it's practical folk knowledge that HAS been tested over the
course of careers by many people.

It is not economical to have many dedicated research scientists trying to
figure out these things. They are figured out by people as they do their job.

The most important thing is mentoring. Paul Graham widely shares the things he
has learned from experience and those things HAVE been legitimately tested
because he has tried ways that failed and tried ways that succeeded. Just
because it is not double blind does not mean that it hasn't been tested.

~~~
groby_b
_It is not economical to have many dedicated research scientists trying to
figure out these things. They are figured out by people as they do their job._

... it is not economical to have many scientists trying to figure out the
cures for illnesses. They are figured out by people as they treat other
people.

Is that what you really want to say? Basic science does not have any immediate
economic pay-off, no. But it often pays in spades later on.

 _because he has tried ways that failed and tried ways that succeeded_

These are ways that worked for PG - and we have no idea if luck is involved.
See his latest article - luck plays a large role in all startups.

I'm not saying mentoring is bad. In fact, it's pretty much the only thing that
produces any results right now. But creating a solid foundation for the
engineering side of computer science doesn't strike me as a bad idea...

------
bbraasch
he makes some excellent points, or should I say hypothoses?

drop 25% of the features to reduce complexity by half. I suppose this is why
templates and re-usable code appeal to managers moreso than software
developers. one sees the clock and the other sees something that would be much
more beautiful after a redo.

my son learned in mechanical engineering school that if you want the best
design and the best device, don't ask the builder to design the device. the
builder will make decisions based on the complications of the build process
and the desire to build. the designer would focus on the utility of the device
and defer thinking to the builder on how to get the thing built.

People with different Myers Briggs score do better in pair programming. Raise
your hand if that makes your brain hurt.

Mythical Man Month is the classic book on this, based on the OS360 project at
IBM.

Complexity drives time to complete toward infinity. Says the moth to the
flame, "you're so beautiful". Says the teacher to the class, "let's not get
ahead of ourselves".

A fellow named Francis Frank used to teach project management as a position
that sits between the hydrant and the dog. The scrum master takes that role
now, looking for a favorable outcome from myers briggs score diversity.

I especially like the point that defects are not a factor of geographic
distance, but of distance on the org chart.

------
hxa7241
If you look at research papers in software engineering, you will notice a
significant thing has happened in the last several years. With the rise of,
first the WWW, and then open-source repositories, there is now a large body of
code available that was simply absent before. It is now usual/common for
research to investigate, test, experiment against that. This must be a
positive change, that we should be hopeful about yielding more substantial
results.

------
adamc
Must have been a great talk, because even the slides are good (and normally
the slides are almost useless alone).

~~~
Maciek416
He definitely got the most thunderous applause of the day at DevDays simply
because so much of the day's talks thus far had been horrible. I was following
the Twitter feed of the conference and after his talk many people remarked
that he had basically saved DevDays Toronto from being a failure.

~~~
9oliYQjP
I was hoping Joel would send around a URL for feedback. I didn't stick around
to see if that ended up happening. The problem wasn't that the talks were
horrible in and of themselves. But it appears that he totally underestimated
some of the audience's abilities. I couldn't have been the only person there
that's used ASP.NET MVC, Python, jQuery, and Ruby. For us, the talks weren't
insightful because they were all introductory.

That said, I chatted with some attendees who thought the talks were great
because they only program in ASP.NET (old style) and have never been exposed
to these sort of languages. I guess they're the developers that ask the
questions on StackOverflow and people like us are the ones answering the
questions? :-) Or maybe I was just unlucky enough to know about the languages
they decided to focus on. Had there been an introductory talk on Haskell or
Clojure, I'd have learned something new.

------
known
Selling Software != Selling Consulting

Hence it depends on type of the company.

------
tybris
Programming is still a craft. If you want to know how it should be done, just
look at the masters. Plenty of successful products out there that can serve as
an example.

Until we develop actual computer science (rather than calling publicly funded
engineering science) that's all we have.

~~~
colomon
Ummm... you do realize that the slides point out scientific studies of
programming with interesting, practical results, yes? And that the author of
the slides is a professor of computer science doing more such studies? One of
his points in the talk was to call for the professional programmers in the
audience to participate in these sorts of studies.

------
russell
A couple of points buried in the talk: defects increase as the code size
increases. The increase is greater than linear. So my question for folks is
why would you program in Java rather than Python, where you are guaranteed to
have more bugs, even with static type checking?

The issue of programmer productivity has been around at least since The
Mythical Man Month, but the generic responses have been better estimates and
better methodologies. There is even the Software Engineering Institute to make
life worse for all of us. I think that except for academics and programmers
themselves, no one cares. Large companies used Cobol and now Java because it
is safe. Everyone else is doing the same. I once asked an executive why he
picked Java over Python for a greenfield reimplementation of his site. he said
it was because there were more Java programmers than any other kind.

~~~
pohl
"A couple of points buried in the talk: defects increase as the code size
increases. The increase is greater than linear."

I went looking for these points in the slides, only because the slides got me
in the mood to question my assumptions and try to follow citations back to
source material - and it wasn't obvious that the presence of something on a
slide meant that it was a known truth. (Example: the slides about Martin
Fowler's claims about DSLs and productivity are in the slides as a skull on a
pikestaff, not as a verified truth.)

After looking, I think you got the first point from slide 15, but I wasn't
able to find a slide that mentioned non-linearity.

I like the question that you're asking, with regard to Python versus Java. I
wonder, though, if that conclusion really rises to the standards of rigor that
this presentation appears to advocate.

Even if you manage to dig up the right Lutz Prechelt citation, has the effect
really been shown to be something that is completely unmitigated by static-
provability?

~~~
russell
Slide 25: "Most Metrics' values increase with code size". The point about non-
linear increase in defects is my (unwarranted) extrapolation. An earlier slide
quotes a 25% increase in complexity gives a 100% increase in size.

No, it really doesnt rise to his standards of rigor. Everything that I have
read, supporting that conclusion is probably anecdotal. The formal studies I
have read have usually involved college students on short projects. Studies
that are probably worth less than useless. Others probably suffer from the
Hawthorne effect.

