
Study of 49 programmers: static type system had no effect on development time - akkartik
http://www.cs.washington.edu/education/courses/cse590n/10au/hanenberg-oopsla2010.pdf
======
fusiongyro
The study may or may not be flawed, but what's really interesting to me is the
reaction. We need more science in our computer science, which means more
experiments and more results like this. We should also be open to the truth
that we use the tools we like because we like them rather than because they're
technically superior, even though we pimp them ad nauseum as though they are.

I once read an article about a technique Intel had developed for improving
cooling of processors by changing the shape of the fan. I related this to some
of my co-workers. One of them proceeded to tell me that this can't possibly
work, backing up his argument with "reasoning" based on off-the-cuff remarks
about the way air and physics "must" work. The fact that Intel had _actually_
done this seemed to have no effect on his eagerness to continue the "debate"
about this scientific _fact_.

I have a hard time believing that I get no benefit from using Haskell over
Smalltalk, but if a body of science were to appear that cast doubt on that
belief, the appropriate thing to do is _change the belief_ , not stand around
debating from imagined first principles why all the science is wrong and can't
be so. Shut up, design an experiment and go prove it!

Perhaps there's little of this kind of actual science in our computer science
because it would mean asking hard questions and accepting difficult truths.
"The prisoner falls in love with his chains."

~~~
ExpiredLink
> We need more science in our computer science, which means more experiments
> and more results like this.

There is an ongoing experiment wrt the usability of programming languages,
libraries, frameworks, concepts, ... It's called the market.

~~~
madhadron
Counterexample: JavaScript. It is the best demonstration I know of that our
choice of languages and tools is mostly based on historical accident rather
than any technical criteria.

------
cromwellian
Studying individuals is the wrong experiment. Static typing benefits tooling
most when you have larger groups of programmers who much collaborate, often
asynchronously. It's easy to hold a mental model of the code you write in your
head and minimize mistakes, it's harder to hold a mental model of a larger
program consisting of code written by many programmers.

I'd like to see them hand a pre-written codebase of say, 10,000 lines to a
bunch of students, and ask them to make large changes/new features to the
codebase.

~~~
pjmlp
You are hitting exactly the main point.

Dynamic languages are very good for prototyping or small scale projects.

But they fail to address the context of programming large scale applications
with teams distributed across multiple sites.

~~~
elderberry
I'm betting youtube is bigger than all of your sites combined, and they use
python.

~~~
xymostech
And good on them. But that doesn't mean it's easy to write large websites in
dynamically-typed programming languages, it just means that they are _very
good at what they do_.

~~~
quadhome
No, it doesn't. It doesn't mean anything except that your belief isn't
supported by any scientific data (as far as presented in this thread).

------
tikhonj
If anything, I think this study is actually _in favor_ of static typing. Why?
Well, most of the proponents of dynamic typing claim it makes development
faster. Most proponents of static typing concede this point, but claim that
while it may slow down initial development, it makes maintenance much easier.

Of course, this study is fairly limited and drawing conclusions from it is
probably premature. I would not base any policy decisions on it alone.

But, if static typing does not slow you down, it seems worth using even if
benefits to maintainability are slight. And if they're nonexistent, using a
statically typed language over a dynamically typed one still doesn't hurt you.

Besides, I think static typing is awesome. And I'm a _college student_ \--you
can trust me! That should be all the validation you need :).

~~~
quadhome
No, it isn't. Please read the paper and not the HN title.

The paper clearly states that static typing _slowed development time down_.

~~~
tikhonj
"In the experiments the existence of the static type system has neither a
positive nor a negative impact on an application's development time (under the
conditions of the experiment)."

Maybe the abstract is lying, but that seems to agree with my original premise.
That is, the type system had no effect on development time.

Now, the paper makes no comment either way--all it says is that the type
system did not have an effect on development speed. My argument is that not
having an effect is actually a point in static typing's favor: the main
advantage of dynamic typing is supposed to be faster development, after all.

~~~
quadhome
Read further than the abstract. It's just an abstract.

To quote:

 _We measured two different points in the development: first, the development
time until a minimal scanner has been implemented, and second the quality of
the resulting software measured by the number of successful test cases
fulfilled by the parser. In none of these measured points the use of a static
type system turned out to have a successful impact. In the first case, the use
of a statically typed programming language had a significant negative impact,
in the latter one, no significant difference could be measured._

~~~
okamiueru
Statistically speaking, 'a significant negative impact' doesn't mean the same
as it does in regular English. It doesn't say much about how large the
difference is, only that they have enough data to show that there _is_ a
difference. When an abstract summarizes that the difference wasn't much to
talk about, that does mean a lot more than you give credit.

Now, judging from their test setup, and also from the very low number of
testers, I find it very hard to agree with what it seems to be you're taking
away from this article. Static typing has more benefit in large code-bases,
with multiple programmers, and for avoiding hard to find bugs related to
dynamic typing. This doesn't seem to be well reflected by the setup they had.

Also, don't think you should repeat the same comments throughout the HN
articles, you are not replying to individuals, but to general readers.

~~~
rbanffy
The part quoted said it clearly

 _In the first case, the use of a statically typed programming language had a
significant negative impact_

Considering the size of the sample, I'd guess the difference is rather large
to considered significant. By looking at the numbers quickly, it seems to be
around 25%. I'm a bit shocked, in fact, because in my own experience, the
difference is much larger, but this experiment controls for language and my
experience doesn't.

~~~
okamiueru
Yes, I was able to read that in both your comments, as well as in the paper.

I suspect you are not understanding my point, or what I said about the meaning
of the use of the word "significance" when used in statistics.

 _edit:_ Say you flip a loaded coin that is 50.1% likely to be heads. Now you
want to test whether this is loaded, and flip the coin a certain number of
times and count the outcomes. If the number of times you flip the coin is too
low, you won't be able to say the coin is 'significantly loaded'. It might be
either way, you don't have enough data. If you flip it enough times, you will
be able to say something about it -- i.e. that either it significantly is, or
significantly isn't loaded.

However, in vernacular English, you would still say that the difference isn't
very significant. Who cares if it is 50% or 50.1%.

~~~
sampo
There is no reason to quarrel about the meaning of 'significance' in
statistical testing vs. in natural language. Just look at figs. 4 and 5 in the
paper, and see that mean times spent for the scanner task were:

5.17 hours (dynamical typing) 7.71 hours (static typing)

And that the difference in statistically significant (p=0.04, Mann-Whitney
U-test). Whether 5 vs. 8 hours is significant in the natural language sense,
everyone can decide for themselves.

------
thomasz
This is an interesting paper, but I'd dismiss the findings as a strong
argument in the dynamic vs static typing debate. I think the setup is a little
bit skewed as it's plays into the strengths of dynamic languages:

* The whole api consisted of 14 classes, which makes the conclusion that static typing doesn't help with API discoverability somewhat moot.

* We are not talking about Haskell, Scala or even Java/C++ flavor of static typing. The statically typed language offers no generics, no support for encapsulation, no type inference etc.

* IDE support seems to be absolutely minimal.

A methodical problem is that they seem to skip over the abysmal success rate:
Only one(!) guy or gal was able to get 100%, more than halve of them was
unable to implement a meaningful parser at all. To be honest, it looks a
little bit like the subjects were overwhelmed by the unfamiliar syntax and/or
most of them where really inexperienced (the paper says that none of them ever
implemented a scanner or parser).

~~~
leothekim
The scope of the study also seems suspect to me. From the paper:

"The introduced study is a relatively large study with 49 subjects and
required for each subject approximately 27 hours pure development time.
Including the teaching time for each subject, between 43 and 45 hours working
time was required per subject."

If I were working on a software project where I expected 30 hours of
development work, I'm not sure static vs dynamic typing would matter at all.
I'd pick the language that would get the job done in 30 hours.

I'm surprised this was accepted in a peer-reviewed conference. I'd give this
study more credence if this involved the same number of subjects working on or
inheriting a much larger, longer-lived code base. Though, based on the
subtitle of the paper, it sounds like the author could be convinced otherwise.

------
Ogre
The conclusion contains a paragraph that matches my initial reaction to the
headline and opening. "It seems that the experiment at least does not
contradict with a common believe that is often articulated in the area of type
systems: that dynamically typed systems have a benefit in small projects while
this benefit is reduced (or even negated) in larger projects." (It goes on,
but I can't seem to copy actual text out of it, I typed that).

My feeling about this has always been that dynamically typed languages will
save time on initial implementation and small projects, and statically typed
languages will win on large projects and things that require more maintenance.
They acknowledge this, and didn't find anything to contradict it.

Their own graphs seem to show that effect too. In some areas, dynamically and
statically typed languages were equal, while in others dynamically typed
languages had lower times. I don't actually see where the conclusion that a
"static type system had no effect on development time" came from, it appears
that it had an overall (if not very large) negative effect. Just like I would
expect for a small (27 hours) project.

I didn't crunch the numbers though, maybe I'm just eyeballing it wrong.
(again, I can't seem to copy-paste out of it, and I don't feel like reentering
all of them)

------
kaffeinecoma
I wonder whether they are actually measuring the right thing here. I think the
real value to static typing is not so much the initial development of small
(tiny, in this case) programs, but rather large systems that have long
lifetimes and maintenance periods involving multiple developers.

I just started diving into Ruby. I'm about a month into it, after having spent
over a decade doing Java and other static languages. The productivity boost
for new development in Ruby is very real; I converted a 10K LoC java program
to < 1K LoC Ruby in about a week, despite not knowing the language well at
all. It almost feels like a "runner's high" to be so incredibly productive.
But there are several things that I find frustrating about dynamic typing.

In dynamic languages it's very hard to navigate existing code and libraries.
In Java, you can click on a variable and instantly navigate to its declaration
and class definition. In a well-configured environment (say Intellij + Maven)
you can even click on imported libraries and drill right down into their
source, even the JRE. It's totally natural to know exactly what a method takes
as arguments, and what it returns.

In Ruby it always feels like a mystery. When you ask the IDE to jump into a
library, it pops up a dialog of 20 choices and makes you guess which one to
open. It feels like the old days of running find/grep over the source tree.
And if you somehow manage to guess correctly and find the code you're looking
for, you often have to read the comments (if they exist) to figure out what it
expects/returns. I'm sure a lot of this is due to my inexperience, but code
navigation is really painful in a dynamic environment. I can only imagine how
hard it would be in a non-trivial project after a couple of years with a few
different devs.

Of course refactoring is way better in a static environment. Even if you throw
out the IDE, you can more or less instantly find where a piece of code is used
by simply renaming it and recompiling to see what breaks. In Ruby you have to
rely on tests for that kind of thing, instead of getting it for free from the
compiler.

Don't get me wrong- I'm excited about Ruby, and I think dynamic languages will
play a major role in the rest of my career. I just wish that code navigation
could be better.

~~~
seunosewa
Depends on the IDE. I use Python with Eclipse. Auto-completion works quite
perfectly. Refactoring too.

~~~
mjallday
I disagree. There's simply no way that the IDE can know everything about a
variable without running the code and I don't believe that eclipse does this.

It's ok at refactoring but it breaks down all the time for me, renaming
unrelated variables or missing cases. I don't believe it can handle __kwargs
for example (no computer in front of me, feel free to prove me wrong)

~~~
Peaker
> There's simply no way that the IDE can know everything about a variable
> without running the code

Or with. With dynamic typing and dynamic name resolution such as in Python,
there's no way to know with certainty, even if you run it.

~~~
tonyarkles
In theory, you're right. In practical, day-to-day programming, an IDE can get
it right the vast majority of times and do exactly the right thing. See a
previous post of mine: <http://news.ycombinator.com/item?id=4037602>

~~~
Peaker
I believe that "works almost every time" is great, but that last inch is also
meaningful. Knowing you can trust your IDE's refactorings is reassuring and
allows you to dare do things you might otherwise not dare.

------
FollowSteph3
My big problem with this study is that the project sizes and teams are too
small. Static typing really starts to get it's power as the team and project
get bigger. This is especially true as the skill levels from larger teams
start to spread from amazing all the way down to terrible. In smaller teams
you can often work with higher quality developers, but with larger corporate
systems, you don't have that luxury. You have to deal with all kinds of
experience levels, etc. You need to be able to modulize the project, etc.

From my personal experience, dynamic typing usually works well with 1-2
developers and continues to be useful until about 4-5 developers (static
typing too for that matter). After that, static typing usually wins pretty
consistently.

My personal preference is static typing. I find it makes the code more
readable and catches a lot of oopses early, at compile time. Otherwise you
just end up writing a ton of test cases and hope to catch the same issues at
runtime. So you have less protection (you have to catch it with a test), and
you just end up spending more time writing tests anyways.

I do understand the benefits of both, I just prefer static typing because of
the way I work. Both work and have their places. Like everything else, you
need to use the right tool for the right job AND the right people.

~~~
jshen
I work for a fortune 20 company and my last employer was a fortune 10 company.
My experience is the opposite of yours.

I've never seen these consistent wins you talk about, despite my years at some
of the largest corporations.

~~~
tmarthal
I don't think the size (or the revenue) of the company matters as much as the
size of the projects that you're working on within those companies.

Someone mentioned maintenance, so I will also ask how old the codebases that
you're working on are? For dynamic typed languages, as the code grows older
(with more bug fixes and add-ons and one-offs) then it gets harder and harder
to maintain as 100% error free (let alone debug).

I will also throw in the fact that doing software library version upgrades on
dynamically typed systems is a pain in the ass. Some code branches have system
level calls changed, and unless you have very good unit test coverages (which
may or may not need to be re-written on a library version upgrade) then you
may or may not catch the problem during the upgrade. With a statically typed
language, a static analysis tool can tell you exactly where and how the
library or interface upgrade affects your codebase.

~~~
jshen
Large projects, at least 4 years old.

"I will also throw in the fact that doing software library version upgrades on
dynamically typed systems is a pain in the ass."

This is very true, but you gain a lot with dynamic languages in other areas.
The last company I worked for had about 300 programmers in the office I was
in. Roughly half did Java and the other half did Ruby. I was one of the few
that moved between both ruby and java projects. I did not see these great
benefits of static typing that people always insist exist for large teams at
large companies.

------
mduerksen
I'm still torn between dynamic and static typing.

The main disadvantage of dynamic typing for me is the lack of _verifiable_
documentation about what data a function needs, and which data comes out of
it. This becomes a problem when

a) the data is complex (e.g. dictionary of lists of items with certain
properties)

b) I haven't looked at the function for a while.

In those cases, I find statically typed code easier to reason about.

On the other hand, it absolutely maddens me that static type systems force me
to spell _everything_ out, even where it's trivial. It slows me down, it
bloats the code (especially when you have to create a new class for
everything) - which again makes it harder to see the _purpose_ of the code
instantly.

That's why I decided to make my own little pragmatic experiment: In cases
where complexity is expected (or experienced) - and _only_ there- I use the
Clojure pre- and post-conditions to check the _shape_ of selected parameters
and/or its return value. It looks like this:

(defn german-holidays

    
    
      "Returns map of german holidays, in the year of d."
    
      [d]
    
        {:post [(like {(date 1 1 2012) :easter} %)]}
    
       ...)
    

(defn calendar

    
    
      [c start-date end-date]
    
      {:pre [(like {:appointments {} :new (list)} c)]
    
       :post [(like {(today) {:occupations [] :appointments 

#{}}} %)]}

    
    
       ...)
    

You can instantly see how the data is supposed to look like, it will be
checked automatically, and it is close to the actual code where you need it
(unlike e.g. Unit Tests).

I'm not yet sure how this experiment will fare in the future, so any
suggestions or warnings are appreciated.

~~~
akkartik
I use tests for documentation. They seem like more work at the start, but
they're more flexible. You can self-document things with tests that you cannot
with just types.

For example, here are some tests from the lisp interpreter I've been working
on: <http://github.com/akkartik/wart/blob/8a8cf96816/030.test>

You're right that tests aren't close to code. But that can be good or bad.
Since it's not next to the code it can be more verbose and thorough without
adding a constant reading burden. And when I wonder, "why is this line here?"
I can comment it out and run the tests to find out.

I think of tests as records of the input space of my program. Most of the time
we go to great lengths to make explicit what we do but not quite what aspects
of its behavior we truly care about. Where are the dont-cares in this program?
If I refactor it am I restricted to a bit-for-bit identical behavior? In
practice that's impossible, and without tests we fudge random parts of the
state space without rigorously knowing what is important and what isn't.

~~~
cageface
Static type systems don't excuse you from writing tests. But they do
effectively write a whole class of tests for you, and run them much faster and
give you more precise errors than manual testing ever will.

People complain about C++ compile times but I've heard of big Ruby projects
with test suites that take 20+ hours to run.

~~~
rbanffy
> I've heard of big Ruby projects with test suites that take 20+ hours to run.

I'd assume those tests go well beyond type checking.

~~~
cageface
Sure but remember most static languages are 20+ times faster than Ruby so even
with compile times you might still come out way ahead.

~~~
aidenn0
Ah, but a lot of bugfixes are developed by making small changes to understand
better what is going on internally during the bug symptom, and there the
compile time is more important than the compile/test time.

~~~
cageface
True, and languages that are slow to compile like C++ really suffer here. But
Java and Go compile so quickly it's hardly an issue.

------
phamilton
I recently talked to a friend of mine who expressed the frustration of not
knowing what a function returns because it's a language without static typing.
It was something I hadn't thought of before personally. I thought about it a
bit and realized that as a Ruby programmer, I depended on two things to figure
out what a function returns, and neither is the source or documentation (most
of the time at least):

1) a REPL. Just give it a whirl and see what you get.

2) intuitive method names and good abstractions. If I call @user.posts in a
Rails app, I can assume I will get an Array-like object that I can treat as an
array without any problems. The actual type does not matter 90% of the time.

~~~
ww520
So you basically put type information in the names. That's typing in a sense.

~~~
lloeki
No, he puts typing in the interface, and that's indeed typing in a sense,
called duck-typing.

~~~
ww520
I think we are talking about different things. Duck-typing doesn't help in
depicting what the return type of a function is when looking at the function
signature only. You need to look at the function implementation (the actual
duck) to tell what the return type is.

~~~
phamilton
My point I guess was that with proper abstractions and duck typing you really
don't have to care about the type. I only notice something isn't a duck when I
try to use it like a duck and it fails - which is surprisingly infrequent.

~~~
ww520
It becomes difficult to tell it's a duck when reading code, which was the
original complaint. Duck type works when you actually try calling it.

------
econner
The study followed 49 undergraduates who had taken Java classes. I would argue
that, because of this, the students were probably more experienced with static
type systems.

~~~
mck-
A good point. Also, 49 programmers is very little. But even with thousands of
developers using all sorts of languages, it's still moot to draw any
conclusion from. It's very subjective and dependant on people's
experience/taste/State-of-mind..

For me, switching from static (java) to dynamic typing (common lisp) has saved
me a lot of development time. Or did it? Maybe it was Emacs? Maybe it was
functional programming? Maybe it was the REPL? Maybe experience?

~~~
heretohelp
I think in a lot of cases programmers are happiest with their new hotness
simply because they became better and more experienced programmers in the
process of learning the new hotness, not because the new hotness is 'good'
necessarily.

Cf. node.js

------
stiff
It seems to be very basic static typing along the lines what Java or C has
stashed on top an Smalltalkesque object-oriented language, so I am not
surprised, the many years I used dynamically typed languages I maybe once or
twice had a type error of the kind "confusing int with string" (e.g. when
selecting a number from database and it being returned as string).

For the benefits of static typing I would rather look into something like
Haskell which has a much stronger notion of type and builds many language
features around this notion. It's probably harder to design a reasonable study
comparing that with for example Smalltalk, though.

~~~
pubby
I'd be more interested in comparing Haskell to one of those even stronger
typed programming languages. I'm forgetting the name of them, but it would be
interesting to see how much is too much.

~~~
njs12345
Are you thinking of Agda/Coq/Idris, which have dependent types? If you
understand a bit of type theory, this is an interesting paper:
<http://www.cs.cmu.edu/~fp/papers/popl99.pdf>

------
jfoutz
so, by swiching from python to haskell, you can get code that's 20x faster for
no extra development time? sounds like a win to me. (cherry picked shootout
data - [http://shootout.alioth.debian.org/u32q/which-programming-
lan...](http://shootout.alioth.debian.org/u32q/which-programming-languages-
are-fastest.php))

~~~
pelotom
Not to mention being able to rule out large classes of errors at compile time,
and ease of refactoring large code bases. You mean I get all that for free?
Where do I sign up?

~~~
comex
Obligatory language argument: If by ease of refactoring you mean having to
change everything whenever anything changes just to avoid a little messiness.
:)

------
fauigerzigerk
I don't think you can show much of anything in 27 hours of development time.
Neither the advantages nor the drawbacks of static typing will have any impact
on productivity for such a small one off, single developer, throw away
project.

It seems the only question this experiment could possibly claim to answer is
whether or not the additional finger typing required by static languages
affects the coding speed of these particular 49 undergrads.

------
Deregibus
49 undergraduate students working independently on a small project over the
course of a single week using a previously unseen language with no
consideration of the performance or maintainability of the resulting code. I
can't see how there are any meaningful conclusions to be drawn from this study
other than "inexperienced programmers find learning new dynamically typed
languages easier".

~~~
Xcelerate
You seem to be subtly implying something about people that use dynamic
languages.

~~~
Deregibus
Not at all. When learning any new programming language there's going to be a
period where you're working by trial and error. Removing compile-time typing
generally means less syntax to learn which makes the feedback loop tighter
which I would imagine gets you the point of familiarity quicker. The study
seems to back that idea, but not much more.

------
AlexFromBelgium
In my own experience: if I write dynamic code I'm still thinking of types.
Especially in Python.. What's mutable, what isn't... This is what my
professors at school say too. Less typing (pressing keys) doesn't make you
think faster!

~~~
aangjie
Seconded. Even though i write python code, i am still thinking of types.
Infact, it's one of my gripes with the language. There are some tired mental
states, when i don't think/forget to think about types, but just write and
call functions. It's at those times that i find having to go back and read the
original function for a type or having to go to REPL for testing the type
painful. I would be happy to have a compiler tell me what type is expected
instead. I guess more experience will lead me to infer this from the actual
error message itself.(ex: Nonetype has no function iter)

~~~
lloeki
You should strive to not rely on type-checking, and trust and rely more on
duck-typing. If at some point you need to ensure something is not None, test
it and act accordingly in the alternate case. If something absolutely must not
be None (or must have a specific method), ever, use assert() so that things
blow up upfront where you can infer (or even read) the reason, and not deeper
in the code where the actual error message, as you mention, has to be parsed
according to the whole call stack to make sense of it.

Sometimes you may need to type-dispatch when you want to do smart functions. A
typical example is a function that takes a sequence/iterable/generator of
strings _or_ a string. Since a string is iterable in python, you have no
choice but to check for BaseString.

------
route66
I'm a bit puzzled which role programmer ability has in this experiment. What
strikes me is the enormous spread in programmer quality within each group.
Varying from 0.7 to 15 hours for the scanner assignment and 50% to 100% of
succeeded test cases. The spread occurs over both groups.

That programmer quality shows way stronger effects than the measured effect of
a typing system would suggest to me that

\- the measurement was flawed?

\- programmer ability beats language choice by orders of magnitude?

Or do I make a category error here?

~~~
rwallace
That programmer ability beats language choice by orders of magnitude, is
already a familiar fact, so that would seem a satisfactory explanation for the
observed variance.

------
Rickasaurus
What about maintenance? I've never heard a static-typing proponent claim they
can develop faster. It's always that they can maintain the software with much
less work and have less overhead at run time.

------
lambdasquirrel
I've wondered about languages that are "hybrid" in the sense that the type
system can be turned on and off, because from an intuitive sense, whether the
language is meant to be used as a dynamic language has an impact on its
design, similarly with a static language.

You take Smalltalk for example, as it is the original archetype of a language
that benefit from blurring the lines of the type system. A language like
Haskell on the other hand was design with the intention that the programmer
would use the type system to do a lot of heavy lifting.

Of course, no real-world language is "pure" in that sense. We've graft type
systems onto Smalltalk's "way of doing things", and the GHC Haskell compiler
now allows you to treat type errors as warnings.

~~~
tikhonj
The change in GHC is just to make development easier. Basically, just imagine
that instead of stopping at a type error, the compiler replaces the incorrect
function with code to throw an error at runtime. So here the poorly typed
function _always_ errors, no matter what you pass it; in a dynamically typed
language, it would work as long as the arguments passed at runtime have the
correct types.

So it isn't anything like dynamic typing, or even quite like making type
errors warnings--it's just a way to compile partially incorrect code to make
changing the code incrementally easier. It's also not designed to be used in
production.

It just automates the existing practice of temporarily commenting out
offending functions and replacing them with `undefined' to get the whole file
to compile.

------
takluyver
Let's summarise what the study actually found. The researchers go in with the
expectation that static languages will have advantages in their study. They
designed a brand new language - 'Purity' - in both dynamically and statically
typed variants, so there was no difference in documentation or other language
features. The only difference is that one forces you to declare variable
types, and checks them at compile time rather than at run time.

They take 49 students and train them in the language - 16 hours for the
dynamic version, 18 for the static, because of "the additional effort for
teaching the static type system". The students have 27 hours to complete two
related tasks. All activity was logged, so they could look at the time from a
test run failing with type errors to a successful test run.

\- Scanner task: overall, students using the dynamically typed language were
significantly quicker to complete a scanner that passed the tests - on
average, about a third faster. Within that, the time spent debugging type
errors doesn't differ between the groups.

\- Parser task (builds on the scanner): The implementations are tested after
coding has finished, showing no significant difference in success between the
languages. In both groups, about half the students didn't manage a meaningful
parser at all. In this part, students with dynamic typing spent significantly
less time debugging type errors.

------
ozataman
I think there are several flaws in this study, but alas, its subject is quite
inflammatory and it is easy for people to gloss over the details and start
drawing conclusions.

Here are a few things to keep in mind, IMO:

\- Good/capable/flexible static languages are much much harder to design than
dynamic ones. Case in point is Haskell, where hundreds (thousands?) of
researches have been working for years to widen the area of expression vs.
area of safety. The devised language for this study seems very minimalistic to
be useful. Most dynamic languages, on the other hand, pretty much let you do
anything even when they have a simplistic design.

\- I find that people take longer to become fast/useful in static languages.
While you can just throw things together and tweak until it works with a
dynamic language, a static one will force you more to know what you're doing.
I had a semi-technical colleague who had a much much easier time getting up to
speed with Python than with any other static language. Therefore, unless the
study involves a long prep time, it will naturally be skewed towards lower
devel time in dynamic.

\- Static languages start shining as the project size/time-line grows. Key
affected areas are refactoring, intra-team communication and incremental
improvements on the code. To test for this, it would be interesting to stage a
time lapse where developers are asked to modify their previous parsers after,
say, a month of total separation.

I think a more interesting study would be to hire 20 highly decorated Python
devs and 20 highly decorated Haskell devs and give them a few different
computational tasks, overall adding up to a large # of hours. Naturally, the
study needs to limit usage of 3-rd party libs and take other normalizing
measures. Then we may get a sense for what the state-of-the-art in both fronts
have to offer.

~~~
swah
Real world tasks or math tasks?

~~~
ozataman
Well, definitely real-world tasks with focus on usefulness and other practical
concerns. Ideas would be:

\- Implement a parser for some protocol/data with testing, etc. \- Implement a
communication protocol/framework/driver/API \- Implement a sophisticated-
enough simulation for a real-world question you're trying to address certainly
with some calculations inside. \- Implement a game AI as defined by some spec
\- ...

------
Uchikoma
No negative effect of static type systems for green field development sounds
great to me.

I do like static type systems (like Scala) more than dynamic ones (like Ruby),
and always thought it would perhaps people slow down. Good to hear it doesn't.

One would need to look into brown field development, if there is any
positive/negative effects (I assume positive ones)

------
quadhome
Looking over the comments, WOW, please please PLEASE read the paper! This
title on HN is very misleading. A static type system had no POSITIVE effect on
development time. The study is casting doubts on POSITIVE impact of static
type systems.

To quote:

 _We measured two different points in the development: first, the development
time until a minimal scanner has been implemented, and second the quality of
the resulting software measured by the number of successful test cases
fulfilled by the parser. In none of these measured points the use of a static
type system turned out to have a successful impact. In the first case, the use
of a statically typed programming language had a significant negative impact,
in the latter one, no significant difference could be measured._

~~~
TheCondor
This study is way to small and limited to show much of anything... but many of
the proponents and historical engineering studies that advocate static typing
don't claim that it reduces "development time" so much as reduce the
times/costs throughout the entire life cycle of the product. In fact, the
"development time" or in this case "prototype time" hasn't been all that
terribly interesting.

What would be interesting is to take these same projects, and same team
members and add some features a year later and see what that looks like and
then take the same project and give it to a new team after 6 months and have
them add some minor feature or fix a bug and measure that. Or double the
performance of the project and measure what that costs, if it's possible.
etc.. It's fun to argue and advocate for what you like though.

------
manmal
Is it not so that development time is always maxed out automatically?
Programmers improve the system as far as they can, and the study explicitly
says that it did not address "design time, readability or maintenance time".
Since software is almost never perfect (unless constructed with proofs, and
you can make optimizations even there), I would think that programming time
depends on the programmer's willingness to invest yet more time into its
features - and this willingness is most likely not related to the language in
use, but rather to financial or temporal constraints. Am I missing something?

------
dleimbach
tl;dr embrace polyglot :-)

If static typing wasn't useful, I don't think anyone would bother with types.
At the same time some static type systems are restrictive while coding, so
you're seeing more and more type inference in languages.

Go does it by allowing one to declare a variable with an expression " a := 1".

C++ 2011 has it with "auto"

Haskell is pretty darned good at it, and a lot of people don't write the types
of their functions, though I think it's good practice to make sure you and the
compiler agree on what you've written.

That said there's languages like Clojure which are dynamically typed in a
sense, and I believe pay a slight performance penalty because they have to use
reflection to behave that way. There's ways to annotate types in Clojure to
get around that. This is an interesting case because Lisp is typically a
dynamically typed language.

At the same time there's awesome languages and environments like Racket that
have statically typed and dynamically typed versions of their scheme dialects.

My feeling is that a ruling on whether types are useful or not is sort of a
pointless discussion. It's not even true that "the jury is out". You're just
choosing tools with different properties and some make sense for some
situations and others for others.

------
awlo
This study, while perhaps not very relevant to real world software engineering
(development, maintenance, refactoring), seems to show that there when
developers design code and applications, there's little difference for them
between using dynamically or statically typed languages. And that's
interesting, because it implies that the way developers think about code is
somehow type independent.

------
yason
The static versus dynamic typing "battle" is kind of from the kindergarten.
You need both, similarly as to how both boys need their fathers even though
both of them get declared the strongest in the world in turns.

With dynamic typing it's much more difficult to create large, reliable
codebases. There's too much implicitness among the interactions of different
actors, modules, and programmers.

With static typing it's much more difficult to create _any_ codebase. The
exploratory nature of programming means that you will have to constantly
change and redefine the static types, signatures and interfaces.

What you want is a language where you can start prototyping dynamically and
later on when the APIs stabilize gradually you can then stamp static type
checking on top of them for cementation without actually having to rewrite
anything, possibly in another language. When you can selectively force a
freeze on a certain piece of code or API then you can gradually move from
dynamic to "static enough" typing as the part of your development process.

~~~
weavejester
If you want to reduce implicit interactions between components, using FP over
OOP will probably give you larger gains that static typing over dynamic
typing.

------
chmike
While interesting, I don't think we can consider the conclusion as the final
word on it. There are many other variables to take in account.

The programmer's expertise is an important one and it is difficult to take in
account. Static typing is to check code validity. If the programmers make no
mistake, then of course specifying types is an overhead. When it comes to
collaborative work, strong typing can save a lot of debugging time, meeting
time and documentation writing and reading time.

Another aspect is the resulting code efficiency. Code is programmed once and
executed many times. It is much easier for a compiler to generate efficient
code when it is given more information to do so by the programmer. This
benefit of the typing overhead is not taken in consideration.

As repeatedly said, use the appropriate tool for your application.

------
Superpig
"According to the known guidelines for empirical research in software
engineering (see for example [16, 17]), it is necessary to make explicit all
elements that potentially threaten the validity of an experiment."

Or, y'know, according to _any basic understanding of science whatsoever._
Jesus.

They're essentially trying to test whether static typing optimises development
time, and as with any optimisation, you only see a substantial increase if
you're optimising the bottleneck. Static typing _won't_ show any speed
benefits if your coders are being slowed down by weird abstractions, poor
tooling, or analysis of the problem.

The experiment didn't focus on use of the type system, but nor did it actually
test anything close to a real-world scenario. All in all, it seems worthless
to me.

------
dbecker
The author appears to misunderstand statistical significance when he makes
claims such as "the existence of a static type system has neither a positive
nor a negative impact on ... development time."

The mean development time for dynamically typed languages is about 30% shorter
in his first experiment, and it is about 20% shorter in the second experiment.

Statistical significance test determines how likely this difference would be
if the difference development time in a large population was 0... It doesn't
mean that the true effect actually is 0... it's just addressing whether we can
completely rule that out.

A more accurate summary of his results would be "Dynamic type systems allowed
20% to 30% faster development in my experiments, but I had too few
participants to reject equality."

------
takeoutweight
Absence of evidence is not evidence of absence.

~~~
Paradigma11
But it is, at least from a bayesian perspective.

~~~
dbecker
Not at all.

Even from a Bayesian perspective, having a small sample that is too small to
reach strong conclusions does not shift your posterior towards 0.

------
arihant
Development time with static languages is not an issue.

Try comparing the time one needs to change the codebase written in ML and the
one written in Lisp and come back. It is experimentation and change that is
slow in static typing.

~~~
dan00
"It is experimentation and change that is slow in static typing."

It's all about the expressiveness of the language. If you can express more
with less code, than it's also easier to change, to experiment.

If you change something in a dynamically typed language you still have to
catch all places, where the change has consequences.

Only If you want to experiment with something locally without needing to
update the rest of the code base, than dynamic typing has an advantage.

I think it's GHC 7.4, which allows something similar for Haskell, to have the
typing checked at runtime, to allow this kind of experimentation.

I think that's the best of both worlds, because for the production code you
can still activate the compile time type checking.

~~~
dan00
"I think it's GHC 7.4, which allows something similar for Haskell, to have the
typing checked at runtime, to allow this kind of experimentation."

It will be in GHC 7.6.1 and the extension is '-XDelayErrors'.

<http://hackage.haskell.org/trac/ghc/ticket/5624>
[http://hackage.haskell.org/trac/ghc/wiki/DeferErrorsToRuntim...](http://hackage.haskell.org/trac/ghc/wiki/DeferErrorsToRuntime)

------
mck-
An interesting study, but really hard to compare. It's like attempting a
discussion of the effectiveness/efficiency of Engligh vs French (using
conjugations or sex).

~~~
_delirium
And in particular, trying to compare the effectiveness of English vs. French
by using 49 undergraduate students who weren't proficient in either language.

------
Tharkun
Considering there are flawed assumptions (ass, you, me) in the abstract, I
can't be bothered reading the rest of the paper.

> [Decreased development time means] that the quality must be improved since
> developers have more time left in their projects.

What kind of nonsense is this? What's this assumption based on? Hearsay?
Shorter development time = better quality? What?

> [Increased development time means] decreases the code quality.

Again: what?

------
tluyben2
Maintenance... Dynamic would be faster for me to write in from 'scratch' (I
find it weird it has NO effect on dev time, but then again the study isn't
really broad), however after your sprint(s) changing things is really much
easier with static typing. That's what I feel at least, especially with
Haskell (and Hoogle) (which unfortunately I don't get to use that often).

------
tszming
Steve Yegg talked about this several years ago [1] and I think it is worth
reading.

Caution: Not an academic research paper, but a real case study from Amazon.

[1] [https://sites.google.com/site/steveyegge2/is-weak-typing-
str...](https://sites.google.com/site/steveyegge2/is-weak-typing-strong-
enough)

------
webreac
This study is flawed: the usefulness of type system is not for initial
development of a small system, it is for the maintenance of huge code base.

The fact that type system had very small cost for initial development of small
system should be seen as an invitation to use strong types.

------
azth
It could also be the case that it is faster to write text parsers in dynamic
languages as opposed to static ones. There are so many different variables to
come to a meaningful conclusion. What about working on very large programs
with other people?

------
1123581321
Type restrictions benefit larger teams, working on larger projects, with
longer periods of time between looking at the code (working with legacy code,
in other words.) A larger study should be tried.

------
gana
Static type system wont have any issue with development time..

Its the bug fixing time that it affects.. A dynamic language is harder to bug
fix than a static one..

------
Lednakashim
Don't we are about the ability of the compiler to optimize? Does anybody
compile code nowadays?

~~~
aidenn0
Meh, dynamically typed systems are within an order of magnitude of statically
typed systems, even on benchmarks that greatly benefit from the static type
information.

------
gouranga
What about debugging and fix time and regression rate? Those are the things
that concern me.

------
papsosouid
This study starts with a bizarre premise. I've never heard anyone, ever
suggest that statically typed languages make initially writing a program
faster. Very much the opposite, java apologists claim that their slower
development time is offset by easier maintenance and less bugs, explicitly
acknowledging that initial development is slower in a java-esque static
language.

Of course, the static typing proponents you find on HN are generally not
advocating incredibly limited type systems like java or the one used in this
"study". I find it hard to believe that a language with such a limited type
system is representative of languages with useful type systems like ocaml or
haskell.

------
gringomorcego
Well I think the article hints at the real benefits from type systems. They
took longer in the beginning understanding the type system, but made up for it
later on by being able to finish in the same amount of time. Static systems be
balling for refactoring.

Hopefully for the next experiment they have them tackle a longer project, and
have the programmers tackle short problems in multiple languages to 'rank'
them.

------
drivebyacct2
1\. They didn't ask me.

2\. They ought to check on maintenance effort and time with different type
systems as well.

------
bitsweet
I tend to find the strongest proponents of static typing are the least
familiar with dynamic languages

~~~
tikhonj
I don't know. I have an equal (equally small :() amount of experience with
both. And, amusingly, I've disliked both.

What I found after using Java, Gosu (basically Java but less stupid), Python
and JavaScript fairly significantly is that all of them have their own
problems. In that particular set, I'd probably value a static type system _a
little_ more than a dynamic one, but it gets eclipsed by other language
characteristics (e.g. support for functional programming).

In fact, thanks to being scarred by Java, for the longest time I really liked
dynamically typed languages more. However, I later realized that it was the
lambdas and the higher-order functions and the clever abstractions and the
terseness that I liked more than the dynamic typing.

Once I learned Haskell (and later OCaml), I've been converted to liking static
typing quite a bit. The beauty in these two languages is not just that the
type system catches _more_ bugs than Java's while being less of a burden, but
also that they type system can actually make the code _more_ expressive.

My favorite example of this is the read function from Haskell. It is the
opposite of toString--it goes from a String to some value. However, the really
awesome thing is that you do _not_ need to explicitly specify what type to
parse to--the type inference can figure it out for you! Imagine being able to
use "5".fromString() the same way you use 5.toString(). In a dynamically typed
language, you would have to somehow specify what to parse to explicitly. So
your fromString code would something like Integer.fromString("5"), which is
less nice and throws away the toString/fromString symmetry.

I have about the same experience with dynamic (Perl, Scheme, JavaScript,
Python) and static (Java, Gosu, Haskell, OCaml) languages and I like _good_
static typing most.

Long story short: use Haskell!

