
Why Johnny Can’t Write Multithreaded Programs - Baustin
http://blog.smartbear.com/programming/why-johnny-cant-write-multithreaded-programs/
======
sz4kerto
> If you need a lock, then you probably have global mutable state that has to
> be protected against concurrent updates. The existence of global mutable
> state indicates a flaw in the application’s design, which you should review
> and change.

No, no, no. If you are writing multithreaded programs, you _will_ have locks,
period. The question is whether you explicitly use them or if they are hidden
somewhere in your libraries, frameworks, etc. Actually, the latter are much
more difficult to debug. Most of us have probably seen situations where
deadlocks or data corruption has occured because of some extremely complicated
interaction of message pumps - UI thread - callbacks - async operations -
etc., without any explicit locking.

> Perhaps most importantly, modern programming languages and libraries make it
> easy to create a producer-consumer application.

Yes, that's great again, until the deadlock-corruption-indeterminism problem
comes back again in a different abstraction level.

There is _no way out_. If you are writing concurrent apps, you have understand
what's happening. By concurrent I mean concurrent on the level you are working
on. Using a parallel 'map' operation is different because the concurrency is
an implementational detail from your perspective. But if you are writing a
massively concurrent quoting/trading engine, then you have to know what's
happening. And if you do, then you're free to use locks, executors, async,
whatever -- it's just a tool to solve the problem.

...

The more I look at this article the more ridiculous it seems...

~~~
temujin
"If you are writing multithreaded programs, you _will_ have locks, period."

Not necessarily. In scientific computing, it is common to encounter
"embarrassingly parallel" scenarios where each thread can solve equal-sized
chunks of the problem, and you can wait till all threads have terminated to
collect results.

edit: Okay, the final wait is usually implemented with a lock under the hood.
Nevertheless, this type of program is straightforward to reason about.

~~~
sz4kerto
And of course that 'wait' operation does not involve locks and/or condvars.

Oh wait.

~~~
lmm
Who cares how it's implemented? Manually using a lock is a smell in the same
way that goto is a smell. Wait for a task to finish is not a smell in the same
way that while is not a smell.

~~~
mannykannot
"Manually using a lock is a smell in the same way that goto is a smell."

The presence of a lock tells you nothing about whether a program is well-
designed. In fact, the whole idea that you can argue about the correctness of
a program from the presence or absence of "code smells" is absurdly
simplistic. I have seen plenty of bad code littered with while statements.

~~~
the_af
Code smells (I don't like the name either) are just warning signs. They do not
necessarily indicate an actual problem.

Incorrect use of abstraction levels, such as using tools that are too low-
level and error-prone for the task at hand (as in the majority of business
applications) _IS_ one of those warning signs. It usually indicates a
conceptual problem, and makes the code needlessly complex.

------
jaimebuelta
To me, saying: "Multithreaded programs are not difficult, you just have to
know how to do them" is extremely similar to "Multithreaded programs are
difficult, and you have to know how to do them"

I've done a lot of multithreaded programs, and, yep, you have to use some good
practices to avoid making the code explode. You know, having to learn them and
stuff is why they are "difficult". Just because you know how to get something
done does not make that "easy". That's what experience is for.

------
notacoward
His argument for "if you use a lock you did something wrong" seems to
presuppose that somebody else created handy concurrent data structures for
you. In other words, he admits that multithreaded programming _is_ hard, so
you should make it Somebody Else's Problem. I don't think that's a very strong
argument.

~~~
Nursie
Half the world seems to operate on the principle that this sort of stuff is
SEP these days, and then sneer at the people who are solving it for being _so_
low-level and not at all modern...

~~~
sophacles
A different take on this: we have a small team. We are solving in a hard
problem. Other people have solved this other hard problem for us. Let's focus
on doing our thing very well, and not spend our limited cycles on redundancy.

I like to drive the point home with this (slightly hyperbolized) statement: X
already does this, and those folks need it to be good if they want to eat, so
why the heck are we doing it ourselves?

~~~
Nursie
Agreed, code re-use is a very valuable and great thing. Where there are well-
tested and well-proven technologies to do what you're trying to do, use them
(if the license fits, it's not too expensive etc etc.)

But _someone_ has to do the 'hard' parts, and do them well, and I don't see
why competent tech folks shouldn't at least be interested in that.

~~~
anaphor
Exactly. Things like this just deter people from becoming interested in "low
level" things like compilers, runtime systems, databases, etc... and maybe
becoming active researchers in these areas.

------
henrik_w
On the subject of multithreading, I like this one by Ned Batchelder[1]:

Some people, when confronted with a problem, think, "I know, I'll use
threads," and then two they hav erpoblesms.

[1]:
[http://nedbatchelder.com/blog/201204/two_problems.html](http://nedbatchelder.com/blog/201204/two_problems.html)

------
timje1
Personally I've found that learning a functional programming language, with
strictly enforced 'no side effects' functions, has simplified multi-threaded
programming for me. I just remember back to those restrictions and force
myself to follow them, whatever the current language allows.

For reference, the language was Erlang, which I believe has the best handling
of threading that I've encountered so far.

~~~
zvrba
So you've basically experienced the benefits of actors (message-passing style,
no shared state). But you can use message-passing paradigm in imperative
languages too, though it's rather cumbersome in non-extensible languages like
C or Java.

~~~
jerf
I think (s)he strongly implied that. I'd simply flat-out say it, the best way
to learn how to program in an imperative multi-threaded environment is to
clock enough time in Erlang, Haskell, or perhaps Clojure (with as much
immutability as possible) that you can get real work done (not merely finish a
couple of tutorials) that you are forced to learn how to program in a minimal-
sharing style, then if you go back to something like Go or something, it's
really not that hard.

I think you can learn how to program multithreaded code well without that, but
you'll learn a _great_ deal faster if you use a language that kicks you in the
face every time you do something stupid (or simply doesn't let you do it). As
in, potentially an order of magnitude faster. Fast feedback is non-linearly
better than slow feedback.

Though this does assume you get to greenfield the multithreaded code. Woe
betide ye who have to retrofit existing poor multithreaded code. I don't even
know what to tell you, except you have my pity.

~~~
timje1
That's correct, I love the actor system in Erlang, and I wish I had access to
it in every other language I've used.

Building multi-threaded applications with functional programming forced me to
write pure functions - or close enough, Erlang does have _some_ side-effects
but this is just to make it more useable than the _mostly_ academic Haskell.

I've carried this approach into C# and Java, and typically end up with methods
that fit a multi-threaded paradigm by default because of their lack of state.
As a freebie, they can be effortlessly unit tested by mocking their direct
parameters and testing the return values (no need to mock or test state).

------
exDM69
I have written some multithreaded code and I can agree that it's not too
difficult to _write_. What is difficult is testing and debugging multithreaded
code.

There are lots of corner cases that you might overlook when writing
multithreaded code, and some of them will only ever happen if the threads get
scheduled in a particular manner.

Multithreaded software development is not easy, partially because in the past
we've acquired practices that don't work very well in multithreaded world.
Partially because it's a whole new world which is not thoroughly explored and
established best practices do not exist.

But all in all, too many people are afraid of multithreaded code and this is
holding us back. It's not scary or difficult, it just requires a more rigorous
approach at times.

------
stevoski
Writing a background thread that does a long-running calculation? easy.

Writing a background thread that can be cancelled immediately, that gives
useful progress information, that doesn't leave other threads indefinitely
blocked if cancelled, and that cleans up after itself if cancelled? hard.

For my single-threaded code I have great static analysis tools, IDE magic, and
testing frameworks. For my multi-threaded code, I'm on my own...

~~~
Pxtl
> For my single-threaded code I have great static analysis tools, IDE magic,
> and testing frameworks. For my multi-threaded code, I'm on my own...

That's a big point. The tooling for parallel code is developing really slowly
compared to the proliferation of multi-core processors. Most of it seems to be
"we'll make the library multi-threaded but all your code will still be single-
threaded" like we see in Node.

------
throwaway0094
Whoa whoa whoa. Slow down. When did I become an Applications developer and not
a systems developer? The kernel I work on uses TONS of threads and locks. And
sure, it's fucking crazy to debug. But I don't think you'll be prying mutable
state from the cold dead hands of kernel developers any time soon.

~~~
the_af
I think it's pretty clear if you're a systems developer (or someone writing a
library with concurrent data structures), this article doesn't apply to you.

The majority of developers aren't going to be systems developers, though. Just
as the majority of developers don't use assembly language either.

------
unclebucknasty
I think there's another simpler factor: some people are just naturally better
wired to think in terms of concurrency.

There's a certain thinking style you bring to writing and understanding
concurrent software. You need the ability to "visualize" the contexts,
behavior, and data access. You need to see the possible race conditions and
understand what resources need to be "protected" and how.

Reminds me of recursion. Back in my CS days, it just made some people's heads
explode. They couldn't think in a nested fashion to understand what was going
on at various recursion levels, nor what was required to terminate, what
should be returned, etc.

Like-wise with proofs in geometry. Some people just couldn't get there. But,
for those who could, it came without too much effort.

You kind of either have it or you don't. You can practice and improve, but
some people just naturally "get it". So, I don't think it's as simple as "they
learned the wrong approach". In fact, if they truly understood the concepts
and possessed the requisite thinking style, they would be able to identify the
"wrong approach", and the right one.

------
rdtsc
> If a multithreaded program is unreliable it’s most likely due to the same
> reasons that single-threaded programs fail: The programmer didn’t follow
> basic, well known development practices.

Well no. It is like saying mt programs are broken because the user didn't know
how to write them. It is not saying much. Everything with bugs is broken
because the writer didn't know how to write it.

The reality is that multi-threaded is hard because it is hard to reason about
and hard to get right. In one large application it only takes on junior
programmer to dip their fingers in and add that one module that will crash the
system 1 out 1000 run on customer's hardware only.

MT issues and problems on a whole new level of difficulty that is
qualitatively different than what single threaded programs experience.
Sequential problems and bugs are still there, but now there are a host of
completely new things happening.

> These programs very often fit nicely into a producer-consumer model with
> three threads:

* The input thread reads data and places it on the input queue.

Well, you also have to make sure the queue is thread safe. Understand what
does it mean to get an item off of an empty queue. Now you have queue-using-
up-all-my-memory problem if you do stream processing with mis-matching
processing rates between parts of your programs. What about objects you put in
the queue. If those have any pointers or are pointers themselves now you are
back to the original problem. So make sure objects are copies, are
serialized/deserialized or there is proper ownership handoff.

In summary, is it possible to write correct highly concurrent programs in a
shared memory model with threads/and-or/events -- yes. Is it easy? No, it is
not easy.

The article correctly points the often common fallacy that using
event/callback-based single threaded approach is superior and avoids all the
concurrency related issue. It doesn't avoid all issues. Now the race is
between callback/errback chains instead of threads. Granularity is much higher
(if incrementing a shared 64 bit int is fine without a lock). But the need to
protect shared data structures is still there. The danger is psychological
because one thinks "hey this is safe, I don't need anything to worry about"

That is why I believe it is better to use a language/system/framework that
uses isolated heaps and rigidly enforces separation between concurrency units.
Erlang has that, D has some of that, Nimrod, Web Workers and Dart isolates to
that. You incur a speed penalty for that. Don't be shy about using regular
Unix OS processes. Those are great for isolation and do it well. The cost of
IPC and cost of creating them can also be high. You in incur a penalty for
copying data between concurrency units, but remember, there is a bigger
penalty you incur when the program has crashed. The converse of that is, the
biggest performance gain your application will achieve is when it goes from
not working to working and staying working.

~~~
rbehrends
_Is it easy? No, it is not easy._

Correction: It's not easy in most programming languages, because most
programming languages fail at implementing mutual exclusion in a way that is
safe. It's basically the same problem as buffer overruns in C, where indices
don't get checked (except that in the concurrent case, most languages don't
require you to associate a lock with shared data and/or don't verify that you
hold the lock before operating on that shared data).

Per Brinch Hansen's famous rant gives the gory details [1]. The basic
technology had been figured out in the early 1970s.

[1] [http://brinch-hansen.net/papers/1999b.pdf](http://brinch-
hansen.net/papers/1999b.pdf)

~~~
Jtsummers
Trapped behind a web filter so your link was filtered as "Personal Site". I
believe [1] is the same document for others in the same situation. Also,
thanks for the link, it's an interesting read.

[1]
[http://cse.unl.edu/~witty/class/csce351/howto/PBHansenParall...](http://cse.unl.edu/~witty/class/csce351/howto/PBHansenParallelism.pdf)

------
yonran
I think the article has some good points, but he should spend more time
discussing what circumstances you can block when waiting for a result, and how
to balance too much blocking against too many asynchronous states.

I think his focus on “global mutable state” is a bit of a red herring. It
makes no difference whether the mutable state is global (protected by a lock)
or owned by a single thread. You can refactor all your synchronized {...}
blocks and replace them with executor.execute(futuretask); futuretask.get();,
where executor is a single-threaded event loop and futuretask.get() blocks for
the operation to complete. This refactoring would make a single thread the
owner of the state, but there would be exactly the same number of deadlocks.
Conceptually, you can think of your critical sections as a separate thread on
which you invoke methods and then block until completion.

After refactoring to use Executor and BlockingQueue, one still has to make
sure there are no cycles in which threads are allowed to block waiting for
which other threads.

------
joosters
IMO the problem with threads is the 'all-or-nothing' approach to shared data.
If you want to make use of multiple CPUs, you either use threads, in which
case _all_ of your process data is shared, or you use multiple processes,
where _none_ of the data is shared. It is the over-sharing that trips people
up with threads.

Ideally, we need an API that allows selective data sharing, so you greatly
reduce the chance of one thread trampling over bits of another. Right now, the
common way to do that is with multiple processes and shared memory regions,
but it is cumbersome to set up and requires the programmer to treat the shared
data very differently (e.g. one process can't just malloc data for a string,
say, and then let the other process access it.

The other route is through message-passing, but again you have to write your
code in a totally different way. Someone please invent multi-threading with
selective sharing!

------
VLM
"Perhaps you’ve accepted the common fallacy that “Multithreading is hard.”
It’s not."

Bzzt full stop epic fail. Think about the other discussion today about the
"naturalness" of zero or one based arrays.

If multithreading was natural and inherent part of all noobs, you'd teach
programming beginning with multithreading and maybe later on you'd have some
weird module which confused most of the class explaining how some legacy
languages and systems only allow one thread which has certain unusual
programming constraints that most of the class would screw up. Also all
automata theory classes would begin with the eNFA as the base class and later
on tell the kids about DFAs being broken forms of a NFA and all that. I'm just
not seeing it.

The article did fail at architectural level. Most non-threaded programs can't
be upgraded to threading because either its not worth the effort or its
impossible to solve the actual problem without threads so the postulated
single threaded solution can't exist therefore you can't upgrade what could
never have been implemented. Why are you multi-threading?

Most likely if you're adding multi-threading explicitly to a single threaded
app for an inherently multi-threaded problem, you're going to spend all your
time figuring out how the single threaded part exploited some other component
of the entire system to create a psuedo-multi-threaded app. Perhaps it relies
on the OS to spawn multiple clients in a client server arch where all the
processes live on the same CPU or something. The problem isn't just adding a
thread so you can checkmark a marketing box that its now a multi-threaded app
so someone can collect an end of year bonus, the problem is you need to scrap
the whole thing and start architecture work from first principles.

Another funny part was the natural assumption that all programs/projects will
be huge enough to contain at least two completely non-interacting subsystems
so no global state is the ideal. That's a dumb architecture design or business
decision. Lets think of an example where you would have threads with no global
mutable state. How about a tetris implementation in one thread and an
enterprise rational DBMS system in another thread and an IRC client in another
thread, all in the same program/project. That's just a dumb architectural
idea, make totally separate projects. Make the smallest usable debuggable
tools and use them together, not the largest undebuggable swiss army knife the
world has ever seen.

------
fatman
This is one of my largest weaknesses. I haven't come across any good resources
for self-study on this topic. I haven't found any online course-type
resources, and the books i've found are hard to follow. Any recommendations
for how to learn this stuff would be very welcome.

~~~
camperman
Would people be interested in a getting started type tutorial using Zero MQ?
As I replied to thewarrior below, the manual is very good and it made
multithreading an absolute pleasure for me but it's not really an introduction
to threads - I had to experiment quite a bit. I can do it for C, Python and
Lua.

What would be useful examples? Multithreaded video decoding? Scaling out
calculations to multiple cores?

~~~
fatman
That would be awesome. I just got on a new project a week ago where the PM
sent me a link to Zero MQ and basically said "here, read this".

------
gnuvince
> The programmer didn’t follow basic, well known development practices.
> Multithreaded programs seem harder or more complex to write because two or
> more concurrent threads working incorrectly make a much bigger mess a whole
> lot faster than a single thread can.

Actually, I prefer to think that the reason multi-threaded programs are harder
to write is because it's harder to compose components together. If I have a
regular old function and another regular old function and compose them, I get
a regular old function and I know that it works how I expect it. With two
multi-threaded functions, then I have no guarantees anymore. If I cannot take
simple components and combine them into more complex ones, how am I suppose to
write correct code? Sorry, but I'm just not good enough.

~~~
Jtsummers
This is where language choice and concurrency libraries become critical.
Erlang's (IME) processes are nearly as composable as functions. With golang's
channels allow something similar. ZMQ seems to provide a similar mechanism in
a language agnostic fashion, though that seems to be about connecting multiple
processes rather than threads (do people use ZMQ for interthread communication
and syncronization?).

------
kunil
I am in no position to call my self I can write good multi threaded programs
but... damn people are bad. Once I worked in an embedded project that had
sleeps in UI thread and then people started to use threads for 'optimizing'.
It didn't go well

------
colinshark
Multithreaded code is more difficult to write and more complex to debug by any
objective measure.

"There are difficult multithreading scenarios, just as there are difficult
scenarios in the single-threaded world. But those are relatively rare."

Is this some sort of flame bait?

------
kasey_junk
A bigger problem than people writing concurrency poorly is that they jump to
concurrency too soon and think it is going save them from their performance
problems. It almost certainly won't.

Contrary to the growing popularity of message passing concurrency, modern
commodity hardware is not massively parallel and jamming massively parallel
algorithms on top of it generally is a mismatch. You are almost certainly
better off having a few well defined threads that are fast than unbounded
threads that are slow.

------
chrismorgan
I like Rust's approach (it's not new, but Rust does a good job of the total
implementation). When I'm using Rust, I know that data race in safe code is
_impossible_.

(I know this isn't quite all there is to the problems of multithreading, but
it's a very high percentage indeed in almost all applications.)

------
jebblue
>> If you eliminate shared state, you have no reason to misuse those
synchronization primitives.

A multi-threaded program that doesn't need to share state probably isn't the
usual case, not from my experience anyway.

Locks are not evil, think of a railway station, there are sections of track
that must be shared by multiple trains.

------
wehadfun
Thanks for the article. When I have free time I will look into the various
tools that modern languages provide to solve problems that are usually handled
with threads.

I would have appreciated an article on the various "libaries" we should be
using instead.

------
JeffJenkins
Is there a list of languages which use separate heaps for each thread? I'm
really interested in learning more about that concurrency model, but haven't
had much luck compiling a comprehensive list of which languages actually use
it

~~~
bwindels
javascript with webworkers

------
mannykannot
The author seems to think it is all about avoiding mutable global state, but
it is more than that: it is about shared resources and temporal ordering, and
these are not issues that can, in general, be avoided in the simple way he
supposes.

------
jheriko
its interesting - but there is something fallacious here:

"Probably the most important lesson to be learned from the past 60 years of
software development is that global mutable state is bad. Really bad."

This represents a misunderstanding about the badness of global variables and
the goodness of encapsulation in my mind. Mutable global state is an
unavoidable necessity to ship your app - every single pixel on your screen is
a bit of global mutable state. I won't dive to far in here, but I cringe
everytime I see these fancy pants words applied because its pretentious /and/
trivially measurably wrong (see everything on your monitor right now).

------
thewarrior
As a n00b programmer I must say that the author dosen't give any examples on
how locks are commonly misused or exactly how to avoid these pitfalls other
than to say "Avoid global mutable state".

~~~
camperman
I am not a n00b programmer by any means but I've never needed to use threads
until recently. Non-trivial threading - like decoding video in the background
and coordinating with audio kind of non-trivial - is hard. Difficult to reason
about, unbelievably frustrating to debug, errors seem random, races, deadlocks
- you name it, I struggled with it.

That all went away when I discovered Zero MQ ([http://zeromq.org/intro:read-
the-manual](http://zeromq.org/intro:read-the-manual)). The manual claims:

"If you've spent years learning tricks to make your MT code work at all, let
alone rapidly, with locks and semaphores and critical sections, you will be
disgusted when you realize it was all for nothing."

It's not exaggerating. ZMQ's multithreading is marvelous. No mutexes, no
locks, no semaphores, no critical sections, no load problems - just perfectly
scalable threads that work with any language on any platform. In fact it's so
cool, that if you don't have a problem that needs multithreading, go out and
find one.

~~~
nly
ZMQ still relies on a message passing model, which is orthogonal to the
threaded shared memory model all our hardware actually presents to us. The
advantage is more being able to go from threads to processes, then to the
network, easily. That advantage is just as useless to most applications as
threading is to achieve concurrency (rather than parallelism).

~~~
camperman
"ZMQ still relies on a message passing model, which is orthogonal to the
threaded shared memory model all our hardware actually presents to us."

So what? There are all sorts of successful language and computing models which
are light years removed from threaded shared memory. Message passing without
sharing state, except for some lock-free primitives, is bloody brilliant for
very scalable concurrency - as Erlang has shown us for coming up on 30 years
now.

"The advantage is more being able to go from threads to processes, then to the
network, easily."

That's the raison d'etre for ZMQ to be sure. But very pleasurable
multithreading is a massive advantage too. If it scales out to nodes as well,
so much the better.

"That advantage is just as useless to most applications as threading is to
achieve concurrency (rather than parallelism)."

Threading using ZMQ message passing is a superb way to achieve concurrency
(given that this is the real world and we can't all rewrite our apps from
scratch in Go or Clojure). I'm a threading _beginner_ and I've got video
decoding, graphics handling and networking for a real time app all in separate
threads going great guns without falling over or messing each other around -
on a very low spec machine too I might add. On a four core workstation, all
threads get a whole core to themselves no problem.

Try it. You'll like it :)

~~~
nly
Any locked (or lock-free) pointer queue with the right ownership semantics
will give you what ZMQ does _within_ a threaded program. A coroutine library
will get you even more flexibility if you just want to setup pipelines. If you
want greater scalability, there are lots of non-blocking I/O event libraries
(libuv/libevent/boost asio). None of these solutions involve moving to a
message passing model.

I'm not discrediting that operating between threads is a handy addition to
ZMQ, but using it for communication between threads seems like a lot of work
(moving to that message passing model) if you never intend to leave one
process.

I do happen to like messaging, but when your data becomes complex it's a right
dog...you're now in the world of protocol design.

~~~
camperman
"Any locked (or lock-free) pointer queue with the right ownership semantics
will give you what ZMQ does within a threaded program."

OK, now I think you're trolling, perhaps by cutting and pasting from random
technical manuals. What you said there makes no sense at all. Could you paste
some example code please?

"A coroutine library will get you even more flexibility if you just want to
setup pipelines."

Now I _know_ you're trolling. Coroutines are completely useless for real
concurrency like decoding audio and video in the background simultaneously -
because only one coroutine can be executing at the same time.

"If you want greater scalability, there are lots of non-blocking I/O event
libraries (libuv/libevent/boost asio). None of these solutions involve moving
to a message passing model."

What if I don't really care about I/O? What if I have a small program that
needs to use multiple threads on multiple cores to efficiently search a large
problem space while doing video decoding in the background and having a
responsive UI? I must just link to Boost? Er, no thanks.

"I'm not discrediting that operating between threads is a handy addition to
ZMQ, but using it for communication between threads seems like a lot of work
(moving to that message passing model) if you never intend to leave one
process."

Compared to getting synchronization right, optimizing critical sections,
debugging random failures under load, it's trivial.

"I do happen to like messaging, but when your data becomes complex it's a
right dog...you're now in the world of protocol design."

I'm going to assume that you're still in school or not actually a programmer.
ZeroMQ is used by AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga,
Spotify, Samsung Electronics, Microsoft, and CERN (I grabbed this list from
its front page). CERN just published a report in which it evaluated ZeroMQ for
a custom middleware solutions running 4000 server processes communicating with
80 000 devices for a total of 2 000 000 I/O points that could be queried. It
aced the requirements. Erlang has used message passing for decades in the
telecoms industry which has enormously complex data and unique realtime,
distributed requirements.

Protocol design is easy. Main thread creates child. Child loads video and
reports its ready. Main thread sends START. Child thread receives START and
starts. Child thread reports DONE when done. Was that so hard?

