

How to unit-test asynchronous operations? - blumenkraft
http://nesteruk.org/blog/post/How-to-unit-test-asynchronous-operations.aspx

======
edgeztv
My solution for multithreaded testing in Java. Been using this code for a
couple of years. Can easily be generalized for any asynchronous task. The main
concept is using a shared barrier construct.

    
    
      /**
       * Executes the task the given number of times in the given number of threads
       * @return All the unchecked exceptions thrown by the task during execution, or
       * an empty collection; never null.
       */
      public Collection<Throwable> run(Runnable task, int nThreads, final int iterationsPerThread) throws BrokenBarrierException, InterruptedException {
        // each thread will await upon the start barrier, then run the task, then await at the finish barrier (where the main thread will be waiting)
        final CyclicBarrier startBarrier = new CyclicBarrier(nThreads);
        final CyclicBarrier finishBarrier = new CyclicBarrier(nThreads + 1);  // +1 for the main thread
        // exceptions raised by threads will be logged and returned
        final Collection<Throwable> exceptions = new ConcurrentLinkedQueue<Throwable>();
    
        for (int i = 0; i < nThreads; i++) {
          new Thread("Thread " + i) {
            public void run() {
              awaitOnBarrier(startBarrier, 5);
              try {
                for (int j = 0; j < iterationsPerThread; j++) {
                  task.run();
                }
              }
              catch (Throwable e) {
                e.printStackTrace();
                exceptions.add(e);
              }
              finally {
                awaitOnBarrier(finishBarrier, 60);
              }
            }
          }.start();
        }
        finishBarrier.await();
        return exceptions;
      }
    
    
      /** Calls barrier.await and supresses all its checked exceptions */
      private void awaitOnBarrier(CyclicBarrier barrier, int timeoutSeconds) {
        try {
          barrier.await(timeoutSeconds, TimeUnit.SECONDS);
        }
        catch (InterruptedException e) {
          e.printStackTrace();
          throw new RuntimeException(e);
        }
        catch (BrokenBarrierException e) {
          e.printStackTrace();
          throw new RuntimeException(e);
        }
        catch (TimeoutException e) {
          e.printStackTrace();
          throw new RuntimeException(e);
        }
      }

------
antonovka
The general answer for event-based systems is: reentrantly spin the event loop
from the test and block waiting on the result.

The general answer for actor-based systems is: wait on a blocking message
receive from the actor task.

The general answer for simple shared mutable state thread-systems is: Write a
complex condition-lock and thread combination (which will probably be subtly
wrong the first time) to block your thread and wait on the result triggering
the condition.

------
cdavies
The most exciting defect that I ever had to debug was a dormant async bug that
had been in the code for many years. I had checked in what I thought was going
to be a fairly innocuous change in a component in our team's subsystem, but it
ended up breaking the full build at the end of the day and was going to be
backed out the next day if I didn't prove I wasn't responsible for it.

It was about 6AM after a night of constantly building debug ROMs with
different instrumentation to try and get one that I could get both meaningful
debug information out of but still suffered from the defect that I finally
tracked down the problem. It turned out that a circular pseudo-dependency
involving 3 different subsystems and five different components was
deadlocking, and all I'd done is change the timing a fraction of a second to
trigger the deadlock.

Asynchronicity is hard to test, and you'll always, always get it wrong. Don't
sweat it too much. Unless you're building flight control software, just make
sure the common use cases work 100% of the time and if broken edge cases turn
up, fix them as and when.

------
clemesha
A worthwhile read might be the unit-testing framework from Twisted (Python
asynchronous networking library), called "trial". See here:
<http://twistedmatrix.com/trac/wiki/TwistedTrial> or poke around the code for
more in-depth details.

------
jerf
To unit test an asynchronous operation, you need an asynchronous language or
library. In Erlang you hardly even notice because the language natively
supports it. In Perl I've hacked some stuff up with POE but it's way more
painful. But once you have the asynchronous capability, it's not much
different than writing a normal unit test.

------
projectileboy
In JBlub, I ordinarily just create an inner class within my test class that
implements the event listener interface I care about, and assert that the
listener got called. This works fine for me when I need to test complicated UI
stuff.

