

Functional Tests As A Tree Of Continuations (Erlang) - TimothyFitz
http://www.evanmiller.org/functional-tests-as-a-tree-of-continuations.html

======
viraptor
One good thing about killing the state each time is that... you kill the
state. I'd be worried about testing an external service through http in this
style. Is it completely stateless? What happens with the session? Can I test
"cancel" on the same step if I already did "continue" on it? What if the order
is different? What information do I have to rollback after trying each branch?
etc.

I know he addresses it in the database example (<<For modern web applications,
all the state is stored in a database. Therefore to make a "copy of the
universe," we just need a way to make a copy of the database to pass down to
the child tests>>), but then he follows with testing over http, where he
doesn't save the state in any way. It's just a bit confusing this way.

If you do testing in the wrecking ball approach, at least those things are in
plain view. Just kill the state every time. If it doesn't take more than a
couple of min. to run all the tests, everything's all right.

~~~
shoover
The state is being saved in the http example, but you can't see it because
it's a functional test at the http client level. Think of the test code as a
script controlling a browser. boss_test:get_request and boss_test:follow_link
are hitting the http server where all the state saving action occurs.

~~~
viraptor
Yes, but after get_request, you've got some data saved in your session (I'd
assume that's the case for any multi-step flow over http). Now how do I
guarantee that when I "continue" from step 2 and then "cancel" from step 2, I
still have the correct session state to act on step 3?

~~~
shoover
That depends on the integration between the web server and the test framework.
In this case the framework is rolling back some in-memory database. If the
session state is in that database, you're good to go. Otherwise the framework
needs to track whatever else you're using for session state and roll it back
like it does the database.

------
ajuc
Cool idea.

In my job we are working on business process engine, and we have implemented
many long business processes that involves user interactions and calling plSQL
code.

Testing these processes manually is pain, we have tests using soapui, but
writing tests in linear fashion is repetative and tedious, our test suite is
lagging us. Also we use big databases and our tests unfortunately aren't
isolated - we have to prepare data manually for tests to run.

This tree-like approach looks great. One problem is - the only way I see to
create "stack of databases" is to use savepoint - rollback to savepoint around
every node in tests tree.

We use hibernate, so rollback to savepoint won't be enough - we'll need to
disable second level cache, and to do sth about first level cache - maybe
flushing after each rollback to savepoint will do.

Anybody has experience with hibernate and rolling back to savepoint?

------
samstokes
This reminds me a lot of "context blocks" as implemented in the Ruby testing
libraries Shoulda and RSpec (and probably elsewhere). A context is a named
code block which can contain a number of tests, along with some common setup
and teardown that's run for each test. Contexts can contain contexts, and
tests in nested contexts get all the setup and teardown from their own context
and its parent contexts, in the order you'd expect.

I've used nested contexts to achieve much the same sort of thing as Evan talks
about here: high-coverage functional testing of a sequence of actions. They
have the same advantage that the length of the test stays linear with the
number of actions, although unlike Evan's callback-style tests, they didn't
stop the test tree as soon as a step failed.

------
loewenskind
I realize the bad example wasn't actually doing anything and the tree example
does, but the tree looks like it takes a great deal more effort to actually
use.

I have a better idea: Erlang already handles green threads extremely well, how
about providing a "test mode" switch in the VM. When a branch is encountered
in execution the VM spawns new threads as needed and just takes them all. If
spawning a new thread would put us over some thread count threshold then just
put it on some queue for when there is space for a new thread. Then you can
just put pre and post conditions on all your functions and you'll find out if
any possible branch traversal causes one of these conditions to fail.

