

Why No One Uses Your Test Framework  - raymondh
http://www.slideshare.net/c.titus.brown/pycon-2012-tip-bof-intro-slides

======
jarrett
My pet reason: Your framework doesn't solve any of the genuinely hard problems
that annoy us when we're trying to test stuff.

Lots of people have written variations on assertions and mocks. But where's
the full-stack test harness for a Rails app with heavy JS usage, that runs at
a reasonable speed, gives us useful traces on the client and server, and
doesn't break when I slightly rework my HTML?

Or, where's the tool for testing integration with REST services outside your
control, that doesn't depend on the real service most of the time, but can be
told to run against the real service as a sanity check when you want to? (Like
Ruby's VCR, but better.)

Or, how about a test framework that accurately emulates mobile browsers and
automatically?

I'm not getting on anyone's case for not having written this stuff. Because
hey, I'm just as guilty of not solving these problems as is the rest of the
community. But if you want to make a testing tool that people will absolutely
love, solve some of these problems really elegantly. (I know, easier said than
done. And yes, some tools have solved some of these problems to some extent.
But I'd consider each of my examples to be an ongoing pain point for
developers.)

~~~
Fluxx
> But where's the full-stack test harness for a Rails app with heavy JS usage,
> that runs at a reasonable speed, gives us useful traces on the client and
> server, and doesn't break when I slightly rework my HTML?

If you're talking Python, Lettuce does this is Python, there also is Splinter.
You can run selenium with a headless firefox quite easily with Xvfb.

If you're talking rails, Capybara has drivers for Webkit (QI), selenium and
even PhantomJS. Their speed is variable, but it's not too bad.

On the "HTML changes break my tests," I'd argue that A) you should write your
front end tests not tied to the HTML if at all possible. Refer to elements by
semantic names, not XPath or CSS selectors. And be smart about your steps.
Make your submit_form() method actually look for a submit button in more than
one way. B) you shouldn't have more than a handful of full stack frond end
tests. They're hard to write, slow, and don't give much help telling you WHAT
went wrong. Most of the time the ROI is only there for verifying a few user
paths, but not much more than that.

~~~
benjiweber
The Page Object Pattern can help reduce the brittleness of automated UI tests.

<http://code.google.com/p/selenium/wiki/PageObjects>

------
jrockway
This is pretty much my impression every time I try to test something in
Python. I've never liked xunit, and I especially dislike it when it's a direct
port from Java. Yes, it's familiar to Java programmers. But when I'm writing
Python, I want something that looks like Python. (XML DOM libraries also like
to think they're Java. That makes Common Lisp my favorite language for munging
XML, since it's the only language bold enough to come up with some other API.
But I digress...)

Also, I don't understand mocks in Python. Say I have a class Foo that has a
method "bar" that I want to override. OK:

    
    
       class FakeFoo(Foo):
          def bar(self):
              print "OH HAI"
    

Done. It's built in to the freakin' language :)

~~~
tdavis
Now make some other code use your "mock" without editing its source. Now
assert on the calls made to "bar()," including arguments. Now make it raise an
exception without writing a second class. And make it return something else
without writing a third. Now patch a built-in. And a context manager. And a
dictionary. When you're done with that, mock a class without writing a new
class _at all_.

If you honestly think that sub-classing is equivalent to mocking/stubbing, you
don't understand the purpose of the activity.

~~~
jrockway
I know libraries can do all those things, but I've never preferred it over the
real thing. As an example, I might do something like this (not Python):

    
    
       my $bar = 0;
       class FakeFoo {
          sub bar { return $bar }
       }
    
       my $thing = ThingToTest->new( foo => FakeFoo->new );
       $foo = 42;
       is $thing->make_bar_two_bigger, 44;
       $foo = -10;
       is $thing->make_bar_two_bigger, -12;
    

Mock enthusiasts would write:

    
    
       my $fake_foo = Mock->new( Foo->class )->instance;
       $fake_foo->return_sequence( bar => [42, -10] );
       my $thing = ThingToTest->new( foo => $fake_foo );
       is $fake_foo->make_bar_two_bigger, 44;
       is $fake_foo->make_bar_two_bigger, -12;
       ok $fake_foo->ensure_exhausted('bar');
    

Yes, it's less lines, but I think it's harder to read because the values are
being set up long before they're used.

~~~
jrockway
Well, this was typo-y. $foo in the first codeblock should be $bar, and
$fake_foo->make_bar_two_bigger should be $thing->make_bar_two_bigger. More
coffee is required :)

------
igorgue
Well there are very good efforts to make this situation better like Konira
(RSpec like Python DSL) and Lettuce (BDD but you can also use it for unit
tests). But nobody really pays attention to it (well maybe just Yipit and me).
And its authors are really cool and active, you can even tweet with them about
problems you have, I even had conversations about testing theory (and
nomenclature) with them.

------
raymondh
Python's unittest.py was a direct port from Java's JUnit.

More recently, a Google contribution to unittest.py added a huge number of
self.assertThisThatOrTheOther() methods. The new methods were designed to have
better diagnostic messages for test failures (i.e. diffs between expected and
actual).

Much needed test discovery was added by Michael Foord.

Third-party testing modules like py.test and nose.py are popular substitutes
for unittest.py. They use a light-weight syntax and are joy to use.

------
pragmatic
There seems to be a lot of subtext here that I'd love to see more about. I'm
only an occasional python hacker and I'd love to know more about this.

One question, is there a web testing framework for python that they do
like/recommend?

~~~
keypusher
For a general purpose testing framework, Nose is great. For web testing,
combine it with Selenium.

------
tmcneal
It's hard to get the full context of this presentation just from the slides,
but I'm assuming the author is talking about Python test frameworks. I've been
using nose in combination with coverage
(<http://pypi.python.org/pypi/coverage>) for a year and have been very happy
with it.

------
kbd
Thanks for the flashbacks from the TiP BoF. I thought we'd never get to the
end, but man that was a fun and nerdy time.

------
jamescarr
I really liked the shot about web testing. It's a crying shame that a lot of
web testing frameworks don't bother supporting javascript.

------
gaius
No-one talk to me about unit testing unless you're already using strong
typing.

~~~
troyjfarrell
Python (the subject of the OP) is a strongly-typed, dynamically-typed
language.

~~~
pbiggar
I think it's pretty clear from context that the OP meant static typing.

Since he's being a dick about it, I very much understand your desire to
pedantically point out his error of terminology. Unfortunately, "strong"
typing is a very overloaded term, which has never had a widely agreed meaning
amongst either academics or practitioners. There are generally two camps:
"strong" === "static", and "strong" === "no dynamic coercions", and neither is
technically incorrect.

