
How we took our tests from 100 seconds to 100ms - kevinburke
http://shyp.github.io/2015/07/13/speed-up-your-javascript-tests.html
======
shanemhansen
So basically their biggest problem was that node.js machine-gun hammers the
filesystem with stat calls when looking for modules. This was made worse by
running on a shared folder.

Vagrant has a mode called rsync-auto which will synchronize directories into a
local vm. Or you could just always have the code local to the node process. It
makes more sense to have your editor work with remote files.

Next biggest problem was relational database. I think that tests requiring a
SQL database are usually a bad sign, and the statement "we'd like to use
transactions but our orm doesn't support them" makes me sad. To paraphrase Ben
Franklin, "those who would give up transactions for a framework deserve
neither". Here's some ideas for speeding up your tests, in order from best to
worst.

    
    
      - Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation.
      - Use sqlite :memory: for tests
      - TRUNCATE TABLE
      - Set your database to running with scissors mode (disable fsync)
    

[edit] I just took a look at the ORM used, waterline
[https://github.com/balderdashy/waterline](https://github.com/balderdashy/waterline)
. Why wasn't step one moving to sails-memory for running tests?

~~~
pyre
> Use sqlite :memory: for tests

Then you run into issues where the functionality works on SQLite, but not on
your real database.

~~~
curun1r
SQLite might be the easiest to configure, but some other RDBMSs have an option
to run in memory too (e.g. MySQL's MEMORY storage engine). Also, as a last
resort, you can always configure the DB to write to an in-memory file system.

This strategy is, in essence, mocking the FS layer in DB tests.

~~~
TylerE
Often those modes don't have the same semantics, which is dangerous for
testing.

~~~
coldtea
After some point it's diminishing returns though. Obviously nothing can be
100% tested.

------
msoad
In my experience unit tests don't catch many bugs. On other hands end-to-end
tests always surprise me with the bugs they catch. Based on that, I'm not sure
advising for more stubbing is a good advise. Maybe I'm wrong but that was my
experience building UI apps.

~~~
DannoHung
In my experience building backend apps, unit tests are way more effective at
catching logic bugs than end to end tests.

End to end tests are great for catching connectivity bugs though.

~~~
michaelmcmillan
An added benefit of using unit tests over integration tests is that unit tests
shows you _what_ went wrong very clearly. If an integration test fails I have
literally no idea where to start hunting for the bug. It could be located at
any level in the stack.

I highly recommend this brilliant talk by J.B Rainsberger on how integration
tests are a scam: [https://vimeo.com/80533536](https://vimeo.com/80533536)

~~~
specialk
If you don't have that much time to write tests I'd recommend integration
tests. Then at least you have some tests. Treat integration tests as smoke
tests. If there is smoke then you can start looking for the fire. Integration
tests are always better then no tests at all. At least they tell you there is
a fire even if it is vague in telling you where it is.

~~~
michaelmcmillan
As someone who always writes unit tests before any production code, I don't
know how I would have the time to _not_ write tests. It drives the design and
reduces time debugging by a ridiculous amount. I work with information systems
though, might be different for other fields.

~~~
lmm
Since I've switched to a language with a richer type system (Scala) I've found
a lot of what I used to do with unit tests can go into the type level instead
(see e.g. [http://spin.atomicobject.com/2014/12/09/typed-language-
tdd-p...](http://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/)
\- but in Scala the techniques are a lot less verbose to apply).

~~~
michaelmcmillan
I am most familiar with Javascript & Python, but I think you're onto something
having fiddled with Java for 6 months or so. Some logic obviously can't be
replaced with only a rich type system, but a lot can.

------
rdsubhas
While bringing down test runtime 1000x is certainly a nice feat, I don't think
this is setting a right precedent. I, for one, would be very afraid. Tests are
not supposed to be written with speed in mind. Metrics seem to dominate the
world of tests, where the spirit is supposed to be catching real world bugs.

For example, the following can be done without destroying the spirit of
testing:

* Stub the sign in (big savings)

* Keep a tab on fixtures (big savings)

* Parallellize (big savings)

* Buy a better machine (big savings)

* Truncate with transactions/container-snapshots/etc (medium savings)

* Tune database for speed than safety (medium savings)

* Tune GC (medium savings), with enough RAM (16GB) disable it entirely (big savings)

* Disable logging and instrumentation, yes including timing measurements, unless intentionally needed (small to medium savings)

* Mount entire codebase, database and resources in tmpfs (small savings)

The below, while they might improve performance, go against the spirit:

* Tautological tests ([http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-tes...](http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-test-driven-development-anti-pattern/))

* Too many mocks/metaprogramming

* Using a different database driver

I'm planning to write a tool, which given a Class, will pump out lines and
lines of tautological tests, line for line (a.a should == a.a), method for
method (a.should receive save, a.save), branch for branch (with some
metaprogramming). Coverage will report a cool 100%, tests will be blazing
fast, and that should calm the lead devs down.

~~~
josteink
> Tests are not supposed to be written with speed in mind.

There's always extremes.

If people write tests with no regard for speed (as I've seen) you can have
"unit-tests" run for 45 minutes before completing.

That might be OK to guard against regressions on a nightly-build, but it will
in no way be a test-suite I'm willing to running prior to every commit I ship.

If the test-suite runs in less than a minute, I'm much more willing to do so
before submitting code, and in that way avoiding regressions which needs to be
fixed later on.

If the tests runs in less than 10 seconds, I'm probably going to run them for
every major change I do, even if I'm not planning on submitting anything yet,
just for peace of mind.

There's IDEs where tests can be run automatically in the background as you
make changes and let you know the second you broke something. For that to work
efficiently, your tests has to be sufficiently fast.

So maybe the aim shouldn't be that tests must be fast above everything else.
But if the test-suite can be run quickly and casually, it's value will
increase tremendously.

------
jkot
Nice problem to have. I work on database engine, and full test suite takes
about 2 weeks. After adding concurrency test, it might take a few months.

I started migrating to AWS spot instances to make this whole monster
manageable.

~~~
Bluestrike2
I just about choked on my coffee. If you're serious, is the issue just the
size of the test suite, or are the tests themselves just woefully inefficient?
By the time you're done, you've forgotten whatever it was that you modified.

~~~
masklinn
I expect they have subsets of the suites working as "fast tests". The full
sqlite test suite is 3 orders of magnitude bigger than sqlite itself and takes
hours on beefy machines, but they're not run "online" by developers.
Developers generally run the "veryquick" suite which takes a few minutes (and
is sufficient to catch the vast majority of issues)

------
steveklabnik
This post is full of great general advice that's not specific to Node. In
Ruby-land, we've been talking about the same things for the past few years,
complete with 'omg how many stat calls are we making again?'

I'm sure others were doing the same before Ruby too. The wheel keeps
turning...

~~~
why-el
Any link to a discussion on these Ruby stat calls? We might look into them as
well.

~~~
steveklabnik
The core issue is that Rubygems adds the `lib` directory of every gem to your
path, so you end up calling stat a zillion times.

I am admittedly a bit out of the Ruby loop these days, so it's possible that
this has been improved. The fastest way of course is to not load a zillion
gems...

------
nraynaud
A decade ago I brought the test time from 86min down to a few seconds.

An intern had thought that synchronizing things with "sleep one tenth of a
second then check that flag again" at the heart of the product and reading and
splitting by line a file where some lines might be 86Mo long were good ideas.

------
Asparagirl
Wait, Shyp uses Sails.js for their framework? Huh, they might be the most
well-known startup to use it. Sails development has really slowed down lately,
especially since one of their core creators/commiters got into YC with a
different project; I wonder how Shyp feels about that...

 _" Avoiding Sails isn’t possible in every situation, but we’ve found it’s a
useful design principle - it helps us separate our business logic and our data
storage layer, and will make it easier for us to move off Sails in the
future."_

Ooops.

~~~
kevinburke
Yep - we've run into a lot of really unexpected behavior with it and with the
ORM (Waterline), so we try to restrict ourselves to "the minimum possible
framework subset that gets the code to run".

For example, we disable `dynamicFinders`, don't use policies, don't use
blueprints, don't use Waterline's associate/populate/destroy, don't use the
"turn any controller function into a named route" setting, don't use globals,
disabled all of the Grunt/websockets junk, and overwrote all of the error
response handlers.

The result is that day to day we're writing something that resembles an
Express app with its own `routes` file. We're looking to move off of
Waterline/Sails entirely, it's slow going though since we're a small team and
it's tough to prioritize that, as it "works" now.

Every company has parts of their tech stack that don't do a great job - we're
somewhat fortunate this one is easy to mitigate.

~~~
iLoch
Sounds like you guys pulled out everything except for Express. I too found the
more I used Sails the less useful it became, as though it was missing a
solution for anything lying outside of the very basics. Compare Sails to say,
Laravel, and you begin to see how much work needs to be put in to bring it up
to date.

~~~
merb
That happens with every Framework. That's why we migrated from Django to
Playframework. Django is great, except for the thing we do / the expertise I /
we have. Play framework is way more "raw" than the most of the framework's
I've seen. Also I use Slick but only the SQL part of it and the Codegen. It's
so great to write RAW Sql while persisting objects easily and type safe
without a big layer in between.

~~~
lmm
To my mind, and as a spray user, play is still huge and does too much.

~~~
merb
You are correct Playframework still does too much for the most people, however
we are using a lot of things from the framework, basically json, forms,
templates, the new DI layer, the ws library (which still is not usable on
spray :(:() and the plugin layer.

Mostly I think we could easily switch to Spray, however currently I really
liked Play in his current version.

------
andresmanz
I don't know what it is with this site, but it made Chrome (Android) crash
four times already. Which means that, thanks to that wonderful browser, my two
billion research tabs are gone since the first crash.

Anyway, I can't imagine how running unit tests could possibly reach 100
seconds. Usually, that means that the tests are coupled to the environment
(database, filesystem, ...). So without being able to read the article, that
would be my guess. There are tons of good (and old) reasons why you should
decouple unit tests from the environment as much as possible. Execution time
is only one of them.

~~~
kevinburke
Sorry to hear that Andres - what version of Chrome are you using on Android?
Will try and see if we can figure out what is going wrong

~~~
andresmanz
It's version 43.0.2357.93.

I don't think it's your fault, though. Chrome loves crashing and throwing away
all of my research tabs. That wasn't the first time. It's only the first time
I could reproduce it with a specific website. Still, as I said, I don't think
it's your fault.

I'll just finish reading the article when I'm back at my PC. Test optimization
is a great thing.

------
zobzu
i think it usually boils down to "its tests, its not performance sensitive"

then tests take 10x the build time and eventually impact productivity like
crazy :(

~~~
tajen
At my startup (35 devs on the main product), we'd spin off a dozen AWS
instances so the tests would run in 20 minutes in parallel. And only tested
commits could be merged into master. We didn't feel it as a bad situation.

~~~
nulltype
It seems like renting computers by the minute is pretty cheap these days. I'm
surprised more people don't just have one machine per test, run them all in
parallel. Testing seems to be embarrassingly parallel in that usually your
tests don't depend on other tests at all.

~~~
tajen
Unit tests are short, but integration tests require the app to start and
setup, which takes around 3 minutes. Then each test is quite independent.

------
henvic
Some npm requires are indeed painfully slow.

For this reason I have wrote a small, but useful tool called require-time.

[https://www.npmjs.com/package/require-
time](https://www.npmjs.com/package/require-time)
[https://asciinema.org/a/21275](https://asciinema.org/a/21275)

~~~
srn_
There's also this: [https://www.npmjs.com/package/require-
times](https://www.npmjs.com/package/require-times)

------
jgalt212
> We run tests 400-500 times a day; a ten second improvement in every test run
> translates to an hour of saved developer time, every day.

Is it just me, or does that seem like running a lot of tests per day. I mean,
how many lines of code do they write per day? 1000? Do they run the test suite
ever time they write two lines of code?

~~~
Fargren
If your suite run in 100ms, why wouldn't you?

~~~
Gigablah
The thing is, after reading all that, I think they're referring to their test
suite _startup_ time.

------
wilgertvelinga
Wallaby.js is another way of speeding up tests during development.

------
dbbolton
What vim colorscheme is that in the screencaps?

~~~
kevinburke
hybrid_material - here: [https://bitbucket.org/kevinburke/small-
dotfiles/src/e4e1baf9...](https://bitbucket.org/kevinburke/small-
dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.vim/gvimrc?at=master#gvimrc-6)
and here [https://bitbucket.org/kevinburke/small-
dotfiles/src/e4e1baf9...](https://bitbucket.org/kevinburke/small-
dotfiles/src/e4e1baf907ae103bedf445e022a795b57d9da9be/.gitmodules?at=master#.gitmodules-130)

~~~
dbbolton
Thanks.

------
wilgertvelinga
For faster tests during development use wallaby.js!

------
supertruth
lucifer, scary

------
throwaway999666
If it so happens that people who are having problems with the running time of
tests have something like on the order of as much test code as they have
"actual" code, I'm not surprised that they have to optimize the test suite.
Apparently that was a trend some time ago (I don't know if it still is).

