Every time you ship a bug to customers, ask yourself "How will we prevent this bug from ever reaching customers again?" and "Why did we write this bug in the first place?"
Almost every time, unit or acceptance tests would have caught the bug initially or prevented the regression. That's not to say unit tests are the only solution.
Sometimes:
* you need better knowledge sharing between engineers
* you should spend more time refactoring your code
* your internal APIs are confusing and inherently cause bugs
Often a follow-up to a bug will involve changes to all of the above.
When you start an experimental project, don't focus on automated tests if you don't need to... I'm all for lazily pulling in the processes you need as you need them. :)
Whether you write unit tests or not is entirely context-sensitive. The author's points might make sense in some context, but they certainly don't apply to every team or even every engineer.
Maybe this guideline is helpful: "I just wrote a new component. What kind of test coverage do I need to allow a new engineer to refactor arbitrarily and still be convinced my code works?"
As far as guidelines go, my personal rule is "test at the highest level of abstraction possible." That could also be phrased as "test those invariants that are the least likely to change." In practice, the two guidelines turn out to be pretty analogous.
What does that mean, exactly? Any code you write is obviously in the service of some goal. Some code, like a method to split a String that's part of a standard library, is its own end: it's a complete API, so it should be tested as such.
A whole lot of other code, though, is in service of some larger component. Maybe you've got 15 classes that all work together to form the e-mail component, and hopefully only one or two of those classes really encapsulate the external API (whether it's an explicit API or just implied) exposed to other components, and the rest are logically part of the implementation. So instead of unit testing every one of those, you get more mileage testing at the level of that API. That lets you change the implementation of that component without having to rewrite a bunch of tests, and the tests themselves then become a much clearer documentation of that API.
But plenty of times, the overall API is so complex that there's a combinatorial explosion if you try to test at that higher level; in that case you have to find smaller chunks that have few enough paths through them that they can be tested thoroughly. Sometimes those chunks are the level of a single class or method; often they're still a collection of things.
Rather than starting from the classes and say "let's test everything," you look for logical groupings that form logical or explicit APIs, say "let's test this API," and then push the tests down a level only insofar as you can't test that API thoroughly enough to execute all the code paths in the related code.
To put that yet another way: test APIs, not implementation details, and the more things you can treat as implementation details, the less friction your tests will exert on your development process and the more value you'll get from them as a result.
Every so often you're right... you certainly should consider the costs of automating the test versus the risk of regressing.
But note that once you you have an organization good at writing tests, the incremental cost of writing a test is very, very small.
We like the term "10x" around here. Design systems for 10x the customers of today. Write code that will survive for a year. (Otherwise it may have been wasted.) Only fix bugs once so you can grow the engineering team without rehashing old issues.
lines of code is very dependent on the language. I have one rails app that's in the 5k range, excluding tests, and it's been running for over a year with no restarts and no need for bug fixes. I think that says something about the effectiveness of my process.
I question the design decisions of any code base with tens of thousands lines of code.
my servers are either debian or ubuntu, and my laptop is a mac. You do realize that the OS isn't a single monolithic code base. It's a bunch of smaller code bases that work together, which is precisely the point I was making.
that's the exception, not the rule right? It's also a example of something that I would absolutely write thorough tests for.
I'm also curious how modular it is. Does a kernel hacker have to keep the structure of that whole set of code in his/her head? I'm guessing there are a lot of clean seams in it that in practice reduce the actual lok related to a project down to a much smaller number.
"With the 2.0 kernel linux really grew up a lot. This was the point we added loadable kernel modules. This obviously improved modularity by making an explicit structure for writing modules. Programmers could work on different modules without risk of interference."
Meaningless. You made an explicit statement that you thought all code bases with more than 10 k lines have "questionable design", using a piddly little rails app as an example.
When someone pointed out that linux kernel had millions of lines now you are retreating to "but it is modular". Programming languages have had modules since the eighties. If someone points out a 25,000 line module what will you say? Each function has less than 10 k lines?
The point is that there are well designed code bases (the design includes modularization) with sizes well above 10 k lines , which directly contradicts your point. I've worked on well designed enterprise systems with a few million lines of code.
You've flipped my statement which changes its meaning. I said I'd question the design of any code base that large. Some would pass that questioning. I did not say that all large code bases have questionable design.
I won't debate a strawman anymore so I'll move on. Btw, are device drivers counted in that millions of lines of code? If so those are very independent things.
This thread was about test coverage, how many of those millions of lines of kernel code are covered by a unit test?
EDIT: so you agree with my original assertions that large test suites may not be worth the cost?
"how many of those millions of lines of kernel code are covered by a unit test?
"
Not many. And yet the kernel is an awesome piece of code. That was the point. I don't have the time to debate test-driven/test-coverage fanatics either.
If I ask the client between shipping a bug now and then and developing twice as slow, I know what he'll tell me.
(edit: twice as slow is about the same as twice as expensive)
Shipping bugs is not a show stopper. It's not even serious. What is serious is developing a live, production app without any consideration for bugs: backups (at least a couple of very different kinds) and control keys. A control key can be anything which will be visibly not ok when there's a bug somewhere.
I do management apps with a web interface: pretty flat architecture, no OOP to speak of, low complexity except the occasional long database query. So bug fixing is not really that time consuming, and definitely not the kind that would be caught by tests. Lately most time has been wasted on configuration issues of one kind or the other.
http://www.startuplessonslearned.com/2008/11/five-whys.html
Every time you ship a bug to customers, ask yourself "How will we prevent this bug from ever reaching customers again?" and "Why did we write this bug in the first place?"
Almost every time, unit or acceptance tests would have caught the bug initially or prevented the regression. That's not to say unit tests are the only solution.
Sometimes:
* you need better knowledge sharing between engineers
* you should spend more time refactoring your code
* your internal APIs are confusing and inherently cause bugs
Often a follow-up to a bug will involve changes to all of the above.
When you start an experimental project, don't focus on automated tests if you don't need to... I'm all for lazily pulling in the processes you need as you need them. :)
Whether you write unit tests or not is entirely context-sensitive. The author's points might make sense in some context, but they certainly don't apply to every team or even every engineer.
Maybe this guideline is helpful: "I just wrote a new component. What kind of test coverage do I need to allow a new engineer to refactor arbitrarily and still be convinced my code works?"