
Ask HN: Can you do TDD in a complex product (data science, AI, etc)? - sandGorgon
I have been considering introducing TDD in my startup . We do a fair bit of non-trivial work in our Python&#x2F;Flask APIs : credit scoring, accounting, workflows, etc<p>I&#x27;m wondering if TDD is even a good idea ? Can one get to a stage where writing testcases is as easy as writing specs or whiteboarding ?<p>Or does TDD become one of &quot;those processes that founders shoved down our throat&quot; ?<p>There was a very interesting discussion about this a few years back on HN - https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=2240595<p>P.S. A lot of responses I get are &quot;whatever works for your team&quot;. Fair enough. But I&#x27;m looking for fairly opinionated answers - how do <i>you</i> do it ? And what would you do better ?<p>what&#x27;s your mental model around writing tests ? Do you write tests after every function, every commit , every day ? When you are building a new product (and maybe hacking stuff together in the middle of the night), would you still write tests before moving forward ?
======
itamarst
I like TDD, it's pretty useful in many cases, but... you may be starting this
the wrong way around.

Your question is "should I use this testing technique?". A better question to
start with is "what am I trying to achieve with testing?" Once you know your
goal you can find a relevant testing technique to achieve it. Your goals will
likely be different for different parts of your product and codebase.

Notice this means different organizations and products (or same product at
different times in its lifecycle) will have different forms of testing,
because they have different goals and resources.

To help with the decision process of choosing testing goals I've come up with
a model, of different goals and the kinds of testing that are useful for each.
I'm going to be giving this as a talk at PyCon in May, but have an essay
version written up: [https://codewithoutrules.com/2017/03/26/why-how-test-
softwar...](https://codewithoutrules.com/2017/03/26/why-how-test-software/)

Since choice of testing strategy is situation-specific it's very hard to give
a general answer without understanding the specifics - ping me (itamar at
codewithoutrules dot com) if you'd like to talk over your specific situation.

------
nanospeck
Here is a good link from Robert C Martin (author of clean code) for the
question : How doeas the presence of neural nets impact the deve lifecycle in
Agile environment [http://blog.cleancoder.com/uncle-
bob/2017/03/16/DrCalvin.htm...](http://blog.cleancoder.com/uncle-
bob/2017/03/16/DrCalvin.html)

------
jimsmart
I generally write tests for the initial/main functionality up-front - usually
just a simple call to some func that's currently just stubbed, assert that it
does the right thing (e.g. "I want to call my new code like this... and get
back this..."). Then implement enough code to make that pass, and perhaps
modify/expand the test during that process (exercising edge cases perhaps, or
adding more functionality - depends on the project).

Then I might write a whole bunch of related code, fleshing out other parts
which rely on the primary part (that I know works well enough for now),
perhaps without writing tests first. But if I do that, I will then go back and
write more tests until I get 100% coverage later. Undoubtedly you will find
corner cases that you didn't cater for when you exercise the code in tests.

It's not done until all my code has 100% coverage, and all the main entry
points, important data structs, tricky bits, etc, have documentation of some
kind. But I might implement a fair chunk of the code before implementing full
tests.

So: some tests up-front (because it makes coding easier), and the remainder of
the tests (for full coverage) have higher priority than writing documentation
on my list of todos.

In no particular order, I've written non-trivial amounts of Go, Javascript,
C#, Python, Java, C/C++, ...

— But your question is not specific enough because the process can vary a lot,
depending on what you're coding (e.g. I don't bother writing tests for bash
scripts: they usually either work, or they don't), and who / what it is for.
e.g. the above process works great if you're building a library (and if you're
not building a library, it can help to try and break apart the project into
some library-like pieces).

Obviously the above would be somewhat different if I were doing TDD for a web-
app, but I'd still apply the same process to the bits and pieces that make up
the app behind the scenes - and then have to also consider front-end testing
if appropriate.

------
rajnp
Not specifically to data science or AI projects, generally test coverage is
very important, otherwise how would you ensure that the functionality works as
expected and handle error/unexpected conditions?

But TDD approach or writing test cases after the implementation is upto the
individual developer/team's preference. Initially TDD approach would be very
difficult for freshers or devs who don't have TDD experience, but in the long
run, it would help.

------
brudgers
In the context of 'my startup' the decision to implement or not implement TDD
or microservices or ReactJS is properly made on the basis of business value.
Does TDD/microservices/ReactJS make money or improve the odds of survival or
provide a competitive advantage or will it create drag on the business?

Good luck.

~~~
itamarst
This is good advice... but it's hard to follow if you don't understand the
costs of particular forms of testing, or what benefits they provide. This was
part of the motivation for creating the model I referenced in a comment above.

~~~
brudgers
Your earlier question does not make a business case and Markham's answer might
be somewhat similar to mine here (unsurprisingly perhaps since I sort of
recycle things I read on Hacker News when it comes to startup specific advice
and my own experience when it comes ordinary business advice). This sort of
falls in the category of both: there is a startup component (survival) and a
general business component (YAGNI based cash flow).

The tricky thing about TDD is that Uncle Bob packages it up with morals and
ethics and craftsmanship. It's probably a big reason why he is a successful
consultant in addition to having coding chops...and I don't disagree with his
moral and ethical opinions but they are about writing code rather than running
a business and here the important decision is running a business and that's
what Markham emphasizes and emphasizes particularly well in his reply to
Cooper's response.

I'd put it this way, the first order benefit of TDD is confidence that
refactoring will pass fewer regressions to production. But a successful
startup does not depend on refactorings because growth drives rearchitecting
and pivoting and those mean that many/much/all existing tests may become
irrelevant. Riffing a bit, it might be that in a startup the way to determine
the benefit of tests is do they get the company to the point where those tests
become irrelevant more quickly? Most startups default on technical debt by
either running out of money or by throwing out the code base because they
acquire different operating parameters due to scale.

~~~
itamarst
Right, so in my model, there's four goals you might have for testing:
understanding users, understanding runtime behavior (of your software),
correct functionality, stable functionality. Which set you choose depends on
business goals.

In a startup environment your goal is often understanding users (does anyone
care about this product/feature? is it usable?) followed by correctness. And
as you say, stability isn't an immediate concern if you're rewriting
everything every two weeks, which is the case for many layers of code.

"I'd put it this way, the first order benefit of TDD is confidence that
refactoring will pass fewer regressions to production." \- I'd say automated
tests, including unit tests and therefore TDD, only give you stable
functionality. Not correctness, since only a human can determine that (see
essay for fleshed out argument - [https://codewithoutrules.com/2017/03/26/why-
how-test-softwar...](https://codewithoutrules.com/2017/03/26/why-how-test-
software/), section on "What means of testing can you use?").

But unit tests (and thus TDD) for code that _isn 't_ going to be a stable is
actually a negative, because you're trying to enforce stability on something
that keeps changing.

\--

Could you link to the Cooper/Markham discussion?

~~~
brudgers
Cooper and Markham were at the top of the link in the question. {edit: I have
been mistaking you for the OP sandGorgon}

\--

I can see how A/B testing might help with understanding users, but to me, A/B
testing is orthogonal to Test Driven Development and early on A/B testing
would probably not have enough test cases to generate deep statistically
significant insights due to small sample sizes and the need to make gross
movements rather than fine tuning a working process.

\--

For Uncle Bob, TDD is a business process because he is selling consulting at
some hourly/weekly/monthly rate. His client base will tend to be looking at
improving stable business processes not those taking a spin of startup
roulette. For a technology startup, what gets developed in house will tend to
be strategic in the sense that it has a big effect on the company's velocity.
Getting 'there' in three months is more than eight times as good as getting
there in two years.

~~~
itamarst
Ah, thanks, will read through.

A/B testing is orthogonal to TDD, yes, and that's really the point: there's
many forms of testing (usability testing, testing user interest, as well as
A/B testing) whose goal is completely different than verifying a
specification. Often they are useful when creating the specification.

And maintaining stable software is very different than spinning up new
product, yes. So you need to start with business goals, work down to testing
goals, and _then_ choose testing techniques based on those.

