

Practical Techniques to Achieve Quality in Large Software Projects - nikita
http://developers.memsql.com/blog/practical-techniques-to-achieve-quality-in-large-software-projects/

======
HeyLaughingBoy
Testing is all well and good, but I worry that software is getting into the
trap that Manufacturing learned to avoid decades ago: you can't test quality
into the product.

Most blog posts about software quality seem to focus on testing. While testing
is certainly a _component_ of a good development process, it's really better
to avoid injecting a defect than trying to be sure you execute tests to find
it later.

~~~
RyanZAG
Can't agree more with this sentiment. Current software engineering practice
has moved from focusing on the software itself to testing the software. Even
peer reviews seem to be taking a back seat these days to 'make it pass the
tests' engineering.

1) Your tests are just as likely to have bugs anyway.

2) The 'bad' bugs are logic bugs - if your logic is wrong, your tests are
wrong, so you are testing the wrong thing.

3) Users often do things you didn't plan for. If you didn't plan for it, you
didn't create a test for it. 100% code coverage does not mean you have tested
every possible input, only that you've tested every possible branch. You can
still go down the wrong branch/get the wrong result.

4) If you design first and test later, you're more likely to have designed
real logic into your system. TDD will let you re-factor more easily - DDD
(design driven development) will make it so you don't need to spend months re-
factoring in the first place.

Basically: software engineering in general needs more architecture and less
ticking checkmarks - we are building space escalators here, we are not
building chair legs. There is no need to re-invent the chair leg, "but this
time with more tests".

~~~
davidcuddeback
I totally agree that tests aren't a replacement for thoughtful design, but
neither is thoughtful design a replacement for tests. I think they complement
each other. Thoughtful design allows you to model how you expect the software
to work and your tests help you validate that model.

 _1) Your tests are just as likely to have bugs anyway._

This is partially addressed by the "red-green" part of TDD's process. I catch
many bugs in my unit tests by ensuring that the tests start out red and become
green only after writing the code that's supposed to implement what's being
tested. Yes, it's possible that some bugs will still exist in your test code,
but I think it significantly reduces the probability that a bug exists in your
application code.

 _4) If you design first and test later, you're more likely to have designed
real logic into your system. TDD will let you re-factor more easily - DDD
(design driven development) will make it so you don't need to spend months re-
factoring in the first place._

I agree with the first part, that putting some thought into design upfront is
likely to improve the overall quality of your software, but I don't see how
that necessarily implies "test last." I like Domain Driven Design [1] (another
DDD) for upfront design, but I still use TDD when implementing the design.
I've not heard of Design Driven Development before. Is that another name for
Domain Driven Design? Its Wikipedia page [2] is lacking substance.

A word about refactoring: I too find that putting thought into design before
implementation reduces the need for large refactorings later. I agree with you
there. However, I've never found that TDD helps me refactor "later." With TDD,
I employ many micro-refactors _during_ implementation. In other words, DDD
provides the outline, TDD fills in the details. Alternatively: DDD is
designing in-the-large, while TDD is designing in-the-small.

[1] <http://en.wikipedia.org/wiki/Domain-driven_design>

[2] <https://en.wikipedia.org/wiki/Design-driven_development>

~~~
RyanZAG
You and nikita below are both right that having tests is much better than not
having tests, and the argument isn't really about choosing design or tests -
you definitely want both.

I was just trying to address the same point that HeyLaughingBoy above was
addressing - tests are not a replacement for design; quality is not about
tests. Tests are a tool to improve productivity in a large code base across
many developers, and they're also a useful tool for a single programmer to use
to focus their development. However, tests themselves are not a mark of
quality, and any actual bugs that tests catch are just nice extras - and if
you follow TDD, they will never catch bugs, since you will code until the test
passes, bugs and all.

So my point is very simple: quality is in design and implementation. Tests are
just for increasing engineering efficiency - you can have the same quality
without tests as long as you have enough time (manual QA). You can't have the
same quality without design, even with all the tests in the world.

~~~
davidcuddeback
_> Quality is in design and implementation. Tests are just for increasing
engineering efficiency._

Yes, precisely. I couldn't have said it better myself. Your original comment
came across as discounting the importance of tests a lot more than I think you
meant it to.

But I was also trying to make a couple points about TDD.

The first is that watching the tests fail before writing the implementation
increases your confidence that there isn't a bug in your test code. This was
just to address your #2 bullet point. But this only treats tests as a
verification tool, which is orthogonal to good design. Since we agree that
tests are for efficiency and we're discussing design, it's not worth dwelling
on this point.

The second point is that TDD is a useful _design_ tool that can be used in
conjunction with upfront design. Upfront design is great for having an overall
coherent design, but there are many finer details that are inevitably
discovered at implementation time. That's where I think TDD really shines.

If you think of building software as sculpting, upfront design is like
chiseling out the shape of your statue and TDD is like switching to a more
precise tool to etch in small details like hair texture or wrinkles. It's not
a perfect metaphor. For example, if you start with a smaller chisel (TDD),
you'll end up with the same statue just taking you longer to build. But if you
do the same with software development, you'll end up with an arm coming out of
the statue's thigh.

------
Ixiaus
I absolutely hate it when I'm reading an interesting article on a
startup/company blog and I go to click on their logo/name in the top left
corner and it...takes me back to their blog instead of their homepage!

------
urbina1985
Test (and Design) are part of the development strategy. Depending on what your
business goals are, you should optimize Design\Development\Test for it.
Sometimes these goals are principals that you want to brand around (e.g.
robustness for a DB server product) or tactical decision that adapt to a stage
(catching up, learning, leapfrogging, supportability, locking a market). These
are scenarios that I've seen (and some of them worked on) over the last years:

* Deliver a strong v1 too late * Deliver a small v1 that outs a new market and allows competirors to catch up * Develop a vnext backcompat-focused product and be surpassed by a new\exisiting competitor * Develop a heavily customer focus product and get trapped in impossible\contradicting requirements * Deliver a weak v1 and lose market credibility * Deliver a secret v1 and have no real market for it

If you have a clear view of the business goals, Test becomes less an opinion
and more of a resourcing puzzle (still hard though). It's a great world of
tradeoffs between layering tests, testability features, customer
research\partnership\simulation, fast development, legacy ownership,
responsibility distribution, exhaustiveness, documentation, community\customer
engagement, reusability of tests, reusability of product code.

------
mahmoudimus
Interesting, thanks for sharing!

Can you share more about psyduck? How much time was invested in building out
your testing systems?

~~~
nikita
I would say it was an initial push on 1.5 months of one engineer with 10% of
time of one engineer for maintenance. As the load increases we have to invest
a few days here and there to scale the system.

------
smm2000
I am curious how many unittests they have - hundreds or thousands...

~~~
nikita
We have thousands of C++ unit tests and hundreds of thousands of functional
SQL tests

------
known
Agile?

~~~
nikita
We are not following any specific methodology such as agile. But we definitely
pay a lot of attention to the varios engineering ideas and practices.

