
Sensible Software Engineering - myth_drannon
https://www.scriptcrafty.com/2019/02/sensible-software-engineering/
======
wildmanx
> Bugs are correlated with lines of code and TDD forces writing more code so
> how can it reduce bug counts? If the test code has no bugs then just write
> the rest of the code in the same style

Nobody has pointed out the fault in this reasoning yet, so I will.

The linear relationship lines-of-code vs. total-bug-count is based on
independence of bug introduction in different parts of the code. That is,
introducing a bug in line 10000 is more-or-less independent of adding line
20000. For product code that is arguably the case, but adding test code into
the mix, this basic assumption doesn't just not hold, but it's turned upside
down.

Test code is not customer-visible, but only dev-visible, with the sole purpose
of finding bugs. Thus, adding test code to a code base _decreases_ the average
probability of a bug for lines of product code. More formally, if you have N
lines of product code and thus c * N bugs for some fraction c, then adding M
lines of test code does _not_ increase the number of customer-visible bugs to
c * (N + M). Instead, it reduces that number to c' * N with c'<c and the
difference being caused by your test coverage. (A 100% test coverage, i.e., an
exhaustive test without bugs or, equivalently, a formal verification, would
bring c' to 0.) Sure, the M lines of test code may well have bugs on their
own, but that only increases c' slightly while keeping it below c, and more
importantly, those test bugs are not customer-visible. They only annoy
developers.

I agree with the rest of the post though.

~~~
ericathegreat
Agree. Consider the situation of data entry professionals, those people who
transcribe audio recordings or enter data into databases.

If you have one person entering data into a computer, then the odds of them
introducing an error and failing to spot it are fairly high.

If you have twice as many people entering twice as much data data, then the
odds of an error getting introduced are roughly doubled.

However, if you have those two people entering _the same_ data, then their
mistakes cancel each other out. If person A and person B both entered the same
thing, it's extremely unlikely that it's incorrect. If they differ though, the
a problem has been identified, and can now be fixed.

The odds of both of those people entering the same piece of data incorrectly
is tiny. Likewise, _accidentally_ introducing a bug into both the production
code _and_ the test is pretty unlikely.

That said though, if those two theoretical data entry people above are given
the wrong data to enter, then they the system cannot protect them. They will
both correctly enter incorrect data. "Garbage in, garbage out".

Likewise, if the requirements of a piece of software are poorly understood,
then it is quite likely that both the test and the production code will
implement the same "bug". Writing tests won't fix a failure to understand the
problem you're trying to solve. And they're not supposed to.

~~~
gdfasfklshg4
> The odds of both of those people entering the same piece of data incorrectly
> is tiny.

This is just not correct. There may be a systematic reason why they are making
a mistake (e.g. a miss-pronounced word) in which case increasing the
confidence intervals does not increase the accuracy. Check out the concepts of
accuracy, precision etc from physical sciences.

------
jasode
_> Bugs are correlated with lines of code and TDD forces writing more code so
how can it reduce bug counts? If the test code has no bugs then just write the
rest of the code in the same style_

I'm not advocating for TDD (the programmer methodology in the IDE) but the
author's explanation about "test code" isn't correct. Code written for
explicit purposes of a _test_ to exercise other code has been shown to
increase correctness. E.g. SQLite database has 711x more test code than the
core engine code.[1] (I made a previous comment why this is possible:
[https://news.ycombinator.com/item?id=15593121](https://news.ycombinator.com/item?id=15593121))

Low-level infrastructure code like database engines, string manipulation
libraries, crypto libraries, math libraries, network protocol routines, etc
can benefit from a suite of regression tests.

It's the high-level stuff like GUIs in webpages being tested with Selenium or
LoadRunner that has conflicting business value because altering one pixel can
have a cascading effect of breaking a bunch of fragile UI test scripts.

[1] [https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html)

~~~
userbinator
The question that always comes to mind whenever testing is discussed is "how
do you know the test code is itself free of bugs?"

I distinctively remember once posing that question in a meeting about testing,
and a manager replying --- seriously --- with "then perhaps the test code
should itself have tests." Someone else must've come up with that before too,
because (at a different job) I've also worked on a codebase where a surprising
number of tests were basically testing the function of another test.

~~~
gjm11
You don't know that the test code is free of bugs. You don't need to.

Case 0: No bugs in the test code. All is well.

Case 1: Bug in the test code that causes some bugs in the real code not to get
caught. That's bad, but you're no worse off than if you didn't have the test
at all.

Case 2: Bug in the test code that causes _correct_ real code to look buggy.
Result: the test fails, you look for problems, most likely you find that the
problem is in the test code and fix it. Going forward, you have a working
test.

Case 3: Bug in the test code that makes something _else_ break. This can
happen and is genuinely bad, but (1) it only affects testing, not your actual
product, and (2) most bugs don't behave that way.

The test code is a net win if the bugs it catches in your real code are worth
the effort of writing and debugging the test code. That's no less true on
account of the possibility of bugs in the test code. It just means that when
you estimate the benefit you have to be aware that sometimes the tests might
be less effective because of bugs, and when you estimate the cost you have to
be aware that you have to debug the code, not just write it the first time.
And then you just ... decide what's the best tradeoff, just like everything
else in engineering.

(And no, you don't need tests for your test code. The test code is a test for
itself as well as for the code it's testing.)

~~~
pintxo
I see 5 cases:

    
    
        Code  | Test  | Result
        --------------------------------------------------
        fine  | fine  | a) We have a regression test, yay!
        fine  | buggy | b) Someone breaks the code to make 
                           the test work. Oops.
                        c) Someone fixes the test, we have 
                           a regression test, yay!
        buggy | fine  | d) We'll fix the code, and we have 
                           a regression test, yay!
        buggy | buggy | e) Bug remains in code. Oops.
    

If the probability of introducing an error is p,

    
    
        Code  | Test  | Probabilities      | p = .01 (one in ten)
        --------------------------------------------------
        fine  | fine  | a) (1-p) * (1-p)   | .9801
        fine  | buggy | b) (1-p) * p * .5  | .00495
                        c) (1-p) * p * .5  | .00495
        buggy | fine  | d) p * (1-p)       | .0099
        buggy | buggy | e) p * p           | .0001
    

So we see, probability for:

    
    
        no harm done                 .9801  (a)
        bugs found + fixed           .01485 (c,d)
        bugs introduced / not found  .00505 (b,e)
    

The above completely ignores the fact, that depending on the code base there
will be significantly more test code than production code. But then test code
is quite often highly redundant, and might actually have a lower defect rate
itself.

Also the probability on introducing an error in the production code and the
test code, might actually not be statistically independent, which I assumes
here. So take with a grain of salt.

[Edit] Actually d) could also end negatively. Guess a working model would have
to take into account that on failing test cases, a sensible developer should
take a step back and reason about why this happened. So the negative outcomes
would be (hopefully) less likely than the positive ones here.

~~~
peteradio
Top notch explanation here. P = 0.01 is 1 in 100 though right?

~~~
pintxo
Oops, yes.

------
james_s_tayler
>So here’s the punchline: if you want to be a good programmer then learn a
technology and language agnostic formalism. Logic, statistics, and game theory
are probably good bets.

I think in an abstract sense control theory is a reasonably good bet.

[https://en.wikipedia.org/wiki/Control_theory](https://en.wikipedia.org/wiki/Control_theory)

I can't say I know it deeply, but a lot of the ideas resonate when I think
about software engineering. If you think of everything as basically an
n-dimensional vehicle with an interface to control it and the control
mechanism is used to set all the parameters relevant to the system then a few
things follow:

    
    
      every system has a safe operating envelope
      parameters are usually linked to eachother such that turning one up turns another down
    

I find there is a lot of mileage to be had about thinking about which
parameters are linked to eachother and when you get excited about turning one
of them up (ie you bring in TDD or a Kubernetes type solution) what is the
effect you are having on the other parameters? That's where a new source of
pain is going to come from. The biggest mistake I see in reasoning about these
kind of things is being blind to the negative side of the trade-off due to the
overwhelming excitement of finally being able to jump on the bandwagon and
join the cargo cult. You have to train yourself to hunt for the parameter that
is being affected indirectly as that's the most important side of the trade-
off. You need to reason about whether or not the indirectly affected
parameter's new value would take you outside of the safe operating envelope.

With every decision we make about systems we build and run we are essentially
trying to steer them, albeit clumsily, in this manner.

~~~
keithnz
good recommendation, if the author understood control theory he'd understand
that code that feedbacks on code is quite different than code with no feedback
loop.

while I find it hard to find anything I'd recommend about the authors article
as most of the reasoning seems a little off, I can understand the sentiment of
the article .

Agile and TDD are really recognition of control theories ideas of feedback
loops keep things in better control and can adapt faster vs long feedback
loops going out of control far easier. This is more targeted at the human side
of creating software. Nothing to say there aren't better strategies than TDD
and Agile techniques, however I think that principle of feedback loops to give
confidence will stay in some form. I think there is a LOT more to be said
about engineering / designing correct, robust, and secure software.

~~~
james_s_tayler
>This is more targeted at the human side of creating software

This in my experience is the most important factor.

------
leetrout
Really buried the lede here. I get that it was acknowledged as a rant /
opinion piece but there's not a lot of really actionable advice for the
general population of programmers, IMO. The article has good points, for sure,
but the ending has the best part, IMO.

> So here’s the punchline: if you want to be a good programmer then learn a
> technology and language agnostic formalism. Logic, statistics, and game
> theory are probably good bets. As things stand that kind of skill is
> probably the only thing that is going to survive the coming automation
> apocalypse because so far no one has figured out a way around rigorous and
> heuristic thinking.

I think there's a lot of support there.

I don't think using Kubernetes as an example of "sequestering the complexity
behind a distributed control system" was a good follow up to TDD generating
more lines of code. Containers are a step in the right direction and
Kubernetes _is not_ the best option for using containers in production but it
_is_ the most popular option and so if you want community mindshare and
support then it probably is the best choice if you can manage it or use a
managed service.

"Serverless" is real, it's here, and containers / k8s are just a step along
the way.

~~~
eikenberry
I agree that Kubernetes is not the best technology out there, but I'm not 100%
on which I would call the best. Did you have a winner in mind (it sounded like
it)? If so why that one?

Also it may not really matter as you said Kubernetes is the most popular and
is only getting more so at this point and the network affect is so strong in
tech like this that the "technically best" most likely will become a moot
point.

~~~
leetrout
I am a big fan of Nomad's scheduler and the code is easy to pull in to an
existing Go project. It doesn't have everything Kubernetes has but I don't
think it needs it, either.

Kubernetes has for sure won the popularity contest but the overhead involved
in running it The Right Way™ on your own is a lot. Given what I've seen I
would advocate for OpenShift if you like RedHat products / projects or
sticking with Kubernetes from one of the well-known cloud providers.

~~~
pytester
I have a sneaking suspicion that Kubernetes is another Ruby on Rails waiting
to happen:

[https://trends.google.com/trends/explore?date=all&geo=US&q=r...](https://trends.google.com/trends/explore?date=all&geo=US&q=ruby%20on%20rails)

~~~
WrtCdEvrydy
Good for me, I must made about 500,000 from moving Rails to other more modern
stacks... I'll learn Kubernetes just as it dies off and start moving people's
shit to whatever's next.

~~~
james_s_tayler
That has to be one of the best strategies out there. So long as you just read
the trends right.

~~~
WrtCdEvrydy
Best of both worlds, I can paid to learn the new tech and the old tech...

------
myth2018
The only part I tend to disagree with is regarding TDD. There is a study from
Microsoft Research showing that TDD results in greater quality -- although you
pay a price for it [0]. I believe this applies as a general rule, although I
recognize I'm not aware of further studies.

However I agree with the article's general idea. In the aviation industry
there already are languages abstracting computers' internals and allowing
programmers to reason about safety-critical programs using more high level
constructs.

Due to its nature, I think there won't be such a technology for general
purpose languages -- in order to be general enough, you can't have too much
things abstracted away. Maybe we couldn't go much farther than what languages
like Basic allows us.

On the other hand, I wish we had such languages for more specific tasks like
ERP-like software, business web applications and so on. It's worth noticing
that many of the biggest ERP companies in the world have their proprietary
domain specific languages.

[0] [https://www.microsoft.com/en-us/research/blog/exploding-
soft...](https://www.microsoft.com/en-us/research/blog/exploding-software-
engineering-myths/)

------
AdieuToLogic
From the article:

    
    
      So here’s the punchline: if you want to be
      a good programmer then learn a technology
      and language agnostic formalism.
    

No.

If you want to be "a good programmer", then learn how to define the problem
for which you are tasked to solve. The technology is irrelevant. The "language
agnostic formalism" is irrelevant.

Unless a person/team knows what must be done, then the rest really doesn't
matter. Techniques which help to elicit repeatable delivery certainly are
worthy to learn, even to advocate for. But without understanding what is
needed, what use are they?

------
sinuhe69
Basically, the author suggested a “back to basic” approach, which is of course
not totally wrong. But ignore the very real challenge of the software industry
is not going to help either. We can not sit down to plan and wait for the so-
called “global optimal” design and then and only then proceed to implement.
Because to achieve a “global optimal” design one has to consider _each and
every_ details and aspects of the problem, which is simply impossible in the
software industry because so many are unknown. The Waterfall model is long
dead. History has proven it does more harm than good. Don’t dig it up.

PS: I don’t disagree that in some cases, academics can use the formal methods
to find a (near) global optimal solution. But I don’t think it’s practical in
a daily context, nor necessary. Our evolution is the best proof that local
optima can lead over time to fantastic solutions.

------
scandox
> The bottleneck has always been in understanding all the implicit assumptions
> baked into the system that for one reason or another are essential to its
> correctness.

> It takes a particular kind of masochist to enjoy reverse engineering a black
> box from just poking at it with simple linear impulses.

These are great observations and brilliantly put. In particular the second one
I think rightly explains why some very smart people definitely do not take to
programming as a profession.

------
painful
I agree with the part that says Agile (and Scrum in particular) is drinking
the koolaid. Managers who don't themselves write any code, but micromanage
their workers with their version of Scrum, are the worst.

~~~
gav
The majority of issues I see with Agile and Scrum is that it's a process added
by people external to the teams and it's treated as a rigid process to be
followed.

I've worked with teams who have had to suffer through a 45-minute standup
every morning that was immensely painful. They were just following the process
as best as they understood it from a couple of days with an "agile coach" and
didn't really understand what being agile really meant.

> Managers who don't themselves write any code, but micromanage their workers
> with their version of Scrum, are the worst.

I don't think anyone would disagree on the evils of micromanaging, but as a
manager that doesn't code, I think that managers who do code are depriving
their teams of the most valuable thing they have to offer--their support. If
you take on a coding task and put yourself on the critical path of that
sprint's work, you have to commit to putting the hours in to get it done. This
is not a good use of your time.

This is the difference between a manager and a Tech Lead.

~~~
Aeolun
Just the fact that a manager doesn’t code doesn’t mean they’re providing their
teams with any support.

------
ed_blackburn
..and I came here thinking I would see an article about an amazing indie games
studio from the 90s :-(

~~~
jlarcombe
I thought exactly the same!

------
mynegation
I enjoyed reading it and agree on many points. One thing that does not seem
right is the claim that introduction of formal specification will increase the
number of programmers. Yes, it takes a special kind of a person to be
interested in fiddling with the code, but I believe the threshold for enjoying
creation of formal specifications is even higher.

------
nishparadox
"Whenever I’ve approached a new system the bottleneck has never been writing
more code. The bottleneck has always been in understanding all the implicit
assumptions baked into the system that for one reason or another are essential
to its correctness". Well.... I know the feels... Assumptions without
"understanding" is like the sword of damocles. As you sway into the program
itself, there's always this notion that "something" is not right. Or maybe our
assumption is not good enough. Most of the time, unplanned assumptions make it
harder to integrate "more" code in the long run.

------
deanalevitt
This was really quite lovely to read.

Like the others, I think the thoughts on TDD are somewhat fallacious but I
certainly can't fault the conclusion.

