
The Servers Are Burning - tortilla
https://logicmag.io/05-the-servers-are-burning/
======
bunderbunder
> _in order to write effective tests, a programmer had to know all of the ways
> that a piece of software could fail in order to write tests for those cases_

No.

In order to write effective tests, a programmer has to think of the piece of
software's entire input domain, carve it up into a set of equivalence classes,
and then determine what the expected behavior should be for a piece of input
from each of those classes. With more careful testing at their boundaries, so
that you can root out edge and corner cases.

The results of those tests will then find the ways your software can fail for
you.

Thinking about it this way, it starts to become clear that a huge part of your
job is finding ways to simplify that input space as much as possible.
Simplifying at this stage makes getting it right so much easier. The fewer
special, edge, and corner cases you allow in the first place, the fewer you
have to code for, the fewer you have to test for, the fewer you might
accidentally miss, and the fewer others might accidentally run afoul of.

This, incidentally, is the key reason why it's good to take some time to
define all your tests before you start writing your implementation. I'm not
necessarily a huge advocate for "red green refactor", but I do think that at
least identifying all your test cases before you ever start implementing can
potentially save you boatloads of time, by helping you recognize opportunities
to simplify your design _before_ you get locked into an unnecessarily
complicated one by a couple hours or days (or months) of sunk cost.

It's also the real reason (IMO) why functional programming - as a style, not a
kind of language - is such a valuable discipline. The challenge with
programming in an imperative style is that it turns your module's entire past
history into one of the inputs you need to consider, and that turns your test
and specification surface into something that is just so much bigger.
Considering that it opens up the possibility of injecting bugs into a routine
without ever actually editing the routine itself, or even any of the functions
it calls, it's possibly even fair to say that it's unbounded.

~~~
pjc50
> think of the piece of software's entire input domain, carve it up into a set
> of equivalence classes, and then determine what the expected behavior should
> be for a piece of input from each of those classes

This is an extremely good way of putting it. Doing some of this at the
requirements stage if possible can be very helpful too, because you can detect
nonsensical or conflicting requirements before you even write any code.

~~~
bunderbunder
I'm rather fond of a book, _Specification by Example_ by Gojko Adzic, that's
partially a manifesto for, and partially a guide to, doing this.

~~~
chrisweekly
Thanks for the rec! Just bought for $5 (kindle ed).

~~~
shady-lady
what site? i can only see manning for $36. not on amazons.

~~~
berti
I'm guessing this one (with a slightly different title: Bridging the
Communication Gap: Specification by Example and Agile Acceptance Testing)
[http://a.co/d/6tGlrUc](http://a.co/d/6tGlrUc)

~~~
chrisweekly
Yes, that's the $4.99 one I found.

[http://a.co/7gJYjIx](http://a.co/7gJYjIx)

------
gred
_> My first employer, the online dating site OkCupid, didn’t harp on testing
either. In part, this was because our company was so small. Only seven fellow
engineers and I maintained all the code running on our servers—and making
tests work was time-consuming and error-prone. “We can’t sacrifice forward
momentum for technical debt,” then-CEO Mike Maxim told me, referring to the
cost of engineers building behind-the-scenes tech instead of user-facing
features. “Users don’t care.” He thought of testing frameworks as somewhat
academic, more lofty than practical._

I found this paragraph interesting because my experience with my one-man side
project is the opposite: I'm so small, and time is so scarce (I have a
separate full-time job), that I can't afford to fix bugs on somebody else's
schedule when _they_ find them... instead I spend a large amount of time
writing tests up front for new features so that I find the bugs on _my_
schedule (i.e. late at night when I don't have other commitments).

~~~
toss1
Yup, classic.

"...no time/resources to do it right in the first place, but plenty of
time/resources to fix it when the customers complain..."

I always thought it better to find the bugs in-house before shipping, but so
many others don't see it...

~~~
megablast
You are acting like most bugs aren't already caught in the dev process. No on
writes code and just deploys.

Writing extra tests that cover everything for a small project that you are
working on yourself is a waste of time.

~~~
LandR
> No on writes code and just deploys.

Oh yes they do!

Hell, I've seen dlls copied from a server, disassambled into code, code
changed, rebuilt and copied back to server....

Most devs here write code then copy their bin folder to a production server
and deploy it.

------
mikestew
_(then-CEO Mike Maxim) thought of testing frameworks as somewhat academic,
more lofty than practical._ ... _Mike the CEO, who was also OkCupid’s best
engineer..._

Mystery of the Melting Servers, solved.

~~~
plopz
Do testing frameworks normally catch memory leaks?

~~~
mikestew
Mine do.

~~~
fhood
You can't possibly say that with any confidence. I literally just finished
reproducing a memory leak caused by an insane combination of circumstances.
This was not something valgrind or any similar tool would find. It was not
something that any rational human being would have thought to write tests for.

Once code is no longer synchronous, and tasks are being juggled around and
swapped between, it becomes virtually impossible to predict what might cause
this sort of bug.

~~~
mikestew
_You can 't possibly say that with any confidence_

In answer to the original question, I stand behind my answer with 100%
confidence. The test infrastructures I build catch memory leaks all of time,
and I do not consider it abnormal for them to do so. If you mean to say that I
cannot be confident that I have caught all of the memory leaks, well duh, of
course I can't. But that's not what was asked, and that's not what I said.

To expand on what was actually asked, though, whether a framework catches all,
some, or no memory leaks is irrelevant. Because if a team is at least testing
for it, I'll bet they're just a bit more rigorous in their coding than a team
that just deploys to production and waits to see what breaks.

~~~
dawidw
> The test infrastructures I build catch memory leaks all of time [...]

Could you elaborate on that, please? I'm curious. Thank you in advance!

~~~
mikestew
It would be very specific to the project. For instance, if it's an iOS/Mac
project, Xcode has nice profiling tools to catch memory leaks. And of course
clang's static analyzer as well. What I'm working on now is embedded Linux, so
valgrind and some test suite to exercise the code.

But in general, no matter the project, you need a few key pieces:

1\. A test suite that thoroughly exercises the code. You'll need this for...

2\. A tool such as valgrind to watch memory as your tests in #1 run. Build
time, ad hoc, stick in the pipeline anywhere you like.

3\. A static analyzer to take a first pass at the code and say, "hmm, this
_might_ leak/crash/kill puppies." So, yeah, the order is off as this should be
the first thing you do. It can run on your dev machine or on the build
machine. Expect lots of false positives.

(optional) 4. A code coverage tool to make sure you're not missing key pieces
of code that need exercise.

------
1_800_UNICORN
If you're an early-stage startup that's still determining product-market fit,
then it's completely fair to eschew tests and accumulate tech debt; you're at
a higher risk of running out of runway before you've even proven that your
business works.

As soon as you have enough users that an outage poses a significant risk to
your business, you need to invest in either refactoring to reduce tech debt,
or a rewrite. And you NEED to add tests, and a robust deployment process that
tests changes at multiple stages with monitoring on the basics (server speed
and memory usage, database error rate, etc).

OKCupid got lucky in this case.

~~~
holmberd
Easy to say, but unfortunately rarely seen in the wild, depending on the
experience of the project manager. Once that product is live and climbing, no
owner wants to hear talks about slowing down for reasons of better test
coverage or refactoring.

~~~
usefulcat
I think the point is more that the slowing down is going to happen anyway if
production systems are repeatedly catching on fire and people are afraid to
make changes. At that point the difference is how having or not having tests
will affect things in the longer term.

~~~
holmberd
Indeed, but prototypes running in production or running without tests, is a
symptom of other underlying problems in the organization. I feel that scope
creep and lack of testing goes hand in hand for projects that are poorly
managed.

------
matt2000
When making something new where you're not sure of the value yet, I've found
that you can get 80% of the benefits of unit tests with around 20% of the
tests you'd write to get "full coverage."

My main goal is to at least have the code run in an expected way and produce
an expected result. This doesn't catch everything, but it does seem to catch
enough problems to be worth it for the time invested.

Edit: I should mention that I also add tests to cover something when I
experience a failure, so it at least won't happen again.

~~~
jerkstate
I'm not a great programmer, I can't usually write code that does exactly what
I expect the first time, so I use unit tests with a debugger in my IDE to run
small portions of my new code until it works the way I envisioned it before I
started writing it.

This is a lot faster than my old method of writing code that doesn't quite
work, and running the whole program over and over with small changes and print
statements each time.

~~~
TeMPOraL
> _so I use unit tests with a debugger in my IDE to run small portions of my
> new code until it works the way I envisioned it before I started writing
> it._

In languages like Lisp, this is what you use REPL for. It's an insanely more
efficient way of working than the usual edit->recompile->run the whole app
again.

However, couple of years working with Common Lisp and (recently) Clojure
taught me that, even with a good REPL at your disposal, properly testing code
as you write it can get pretty unwieldy - especially if the inputs are
complex/large. I found it beneficial to write unit tests and call them from
REPL instead of testing the code in REPL directly, as it's easier to maintain
and develop individual test cases, as well as re-test all of them when I alter
the code I'm working on.

(That's of course apart from the fact that unit tests are more permanent, and
provide value later on.)

TL;DR: thumbs up for writing unit tests even when working in an interactive
programming environment.

~~~
Sophistifunk
REPLs are a great tool for exploring your ideas, and poking your software to
test it "right now" but once you're happy with a small piece of code, nothing
beats a suite of unit tests to help you refactor with confidence as your
implementation gets a bit too hairy and you need to rejigger it in order to
progress in a direction you didn't see coming.

------
evancox100
"If you wrote buggy software, why would the software you wrote to check that
software be any less buggy?"

I work in chip design, where bugs can be rather costly, e.g. $1mm for a new
mask set, not to mention the months it takes to get back new hw. The situation
described above is why we try to have one person do the design and a different
person do the verification/testing. A lot of the time, the test writer will
treat the design as a black box, not even look at the code, and instead verify
to an external specification. (There's also white box testing, where you try
to target specifc areas of the design that might be especially problematic.)

This does protect against issues like misinterpreted requirements, invalid
assumptions of valid input and/or operating scenarios. But I think it also
ends up as a lot more work, to have two people get familiar with whatever it
is you're designing. The best method is probably a combination of both.

~~~
ereyes01
When you have a well thought-out external specification, it really gives you
lots of good material from which to derive all the right tests while still
treating your implementation as a black box.

Unless you are implementing an established RFC or some other well-defined
system, such a tight specification rarely exists in a software setting, and is
very difficult to effectively design a priori. For this reason, techniques
like test-driven development have arisen to try to make the programmer
repeatedly and consistently assert their understanding of the specification
via unit tests, while evolving those assumptions as needs and understandings
change.

------
3pt14159
I appreciated this article because I so infrequently get to hear from people
that hold this view of testing, which I do not share.

To me, the importance of tests are a function of the consequence of failure
and the likelihood that failure will happen. It's a question of hazard and
risk.

One thing to realize is that code that lasts longer is more likely to fail
because the people and libraries that support it are prone to change and
assumptions can break. So risk goes up over time.

The other thing to realize is that code that's part of a growing product is
going to impact more people if it fails. There might be only 400 users today,
but 40k three years from now. So hazard goes up over time.

When I heard the technical debt arguments I try to triage them into "grows
with the company" vs "doesn't grow with the company" so I can figure out what
to accept as debt and what to pay. A complex deploy infrastructure doesn't
scale. Just SSHing into a box and doing the deploy manually is just as fine
for 100 users as it is for 1000. But tests do. So I write lots of tests;
especially for ACL and I punt on the infrastructure stuff until I absolutely
need to.

Note that this leaves aside the entire argument over whether tests slow stuff
down. I think that on average they make refactors easier and features slower,
so for the projects I'm on it's a wash. But even if I grant the point my
conclusion wouldn't change. Tests are good.

~~~
ozim
I think just SSHing to box and doing deploy is not function of users as you
noticed but function of how many devs are working on project. For 1-2 devs
manual deploy might be ok, when 3rd comes in, it is time to get rid of any
magic that might happen when deploying by hand. Because you know someone will
do something special and not document that, when he updates automated deploy
he does not have to document or talk with other guys, it is documented in
deploy tool.

~~~
__david__
It's also a function of how many servers you have. If you have to ssh into
more than one box to deploy then you should probably have automation, even if
you only have an ops team of 1 or 2.

------
marcus_holmes
This is one of the many reasons why I have an allergy to dependencies.

I know other people's code is probably better than mine. But I understand
mine. If something goes wrong (and something always goes wrong) I know how to
fix it. When something goes wrong in someone else's code I either have to
start working out how their code works, or report it and sit there like a
lemon with a broken system until whoever wrote it has the time, energy and
inclination to fix it.

Also, testing. I wish I was better at testing. At least I've got into the
habit of writing tests to cover whatever bug I just discovered in production.

~~~
azeirah
All I know is that if enough people rely on the same dependency for long
enough, the chance of encountering large bugs becomes smaller and smaller.
Especially if the dependency has a stable interface.

Good software gets better the more it gets used and abused, so I tend to stay
away from small dependencies, they're usually not worth the time.

------
noelwelsh
_How could such a tiny change have such an outsized impact on the site? “That
same story happened so many different times,” my old boss David told me.
“Someone launched a small, relatively innocuous change that did one of the
millions of unexpected things it could have done, which then happened to break
some part of the site, and then bring it all down—sometimes bring it down to
the point where we couldn’t recover it for hours.”_

Preventing this is one of the basic motivations of functional programming. If
a function is "referentially transparent" or "pure" it can't do anything
unexpected. It seems to me this is the only reasonable way we can reason about
software at scale.

~~~
mreome
While I can understand some of the benefits of functional programming within a
single sub-system, or within academic research, I struggle to understand how
pure functional programming can address the need for the large amounts of
state information and abstraction layers required for a large scale system. I
work with systems where (large/numerous) sub components require very
specialized domain knowledge to understand internally, so multiple levels of
black-box abstraction are required to integrate/interact with the sub
components. Additionally, both the client systems and the sub components may
have huge amounts of persistent state information.

Could you suggest any resources that might help me understand how this can be
accomplished in a pure functional paradigm? Something that assumes only a
casual understanding of functional programming would be especially helpful.

~~~
TheDong
> Do you have anything that explains this complex topic?

> Something that assumes I don't know anything about the pre-requisite topic
> would be helpful

Unfortunately the only way to gain a deep understanding of the powerful tools
functional languages do give you is to gain a deep understanding of functional
paradigms.

Me saying "Monads can help with state abstractions" doesn't help you.

The fact that large non-trivial applications have been written in haskell
should be enough evidence that this is possible.

The fact that the majority of people involved in such endeavors have claimed
that it made their code safer and easier to refactor and had relatively few
bugs should be enough evidence that it's a good idea.

The fact that those who decry it more often than not do not know the subject
should give their criticisms no weight.

I don't see why you need any more evidence that you should simply learn
functional programming so that you may first-hand answer your own question.

~~~
kbenson
> Unfortunately the only way to gain a deep understanding of the powerful
> tools functional languages do give you is to gain a deep understanding of
> functional paradigms.

> I don't see why you need any more evidence that you should simply learn
> functional programming so that you may first-hand answer your own question.

It is possible to interpret the GP comment as a request for good resources to
learn FP. You seem like you could probably point one or two out. Since they've
already expressed a desire to understand, that might be more beneficial for
them than the equivalent of "just do it".

~~~
mreome
Thank you. Basically I was looking for the functional-programming equivalent
to the OOP factory-equipment analogy, or asking if such an analogy does/can
exist (if it can't, the why might also be a valuable explanation). That
analogy can be used to explain the main ideas behind encapsulation,
interfaces, and internal state, and why they are valuable, before having any
understanding of how those are implemented in practice, and even without any
programming knowledge at all. So far everything I've found on functional
programming starts with implementation details and provides no initial
framework for understanding the broad-strokes of organisation and value
proposition.

~~~
lavayya
You might like Rich Hickey's talks, especially "The value of values", "The
language of the System", and the 2018 conj key note. The Talk in transducers
also has a nice analogy: the conveyor belt.

------
JackFr
Wow.

1\. We don't test. 2\. We don't code review (or rather if we do, we do it so
poorly swallowed exceptions don't raise red flags.)

That's an outrageously unprofessional software process.

~~~
andrenotgiant
An outrageously unprofessional software process that created a product worth
$50M!

[https://techcrunch.com/2011/02/02/match-com-acquires-
online-...](https://techcrunch.com/2011/02/02/match-com-acquires-online-
dating-site-okcupid-for-50-million-in-cash/)

~~~
danesparza
The OP is about technical process in the software world. I'm not sure how a
company's valuation comes into the conversation...

~~~
blacksmith_tb
I assume the implication is "if OKC can fly by the seat of their pants and
still make millions, anyone could!" Which is true, but still not advisable...

~~~
stcredzero
Given that technical debt is quite hard for non-technical people to quantify,
why wouldn't we expect that most startups would tend to accrue technical debt?

~~~
theseatoms
Can _anyone_ quantify technical debt? Best case is something resembling a
medical diagnosis.

~~~
stcredzero
I think that could be enough. Actuaries have a good idea about when you are
likely to die in the next two years. Basically, when you are old and your
medical costs suddenly double. Doctors use the same sort of inductive
reasoning for their prognosis. The industry probably collectively has all the
information to put together useful actuarial tables on technical debt, but
it's all silo'd in individual companies.

~~~
theseatoms
Interesting. Are there IT consulting firms that specialize in this field of
study?

------
stcredzero
_Yet what I found even more troubling was that in order to write effective
tests, a programmer had to know all of the ways that a piece of software could
fail in order to write tests for those cases._

So the programmer needs to think about all the ways the code can fail. Writing
the tests concretely documents that thinking in the source code repository and
automates re-checking that thinking. A really good programmer then tries to
organize the code, such that it's easy to think about how any given unit of
code can fail. Writing unit tests rewards such organization and penalizes bad
organization.

That said, TDD doesn't seem to work out that way in practice, most of the
time.

------
hnruss
After reading the introduction, I thought that the author was going to use a
story of software failure as an example of why you should write unit tests, or
at least why you shouldn't deploy untested software.

However, the moral of the story was essentially: "software is so complicated
that it is bound it break, so you have to be good at fixing it". While that is
certainly true, I think that developers have a responsibility to use whatever
tools they can to write high-quality software.

~~~
jschwartzi
> However, the moral of the story was essentially: "software is so complicated
> that it is bound it break, so you have to be good at fixing it". While that
> is certainly true, I think that developers have a responsibility to use
> whatever tools they can to write high-quality software.

I actually disagree that we have a responsibility to write high-quality
software. As engineers our job is to create software that is good enough for
the task at hand and that can be improved later.

Sometimes that means that you invest in 100% coverage plus integration and
system-level testing because "good enough" means that it has to never fail.
Sometimes that means you don't spend much time automating tests and instead do
informal development testing before you check something in.

It's entirely up to your judgement based on what your code is going to be used
for, and I don't think it was inappropriate for OkCupid's engineering staff to
avoid testing in this case. After all, their business was really successful
through the period where the error happened.

~~~
jhayward
> As engineers our job is to create software that is good enough for the task
> at hand

This is the essence of engineering. As the saying goes, "Any fool can build a
bridge that won't fall down. It takes an engineer to build a bridge that _just
barely_ won't fall down."

In other words an engineer's contribution is not perfection, it is being able
to know the difference between "good enough" and "needs more work" and to
apply that knowledge to meet time, cost, and material constraints.

~~~
user5994461
Every bridge will fall down, it's just a matter of time.

------
drblast
The reactions to the outage on Twitter are hilarious and make me glad this
happened:

“@okcupid how am I supposed to get my daily dose of crushing rejection and
emotional humiliation if your site is down????”

“Okcupid stops working over lunch hour, NYC wonders if we're committed to the
people in our phones from now on, panic in the streets”

“@okcupid How can I continue to be ignored by the females of the world if they
don't know where I am to ignore me?! #panic #freakout”

------
ahallock
> ...in order to write effective tests, a programmer had to know all of the
> ways that a piece of software could fail in order to write tests for those
> cases

That's impossible. You reason about the problem as best as you can and create
tests appropriately; and then if there are edge cases/failures in production,
you go back and add those to the test suite.

------
gonzo
> If she forgot the square root of -1 was undefined, she’d never write a test
> for it.

Sigh. sqrt(-1) is not “undefined”.

------
Jeff_Brown
Static typing helps.

Higher order programming helps.

These are two reasons I love Haskell. The compiler does a lot of work of
understanding your code for you, and makes it easy to query that metadata
interactively, and refer to it from other parts of static code.

Haskell is of course not the only statically typed language, but its type
system encompasses more, and more uniformly, than any other I am aware of. A
lot of things that used to have to be part of the comments can now be first-
class citizens.

------
mirimir
I'm reminded of the day that I discovered SQL cross joins. Some decades ago,
working in Access on WinXP. So I executed, and nothing much happened. So I
went for lunch, and when I got back, the machine was frozen. So I went WTF?,
and hard reset. And then did it again, a few times.

And the cool thing was that doing that, and running out of disk space,
apparently didn't damage the system. Microsoft must have designed WinXP to
tolerate that.

------
heisnotanalien
You write tests not only to hopefully catch bugs but so that you can change
the software reliably significantly in future and not lose forward momentum.

------
ww520
Yep. I had caused a similar problem in the past that brought down a live site.
It's a cascade failure on error handling that caused an avalanche of retrying
requests piling up that eventually more and more servers failed under the
load. Not fun. Luckily we had well defined deployment and rollback procedures
and was able to roll back the change easily.

------
stephengillie
> _“Users don’t care.”_

This is the counter to every technical bikeshedding post. Either it creates
revenue by being what users want, or it's a waste of time. And people seem to
love wasting time instead of delivering.

~~~
TeMPOraL
Programming would be trivial if things were so simple.

The reality is, that "either it creates revenue by being what users want", or
it enables creating more of that later on, or prevents the speed of creating
more of that from dropping, or prevents the whole thing from catastrophic
errors (like in this article), ...

Those are the trade-offs engineers have to consider daily.

------
08-15
Is this really about testing?

> If (the database throws an error) { do nothing }

It's paraphrased, the original was probably the C++ anti pattern of

> try { / _some code_ / } catch {...} {}

Whoever wrote this should be disciplined. This kind of code doesn't do
anything, it just makes a bad situation worse. (It's also never tested,
because who writes tests that provoke failures?) This kind of idiocy has no
technical solution, unless the two-by-four counts.

If you're ever tempted to write something like this... don't. Especially in
C++, the correct approach is to do nothing. Someone else will handle the
exception. Code that isn't written doesn't cause trouble and doesn't need to
be tested either.

------
franzwong
Sometimes I want to see how a piece of code work, I can just run the unit
test. Otherwise, I may need to start the server, prepare some unrelated data,
send a request to the server.

------
xupybd
A very well written piece that sums up the state of play without any sense of
panic. That’s pretty rare for this sort of article.

------
planxty
Test your code, or stay the hell away from me. :)

------
TheCapn
_(Off Topic -- Rant regarding Webpage)_

Can I take a short moment to bitch about "sidebar" frame or whatever the hell
this domain is using? The webpage looks like a fluid single frame but the left
side is static. The browser page has a scrollbar but I need to have my mouse
over the correct div before my scroll wheel will move the page, its a bit
aggravating since there's no visual break between the "sidebar" and the
content.

~~~
pwg
I just turned off CSS in Firefox (View menu -> Page Style -> No Style) and the
fixed sidebar was no more and the odd scroll bar behavior was gone.

~~~
pritambaral
I used Firefox's Reader Mode.

