
Observations on Observability - jlward4th
https://blog.colinbreck.com/observations-on-observability/
======
jacques_chester
What I wish I had is a book that introduces control engineering in some depth
for software folks like me.

I wanted to write such a book, or half of such a book, as a second part of
_Knative in Action_. I wrote a simulator[0] and did a ton of reading which
felt like I'd barely scratched the surface of the surface.

But really, there's just no room for what I wanted to do. Based on a simple
estimate it would have run to about 200 pages on top of another 250 pages of
actually-about-Knative material. So that whole section wound on the cutting
room floor. Maybe another time.

In terms of books I felt helped me the _most_ down this road, here's a
reasonable reading list I'd point to:

 _Feedback Control for Computer Systems: Introducing Control Theory to
Enterprise Programmers_ , by Philipp Janert. Short, to-the-point introduction
to basic classical control theory using PID controllers.

 _Matching Supply with Demand: An Introduction to Operations Management_ by
Cachon and Terweisch. A short, meant-for-MBAs textbook on operations
management (ie applied operations research). Simple and approachable. Very
good for warming up to...

 _Factory Physics_ by Hopp and Spearman. Magisterial in scope, also
magisterial in grumpiness and idiosyncracy. I've directly applied material
from this book and _Matching_ to research work.

 _Business Dynamics_ by Sterman. Still the best system dynamics book I've ever
read. I'm anxiously waiting for the 2nd edition, expected circa 2021.

 _Performance Modeling and Design of Computer Systems: Queueing Theory in
Action_ by Harchol-Balter. Approachable and enlightening. I stumbled on a few
of the proofs and got lost once or twice in thickets of notation, but that's
due to my own mathematical immaturity more than the book itself.

But I still don't have _a_ book to refer to for the specifics of software
dynamics, spanning its physics and economics. And I wish I did. I would love
to read that book, if folks can make recommendations.

[0] [https://github.com/pivotal/skenario](https://github.com/pivotal/skenario)

------
floatrock
> At scale, digital becomes analog. — Michael Feathers

> Fortunately, as discrete pieces aggregate, they start to look continuous. At
> scale, we can take advantage of this, embracing higher-level metrics that
> can be treated as continuous signals.

He's saying that rather than look at request-level metrics in your Big Data
Message Queue (request ID, etc.), your message queue _in aggregate_ can be
seen as an industrial process, and there's an entire discipline built around
managing those called Statistical Process Control.

Seen in this way, the next NewRelic is going to be all about creating p- and
c-charts [1] where the limits algorithmically turn knobs on your kubernetes
cluster.

In other words, the next big dev-ops trend is going to be the return of Six
Sigma.

[1]
[https://en.wikipedia.org/wiki/Control_chart](https://en.wikipedia.org/wiki/Control_chart)

~~~
jacques_chester
I've occasionally pitched SPC as a source of concepts to mine, starting with
XmR charts and the Nelson or Western Electric rules.

I think that actually, observability has always been one of the weakest areas
in tech -- we reinvent a lot of it ourselves while the wider world _abounds_
in rich concepts of measurement, inference and control. SPC, control theory,
inference engines, classifier systems, measurement theory, design of
experiments, fuzzy sets, Dempster-Shafer theory, avionics, system dynamics,
cybernetics and frankly a bajillion other disciplines that I've never heard of
and never will.

Sometimes in R&D, the R is much more profitable than the D.

Which point, I should say, the article makes reasonably well.

------
tiedemann
I can not say that I really understand anything but I like it; any chance of
using the term factorial in meetings outside of gaming is a win.

