
Open-sourcing Riskquant, a library for quantifying risk - el_duderino
https://netflixtechblog.com/open-sourcing-riskquant-a-library-for-quantifying-risk-6720cc1e4968
======
batterseapower
If you like the idea of this library you'll probably like the book "How to
Measure Anything" by Douglas Hubbard
([https://www.goodreads.com/book/show/20933591-how-to-
measure-...](https://www.goodreads.com/book/show/20933591-how-to-measure-
anything)). It's all about how to get sensible confidence intervals for things
that are often considered unmeasurable such as the value of IT security. The
book mostly uses Excel to do this modelling, but it looks like riskquant would
be an excellent alternative on that approach, that for the more technically
minded practitioner.

~~~
thanatropism
I have that book. Basically I’m the last in a giveaway chain and can’t
honestly recommend it enough that someone should lug it home. Next time I move
it’s going on the trash.

It’s really not very good, even for executives who shouldn’t care for
technicalities. The best thing are the calibration exercises. But my advice
is, skip this one.

~~~
qznc
Can you elaborate?

I'm half-way through it. I know most of the general stuff already but my
knowledge is from lots of sources I mostly forgot. This books seems to be a
good collection for this topic. At least, I don't know any substitute.

~~~
thanatropism
It overstates its claims. An admittedly unfair "take" would be that it
encourages people to pin made-up numbers and take solace in having quantified
the qualitative.

For example: it tells you to do Monte Carlo simulations with made-up
probability distributions, but silently lets the issue of the joint
distribution -- how the made-up random variables correlate. But so much risk
is driven not by marginals (say, I know the physical parameters for my chair
and have a certain confidence that it won't crumble like carboard) but by
correlations, even apparently distant ones (there's fracking or whatever going
on and destabilizing the upper layers of the Earth, increasing the chances of
earthquakes).

An even broader critique is the McNamara fallacy:

> "The first step is to measure whatever can be easily measured. This is OK as
> far as it goes. The second step is to disregard that which can't be easily
> measured or to give it an arbitrary quantitative value. This is artificial
> and misleading. The third step is to presume that what can't be measured
> easily really isn't important. This is blindness. The fourth step is to say
> that what can't be easily measured really doesn't exist. This is suicide."

\----

There's this book I like called "Guesstimations in the back of a napkin" or
something -- it's a book of exercises in Fermi-type estimation. It encourages
you to consider the whole, to think bottom-up, top-down and core-out. It
preserves a keen sense of the qualitative complexity of problems even as it
encourages you to, well, make wild guesses.

~~~
mindcrime
_There 's this book I like called "Guesstimations in the back of a napkin" or
something_

This? [https://www.amazon.com/Guesstimation-Solving-Worlds-
Problems...](https://www.amazon.com/Guesstimation-Solving-Worlds-Problems-
Cocktail/dp/0691129495)

------
juskrey
You can quantify in normal domains e.g. probability of seeing a human over
2.05m tall. You CAN'T quantify tail risk in unbounded (for practical purposes)
domains, period. This is just ridiculous: "For this example, there’s about a
2% chance losses would exceed $60 million in a year." If you see you can lose
everything, you are not quantifying, you are simply not going that way. If you
don't know if you can protect yourself from losing everything, you are not
going that way. If you can afford losing some limited amount you just write
down the loss from the very start. That is that simple.

Netflix is just going straight to hell with such prediction approaches. And,
mind that, this is not a prediction, this is a fragility statement.

~~~
bernardv
Oh yes you can. The insurance and reinsurance actuaries, and regulators, have
done this for a very long time.

~~~
juskrey
Insurance and reinsurance just clip the tail contractually, they never take
"infinity multiplied by some small percent" risk! They don't go underground
when their client does.

~~~
Chris2048
> they never take "infinity multiplied by some small percent" risk

At what point does the tail represent "all humanity dead"? No one needs a
contract for the humanly unrecoverable.

~~~
juskrey
We are talking about business death

~~~
Chris2048
then the tail ends even before then, but that doesn't mean insurance companies
don't pay out billions in case of an economic crash.

~~~
harperlee
They are typically exempt in case of natural disasters, war, force majeur,
nuclear effects, and a well-thought-out plethora of tail risk events.

~~~
juskrey
Yes, to be exempt from tails is basically the definition of their business.

~~~
Chris2048
Whose business? Those that don't cede such risk to larger entities?

------
pps43
Not clear why the losses should follow lognormal distribution rather than some
less pleasant to work with distribution like Cauchy.

Also not clear why loss frequency is assumed to be independent from loss
magnitude.

Instead of "percent chance that the loss will exceed X" (VaR), expected
shortfall would be more useful.

~~~
jacques_chester
The distribution comes from PERT 3-point estimates.

LEF and LEM are built of lower factors that roll up, which to my mind at least
seem independent.

I'm not sure what you meant by expected shortfall.

------
thenightcrawler
Nassim Nicholas Taleb is enraged by this!

~~~
rq1
Why would he be?

I suppose it is tempting, if the only tool you have is a hammer, to treat
everything as if it were a nail.

~~~
syndacks
>Why would he be?

Read Black Swan, by NNT. His thesis is that the most significant impacts on an
individual and/or a collective are the function of outliers or "Black Swans".
In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his
thesis. It's a pretty good book I suppose, but it was also before the all the
faux-philosophy Malcolm Gladwell type books became a thing.

>I suppose it is tempting, if the only tool you have is a hammer, to treat
everything as if it were a nail.

What exactly do you mean by that? In all sincerity, I think you misused an
idiom or at least didn't make a clear connection as to why you chose that one.

~~~
tajd
> In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his
> thesis.

In my opinion this is perhaps the best 1 line description of his work in
general. I think there are some interesting ideas which are worth discussing,
but sometimes his attitude and use of "anecdata" tires me out.

------
caseyf7
I wish there were detailed scenarios of how they were using it.

------
fancyfredbot
It doesnt seem to have an input for correlation between the loss scenarios?
That would affect expected loss a lot.

------
veeralpatel979
Link to the GitHub repo: [https://github.com/Netflix-
Skunkworks/riskquant](https://github.com/Netflix-Skunkworks/riskquant)

------
rubyfan
This seems like applying insurance modeling to security. Is this a new way of
looking at risk or a reinvention?

~~~
Eridrus
This is part of an attempt to make information security risk modelling more
quantitative that has been going on for a few years. There's very little in
the way of data to really back most of this up, but actually putting numbers
to things is significant progress IMO.

~~~
rubyfan
Agree it seems like a better way to inform decisions and manage risk. Is it
backed by any sort of real understanding of litigation, settlement, statutory
experience? I often hear reputational risk cited by security teams at public
companies... it usually follows some sort of indecision or appeal to a higher
level of management or beuqacratic tool.

~~~
Eridrus
Not really. But it's significant progress compared to sticking charts like
this into reports [https://www.shipownersclub.com/media/2017/02/Risk-
matrix-1-7...](https://www.shipownersclub.com/media/2017/02/Risk-
matrix-1-768x386.png)

I think it's also worth saying that when people say reputational risk they're
generally not thinking about litigation, they're thinking about consumer
perception, which it's not crazy to believe would have large impacts for large
companies.

------
veeralpatel979
I chatted with Markus de Shon, one of the creators of riskquant, a few days
ago. I would also check out Ryan McGeehan's writings on this at scrty.io.

I'd be happy to discuss this topic with anyone; my email address is in my
profile.

------
maxymajzr
I see an interesting library, risk quantification (relates to what I work
with).

I click the link.

Sniff/Accountwall stops me. I hit ctrl+w before I even thought of doing it
(yes for muscle memory!)

For people who went through the trouble to see the content, is the library
worth it?

These days, anything behind paywall/sniffwall/must-be-authenticated-to-read-
this-wall warrants a ctr + w from me.

------
Dowwie
Focus on the stress tests and worst case scenarios, Netflix.

~~~
Jugurtha
I suppose this is the impetus. I think that this gives them a way to
prioritize which issues to fix in a better than the arbitrary hunch way. We
assign "weights" all the time to issues either based on expected "time to
complete" or on the severity of not fixing this. However, someone's severe is
another's meh. This could give a score in a systematic way and help alleviate
the guesswork.

~~~
Dowwie
Risk is managed differently in places that have experienced actual loss. No
need to try to educate me about weighted decision making.

~~~
Jugurtha
I was not trying to "educate" you, not that I see anything wrong being
educated. I was brainstorming with you, hence the "I think" and "I suppose". I
exposed my thoughts in writing.

Your reply could have quoted my reply, with a comment that said "maybe, but
[insert counter argument, or poke holes in my statement's logic]".

------
Chris2048
I feel "riskquant" is a little vague for a project name. It's like there are a
million internal libraries called "riskLib" \- "risk" is a very general
concept!

------
leecb
Requires registration to read.

Tab closed.

~~~
muglug
If the authors are reading, I think you mistakenly marked this as a paywalled
article. I can't believe this was done deliberately from someone on the
Netflix side.

------
sheerun
Can someone explain to me why Netflix puts their posts behind a paywall?

~~~
nurettin
It looks as if they just registered for medium.com and slapped their domain on
it.

