
Analysis of the Imperial College Epidemiological Model - barry-cotter
https://m.facebook.com/scarlett.strong.1/posts/252437219500976
======
knzhou
We could argue about code smells all day, but ignoring that, I think there's
only one essential point made in the post. It claims that without a lockdown,
things would be just fine, because people would automatically isolate
themselves on their own. And therefore the model is useless.

This is a completely unfair criticism. When the relevant decisions were being
made two months ago, a huge number of influential people in the UK were
seriously proposing to do _nothing_ to get herd immunity. This model, which
intentionally depicts what would happen if people indeed did nothing, was a
big part of why people changed their minds.

You can't say the model is wrong because people would take COVID seriously
anyway, when the model is half the reason people take it seriously in the
first place. We're somehow forgetting history that's only months old now.

~~~
Traster
According to the Atlantic

> Johnson instead offered a suite of soft advice—people with symptoms should
> stay home; no school trips abroad; people over 70 should avoid cruises.

So to be clear, even the most "do nothing" approach was not "do nothing", and
at that time, people were alread voluntarily starting to socially distance.

~~~
knzhou
These options are so soft that they really are equivalent to doing nothing.
Sick people have always been told to stay home; trips abroad and on ships
might cause visible clusters early on, but they're a drop in the bucket later
when the disease is already at home.

The real issue is that a strong response causes people to take things
seriously, and a weak response causes them to not take it seriously. In other
words, just like the model, we have a situation where predicted results inform
the actions taken, which affect popular opinion, which in turn change the
predicted results.

Yes, the optimal outcome is to somehow have the government do nothing, thus
restricting nobody's freedoms, but simultaneously have society take it so
seriously that they eradicate the disease themselves in two weeks. But that is
not a genuine policy option.

~~~
Traster
You say that, but your intuition about those measures isn't good enough, which
is why we needed a reliable model to tell us whether those measures were going
to be effective, that's the entire point of the research. If our starting
point is "Oh well obviously we need to do lock down" then why are we wasting
money on modelling it? And since we haven't modelled the system without public
policy intervention, none of the results mean anything.

------
savara
Some questions I ask myself when reading random posts with grand and important
claims on any subject:

Where is this from? Who originally wrote it? Is this text’s origin really a
random Facebook post, from a pseudonymous author with a cartoon profile
picture and no claim of any serious credentials in the subject at hand?
(Whether epidemiology or anything else)

Regardless of the merits of the text’s post (which I do not claim to be able
to judge) all evidence has to be analysed for context as well as content.
Simple “common sense” claims (with a couple of big words to impress non-
epidemiologists like me) are made to debunk the models: where is the evidence
rather than rhetoric, even some basic citations, and/or examples of or links
to counter-modelling? The post doesn’t even link to the original model files
from Imperial that they’re claiming to critique.

It’s perfectly _possible_ that the claims made in this Facebook post are
correct, but it doesn’t mean anyone should take this post (and its
conclusions) remotely seriously without asking some very robust questions of
it.

~~~
Gibbon1
I have another view really. Which is reading a paper analyzing pandemic
overshoot. The authors covered a half dozen models. From very simple ones
built on a simple differential equations to complex ones that factored in
network effects. My take away is actual models[1] in this space tend to be
robust.

The Imperial College Model is a complex model designed to answer subtle
questions about the spread and containment of an epidemic. But there is
nothing subtle about COVID19. The model predicts catastrophe unless you turn
all the knobs to contain it to 11 and do it now.

The model is validated by real experience. Italy, New York both blew up. And
elsewhere half hearted measures merely slowed the virus down, not stop it.

The true the deniers can't escape is. If what's been thrown at the pandemic is
unnecessary. Then if so, why hasn't the pandemic just collapsed?

[1] As opposed to models that fit a prior defined curve to data. Those are
shit.

~~~
lbeltrame
> The model is validated by real experience. Italy, New York both blew up.

Do you know where the first clusters in New York where?

Because in Northern Italy, and especially around Bergamo, hospitals and then
nursing care homes turned into infection centers, and with the population
there so skewed with the most vulnerable (along with imperfect knowledge on
the pathology) it was easy for the virus to kill them.

In fact most of the initial clusters were in hospitals, and negligence turned
Alzano Lombardo in a nice spreading place.

Would a model tuned on what we know now, taking into account different
infection routes and places, work the same way? I don't know, but it is a
question worth asking even if in the end the model proves to be absolutely
correct.

------
standardsam
"the model that shut down the world economy"

The model was released on the 16th of March. Large parts of the world had
already "locked down" prior to this date.

edit: of course there is no doubt this model was influential in other
countries' decision to lock down.

~~~
ImaCake
Australia's influenza modelling had plans for this style of lockdown prepared
long before the 'rona hit. So you are right, this model did not shut down the
global economy. If anything did, it was China's demonstration that such a
lockdown could actually work.

~~~
lbeltrame
To be fair, not all of China locked down as hard as the Hubei province.

------
sampo
John Carmack, who worked on the Imperial College model code, has a pretty
positive attitude about it:

[https://twitter.com/ID_AA_Carmack/status/1254872368763277313](https://twitter.com/ID_AA_Carmack/status/1254872368763277313)

------
Traster
I think there are some really important points here, and one of the most
shocking is that Imperial put out in their summary that optimal mitigation
policies could reduce deaths by half when their model just clearly doesn't
model the base case.

> The social distancing policies are being compared to a zombie alternative
> that cannot happen and will not happen.

This is absoutely correct, my company made us work from home signficantly
before the lock down because they were quite rightly worried about business
continuity. In the Imperial model, that sort of thing just isn't accounted
for. We literally cannot say whether the natural societal reaction would've
reduced that base case by 1% or 99%.

The second point I would make is that epidemics aren't different every time.
Or rather, the underlying fundamentals of how diseases spread aren't that
different every time. So how come, all of these institutions that are _meant_
to be studying the spread of diseases are putting out papers based on hacked
together research code from the 1980s? Like, what were you doing in the 17
years since SARS?

Where are the dozens of open source, high quality, back tested,
parameterizable models sitting on the shelf ready to go?

In many ways putting out the results of some trashy model that you haven't
even properly tested and trying to shape public policy with it is actually
quite a damaging thing to do. Not only is it likely to lead to bad decision
making, it's also going to destroy trust in institutions that desparately need
to be trusted in these situations.

------
iagovar
Everything described in the review is common practice in science.

Scientific publishing is broken. Is not only about the methodologies and how
some reviews are conducted, but you can't get the files needed to reproduce
them, not to mention datasets that are almost impossible to obtain.

------
raverbashing
I've seen a couple of criticisms of the model, let's say, some less charitable
(condescending) than others. This seems an ok criticism, though it is missing
one of (IMO) important points of the other criticism (which I won't link here
but some might have seen in some "skeptical" places)

I've worked briefly with simulations, I think most of the items here are
valid. The RNG especially is a very easy place to shoot yourself in the foot,
especially if your main RNG is used to derive random numbers with different
distributions, and your numbers might look off and you don't know why.

Example, your RNG is biased or does not have all the range/precision you think
it has (similar to throwing a die when you are expecting a number between 1
and 10 for example - this sounds stupid, but think for example, your results
come as 0-100 but biased towards smaller numbers, it will be a pain to find)

Now, some other criticisms were pointing to a potential multithreading problem
in the model, which sounds like a worrying problem.

------
Mvandenbergh
This is not really a great analysis. I would like to see an analysis from
someone who:

-Is actually familiar with academic code so that the usual pearl clutching can be tempered with realism. That doesn't mean throwing up one's hands and just accepting that all academic code is like this, but it does mean coming from a place of knowing what normal academic code looks like (much worse than this).

-Is actually familiar with production forecasting in other contexts (outside of the small bubble of SV) the writer says they have "done some model review of financial models". I have done a lot of such reviews and many models are implemented in idiosyncratic Excel. Yes, models that drive policy decisions in government and large transactions in finance, models that are used in internal forecasting in many places. All Excel.

A recognition of the difference between should and is. There are claims here
about what should be done in modelling which are correct but with little
recognition that they almost never are.

Really I would like to see a review by someone who is familiar enough with epi
modelling _and_ with modern software engineering practices to give a balanced
assessment.

The only material criticism of the model that isn't just "this should have
been implemented better" is that it assumes that people will not voluntarily
implement forms of social distancing when many people are dying and that this
distorts upwards the death count. An agent based model would model people's
time varying contact matrices based on what they can observe (so as death
rates go up, voluntary social distancing goes up as well).

Presumably this leads to a higher peak than mandatory measures because deaths
lag measures which are visible to policy makers but not to the public,
therefore people will distance later.

This is indeed not addressed in the model. It could be addressed exogenously
by making interactions dependent on the rolling average death count of the
last several days as that is probably the signal that will the cause the most
widespread panic.

However. There is some important context here which is that:

-This was not even the only model used by SAGE. The LSHTM model was run alongside it and gave similar results.

-The "no intervention" case was sense checked using the basic equations of epidemiology.

Models that drive decisions can be criticised on technical-procedural grounds
(ideally this model should use fewer globals, and use psuedorandomness better
so that it gives the exact same result on every run rather than more-or-less
the same result on every run) and on model concept basis (the model assumes
zombies) but the most important assessment of real models is how they drive
decisions.

In that regard the model was successful and drove the same decision as the
very different LSHTM model, the models used by almost every other country, and
what could be estimated using Epi 101.

That is the necessity of mass social distancing. Even the Swedes,
incidentally, accept this. They have chosen to make everything except for the
closure of the highest risk establishments voluntary. I wonder though if isn't
the case that these restrictions are in practice voluntary in many cases even
in the UK. I could easily visit my family and friends - the police would never
know. My employer could continue having us come into work - there is no
absolute ban on office workers being required to come in, just guidance that
people who can work from home should. The actual legislation can be considered
not only as enforceable rules but also as high-cost signalling. There will be
people who simply will not take it seriously otherwise on the grounds that "if
it were really that bad, they would ban it"

So to come back to the one substantive criticism - what different advice would
a model that implemented behaviour changes have led to? I don't think it would
have changed anything, if you tell policymakers: "listen, we don't necessarily
need to make these things mandatory, once the coffins pile up people will stay
home by themselves" what do you think that is going to drive? They're hardly
likely to decide that we can leave off the restrictions. We came up to 80% or
so of ICU capacity as it was. That tells me that anything that led to less or
later social distancing would have put us in the position that Bergamo or
Madrid were - at least localised and temporary overwhelming of hospitals.
Potentially much worse.

~~~
Traster
>-Is actually familiar with academic code so that the usual pearl clutching
can be tempered with realism.

I think this is a real and systemic problem. Academia has driven itself into
this horrible cultural rut where they've defined for themselves this attitude
towards code that it's not important and it's fine if it's bad, hacky,
undocumented, not version controlled, copied and pasted over and over. Its
just research code!

It's not pearl clutching to point out that this is the coding version of a
teenager saving final_report.docx, final_report_final.docx,
final_report_3/23/2020.docx, final_report_v3.docx. It's basically everyone
outside of academia looking at academics and wondering why they don't just
grow up, and it's a cultural issue.

The problem is that what you describe as pearl clutching is just the very
reasonable observation that an entire profession has chosen to just refuse to
do a core part of their job.

~~~
Mvandenbergh
The problem is that it is _not_ everyone outside of academia. The vast
majority of the world's modelling code for various values of modelling is
written and "maintained" by subject matter experts who are not software
engineers. Finance, insurance, you name it and there's a horrible Excel model
driving it.

~~~
Traster
I think the core difference is that most places outside of academia there is a
closed feedback loop. You make a crap prediction, you lose a bunch of money,
you get fired. There's a self-correcting mechanism for insuring the models
work correctly.

In academia you run your model, you write your paper, you publish your paper,
you get your promotion. Because no decisions are made based on your model,
there's no accountability. Even crappy excel models in business are more
reliable than academic ones because people actually have to use the results-
and that compounds over decades.

But the impact of lots of academics generating rubbish and publishing it is
that we're literally polluting our knowledge pool. Which is why it _should_ be
of as important for your code to be right as it is for your experimental
method to be right.

~~~
Mvandenbergh
Sure, models that break along their common paths will lose people money.
Models that don't produce the results they should won't lead to high impact
publications.

In both cases, models can break in unfortunate ways along rarely used code
paths or under unusual conditions. If a model just doesn't work, then of
course it will be fixed. That's in any context. If it works most of the time
until it breaks badly once, then it may well not be fixed before it has caused
massive damage.

------
renewiltord
This whole thing has made me grateful for the fact that only engineers at my
company will know the horrors I have perpetrated in my career.

Must be awful for the poor folks whose work is being savaged by everyone.

------
octonion
Who exactly is this person and what are her credentials?

