
Show HN: Covid-19 Interactive Model - nbnoll
https://neherlab.org/covid19/
======
nbnoll
Hi everyone, My name is Nicholas Noll, a computational biologist and co-
developer of the above tool. As of now there are two other people actively
working on this project: Richard Neher, Ivan Aksamentov, and I. Richard and I
have a background in theoretical physics but we have been working on
quantitative biology and epidemiology. Richard is also one of the founders of
nextstrain.org which leverages phylogenetic analyses to track the global
spread of many pathogens -- including SARS-CoV-2. The other person, Ivan, is a
super talented software engineer who just recently joined our group.

We started working on this a few weeks ago, in collaboration with Jan Albert
and Roberk Dyrdak, as a helpful model for public health officials and
hospitals to help predict the usage of local community hospitals, and explore
the utility and immediate need for quarantine measures.

We are very much still actively developing the tool, now with an eye to help
global communities at large. If anyone is interested to help, we are trying to
organize a road map and put it together on our Github page. I'll be around to
answer questions and listen to feedback.

~~~
starpilot
Interested in the backend. What is this coded in? Python/dash?

~~~
nbnoll
It's all coded in typescript and javascript. I've been toying around with a
more in-depth model in web assembly, but as I alluded to elsewhere in this
thread, the coding is slow going.

~~~
Lukas1994
Hey Nicholas, love the tool - great job! I'm working on a new modelling tool
([https://causal.app](https://causal.app)) and built a COVID-19 model myself:
[https://my.causal.app/models/1432](https://my.causal.app/models/1432). This
lets you easily run Monte-Carlo simulations. Might be useful for you. But
maybe your model is too advanced for Causal.

------
jupp0r
It would be really nice if the uncertainty of fatality and severity
information we have right now could be incorporated and visualized. Current
reports by the CDC and info from France and Spain suggests that the age
distribution of severe/critical cases might look very different than data from
China made us believe. On the other hand, we don't know enough about the
prevalence of asymptomatic cases, which have huge impacts on spread and
severity, potentially.

It would be really nice to model the probability distributions for each of
these and see the resulting probabilities of simulation outcomes.

~~~
jansan
Also, the huge uncertainty regarding the ratio of symptomless/unreported cases
should be visualized. Guesses are in the range from 30 to 90%.

------
guscost
Shouldn't this tool allow for modifying the numbers in the "Severity
assumptions" section, too? This uses the same estimate from the Imperial
College model, which is basically an educated guess at this point:

> A total of 72,314 patient records—44,672 (61.8%) confirmed cases, 16,186
> (22.4%) suspected cases, 10,567 (14.6%) clinically diagnosed cases (Hubei
> Province only), and 889 asymptomatic cases (1.2%)—contributed data for the
> analysis.

Another study guesses that _86%_ of cases are asymptomatic and/or otherwise
undocumented:

[https://science.sciencemag.org/content/early/2020/03/13/scie...](https://science.sciencemag.org/content/early/2020/03/13/science.abb3221)

~~~
nbnoll
All numbers in the table should be editable by the user -- apart from the
fatality percentages which are computed from the values given. We've been
working on the design to make this more obvious at a glance.

~~~
guscost
Aha, thank you! Nice work putting this together, it's much easier to
understand than a paper.

~~~
nbnoll
No worries. I think it's also important to note that the Imperial College
study that's gotten a lot of attention recently is an agent-based model - i.e.
they simulate actual individuals - which allow them to look at specific
quarantine measures like school closures. You can accomplish an approximate
version of this with our model by controlling the isolation column of this
table. This allows one to selectively quarantine age groups quantitatively.

~~~
guscost
Cool - again, nice work!

------
gus_massa
Someone notices in another thread that in this model the death number does not
depend on the number of beds or ICUs.

~~~
nbnoll
We are working on this! It originally started out just as a visual
representation however now it factors into the model. Check out branch
feat/hospital-bed-model

~~~
gus_massa
Nice to hear that, because one of the important factors is if the health
system gets oversaturated.

Edit: For someone else that is interested, the link to the repo in GitHub, is
in the "About" section at the top.

Edit 2: The About section has a lot of info. Perhaps you should copy part of
that info to the main page.

Edit 3: When the site loads, it should show the graph of the simulation with
the default parameters.

~~~
nbnoll
I just wanted to reply that the finite ICU capacity was merged last night and
is now on the main webpage.

------
chisleu
Does it scare anyone that the North / Fast setting for the USA puts the death
count at ~1/2 reality for the past few days? Does this indicate that it spread
faster than fast?

This is going to be really really bad isn't it?

~~~
spookthesunset
Consider that in the US testing has barely started and even then there are a
ton pre-conditions to get tested. Any “scary growth curve” of any absolute
number you see is almost certainly a reflection of how much tests are being
done. More tests means more confirmed results which means a nice scary looking
curve.

If you start with a hypothesis that this virus is already in widespread
circulation and possibly has been for a while (which is the simplest
explanation, mind you), odds are good many people already had it or have it
right now.

Only way to prove/disprove if it is widespread is by testing a random sample
of the entire population. Something we aren’t doing in the US.

~~~
jhncls
What's really disturbing and hard to doubt is the growing daily death count in
a country as Italy [1]

[1]
[https://www.worldometers.info/coronavirus/country/italy/#gra...](https://www.worldometers.info/coronavirus/country/italy/#graph-
deaths-daily)

~~~
badfrog
It looks like the number of deaths per day hasn't really been growing in the
past ~5 days. That's somewhat comforting.

------
robhunter
Just a suggestion, but some general thoughts on what activities correspond to
what levels of mitigation - eg. are schools closing at 80%, restaurants at
70%? What activities result in which %'s?

------
billions
I built Sneezemaps.com to correlate symptoms & confirmed cases

US:
[https://sneezemap.com/?zone=eyJjZW50ZXIiOls0MC4yNzk1MjU2Njg4...](https://sneezemap.com/?zone=eyJjZW50ZXIiOls0MC4yNzk1MjU2Njg4MTI5MSwtOTguODc2OTUzMTI1MDAwMDFdLCJ6b29tIjo1fQ)

Europe:
[https://sneezemap.com/?zone=eyJjZW50ZXIiOls0Ny42NjUzODczNTYz...](https://sneezemap.com/?zone=eyJjZW50ZXIiOls0Ny42NjUzODczNTYzMjY1NCwxMC4zNzEwOTM3NTAwMDAwMDJdLCJ6b29tIjo1fQ)

~~~
oehpr
this site isn't working for me, I grant location access and it says I need to
turn on location access to see the map. Doesn't seem to be working?

Fails on Chromium and Firefox.

------
patrickbuckley
Thank you for making such a great tool. Do you have any plans that will allow
the death rate to be adjusted once ICU overflow kicks in? There is going to be
a much higher death rate once we run out of ICU beds and it would be
interesting to see how that changes things

------
artur_makly
503 ERROR

------
reubens
I’d be interested in helping out with the front end. Having some issues with
display on my iPhone SE. Thanks for the awesome tool!

------
titzer
Why is it when I reduce the infectious period, the steepness of the infection
curve increases? That seems wrong.

~~~
gridspy
Possibly because you forgot to reduce the R factor. If you infect people over
1/2 the time, perhaps you need R to be 1/2 what it was before.

I assume that the model is spreading the people you infect across the time you
are infectious.

------
altaaf_baatli
The link is not working right now

------
toddm
Thank you for providing a very nice tool that is of use for all of us. Much
appreciated!

------
Ameo
After reading about the implementation of this model and experimenting with
different a while, I've made some observations about its behavior and
especially about the values that have been chosen for its for its parameters
which I feel are important to consider when interpreting its output.

TL;DR: Many of the critical inputs to this simulation are based off of almost
entirely unknown values, and relatively small changes to these inputs can
swing the results of the simulation by orders of magnitude. The defaults that
the website creators have selected seem pessimistic on average compared to
even the source research which can have the effect of misleading users as to
what the expected outcome of the pandemic is.

Since under-reacting and over-reacting to this situation both have huge real-
world ramifications, making assessments based off of even a fully valid and
well-designed model can lead to incorrect decisions being made if its
parameters are set unrealistically.

\----

First of all, there are a TON of variables and many of them pull values from
extremely thin data. Although it looks like the researchers didn't just make
up any of their values, there are several instances of very impactful
parameters being hand-waved:

## MITIGATION CURVE

The "Mitigation" ratio curve, an unsurprisingly critical variable for the
simulation, seems too high even for the "strong" mitigation preset. Even after
looking extensively, I couldn't find any explanation or justification from the
simulation's creators as to how they created their default curve. As this is
possibly the most impactful parameter in the entire simulation, that just
makes it more critical to correctly pick its value.

Judging by the degree of dramatic measures that society has taken so far and
the actions of people that I've observed, I'd personally estimate that ratio
to be much lower than even the one included in the "strong" mitigation curve.
Of course this is random guesstimation of my own here, but it goes to
illustrate the massive changes in results that can stem from small changes to
the input values they've chosen.

After starting at default settings, setting population to USA, and setting
mitigation factor to "strong", further adjusting the March 15 mitigation
factor value down 10% from 60% to 50% (a value that seems extremely reasonable
and even generous to me) causes the total number of cases to drop by nearly
66% holding all other values constant. Adjusting it to 40% reduces the total
infections by another nearly 50% from there.

Another paper that is cited by the simulation's authors writes that "[...] by
taking drastic social distancing measures and policies of controlling the
source of infection, with the tremendous joint efforts from the government,
healthcare workers, and the people (Fig. 1), Rt was substantially reduced
[from 3.58] to 0.32 in Wuhan after February 2, which was encouraging for the
global efforts fighting against the Covid-19 outbreak using traditional non-
pharmaceutical measures [...]" [4] This would seem to represent a mitigation
ratio of 0.1 after just around a month. Clearly there are differences in the
way that the pandemic was handled in China compared to how it's being handled
now in the rest of the world, but China's ability to combat it that
effectively seems to lend credence to the idea that mitigation ratios of less
than 0.5 are possible if not already in place in the US and across the rest of
the world.

In any case, claiming nearly 80% of the transmissibility of this virus
remaining at this point in time and maxing out at 60% as the "moderate" case
seems dangerously exaggerated to the upside and potentially skews the results
of the whole simulation due to how critical this one variable is in
determining the output.

## ANNUAL AVERAGE R0

Another foundational variable is R0: The number of additional infected
individuals per infection. The linked research paper states "the early human-
to-human transmission of 2019-nCoV was characterized by values of R0 around
2.2 (median value, with 90% high density interval: 1.4–3.8)" [2] Although the
distribution from the research paper has a long tail towards higher values,
setting a value of 2.7 for "Moderate/North" feels somewhat disingenuous given
that the mean is 2.2.

Of course other research papers list a wide range of different values for this
variable, so perhaps expanding the range of the presets would be a better
option. In any case, this variable is dominated by the mitigation factor in
cases of high mitigation, so its precise value may not matter as much in those
situations.

## SEASONAL FORCING

The "seasonal forcing" factor varies from 0 to 0.2 in all of their
"Epidemiology" presets. On their about page, the example they provide in their
graph seems to have a value of ~0.6, but that may just be an illustration
using non-realistic values for visual effect. [1]

One other thing to note is that their implementation uses the selected R0 as a
mean value for their function, meaning that at the peak month the true R0 is
`(1 + <seasonal forcing>) * R0`. This doesn't agree with the data from the
research paper, which estimated their R0 value using data from the most
infectious period (winter). Assuming a seasonal forcing factor of 0.2, that
means The R0 values they provide are actually 20% inflated in January on top
of their already high values.

Of course, their very choice of a cosine wave to model that impact seems
largely unfounded, but of course given a lack of data pointing to a more
accurate option it's as good as any. That being said, even a small change to
that function could have massive impacts on the simulation's results.

## ICU OVERFLOW

In the most pessimistic scenarios, their implementation of additional
fatalities caused by ICU overflow fail to take into account potential measures
such as emergency hospitals, things like the hospital ship being sent over to
New York by the navy [3], and temporary increases to hospital capacity. In
most cases, the peak of ICU overflow doesn't occur until months down the line,
giving a lot of time to scrape up resources to increase capacity.

Perhaps providing an additional value to adjust the hospital capacity over
time would be useful for accurately representing the impact of ICU overflow.

## OTHER VARIABLES + NOTES

\- "Imports" is held constant through the entire simulation. It seems unlikely
that it would remain this way; setting this as a curve instead would make more
sense in my opinion. Also, I couldn't find any justification or explanation
for the values they picked for that parameter.

\- Improvements to treatment causing ICU/hospital stay time to go down aren't
accounted for. Of course there's no guarantee that it will change
significantly as the pandemic progresses (or even that it will not get worse
due to some mutation of the virus or other circumstance), but recent research
and experimentation seems to already be making progress in creating more
effective treatments for the virus.

\- Due to how incredibly vast the system this model models is, there are an
uncountable number of possible external events that could, in some situations,
invalidate the whole thing entirely. At least mentioning the possibility of
things such as a vaccine being released in the coming months, mutations with
developed antiviral drug resistance, etc. seems prudent to me.

\----

NOTE: I'm not a trained statistician or epidemiologist - I just have
experience working with data and compute models. I'd appreciate any feedback
about the accuracy of my comments here or expansions/refutations of my
thinking.

[1] Link from about page:
[https://neherlab.org/covid19/assets/seasonal_illustration.15...](https://neherlab.org/covid19/assets/seasonal_illustration.151cec1.svg;)
Screenshot at time of writing:
[https://ameo.link/u/7pk.png](https://ameo.link/u/7pk.png) [2]
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7001239/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7001239/)
[3] [https://www.nbcnewyork.com/news/local/navy-medical-ship-
comi...](https://www.nbcnewyork.com/news/local/navy-medical-ship-coming-to-
relieve-nyc-hospitals-amid-virus-was-last-in-city-after-9-11/2333767/) [4]
[https://www.medrxiv.org/content/10.1101/2020.03.03.20030593v...](https://www.medrxiv.org/content/10.1101/2020.03.03.20030593v1.full.pdf)

------
llamataboot
The biggest thing I've been trying to get friends to understand (from 2 weeks
ago on the we have to shut things down train, and now on the opposite "look
not interacting with any other human being for 2 months might not be the best
course of action train") is the non-linearity of how infection rate responds
to decreasing the amount of social contacts.

ie; in most models you can get about a 50% gain with just a 25% reduction in
social contact, after that it is diminishing returns all the way down to the
bottom of trying to wring small population level effects out of reducing
contacts from "very little to very very little"

In my mind, people are still underestimating the effects of large gatherings
(and by large here I even mean 100 people in a nightclub) and overestimating
the effects of seeing one friend in your house that sees another one friend
the next day etc.

Don't want to get into an overall debate here about what level of distancing
is appropriate or acceptable, would like to see an interactive model that does
a good job showing how different types of distancing interventions have very
different effects and that once you get to a certain point it gets very hard
to get much more overall connection out of the system...

~~~
listsfrin
There is official health institute report from Italy:
[https://www.bloomberg.com/news/articles/2020-03-18/99-of-
tho...](https://www.bloomberg.com/news/articles/2020-03-18/99-of-those-who-
died-from-virus-had-other-illness-italy-says)

This is just a panic created on social media.

~~~
chimprich
> This is just a panic created on social media.

Don't be ridiculous. You can discuss the distribution of deaths, but the
deaths are real.

A 2% death rate is still 2%. That's huge. It still has the potential to
overwhelm any country's health system.

~~~
listsfrin
The deaths were always real. Putting the whole planet on hold because some 80
years old with three associated illnesses died is a bit borderline SF and a
bit borderline stupid.

~~~
deelowe
It's not going to be 80 year olds dying if the hospitals are overwhelmed. 40%
of people in ICU in italy are 19-40, basically, those who make up the majority
of the workforce.

~~~
jansan
I highly doubt that number. Do you have a reliable source for that?

~~~
chimprich
It looks startling, but could be true. Reportedly, the health workers in Italy
had to do some fairly brutal prioritisation in triage. If you have a lot more
patients requiring ICU than you have stations, you end up with a high
proportion of younger patients in ICU even if they are a low proportion
overall.

~~~
listsfrin
The classic fake news model: post some shit with no evidence that is
immediately validated by some other worried internet person.

~~~
spookthesunset
Basically the current state of the internet in a nutshell right now. People
assume any valid question of any data means that person is a denialist nutter
or something. Fact is, we are working with very biased data and people, even
the smart folks here, are misinterpreting it. The virus exists, but there is
no data that tells us how widespread it is in the population. Odds are very
good this sucker has been adrift for weeks or months and many of us already
got it. But without good, random sampling we cannot prove or disprove that
hypothesis.

About the only place I’ve seen where rational talk is allowed without getting
flame to death is /r/covid19.

Hacker news, unfortunately, appears to have devolved into yet another place
full of panic stricken people.

------
ouid
I closed this after "based on data from China". That's almost as bad as basing
it on US data.

~~~
dang
Please don't post unsubstantive and/or flamebait comments to HN.

------
throwaway_pdp09
Interface is a not great. I wanted to compare US model to what's actually
happening (I've been graphing the US curves with public data).

Top dropbox is Scenario. Default is custom. I pick 'country - no mitigation'.
Box below labelled 'Population' (eh?) now says 'Germany'. I pick 'United
States' from it. The box below that also is labelled 'population' but has the
value with no digit separation to make it readable (shows 330000000, which I
initially misread under by an order of magnitude) but it does have an up/down
arrows beside it to increment/decrement it by 1 (double eh?).

The labels are not clearly associated with the dropdows, so it's easy to get
confused over whether they refer to the box above or below.

The help buttons are (blue show not hide text I think. The text they do have
is often unhelpful eg. "ICU/ICMO (est.)" ask for help on that and you get
"Number of ICU/ICMO available in health care system". Could be worse, could be
better.

The text box that pops up when you hover over the produced graphs are hard to
read - you've got light text on a white background. Also the larger numbers
are hard to read without digit separators. Also, don't get clever with the box
following the pointer, it's not slick, it jumps around and is distracting.

The graph axis is very logarithmic which is hard to get a feel for. Also the
top figure shows for me as 0000000 because the leading '1' is hidden.

I guess it's for professional of this area, which I'm not, so I'll step back
now. HTH anyway. Not intended as dismissive if it sounds that way.

