
The Metric We Need to Manage Covid-19 - sethbannon
http://systrom.com/blog/the-metric-we-need-to-manage-covid-19/
======
dan-robertson
If you look at test results you probably won’t learn much. There will be far
fewer tests than cases and most cases will never be tested. It seems more
likely that you’ll just be measuring some other effect of how testing is done,
for example if the number of tests per day ramps up slowly (and non
exponentially) then it seems likely that you will observe Rt to be decreasing
(as the number of confirmed cases isn’t shooting up), especially if people are
mostly only tested when they turn up in hospital and are expected to be
positive. I don’t know how much the author’s model accounts for that but I
find the confidence intervals very unconvincing. I would expect them to be
much larger.

This study tries to estimate Rt in several European countries from the death
rates though there are issues with those statistics too:
[https://imperialcollegelondon.github.io/covid19estimates/](https://imperialcollegelondon.github.io/covid19estimates/)

~~~
henrikschroder
I saw an interesting alternate statistics where you measure the overall
mortality rate and compare to previous years. EU only, but something like this
might exist elsewhere as well:

[http://euromomo.eu/](http://euromomo.eu/)

The data lags behind a bit, but since it measures _total_ number of deaths, it
doesn't matter if different countries measure coronoa-related deaths
differently. Everyone measures actual deaths. The numbers are hard to fudge
for politicians who wants to boost their results.

As a bonus, this also measures how your society is handling the pandemic
overall. We know that people will die of the lockdowns as a second-order or
third-order effect, and this way we can measure that as well.

~~~
alkonaut
How do you interpret a zero negative excess death rate? Sweden has over 1000
Covid attributed deaths, but almost no difference in weekly deaths compared to
the same weeks in 2019. Death numbers are extremely noisy and especially in
the flu season with the flu one year perhaps twice as deadly as the previous
year.

Covid restrictions have also reduced (a _lot_ ) some other seasonal diseases
like flu and noro/caliciviruses, so you get an effect in the other direction
too. When those that die from Covid are in the highest age groups it's not
unthinkable that a part of them would have been killed by the flu too, only
the flu didn't come this year because of Covid. Perhaps that's why we have a
lot Covid deaths but no excess deaths? Who knows. It's too noisy and has too
many dependencies to be very useful I think.

~~~
malkuth23
Less people driving?

~~~
throw_away
Also perhaps fewer elective medical operations, which result in fewer
iatrogenic deaths (3rd leading cause of death, at least in the US).

~~~
alkonaut
This is a good candidate. Although some of this effect might be countered by
people dying because they didn't get elective treatment.

Regardless: when you change pretty much every parameter of society (risk
behavior, driving, crime, health care, ...) I don't understand how one can use
the overall death statistics and try to attribute them to Covid.

~~~
malkuth23
I think the elective treatment theory is a better point than my driving guess.
If you have a disease and your options are to get surgery or take medicine,
you are probably being pushed towards non-surgical options, even if that would
have overall worse results on average. This might ultimately raise the
mortality rate, but would temporarily depress it.

Anecdotally, I definitely find myself trying to be safer right now. I have
avoided ladder work on my house and been super careful when using a knife. I
just do not want to go to a hospital right now.

I think looking at the overall death statistics are super interesting though.
If traffic deaths fall, that is in some ways a byproduct of Covid. If more
cancer patients die, that is also a byproduct. All of this goes into factoring
how much damage and protection both the disease and the quarantine orders have
caused.

------
zuhayeer
Jupyter notebook for this available here:
[https://github.com/k-sys/covid-19/blob/master/Realtime%20R0....](https://github.com/k-sys/covid-19/blob/master/Realtime%20R0.ipynb)

------
MiguelVieira
This site does this [https://epiforecasts.io/covid/posts/national/united-
states/](https://epiforecasts.io/covid/posts/national/united-states/)

~~~
alkonaut
This site also uses "confirmed cases" instead of deaths. That would be useful
many weeks ago, but now in most countries it's no longer a useful number. It
should use deaths _and_ a country specific reporting factor for e.g. whether
deaths include only hospital deaths or also other deaths, if only deaths with
positive tests or includes assumed Covid deaths.

------
zaroth
How can a confidence interval possibly be calculated without accounting for
the number of tests—not just the number of positive tests—and the total
population size?

What TFA shows us is the Rt _assuming that all cases are known_.

The model which knows nothing about testing rate, cannot possibly be used to
predict R0 or Rt.

As a simple proof, consider what the model would output I f testing simply
stopped. 0 out of 0 tests positive. After several days, the model outputs an
Rt=0 with 100% confidence.

If you could account for testing rate, and then attempt to account for missed
cases through proxies using hospitalization rates, ILI surveillance data,
excess fatality, as well as incubation period, and an asymptomatic rate...
maybe then you could draw error bars around an estimate of Rt that would pass
the smell test?

If testing was run against 1,000 randomly selected people per geographical
area each day, with a positivity rate reported over time, that would provide
the perfect basis for making these calculations. The time series of test
results that we actually have today require enormous amounts of unpacking
before they can be used safely to extrapolate much of anything.

~~~
richk449
Rt is based on the rate of change, not the absolute value, right? So if
testing coverage is constant in time, then doesn’t it drop out?

Of course, it isn’t constant with time, but it is also probably not changing
that drastically over short time frames (like the seven days the author uses),
so it shouldn’t throw the results off too much.

~~~
zaroth
Even if the number of tests stay the same, but just the testing criteria shift
over time, it would invalidate the daily positive test count for use in
determining Rt without somehow adjusting for the resulting bias.

------
corpMaverick
For lack of better charts. I have been calculating by hand the number of days
since total cases(or deaths) were half as they are today. Example there are
586k cases today. How many days ago we had 290k ? Almost 10 days. So it is
still doubling, but not as fast. A week ago it was doubling every 7 days.

~~~
Taek
I've been following the the death rate, because I think deaths are probably a
more accurate statistic than cases - less likely that a covid19 death goes
unreported.

This of course is a significantly lagging indicator, but it has been long
enough that we can see social distancing is working.

~~~
_bxg1
> less likely that a covid19 death goes unreported

I don't think this is true at all. Many people die at home. Many die without
having ever been officially tested. I've read that in some places the death
toll is only counted from those who were tested and then died in a hospital.

~~~
MikeAmelung
It's definitely still "less likely" that a death goes unreported versus an
infection without a death. Both numbers are shit, but deaths likely less so.

~~~
aledalgrande
I think deaths might be unreliable too, because 1) we don't know the mortality
rate, we're extrapolating it from bad data and then using an extrapolated
number to calculate a trend and 2) mortality rate would change between
different groups, e.g. sick and old people vs healthy people, depending how
infections spread.

~~~
spuz
That's why we're talking about deaths, not mortality. Mortality requires you
to know the number of people who are infected which as stated above can be
inaccurate because many people are not getting tested. However, it's less
likely that a person who dies of pneumonia in hospital won't be tested. Also,
there is likely to be a smaller proportion of deaths happening outside of
hospitals than there are infections so you should consider the number of
deaths to be more accurate than the number of cases.

~~~
akiselev
_> That's why we're talking about deaths, not mortality._

I believe you're confusing fatality and mortality. The correct version of that
statement is: _" That's why we're talking about mortality, not fatality._

The case _fatality_ rate is the ratio of number of deaths to the number of
people who are confirmed to have the disease (which depends a lot on standard
of care, reporting, and bureaucratic integrity) whereas mortality rate is the
rate of deaths in general.

------
nabla9
Without systematic random testing these numbers are describing testing
activity, not effective reproductive rate.

Infection rate is not the same as number of cases.

------
KaiserPro
Again, much as these are pretty, without decent test data, they are a
sideshow. The emphasis really should now be on proper sampling of a cross
section of the population.

That should really be weekly testing of 1000 or more, over a range of
backgrounds and wealth. Only then can we really see the impact.

Using mortality is only really useful when we know what the r0 is, or the
mortality rate. We don't know either.

~~~
lukeschlather
Looks like we're testing 10000 a day on weekdays, and over 5000 on weekends.

I find it difficult to believe with all that testing going on that no one is
reserving 3% of the weekly tests to do proper estimation as you suggest.

[https://www.cdc.gov/coronavirus/2019-ncov/cases-
updates/test...](https://www.cdc.gov/coronavirus/2019-ncov/cases-
updates/testing-in-us.html)

~~~
ceejayoz
Locally (upstate NY) per our pediatrician they're only testing people with
symptoms _bad enough to put them at risk of hospitalization_. We're still very
short.

~~~
lukeschlather
Isn't that what you would expect any clinician to say, even if they were doing
sampling? They're not going to use people asking for tests, they're going to
select a group of people at random from the general population and invite them
to be tested. Everyone else who has no symptoms they're going to politely say
they're not testing people with no symptoms.

~~~
ceejayoz
The point is that tests are still in extremely short supply, at least around
here.

------
squidproquo
Interesting article. In late March I built a simple SEIRD model for a few US
states with python. It was very difficult to estimate R0. R0 is essentially
Beta/Gamma, where Beta^-1 is is time between contacts and Gamma^-1 is the time
until recovery. I think what should be added to this analysis is population
density. Each state has a different rate that people come into contact with
one another. It would make sense that R0 in NYC is much higher than in Iowa.

Also this website is crowdsourcing US COVID-19 mortality forecasts if anyone
is into data modeling:

[https://www.unitarity.com/app/challenges/us-coronavirus-
outb...](https://www.unitarity.com/app/challenges/us-coronavirus-
outbreak/events/apr-20)

This website

------
LatteLazy
It's really too late to manage this. Testing hasn't been made available in
time mostly due to total failure to plan and political decisions not to manage
any of this.

All we can do now is turn up and down the level of isolation depending on how
many ventilators and ICU beds are available. We will have to do that on a sort
of feedback, trial and error basis because we lack capacity and what capacity
we do have is being wasted.

The only question is whether anyone will be held accountable for this colossal
mess and the preventable deaths its caused, or if we will just accept this
happening ever 5 or so years...

~~~
arrrg
Depends.

If active cases can be pushed down to a mange-able level (i.e. a level where
contact tracing is a realistic option and outbreaks can be, by and large,
contained) a return to relative normalcy could still be possible.

So that’s another one, two months of tight lockdown to push down active cases
(to, say, 500 in the NY area). Then it becomes feasible to throw massive
amounts of people at those 500 cases and to do aggressive contact tracing and
quarantining of all contacts.

Maybe just by people interviewing the infected, calling around, playing
detective (a bit slow), maybe also with the help of contact tracing through
mobile phones. Also, you obviously would need to be able to test for the virus
(ideally) whenever someone shows the slightest hint of related symptoms (and
since those are so unspecific and not exactly rare you would have to expand
testing capability).

Some measures will obviously stay in place, but many of the most drastic and
impactful ones could be reduced since there wouldn’t really be uncontrollable
community spread.

That, to me, sounds like a realistic plan (unlike a complete lockdown until
then or trying to let the wave wash over us) for the time until we have a
vaccine and until that vaccine is availible in sufficient quantities.

Of course, this depends a lot on whether we can push down active cases so low.

South Korea seems to be nearly or already there. Of course we do not know
whether China tells the complete truth, but even if they lie quite a bit they
are also probably nearly or already there. Outbreaks with exponential growth
haven’t happened there at least, of that we can be relatively certain (because
that would be hard to hide).

So, yeah, it’s probably possible if we can push effective R a bit below zero
for a couple of weeks to get into a position where we can regain control.
Maybe. Hopefully.

~~~
LatteLazy
I don't disagree with the plan. It's our only option. But how will we
implement it? We have no idea how many active cases their are in the NY area
(I assume you mean NYC). And we don't know how many people require
hospitalization. So you cannot estimate back from the hospitalized numbers.
Plus, that would only tell you how many people we infected a week ago. Except
that even that isn't certain because we don't know how long between infection
and hospitalization. And all of these numbers depend on what pre-existing
conditions (including age) a patient has.

So are we over the worst of it or is it just starting? Should we increase or
decrease our lock down right now?

And before you answer, getting it wrong won't be noticed for at least a week
(or more if the 1 week incubation figure is wrong, which it is), and in that
time any mistake will kill tens of thousands or burn $100bn dollars.

This is what we get for testing senators and NBA stars when we should have
been studying populations. China blind sided us with this once by lying about
it and covering it up. But we have blind sided ourselves as well by lacking
any leadership and ignoring experts. This whole thing as been a huge lesson in
how badly western governments are currently running. The slightest hint of a
problem and it's every man for himself.

------
usaar333
Good post discussing Rt (rarely brought up), but a few points are lacking:

* in the face of increasing (and volatile) testing capacity, Rt is being significantly over-estimated by looking at case counts. It's improbable CA still had an Rt above 1 the first week of April with covid deaths linear by the second week. More likely Rt was 1 about a week after the SIP (late march).

* Ignoring "herd immunity" effects of Rt. Some of the "under control" states (NY, Louisiana) have had very high infection rates which in their own end has dropped Rt down. NYC is a strong example of this - with > 20% of the city having been infected, well, Rt will drop dramatically just by so many contacts already having immunity.

* Implying this argues for states locking down now. I agree there is heavy evidence favoring lockdowns weeks ago to have avoided many deaths, but at this point, any lockdown is going to take a week to have an impact on even confirmed cases. The most susceptible populations (essential workers) are the very ones exempted from lockdowns (and less susceptible ones have already voluntarily socially distanced), so there's likely not much gain to be had. Sweden is conveniently our low density control group using mostly voluntary measures (WA State is similar through March 23) and its peak passing a week ago is a sign that Rt drops under 1 faster than you think.

------
birdyrooster
I also would like to see this broken down by city because the state highly
dilutes the signal when trying to rate the responses of major metropolitan
governments.

------
standardUser
Are there similar calculations for European countries? Several countries are
easing lockdowns based on the idea that the transmission rate has become
manageable. I'd love to see how those numbers stack up to US states.

EDIT: Turns out there is...
[https://epiforecasts.io/covid/posts/global/](https://epiforecasts.io/covid/posts/global/)

------
drtillberg
I would be really impressed if it were possible to control for the differences
that would occur if one state did a very good job tracing potential cases and
_then_ testing them (Massachusetts), versus more ordinary blanket efforts
elsewhere.

~~~
corpMaverick
I think we will see the overall work when we look at deaths. It is lagging
though.

~~~
CydeWeys
The author needs to re-run the analysis using only deaths, or deaths plus
hospitalizations (if those figures are available). I fear that all the model
reveals as currently written is how well states are getting a handle on their
testing, and paradoxically showing states with better testing as worse off
because of the growing number of cases.

------
jrd79
Your second rank ordering chart, the one by the upper bound of Rt, is
deceptive and misleading. North Dakota and Arkansas only have a high upper
bound because there are so few cases that the uncertainty is high. That chart
and any analysis associated with it are completely without merit.

------
glofish
Now ask yourself - do you believe that Louisiana has a more stringent, more
accepted and more enforced lockdown than say Massachusetts or Pennsylvania?
Yet the epidemic seems to slow more drastically in Lousiana than most other
states.

Why is that? Investigating that seems to be more fruitful avenue.

Perhaps there is more validity to the initial theory that the virus is a lot
less virulent in high humidity and heat. Many more observations seem to back
that. Low number of cases in India or Brazil etc.

~~~
knzhou
You're cherry-picking. Louisiana is on the low side, but Alabama, Texas, and
Georgia are on the high side.

In noisy and unreliable data it's _always_ possible to pick a single data
point that supports any desired conclusion. I'd say that all we can conclude
is that everybody should stay vigilant.

~~~
doodlebugging
>In noisy and unreliable data it's always possible to pick a single data point
that supports any desired conclusion.

Reminds me of the Principle of Propaganda - You can prove anything if you
ignore enough facts or data.

------
caseyf7
So cool to see Kevin doing a deep dive on this crisis with the goal of making
an impact. It says a lot about his character that he chooses to spend his time
doing this.

~~~
gpu_explorer
But also I think that it's tiresome for people without background in study of
disease (or long experience like Bill Gates) to get involved. Not only this
but it can be dangerous with any area like diseases.

This is the same as problem that you see in the White House of America with
this son-in-law of president. Systrom has no experience with infectious
diseases or study of diseases (or I did not hear about it yet and he does not
say about it), and human society needs very much people who are experts right
now.

Maybe Systrom could also use tech to help as this is his area of expertise.

------
johnchristopher
Totally off-topic: interesting, it's the first usage of the default 2020
wordpress theme I see in the wild.

------
cityzen
Does having a popular mobile app make people scientists? This elitist shit is
annoying. I would much rather see him write up something about how he is
distributing his wealth to help with the coronavirus rather than this stuff.

~~~
dna_polymerase
He sold Instagram years ago, so it's no longer his. Also, instead of blindly
launching an ad hominem attack against him, judge the article on its contents.
Of course, having success in one area doesn't mean he is an expert in another,
but it doesn't exclude the possibility as well.

------
_nalply
Edit: Sorry, I made a mistake.

Trying to measure Rt seems to be a good idea.

However it's not clear how the proposal calculates Rt. No formulas and no
references to raw data. So I am afraid this is not very useful.

This said, I could imagine that an approach which takes the number of
confirmed cases and the total number of tests could work to estimate Rt. The
idea is that while the number of cases is not a good estimator we can try to
get a better estimator using the number of tests as well. It's a bit tricky,
however, because a low number of tests could mean that only the probable cases
are tested which skews the numbers.

~~~
Aeolun
Did you miss the link to the notebook containing all the code and
calculations?

~~~
_nalply
Yes, I goofed up. Sorry.

