
Concerns with that Stanford antibody study of coronavirus prevalence - benchtobedside
https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanford-study-of-coronavirus-prevalence/
======
strangeloops85
Another odd thing: One of the co-authors of the study, Andrew Bogan seems to
manage a hedge fund, and then published a WSJ op-ed on the study. He didn't
note in the op-ed that he was involved in the study (and a co-author no
less!). A lot of conflict of interest potential here. These authors seem to
have an agenda and end-result in mind, no matter the data.

Edit: here's the WSJ op-ed (paywalled): [https://www.wsj.com/articles/new-
data-suggest-the-coronaviru...](https://www.wsj.com/articles/new-data-suggest-
the-coronavirus-isnt-as-deadly-as-we-thought-11587155298)

The more I think about this, the more outrageous I find this. It's a form of
astroturfing to advance the beliefs of the authors of the study. (Note that
the senior author Bhattacharya and of course Ionnadis were advancing this
theory before data collection began. This means their analysis deserves even
more scrutiny)

~~~
pacala
They made a hypothesis and conducted a study to validate it. Perhaps they are
wrong. Would be far from the first time studies are proven wrong. At least
they did something. Make a better study. Make a hundred better studies. Prove
them wrong!

~~~
fao_
I think the user is referring to the conflict of interest and the lack of
interest in proper ethical conduct by one of the co-authors. If they won't
follow generally accepted ethical guidelines, what other rules will they
ignore or bend?

~~~
pacala
The conflict of interest part is fair.

The "Note that the senior author Bhattacharya and of course Ionnadis were
advancing this theory before data collection began" is not. That's how science
is done: you advance a theory, then you conduct experiments to validate /
invalidate the theory.

~~~
mediaman
I think there's a difference between postulating a hypothesis, and then
testing it, which is what I believe you're referring to, versus going around
advocating a particular position as being true and then "testing" it.

When people are strong advocates of a particular position, they tend to find
ways to design or process experimental data in ways that support that
position. The ways in which bias can seep into the experiment are not always
easy for outsiders to see.

So it comes down to whether their "advancing this theory" went beyond what a
reasonable scientist would do in postulating a hypothesis that needs testing.

------
gryson
In regards to the issue of selection bias, this is what the ad looked like:

[https://twitter.com/foxjust/status/1251270848075440133](https://twitter.com/foxjust/status/1251270848075440133)

Quote from ad: "We are looking for participants to get tested for antibodies
to COVID-19."

A quote from someone who participated in the study:

"I participated in the study because I had been sick the week before and was
very curious. In the intake questionnaire they asked if I had recent symptoms.
I'm unpleasantly surprised that they seem not to have made an effort to use
that data to unbias the study."

[https://twitter.com/mattmcnaughton/status/125132223548416819...](https://twitter.com/mattmcnaughton/status/1251322235484168192)

Another:

"I was part of this study and that is totally why I signed up! People I talked
to who tried to sign up had similar reasons. Lots of subjects at the testing
site wearing masks, more than you see at the grocery, more evidence that a lot
of us were more conscious about transmission"

[https://twitter.com/McSalter/status/1251511091294691328](https://twitter.com/McSalter/status/1251511091294691328)

It's very hard to deny that selection bias may have played a part here.

~~~
Fjolsvith
So, I'm not getting why selection bias would matter here. If they are counting
the total number of participants who were positive for the antibodies, and
this number was out of the total number of residents in the county, big deal.

Why would they ask all the healthy people to come get tested? Just assume that
all the untested people are negative for the antibodies.

Otherwise you would get a _higher_ infection rate count, which would of course
result in a lower mortality rate for the disease.

~~~
PierceJoy
If the sample was biased, then the results cannot be generalized to the whole
population. The purpose of this study was not to find positive cases to treat,
it was to estimate the prevalence in the general population.

------
tptacek
The original study was on the HN front page twice; biggest thread:

[https://news.ycombinator.com/item?id=22899272](https://news.ycombinator.com/item?id=22899272)

This analysis is pretty damning, and is more credible in context (the topline
findings of the original study constituted extraordinary claims, which, if
extrapolated, could imply that a majority of all New Yorkers had C19
antibodies).

Some of the underlying ideas here are pretty straightforward. For instance,
the fact that even with 90+% specificity, if your rate of false positives
exceeds the true positives in the population (as can happen even with good
tests when the underlying condition is rare, as it is with C19), you're going
to have problems.

~~~
SpicyLemonZest
Other antibody studies have consistently found similar results to the Stanford
one. The Stanford numbers are very high - most have found closer to 20-30x
reported cases than 50-85x - but it's virtually guaranteed that a huge
fraction New Yorkers have C19 antibodies today. I'd be pretty surprised if
it's less than 20%.

~~~
sjg007
We don't know if these antibody tests are specific for covid-19 only or in
fact for any coronavirus. We don't know if they randomly test positive either
due to something else. There are a lot of unknowns. These antibody tests are
not FDA approved.

~~~
SpicyLemonZest
We lost a month at the beginning of this pandemic waiting for FDA approval of
the tests confirming coronavirus was spreading in the US. I am unconvinced
that this approval offered any value at all, and we can't afford to lose
another month waiting to see how far it's spread.

~~~
ineedasername
Your statement amounts to "we have to do _something_. This is _something_.
Therefore we have to do it."

Doing the wrong thing can be extremely dangerous in this situation.
Consciously choosing to wait until you have greater certainty that the action
taken is correct isn't the same as inaction, not when the downside risks of
getting it wrong are so high.

~~~
SpicyLemonZest
I disagree that theorizing and gathering data can be extremely dangerous in
this situation. If we're worried the Stanford study is flawed, the right
response is to urgently do more studies, not sit on our hands waiting for some
ponderous FDA approval process.

~~~
ineedasername
There's a certain limited amount of qualified research capacity representing a
bottleneck. So, urgently doing more studies to refute one that has flaws and
bias by the author isn't the right type of action to take if it will take
resources from other, more productive endeavors.

Theorizing and gathering data is fine; going on to write a WSJ op-ed pushing
it in a way you hope will influence policy goes a far step beyond proper
scientific research protocols.

~~~
SpicyLemonZest
Standard scientific research protocols are heavily biased towards inaction. We
can't afford that bias in the middle of a global emergency; it matters a _lot_
if necessary action gets delayed for a month or even a week. Imagine how bad
things would have gotten if we had made the same demands a month ago - if we
had refused to institute social distancing until randomized controlled trials
proved it helped for this disease.

~~~
ineedasername
There was no need to delay social distancing on account scientific uncertainty
over the practice. We already knew social distancing was an effective method
of preventing spread. Heck, we've known it since the 1918 spanish flu, since
quarantines themselves begam. (and still, it _was_ delayed, still is delayed
in some places)

The point is that there is a resource bottleneck on "action". There are
limited resources. You qualified your own statement as _necessary_ action. How
do you know what is necessary? Options must be weighed, the most promising
chosen, as many, but not all paths can be pursued. I'm not saying "do
nothing", I'm saying that we can't do _everything_. And then, yes, in areas of
great uncertainty, where _wrong_ action can cause more harm than _choosing_ to
wait, then we should not take action merely for the sake of "we can't do
nothing!!!" emotional response to the crisis.

------
snapetom
As flawed as this is, it's in line with other studies around the world. You
can nitpick and critique each one for something, but we now have a whole body
of evidence using different techniques and different methods that are all
stating the number of cases is vastly undercounted, and the IFR is under 1%.

Scotland:
[https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v...](https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v1.full.pdf)

NYC Pregnant Women:
[https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19...](https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19&cid=DM90482_NEJM_COVID-19_Newsletter&bid=186123144)

Finland: [https://thl.fi/en/web/thlfi-en/-/number-of-people-with-
coron...](https://thl.fi/en/web/thlfi-en/-/number-of-people-with-coronavirus-
infections-may-be-dozens-of-times-higher-than-the-number-of-confirmed-cases)

Germany:
[https://www.land.nrw/sites/default/files/asset/document/zwis...](https://www.land.nrw/sites/default/files/asset/document/zwischenergebnis_covid19_case_study_gangelt_0.pdf)

Chelsea, Mass.: [https://www.bostonglobe.com/2020/04/17/business/nearly-
third...](https://www.bostonglobe.com/2020/04/17/business/nearly-
third-200-blood-samples-taken-chelsea-show-exposure-coronavirus/)

~~~
wahern
No professional was ever under the delusion that the IFR was >= 1%. Perhaps
you meant CFR[1], which by definition does _not_ reflect total number of
infections. See, e.g.,
[https://en.wikipedia.org/wiki/Case_fatality_rate](https://en.wikipedia.org/wiki/Case_fatality_rate)

[1] Where 'C' means "case" or "confirmed", which in the context of medical
terminology and despite some ambiguity categorically is not synonymous with
infected.

~~~
argonaut
Anthony Fauci has stated multiple times in the past that he thought the
"fatality rate" of Covid-19 would end up "around 1%." He didn't make a
distinction between CFR and IFR, but the general public's understanding of
"fatality rate" is obviously that of IFR.

~~~
snapetom
> but the general public's understanding of "fatality rate" is obviously that
> of IFR

Looking at my nextdoor group. I'm going to disagree with you on this. There's
a lot of people screaming, "4.7% of people will die!" Even for a CFR, that's
high.

~~~
argonaut
What you're describing is consistent with the nextdoor group thinking there is
a 4.7% IFR.

------
tbenst
I think the criticism in this thread and elsewhere is a bit too harsh. It’s by
no means a perfect study, nor the last word, but hopefully will motivate
further studies.

I volunteered on this study and talked with hundreds of the participants, at
least 200 and possibly as many as 400. Two reported previous COVID symptoms,
unprompted.

The bigger problem was socioeconomic bias. Judging from number of Tesla’s,
Audi’s, and Lamborghis, we also skewed affluent. Against the study
instructions, several participants (driving the nicest cars I might add)
registered both adults and tested two children. In general, these zip codes
had a lower rate of infection. It’s very hard to understand which way this
study is biased, and a recruiting strategy based on grocery stores might be
more effective, but difficult to get zip code balance

There has been additional validation since this preprint was posted and now
there’s 118 known-negative samples that have been tested. Specificity remains
at 100% for these samples. An updated version will be up soon on medrxiv.

~~~
jariel
"The bigger problem was" ... conflict of interest.

~~~
tbenst
This strikes me as classic, Reddit-style conspiracy theory. I have no idea who
Andrew Bogan is, but he didn’t play a major role in the execution of the
study. I’m sure on the paper’s published the contribution section will specify
what he did. Remember: this is a preprint!

~~~
tempsy
I hate that the word “conspiracy” is now lobbed at others to immediately
invalidate what they have to say, even with evidence.

Did you skim through Bogan’s op-ed? He references the study without ever
mentioning once that he played a role in it (no matter how small it was).

You don’t need a tin foil hat to ask yourself what incentives a hedge fund
manager would have in downplaying the severity of the virus by participating
in the creation and media distribution of a study that does exactly that.

~~~
tbenst
We all gain more insight by discussing the study’s or analysis’s flaws, and
improving future studies.

I learned nothing of interest scientifically from the discussion of Bogan.
Attack the substance, not the author.

~~~
tempsy
The author provides an insight into the motive for the study. If these authors
decided to come together to do this study with the intention of downplaying
the lethality of the virus then it poses a very serious ethics violation.
Stanford should investigate.

~~~
mieses
Is it wrong to have a theory that the virus is less lethal than widely assumed
and to investigate this theory? Are scientists not allowed to investigate
theories that are contrary to dominant political or religious dogma? What if
they end up being right?

------
gnicholas
I have wondered about the selection bias issue, and I was hopeful that this
writeup would give a good look at this and other potential issues with the
study.

But when I read it, I was a bit turned off by the author's attitude — which
seems to be that he or some of his colleagues should have been consulted by
the study authors because they are "statistics experts".

He refers to this apparent omission multiple times, and he also seems to think
he's dunking on the authors when he references Theranos (and the fact that its
advisors came from government/law/military). But this study is completely
unrelated to Theranos (though they both involve blood and Stanford). Off-topic
comments like these left me wondering if his analysis is a fair critique, or
if he has an axe to grind.

~~~
paultopia
But he's right though---the authors should have consulted some survey data
experts. And the other people he names are, in fact, well-known survey data
experts who could have made the work better.

~~~
buboard
the authors list had plenty of epidemiology data experts

~~~
ineedasername
That is not the same as survey experts.

~~~
SpicyLemonZest
There's always going to be some subcategory of experts they didn't talk to. Do
medical studies typically include survey experts signing off on the surveys?

~~~
ineedasername
When a survey is a critical aspect of your study & data gathering, then yes:
consultation with someone familiar with best practices in survey design is
pretty common.

------
Mikeb85
Every study on coronavirus antibodies that's released gets panned here, yet
every single one of them show strong _enough_ evidence that infection is more
widespread and death rates lower than widely assumed.

Even if every one of them is flawed, all the information taken as an aggregate
paints a picture. In my province (Alberta, Canada) the health authority has
recently expanded tested and as a result there's more confirmed cases and a
lower death rate. Other health authorities in the country have strongly
suggested infection rates are much higher than confirmed cases (which lowers
the death rate since every single death is being accounted for in our
country).

So there's concerns with this study, and maybe another, but there's no
evidence to counter the conclusion we're seeing again and again.

~~~
blago
I don't think anyone doubts that infections are much more common than the
recorded rate - we all know that a significant number of people are
asymptomatic, hence never tested and recorded.

The problem is that those observations are typically used to conclude that
"this is just a bad flu" and advance political demands such as "liberate X"
and "immediately reopen the country".

The question that truly matters is "which model is less wrong" and the
overflowing ICUs in NYC and Europe (that's the bottom line) provide a clear
answer - we should err on the side of caution.

~~~
Mikeb85
> The problem is that those observations are typically used to conclude that
> "this is just a bad flu" and advance political demands such as "immediately
> reopen of the country".

And why not? Should studies and observations be dismissed just because the
result isn't what some people want?

>The question that truly matters is "which model is less wrong" and the
overflowing ICUs in NYC and Europe

No overflowing ICUs here. Emergency rooms are way under capacity.

Where I am, we have a population of 4 million. Only 40 ICU visits during the
whole pandemic. Only 59 deaths and the majority were individuals over the age
of 80. Over half were in nursing homes. And yet there's a vocal group who
don't want anything to reopen.

Where is the line where we reopen? 25,000 Albertans die every year. 275,000
Canadians die per year. 2.8 million Americans die every year. It's not
reasonable to wait until coronavirus is completely eliminated when economic
hardship itself is correlated to a higher mortality rate.

Edit - just looked at some closer stats for my region: only 3 deaths under the
age of 60. Population 4 million. With 3k official cases and likely 30k or more
total cases.

~~~
blago
Congratulations! It's working. As to when and how it is appropriate to reopen
without spiking the curve and killing millions - many of whom would die from
easily treatable and unrelated conditions because they wouldn't be getting the
attention they need - I think we should defer to the experts.

~~~
Mikeb85
This thing was never on pace to kill millions. Maybe if it was actually as
deadly as SARS and that was a valid concern, but it's proven to be nowhere
near as deadly as SARS.

~~~
blago
It killed hundreds of thousands and counting AFTER we did the unthinkable -
shut down the world.

~~~
Mikeb85
It hasn't reached 200k so no, not _hundreds_ of thousands.

Swine flu killed ~500k, it doesn't look like this'll get there. It might, but
it's not even the worst pandemic of the 2000's yet.

[https://www.nytimes.com/article/coronavirus-vs-
flu.html](https://www.nytimes.com/article/coronavirus-vs-flu.html)

~~~
Lewton
> Swine flu killed ~500k, it doesn't look like this'll get there.

... Are you willing to put money on this? It'll be the easiest money I'll ever
make

The US is already at 3x more deaths from Covid-19 than the most pessimistic
estimates of how many died there from swine flu

Those 500k deaths you're talking about? All happened in Africa and Asia. This
pandemic has barely begun to hit India and Bangladesh. Same with Africa

~~~
Mikeb85
India was one of the first countries where Covid-19 reached and Thailand was
the first. Both had limited spread despite being very densely populated. If it
was going to spread further it would have already.

Based on the available evidence it doesn't spread well in hot climates so no,
I don't think it's reaching 500k. Maybe there's a chance if the US keeps
fucking up.

~~~
Lewton
> If it was going to spread further it would have already.

I gotta say, your capability for wishful thinking is impressive.

Look at Singapore the past week. They had everything under control for months.
Or what. Singapore's climate is too cold?

Again, willing to put your money where you mouth is? Let's say 500k deaths by
the end of the year excluding the US

~~~
Mikeb85
Singapore has 11 deaths total from Covid... In a city state of 5 million.

Also, warm climate doesn't mean there's no air conditioned buildings or
absolutely no spread, but the stats do suggest it makes spread more difficult.

500k excluding the US seems pretty good for me. That means you expect 5x more
deaths than up to now, despite deaths peaking in all the hardest hit countries
that aren't the US...

~~~
Lewton
> 500k excluding the US seems pretty good for me. That means you expect 5x
> more deaths than up to now, despite deaths peaking in all the hardest hit
> countries that aren't the US...

Yes. You seem to be under the impression that the virus has saturated the
world and has stopped spreading for some, to me, completely unfathomable
reason

So. What'll you say.. $50? $100? $1000? Money goes to charity, or for personal
gain?

Me betting that
[https://www.worldometers.info/coronavirus/](https://www.worldometers.info/coronavirus/)
will report more than 500k deaths excluding the US by the end of the year

~~~
Mikeb85
$100 works for me for an internet bet. I think charity, can just provide proof
so don't have to bother with transferring money. And I'm ok with the terms. If
you have a favourite charity can go with that.

~~~
Lewton
Deal :)

I suggest Give Directly [https://www.givewell.org/charities/give-
directly](https://www.givewell.org/charities/give-directly)

If you prefer something else, I suggest anything on
[https://www.givewell.org](https://www.givewell.org) list of charities

------
sct202
Also of note, Premier Biotech, the 'manufacturer' of the antibody tests in the
Stanford study, has been accused of distributing non-FDA approved Chinese
antibody tests. [https://www.nbcnews.com/health/health-news/unapproved-
chines...](https://www.nbcnews.com/health/health-news/unapproved-chinese-
coronavirus-antibody-tests-being-used-least-2-states-n1185131)

~~~
acqq
More about how the antibody tests aren't doing what one would want to believe
they do:

[https://www.theguardian.com/world/2020/apr/09/uk-
government-...](https://www.theguardian.com/world/2020/apr/09/uk-government-
urged-to-abandon-poor-finger-prick-antibody-tests-coronavirus)

" _None_ of 3.5m home tests ordered have so far been accurate enough to detect
coronavirus immunity"

UK got 3.5 million(!) unusable antibody tests.

[https://www.bmj.com/content/369/bmj.m1449](https://www.bmj.com/content/369/bmj.m1449)

"John Newton, Public Health England’s director of health improvement, said:"

 _" A number of companies were offering us these quick antibody tests_, and we
were hoping that they’d be fit for purpose, but when they got to test, they
all worked but _were just not good enough to rely on_.

“The judgment was made [that] it’s worth taking the time to develop a better
antibody test before rolling it out, and that is what the current plan is.”"

"Newton told the committee that the tests trialled so far had lacked
sufficient sensitivity to identify people who had been infected. “We set a
clear target for tests to achieve, and _none of them frankly were close_.”"

------
nabla9
So far the best tests are from Iceland and Italian town Vo.

In Iceland over 11% of population is tested, 4% positive. Results indicate
that over 50% of infected are asymptomatic.

Italian town Vo tested everyone in their village with similar results. Over
50% of those who test positive are completely asymptomatic.

~~~
kgwgk
... at the time of the test. Or there was a follow up?

~~~
DennisP
And what were the false positive rates?

~~~
andrewla
If I could upvote this a million times I would. If 50% of those tested
positive are asymptomatic, then a good first-order approximation is that the
false positive rate is 2x the actual prevalence, and the results say very
little about prevalence that actual hospitalizations don't say.

People, and to a large degree the media, are putting a huge emphasis on
testing, not always in a way sensitive to the essential difficulty of
evaluating test accuracy, especially in epidemiological settings.

~~~
ghastmaster
Would it be disingenuous to say that false positives in the current case
fatality rate would bring the rate down because hospitals are using symptoms
combined with PCR for deaths?

~~~
ineedasername
From what I've heard, at least in some areas, a Covid test is often performed
only after a (usually very fast) flu test is performed. So a death resulting
from a person negative for flu but also false-positive for Covid would seem
very unlikely. As would any death of this sort from a person who came up
positive. Much more likely would be that false positives cluster among those
with mild or no symptoms (with the mild symptoms instead coming from a more
mundane infection)

If false positives _were_ pushing the published death rate up, it's important
to remember there are also deaths among people infected but _never_ tested
that would be pushing the rate down. I don't know if they would cancel each
other out though. I would however really like to see published data from the
CDC about reported pneumonia deaths just prior to the "official" outbreak to
see if there was an otherwise unexplained increase.

~~~
ghastmaster
Agreed. It will be very nice to have more data. There are too many ifs right
now.

------
dr_faustus
Well, all the antibody studies around the globe have several things in common:

* relatively small numbers of participants (< 5000 and therefore only dozens of partifipants with postive results)

* focusing on relatively small geographical areas

* working with antibody test with a high uncertainty regarding the specifity

One argument that strongly contradicts the narrative that a huge number of
people already are/were SARS-COV-2-positive is the of positive PCR tests in
Germany. Germany performs hundreds of thousands of PCR tests per week but
still mainly tests people with some symptoms. If SARS-COV-2 were that
prevalent, you would expect a large proportion of the tested to be positive
but its only 4% by late March [1]. Every expert I heard admits that there is a
significant number of undiagnosed cases. But 30x-60x seems to be quite
unrealistic if even only 4% of people with symptoms are positive.

[1] [https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-
de...](https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-deutschland-
coronavirus-covid-19-who-pandemie)

------
HandsomeMD
I get this study may be flawed, but there are several upcoming studies showing
similar findings in respect of likely lower CFR than expected. The issue is,
with what magnitude

------
i_am_proteus
Ioannidis warned us in advance that he'd publish bunk. "Why Most Published
Research Findings Are False" [0] is his most famous work.

[0]
[https://journals.plos.org/plosmedicine/article?id=10.1371/jo...](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)

------
seemslegit
Why is "suggesting that nearly 2% of the population has been infected “under
the radar” even remotely shocking ?

~~~
vkou
Because when you claim that 2-3% of the population has been infected in an
area where 1/30000 of the population died from the virus, it implies that the
virus has a 1/600 infection/death rate.

Since ~1/500 New Yorkers have died from this virus, that would imply that 120%
of the population of NYC have been infected, and that new NYC cases will drop
to zero in two weeks.

Which is, obviously, nonsense. One of the numbers here doesn't fit the facts,
and it's probably the one that claims that 2-3% of the population of Santa
Clara was infected, with the overwhelming majority not having any symptoms.

Data from the Diamond Princess points to something between 20% and 50% of
infected people being asymptomatic - not 95%.

~~~
irq11
Try 1 in 600. There are also huge error bars on that ratio, because the
denominator (population size of New York), is a rough estimate, at best.

In any case, this is nit-picking: the obvious source of error here is trying
to cross-apply the IFR from a limited sample in one city to another,
completely different city. If you’re only off by a factor of 20%, that’s a
pretty strong indication that you’re on to something real.

New York is testing a lot more than California, so I expect their ratio of
undiscovered cases to be lower. But it’s becoming clearer and clearer that the
true IFR for this virus is substantially lower than previously estimated.

~~~
mcbits
Nobody is estimating that 100% of New York has already been (or even will be)
infected, so something is off by much more than 20%.

With an optimistic, but at least plausible, estimate that 40% of NYC have been
infected, then the IFR is around 0.4% which actually is within 20% of the 0.5%
conservative guess that people have been making.

~~~
irq11
Who are “people”? Who is defining “optimistic”, and “plausible”? Where are you
getting these numbers?

The LA study implies a ratio of 30-50x the number of confirmed cases.

Even at the high end of that range, given the current confirmed infection
count in nyc (141.2k), then about 7M people would have been infected. That’s
not 100% of the population, and it’s entirely plausible.

~~~
mcbits
e.g.

> Comparing deaths onboard with expected deaths based on naive CFR estimates
> using China data, we estimate IFR and CFR in China to be 0.5% (95% CI:
> 0.2-1.2%) and 1.1% (95% CI: 0.3-2.4%) respectively.

[https://www.medrxiv.org/content/10.1101/2020.03.05.20031773v...](https://www.medrxiv.org/content/10.1101/2020.03.05.20031773v2)

7 million is 80% of the population, which is at the high end of high estimates
for the total portion of the population expected to be infected in the end.
That would be more plausible if daily deaths were well into the long tail, but
NYC appears to be just past the hump and hundreds/day are still dying.

I would say 30x undercounting (48% infected) is highly optimistic but still
plausible if you want to embrace that.

~~~
irq11
Ok, so you’re taking one paper and assuming it is correct, and using that to
dismiss more recent data.

Keep in mind that not even two weeks ago, the best estimate of IFR in China
was 0.66%. It keeps dropping. Also, the confidence interval on the paper
you’re citing extends to 0.2%.

I’m not saying that the factor is 50x, just that it’s plausible.

~~~
mcbits
I don't assume that paper is correct. It's the first example I found of people
estimating 0.5% since you apparently hadn't noticed that number being bandied
around for the last month. It's a number I've been seeing since early March,
along with 40-70% of the population ultimately being infected. I saw one guy
suggesting up to 80%. If NYC is already at 80% and rising, that's pretty
surprising.

I would like to get an idea of what the IFR really is, not which extreme end
of the range of uncertainty looks the best or worst. If it's a little less
than 0.5%, great. But when some paper comes out of left field and suggests
it's closer to 0.05%, sure, I'm skeptical, especially when mortality is
already above that fraction of the entire population in several places.

------
mxcrossb
> But we didn’t then write up a damn preprint and set the publicity machine
> into action.

The latter part is crucial here. I don’t think you should have to apologize
about mistakes in a preprint. Papers get improved by the review process. And
most of us just upload them to get around journal paywalls, or to make it
easier to share with our colleagues what we are working on.

But when a University hears that you’ve got a result on a hot topic, dollar
signs light up in their eyes, and they go to work. Scientist beware.

I hope we can find a balance where scientists don’t rush something out just
because it’s a hot topic, yet are also not paralyzed from working on something
because of the dangers of the spotlight.

~~~
sct202
The lead professor (Eran Bendavid) in charge of this study wrote an op-ed a
month ago in the WSJ (with another contributor Jay Bhattacharya to this paper)
about how we shouldn't shut down over coronavirus:
[https://www.wsj.com/articles/is-the-coronavirus-as-deadly-
as...](https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as-they-
say-11585088464)

I would be very surprised if they weren't aware of what they were doing by
releasing their pre-print.

~~~
mxcrossb
Thanks I was unaware of this. It looks like I was wrong.

------
eanzenberg
Its wishful thinking to overlook the mountain of evidence building for an IFR
~ 0.1-0.5% of this disease.

~~~
guscost
"Wishful"? You must mean those of us who are still holding out for less than
0.1%...

------
_cs2017_
All these confidence interval discussions (both from the original study and
from the critique) have no value besides entertaining professors who have more
knowledge of math than common sense. Nothing good will ever come from
buzzwords such as "Agresti-Coull 95% interval".

Bayesian techniques are not much better since no one will ever agree on the
prior.

Just treat the study as some super rough point estimate. Adjust for biases
such as selection bias if you can. Look at other studies too. Add your
personal opinions (e.g., on whether conflicts of interest are relevant here).
Complex statistical arguments won't buy you much more than that.

------
deburo
The comments below the article are also incredibly interesting.

I'll now wait on feedback from the authors to the concerns expressed here. But
also, the focus will be on many more serology studies in the coming months.
Looking forward to their results.

------
Kiro
In Sweden they did antibody tests on blood givers and found that 11% had
antibodies. There was no ad or survey so there couldn't have been any
selection bias.

~~~
craydandy
Do you have a link for the study or news article?

~~~
Kiro
In Swedish: [https://www.svt.se/nyheter/inrikes/11-procent-av-
stockholmar...](https://www.svt.se/nyheter/inrikes/11-procent-av-
stockholmarna-har-antikroppar-mot-covid-19)

~~~
craydandy
Thanks!

------
fapjacks
Frankly I find Andrew Gelman _at least_ as invested in a particular outcome of
this political issue as a hedge fund manager.

------
pacala
The OP is, for lack of better words, so academic. He wants an apology? OK, the
study has flaws X, Y and Z. How about propose and conduct a better study?
ASAP? Throw darts on a map if you have to.

There are millions of people kicked out of a job. People defer medical
procedures indefinitely. Kids skipping school for months on end. We will,
sooner or later, run out of basic necessities as well. The world doesn't run
on money or theories. It runs on us, real people, shuffling our hands and
turning sun and soil into food and heat and clothing. Right now we are
grounded at home. This can't go on forever. We are running against the clock.
Do something about it!

~~~
DangerousPie
The OP wants people to stop spreading misinformation.

~~~
pacala
We are already drowning in misinformation. We have no clue how deadly this
disease is, and yet we are using what appears to be [gross?] overestimations
to drive public policy to the tune of trillions of dollars printed with a
flick of a pen.

Produce and spread good information instead.

~~~
mcphage
> Produce and spread good information instead.

Also: identify bad information and seek to stop it from spreading! That’s what
this article is all about.

~~~
pacala
January. February. March. April. It's been almost 4 months since we know about
Covid-19. From the rock I'm living under, it appears that the scientific
community at large has produced remarkably little reliable data. Where are the
epidemiological studies?

Producing reliable data is the highest priority.

~~~
URSpider94
Epidemiological studies are known as “models”. Those are being produced at a
rapid clip, and are being updated daily as we get more data. As with most
models, they start out with huge error bars and get better over time.

Every epi would LOVE to have widespread test data, but we don’t. Oh well.

These antibody tests only became feasible in the past two weeks, when the
tests were actually developed and validated, and then this test was run.

There’s a ton of science being done, if you stop and listen to what is being
published.

~~~
pacala
Conducting experiments is expensive in both money and effort. Let's all sit in
our comfy chairs and build fancier models with garbage data instead. If we
could divine just the right formula, it will magically paper over the holes in
the data. Sounds like every other scientific area I'm aware of. For whatever
reason, the experimentalist is a dying breed.

Edit: Do you happen to have a link to a good aggregator for listening to
"what's being published"?

~~~
URSpider94
You do know that epidemiologists are statisticians, right? They, um, don’t do
experiments and never have and never will. They crunch numbers and fit them to
models. Which is what they are doing.

If you want to see more data, well yes, so does everyone. What magic tool
would you like people to use to get that data? As I explained above, we seem
to have a shortage of swab tests to determine infection, and antibody tests
only became available about a week ago, at which point people started to use
them.

Not quite sure what more you are asking to be done here?

~~~
pacala
That attitude reminds me of certain type of software engineers: We are
engineers, our time is too valuable to write tests, hire someone else to do
so. When I hear that, I run away. Fast. The world needs results, and there are
little. How about we stop the excuses and roll our sleeves instead?

