
Problems with Japan's Covid-19 reports - hardmaru
https://stdio.sangwhan.com/wtf-japan-covid-19-report/
======
reustle
The team behind [https://covid19japan.com](https://covid19japan.com) fights
with this nonsense every day. We're open ears to improvements, and are playing
with ways to possibly automate bits.

Our GitHub:
[https://github.com/reustle/covid19japan/](https://github.com/reustle/covid19japan/)

~~~
xrd
This is fantastic, terrific way to visualize this information. It's
interesting that Hokkaido is red for infections but the growth rate is really
slow. Are the distant prefectures less likely to follow guidelines from Tokyo
and reporting more honestly?

~~~
panny
>It's interesting that Hokkaido is red for infections but the growth rate is
really slow.

Hokkaido is about the size of South Carolina and approximate population of 6
million people. ~2.5M of which live in Sapporo. Most of the island is more
rural with many tourist destinations. It's a popular destination for Chinese
tourists in winter.

>Are the distant prefectures less likely to follow guidelines from Tokyo

Tokyo was seen as mostly following Hokkaido governor's lead early on. Hokkaido
was the first to close schools, etc.

>and reporting more honestly?

Could HN dwellers stop implying Japan is handling this dishonestly? That is
extremely rude coming from the nation whose CDC was angry to accept 14
infected citizens from the only ally nation that would accept their US Diamond
Princess cruise ship and provide care for passengers.

~~~
reustle
> Could HN dwellers stop implying Japan is handling this dishonestly?

They are. The incredible rate of test refusal here is keeping the numbers very
low.

~~~
panny
Really? What is the rate of test refusal Shane Reustle?

~~~
dang
Please do not cross into personal attack.

~~~
panny
Asking for the "incredible" rate of test refusal is not a personal attack.
Calling Japanese stupid is not okay, but you don't warn the person who does
it. You warn the person who takes the bait. You then retaliate by warning me
only after I call you out on it.

Shouting fire in a crowded theater is not free speech. What you are doing here
is not protected.

~~~
dang
Nobody called the Japanese stupid. You chose to interpret it that way.
Actually, I understand how it could feel that way, and I empathize. But take a
look again:
[https://news.ycombinator.com/item?id=22730110](https://news.ycombinator.com/item?id=22730110).
It didn't say that. What it did do was repeat an internet cliché [1] in a lazy
way. That made it an unsubstantive comment, and an unsubstantive comment on an
inflammatory topic is basically always flamebait (as I pointed out to another
user earlier today:
[https://news.ycombinator.com/item?id=22739350](https://news.ycombinator.com/item?id=22739350)).

You've unfortunately been posting to HN quite a bit in the flamewar style.
Would you please stop doing that? I realize that these topics are sensitive
and hit sore spots all too easily. That's one reason the site guidelines
include: " _Please respond to the strongest plausible interpretation of what
someone says, not a weaker one that 's easier to criticize. Assume good
faith._" It isn't easy, but it's necessary that we all do it, because
otherwise we just end up fighting with each other. Fighting each other feels
urgent, but is not interesting, and the purpose of HN is to be an
intellectually interesting place on the internet, if possible. This requires
that we sort of resist the worst interpretations of each other's words,
because they're so often vulnerable to misunderstanding.

By the way, in case you're not familiar with so-called Hanlon's Razor [1],
it's not primarily about stupidity. It's really a way of saying that most
human outcomes are not the result of malice. So in a way, that post was a
(weak and unclear) defense of the Japanese institutions and reports that were
being discussed above—a way of arguing that they weren't conscious bad action.
The word 'stupidity' appears in that proverb partly out of ironic humor which
is typical of the internet subculture that produced it, since it's ironic to
defend someone against the charge of malice by making the charge of stupidity
instead. It's not intended as the deep insult that the word 'stupidity' would
imply in many cultural contexts. If you already know this, I apologize.

There is an additional level also. The word 'stupidity' in that proverb would
not normally be a national or racial slur. It usually refers to the
'stupidity' of large institutions, which tend to produce inefficient or
inadequate results even when the individuals who make up the institution are
doing their best—or to the 'stupidity' of bureaucratic functionaries, who
hacker-type people tend to feel superior to and smarter than. If you look at
[https://news.ycombinator.com/item?id=22556236](https://news.ycombinator.com/item?id=22556236),
you'll see exactly the same argument applied to Spain's Covid-19 response. If
anyone is being criticized as stupid in those comments, it's not peoples like
the Spanish or the Japanese, it's mid-level government officials everywhere.
Whether that's fair or not, we can leave as a separate question.

[1]
[https://en.wikipedia.org/wiki/Hanlon%27s_razor](https://en.wikipedia.org/wiki/Hanlon%27s_razor)

------
guardiangod
Knowing Japan, I bet some poor freshly graduated but elite sod in the health
ministry was, out of the blue, told by his boss "here's a bunch of reports
from different prefectures. Make a report out of them." So he goes off and try
to make sense of the data, which is why you see those magic and unexplained
equations in the report.

The kicker is the most complex computer thing he ever did in his life was that
Word presentation in university. HTML? Isn't that something you write by hand?

~~~
kenhwang
My mom is responsible for updating all of Los Angeles' numbers. It's literally
"someone sent an email with updated new numbers, put them on the government
website/in the database". There's no "making sense of the data" phase, just
copy, paste, and hope whoever delivered those numbers did it correctly. She
described the situation "all hands on deck, extra shifts, and losing staff to
sickness daily"; the last thing they care about is some number on a screen.

~~~
jariel
There's nothing wrong with that - in a peculiar situation, wherein there is no
'standard web site' something ad-hoc is perfectly reasonable.

In the 'old days,' it would be a public number board or chalkboard, but it
would be accurate. This is what a lot of ladies in the military or in
operational centers would have done.

And definitely nothing wrong with hand-tweaked HTML. I wish sites were basic
enough and HTML clear enough that it could be more pervasive.

You can dig up ledgers from 'old-timey' 100-year-old from little stores in the
middle of nowhere that were handwritten and _perfect_. Perfect handwriting,
perfectly updated, perfect math (i.e. adds up after corrections).

What matters is that the process is legit, that people are responsible for how
the numbers are propagated, and they do it professionally.

It's almost weird to think that people expect that there is a 'robust and
specific electronic process' for everything.

The level of organizational planning in WW2 was pretty shocking: all docs,
notes, cue-cards, human process.

------
cmehdy
While things like masks and social distancing are more likely to be already in
place in Japanese culture, there definitely seems to be a bit of a surprise
regarding actual numbers vs. means put in place to prevent the spread.

The reports seem to reflect the opacity of communication of Japan's situation
about the Covid-19. It's good that the author put out actionable suggestions
at the end, but I'm wondering if the opacity is a result of internal issues or
intended opacity. Makes me glad to see Canada's stats[1] being so clear in
comparison.

[1] [https://www.canada.ca/en/public-
health/services/diseases/201...](https://www.canada.ca/en/public-
health/services/diseases/2019-novel-coronavirus-infection.html)

~~~
btilly
I am extremely suspicious that Japan's numbers look good because they have not
been testing enough.

Given that honesty in testing consistently precedes the eventual slowdown in
deaths by several weeks, I strongly suspect that Japan is maliciously
incompetent and a lot of dead Japanese will be the result.

~~~
standardUser
" I strongly suspect that Japan is maliciously incompetent and a lot of dead
Japanese will be the result."

When? Japan has had this coronavirus a lot longer than most places. If we
expected the transmission rate to explode, we would have expected it to happen
already, and we would be seeing the consequences.

~~~
hilbertseries
It looks Tokyo is starting to have an out break

[https://www.chicagotribune.com/coronavirus/ct-nw-tokyo-
olymp...](https://www.chicagotribune.com/coronavirus/ct-nw-tokyo-olympics-
coronavirus-20200330-fwlj7xw7rfb6jfkjh5iafplzsq-story.html)

~~~
btilly
The most interesting point in that article is that Japanese citizens and
politicians are suspicious of the timing. All numbers looked good until the
Olympics were called off. Then Japan started reporting climbing cases and
talking about emergency.

I certainly hope that the politicians in Parliament who asked questions got
satisfactory answers. I doubt that they did, but there should be time to ask
those same questions later when there is a clearer picture.

I just hope it isn't as bad as I fear it will be.

~~~
unishark
> All numbers looked good until the Olympics were called off. Then Japan
> started reporting climbing cases and talking about emergency.

Typical causation versus correlation problem. Exponentially increasing
pandemic outside Japan can also be the cause for everything: the olympics
cancellation, the increasing govt concern and testing, and the sudden surge of
cases within Japan from travelers.

No conspiracies needed.

~~~
btilly
First of all, lack of testing and government denials that there is a problem
here been a hallmark of the response worldwide. It is hardly a conspiracy
theory to say that it happened in Japan as well.

Secondly for a couple of weeks now I've been running across articles of the
form, "Japan should have had a problem by now, why are they an exception?"
Which is more likely? That they were an exception until now, or that the
Olympics provided extra motivation to stonewall?

It is clearly not proof. But extra motivation to fail to try to detect their
own problem combined with reason to doubt their version of reality (in the
same way that happened over and over again elsewhere) makes my estimate of the
conditional probability very high that their government was simply a bit more
deliberately blind than elsewhere.

And, as with everywhere else, the cost will be dead citizens. Only the further
you let the problem go and the older your population, the worse it is. Japan
has the highest concentration of truly old in the world. And may have let it
go longer than anywhere else. The result? The final body toll won't be pretty.

------
svara
At the end of the day, hiding exponential growth of this is impossible.

That's what really bugs me about those who love to point fingers at how this
or that country is counting things wrong (often revealing their biases in the
process).

Maybe every country has a random constant factor between actual cases and
reported cases. But it doesn't matter - if it goes exponential, we'll know.

~~~
gwern
Yes, obviously once it goes exponential, exponentials are hard to hide. The
point is to stop it _before_ it goes exponential. That's why people were
discussing, say, Iran or NYC testing problems and lack of social distancing,
rather than saying "well, at the end of the day, hiding exponential growth is
impossible, so who cares"...

------
nemetroid
My hypothesis is that the "with no symptoms" and "with symptoms" items
actually do include deaths.

> Remember to not include death, because this is a quasi-hierarchy. The table
> tempts you to, but it's excluded. (The quasi-nature is because it is under
> the PCR tested positive umbrella on level 2.) So what about integrating
> level 3?

> That's simple. You don't. Because no matter how hard you try, the numbers
> won't add up. (e.g. Give it a try - you'll end up with 129 != 131 and 1147
> != 1191. We'll need one of these numbers later.) Level 4 adds up though, so
> the plot thickens.

Note, however, that (131 - 129) + (1191 - 1147) = 46, i.e. the number of
deaths.

So if you adjust the 131 and 1191 figures to match their subdivision,
everything would sum up correctly, including the top level
(pcr_tested_positive), which _would_ then include deaths.

This would imply that 2 people without symptoms died, and 44 people with
symptoms did.

Taking a guess, "deaths" used to be two separate columns: one under "with no
symptoms" (2 deaths), and one under "with symptoms" (44 deaths). It was then
decided that "deaths" should be its own top-level item, and someone decided
that this was the best way to achieve that goal.

I can't find a way to make sense of the daily increases though, so this might
be entirely wrong.

------
dataatemytime
Has anyone applied Benford's law to the raw data?
[https://en.wikipedia.org/wiki/Benford's_law](https://en.wikipedia.org/wiki/Benford's_law)

~~~
kevinventullo
I think this could make an interesting post!

------
rurban
Looks mostly right but he is wrong about the Diamond Princess. Those people
all went off home, which is not Japan. Mostly western states. So they don't
count to the Japanese numbers, only did in the very beginning.

The rest looks indeed cooked, on purpose. Pasting images instead of tables is
the usual trick.

~~~
longtom
He is also wrong about there having been no follow-up. Here it is, though only
about a smaller fraction of the passengers.

[https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3551331](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3551331)

------
LorenPechtel
I think this is a case of someone simply scanning various reports into the
website--doing it the easiest way possible, not the right way.

~~~
noobermin
Seriously, why all the skepticism regarding this? I understand it but it's
strange to me. Hanlon's razor seems like an easier explanation, why does there
need to be a nefarious plot under every bed.

------
lainga
_The reason why I used a two day old report on March 29 is simple, there was
no full report yesterday nor today. I 'm guessing the Tokyo lockdown means
that nobody can get to work, and PCs are too expensive to buy for home use so
the people at the Ministry of Health can enjoy a nice weekend with hoarded
pasta._

How's corporate Japan as a whole doing with remote work?

------
icoxfog417
The official format is terrible but you can get CSV data from here
[https://github.com/kaz-
ogiwara/covid19/blob/master/README.en...](https://github.com/kaz-
ogiwara/covid19/blob/master/README.en.md).

------
zadkey
I've been trying to access this website for the last 2 days, it is either down
or overwhelmed.

------
noobermin
I know, data science (was it called that?)...didn't exist until tableau

------
phoenixstrike
Popular opinion in the US regarding the integrity of various countries'
coronavirus reporting is hilarious. Whatever China says must ALWAYS be a lie,
on the other hand, the US numbers are not "suppressed," it's just
incompetence. Meanwhile, nobody is batting an eye at Japan's ridiculously and
obviously doctored numbers.

Hey guys, maybe China was just having literally the same organizational
problems the US is having now leading to underreporting of numbers, back in
Jan-Feb when every commenter on reddit and HN were saying China was covering
it up.

Meanwhile people just assume Japan's numbers must be honest because... why?
Anyone who has the slightest awareness of how Japan works knows that this is
par for the course. And no, it's not just the government brainwashing the
people. The people are complacent in this and defend their government by
rejecting criticism and continuing to vote them in office.

~~~
usaar333
> Whatever China says must ALWAYS be a lie, on the other hand, the US numbers
> are not "suppressed," it's just incompetence.

Well, this jibes with reality.

I could care less about actual case counts - few countries are close to having
the capacity to test everyone so who cares.

What is a huge problem with China is the death statistics. Looking at Lombardy
with 6k+ deaths with the outbreak not yet peaked, am I really supposed to
believe that only 2.5k died in Wuhan (similar population)? The CFR outside
Hubei is ridiculously low - only 1 death out of 1200 cases in Zhejiang (!)?

During China's outbreak, there were additionally lots of accusations of case
data being completely made up (insufficient variance in daily numbers) and
looking at basically every other country's quite noisy reports today, that was
almost certainly true.

The implications are pretty vast: China's reporting actually made the disease
seem less urgent than it really was. Perhaps tens of thousands of lives would
have been saved had they been more transparent.

~~~
bogomipz
>"Well, this jives with reality."

You want "jibes" here. Jive means to talk nonsense:

[https://www.grammarbook.com/homonyms/jibe-
jive.asp](https://www.grammarbook.com/homonyms/jibe-jive.asp)

~~~
StavrosK
Plus, "couldn't care less", though I'm fighting a losing fight.

~~~
neltnerb
Maybe they _could_ care less? Yes, it's a losing fight.

Of course, it's probably more literally true that they could care less, after
all, they care enough to post about it on the internet. They probably don't do
that for... um... articles about the 1984 Honda Accord after all?

Oh English.

~~~
StavrosK
Think of "I could go out less", "I could eat less", "I could spend less", etc.
All these phrases clearly imply that you're overdoing it now and need to stop.
"I could care less", as used, means the opposite, that they don't care almost
at all.

Your explanation is a retcon.

~~~
neltnerb
... that was the joke. But I agree jokes are better when they're explained
(not kidding).

Of course, if enough people start saying "I could care less" I guess they get
to decide what it means. Living languages are literally awesome.

------
yufeng66
Never attribute to malice that which can be adequately explained by stupidity.

~~~
objektif
Oh man, china would like to have a word with you about that statement.

~~~
dang
No nationalistic flamebait on HN, please.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
panny
It's okay to call Japanese stupid.

It's not okay to call Chinese stupid.

??

~~~
dang
Obviously it's not "okay to call Japanese stupid".

------
hujun
now days hacker news is often filled non-tech related discussion, specially
anything relate to China or politics; come on guys, this is _Hacker_ news, not
political news; if you are so passion about politics, go to some forum
dedicate for those things;

~~~
dang
This isn't so different from how HN has always been. Many topics have
political overlap. That's inevitably the case on a site that's dedicated to
intellectual curiosity
([https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)).

This is a challenge for moderation, but the approach we take has been stable
for a long time. I've written about it here:
[https://news.ycombinator.com/item?id=21607844](https://news.ycombinator.com/item?id=21607844),
and there's lots more explanation at
[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20political%20overlap&sort=byDate&type=comment)
for anyone who wants it.

Concerns about HN getting more/too political go back almost as far as the site
itself:
[https://news.ycombinator.com/item?id=17014869](https://news.ycombinator.com/item?id=17014869).
There's no systemic breach, just topical fluctuation.

How we handle major ongoing stories (like this crisis) is mostly by
downweighting routine follow-ups, so that just the stories with significant
new information get primary coverage:
[https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20major%20ongoing%20significant%20new&sort=byDate&type=comment).
That's the intention, at least.

Of course there are always exceptions for submissions that have intellectually
curious angles, like the detective aspect in the OP here.

