
Bloomberg’s ‘The Big Hack’ - okket
https://daringfireball.net/2018/10/bloomberg_the_big_hack
======
tristanj
Regarding Apple's denial, there are other publications that corroborate the
Bloomberg story. Previously, Apple has denied security incidents even when
multiple outlets report it. For example, last year, The Information reported
Apple discovered malware on Super Micro servers in their development and
production environments [1]. As a result, the Information claimed that Apple
ended up terminating its relationship with Super Micro.

In response to the report, an Apple spokesperson denied there was a security
incident, stating: _" We’re not aware of any data being transmitted to an
unauthorized party nor was any infected firmware found on the servers
purchased from this vendor."_

However, based on sources from within Apple, Ars Technica claimed Apple
employees did find compromised firmware in Apple's design lab. Super Micro SVP
of Technology also reported Apple terminated its relationship with them.

I believe we are seeing the same situation here.

[1] [https://arstechnica.com/information-
technology/2017/02/apple...](https://arstechnica.com/information-
technology/2017/02/apple-axed-supermicro-servers-from-datacenters-because-of-
bad-firmware-update/)

~~~
IBM
Apple's denial references that report.

~~~
tristanj
Which makes it more curious. Last year, Apple PR denied existence of the
security incident, but today they admitted it actually happened. That is a
contradiction.

~~~
ralfd
No contradiction.

Read it closely: The old statement said that no infected firmware was found on
servers purchased from supermicro, meaning freshly purchased servers delivered
from supermicro were clean. The ArsTechnica clarifies in an update to the
article that the infected driver was installed later.

The new statement says: "That one-time event was determined to be accidental
and not a targeted attack against Apple."

------
lacker
It does not seem like this story is true. If it's true, it makes absolutely no
sense for Apple and Amazon to attack Bloomberg. Sure, a national security
letter could force them to stay quiet, or maybe even to lie to the public and
say it didn't happen, but it can't make them criticize Bloomberg. Attacking
Bloomberg if the story is true is only going to convince Bloomberg to dig
deeper. And the story isn't even _that_ bad for Amazon or Apple - it's much
worse for US-China relations than it is for either of those companies.

The key technical detail of what these chips are allegedly doing also does not
make sense. From the article:

 _the chips allowed the attackers to create a stealth doorway into any network
that included the altered machines_

How can you get around a firewall by using a compromised machine that's part
of the internal network?

I don't think Bloomberg reporters are just making stuff up. But the technical
confusion here makes me suspect that the government officials who leaked this
story just didn't understand the details of a real incident that happened, and
in the leaking the story got mangled into inaccuracy.

~~~
QuinnyPig
It goes beyond that. If this story were in fact true, what we're witnessing is
large companies whose entire business is built upon user trust setting fire to
themselves. Who'd ever trust AWS again with anything of consequence if it
turns out the denial is false?

I don't buy it as written. There's something else here.

~~~
Jach
> Who'd ever trust AWS again with anything of consequence if it turns out the
> denial is false?

I agree it makes little sense to attack and deny rather than just stay quiet
or keep it to a minimal PR blurb. But I think you're overestimating the amount
of blowback they would get if they're lying. Most people just wouldn't care.
It might be the decisive factor in a bigger company deciding where to put
something new and them going to Oracle Metal or something, but I am really
skeptical their bottom line would suffer because of this one thing.

~~~
394549
Also, almost everyone who lies thinks they'll get away with it. Otherwise, why
lie?

Maybe AWS and Apple are counting on the top-secret nature of all this to
enable their lies. Would US intelligence agencies really come out on the
record to contradict the denials and confirm the hack if it meant destroying
the credibility of a couple of American companies? The Bloomberg article
detailed how the government was _not_ willing to sound the same alarms against
an American company as they were against Huawei and ZTE for just that reason.
I'm guessing that creates enough ambiguity for false denials to be effective.

~~~
morpheuskafka
Yeah I think people don’t understand how big of a deal this would be. Remember
Joseph McCarthy? Now picture that, in today’s politics, only with even a shred
of real evidence that US companies and the military were infiltrated. That’s
an act of war between two nuclear powers. There is literally no telling what
kind of chaos could follow.

I’m not saying it’s guranteed that this story is good—-for one, I find the
glossed over technical details a little questionable. But it is definitely
plausible that this is a false denial, under orders or not.

Regardless, two points are very clear. Remember Stallman who uses that old X60
with a free BIOS? We all called him crazy when he insisted on not having
management firmwares and secretive back doors. Now we have a story claiming
they were used in an attack by a US adversary (as well as a long chain of
confirmed recent security flaws). The second point is that we have to stop
trusting China as the sole source for all electronics. It’s a national
security crisis that there isn’t a single US facility making many of the
components needed for basic computing platform. The federal government needs
to immediately and strongly incentive more domestic foundries. And not the
trump style “here be factory, jobs good” but actual targeted planning to bring
the whole supply chain back, not just the final assembly everyone talks about.

------
propman
For major major reports like this, reputed newspapers are rarely wrong. See
WSJ John Crareyou of Theranos. Theranos vehemently denied it, MSM was with
Theranos and at least they weren’t outright against them. Then they did more
digging etc.

The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc.
these are experienced reporters having 15-30 credible sources, evidence etc.
every time this happens, everyone denies and slowly little by little the story
finds more evidence and facts and it becomes true.

This is why a free press is soooo important

Why would Apple disclose that millions of their products could be hacked and
you’ve lost all your privacy. Who would trust them? They’d lose billions,
regulations would come etc. it’s in the best interest for every party to deny.

~~~
lacker
On the other hand, Newsweek was wrong about Satoshi.

My suspicion is that there are many cases where the Chinese government is
actually trying to insert backdoors into things, and that in particular
Supermicro really has been compromised by the Chinese government, but the
technical details of this chip are incorrect. That explains why so many
government officials are eager to leak information to Bloomberg, but at the
same time the technical details don't really make sense.

It isn't really in Amazon's and Apple's best interest to lie about this. When
Gmail got hacked by the Chinese government, Google was pretty honest about it.
China has a lot of resources so you can't really expect companies to fend off
100% of attacks on their own; it makes sense for them to acknowledge this
publicly and get help from the US government when needed.

~~~
makomk
Note that Facebook and Apple have confirmed that they had heard about and
actually saw, respectively, security issues with compromised software updates
from Supermicro - it's only the chip story they're denying.
[https://www.bloomberg.com/news/articles/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-the-
software-side-of-china-s-supply-chain-attack)

~~~
smsm42
There's a huge diff between "your web management software is buggy" (pretty
much every web management software is kinda crappy and significant number
probably would dissolve like wet paper under a serious security audit) and
"your motherboard has extra chip which grants full hw level access to outside
parties". The former is completely routine and happens nearly every time
anybody bothers to do a security audit. The latter, if true, is one of the
biggest stories of the decade.

------
notatoad
Can National Security Letters can be used to require companies to issue
outright lies to the public? The bloomberg article indicates that the
investigation is not complete, so that could be on explanation for the
apparent disconnect between a seemingly well-reported story and the unsually
forceful denials.

~~~
ProAm
No but the gov might be unhappy that the secret is out while they were trying
to use the chips to infiltrate/reverse engineer wherever the chips were
reporting to? Amazon has gov contracts it wants to protect so Im not surprised
they would deny this at the govs request, and Apple's #1 product is privacy so
having something like this show up on their servers would undermine that
product strategy so I can understand their denial too (homegrown or at the
govs request)

~~~
thebooglebooski
This seems like the most likely reason for domestic companies to deny
everything, and for their supply chain distributors to play along.

1) Big company partners with distributor. 2) Distributor has security issues.
3) Gov is already aware of security issues, says nothing. 4) Big company
becomes aware of issues. 5)Gov steps in and pitches a deal: i)Both big company
and distributor must deny. ii)In return, gov gets to: iii)Preserve any
existing contracts iv)Protect the big company and distributor, with any legal,
trade, or commercial benefits

------
whydoineedthis
Bloomberg reported that the startup I worked for was for sale and the founders
were pitching it to potential buyers. Internally, they vehemently denied the
report and said Bloomberg completely made it up. Bloomberg was dead on the
money. That's not to say every journalist/article they publish is going to be
spot on, but I definitely give them enough credit that I will believe this
report until proven unequivocally not true by the accused.

edit: spelling.

~~~
berberous
A CEO shopping a private startup has every incentive to lie to his employees
in that scenario and little reason not to.

On the other hand, Apple and Amazon have huge reasons not to blatantly lie
about this. The plaintiffs bar would be all over both companies for material
false and misleading statements if this turned out to be true (see also Elon's
recent experiences with the SEC).

Apple and Amazon are certainly incentivized to not admit bad facts, to spin
facts, to issue misleading unclear statements that read as a denial but are
not, etc. But I really don't think the legal teams at either company would let
executives get away with issuing such full-throated and clear denials if they
were untrue.

~~~
whydoineedthis
yeeaaah...on the other hand, they could be pulling a hat trick from the FCC
playbook. "I had no idea it was legitimate users leaving feedback and the guy
I trusted to inform me no longer works here. I was told everything was fine!"
Plausible deniability goes a long way when you have enough money/political
support or are "too big to fail". It happens. Often.

------
sgwealti
What if Apple or Amazon didn't find the chips but other investigators found
them in Apple/Amazon servers. That would make their denial wording technically
true. They didn't say no chips were found, they just said they didn't find
any.

------
edoo
This is this is similar to all the backdoored Cisco devices the FBI found all
over the government. If they are doing it at all they would have a complex
plan and this would be one of the many approaches. Even scarier IMHO are the
CPU fabrication hacks that add in an imperceptible backdoor directly in the
chip logic. A recent report showed how CPUs can be backdoored at critical
points in the fabrication process by a single operator that would be
incredibly hard to detect. We are talking instruction patterns that charge
capacitor buffers that allow privileged access once a threshold is reached.
Amazing really.

------
saudioger
>But they don’t lie, because they understand that one of Apple’s key assets is
its credibility. They’d say nothing before they’d lie.

This is so typical of Gruber as an evangelist.

There's no way that Apple would remain silent on this even if they KNEW it
were true. The only possible move is denial.

Silence is validation or uncertainty, a statement of ambiguity will tank the
stock and reputation as experts, recognition of even partial truth could
possibly destroy their supply lines overnight.

I honestly think the corporate denials here need to be outright ignored
because they have so much to lose. A story of this magnitude is basically like
pointing a gun to someone's head and asking them for permission to pull the
trigger.

~~~
fermienrico
I see Apple and Amazon has a tremendous incentive to lie.

What incentive do you see for Bloomberg to report a major story and lie!? Why
would they do that?

~~~
saudioger
You can theorize all sorts of conspiracies, like the government planting the
story to distance China... so it's possible Bloomberg was mislead.

I think at this point it's more believable the story is true, because
Bloomberg is the most credible participant at this point... they're anonymous
sources, but as long as Bloomberg did their due diligence as journalists,
they've validated their (numerous) sources as credible.

...and like you said, all these corporations have MASSIVE amounts to lose.
Just check out what Trend Micro's stock is doing today.

~~~
fermienrico
That's exactly my point. They have MASSIVE amounts to lose - that's why they
don't want to admit that their datacenters are compromised. Super Micro's
stock is down -41%.

------
wgerard
"But in my experience, Apple PR does not lie. Do they spin the truth in ways
that favor the company? Of course. That’s their job. But they don’t lie,
because they understand that one of Apple’s key assets is its credibility.
They’d say nothing before they’d lie."

As a few people pointed out in the other thread, didn't they pretty explicitly
deny they were involved with PRISM?

~~~
mercutio2
PRISM was an NSA-internal codename for a really innocuous automation of
servicing legal warrants.

Apple knew nothing about anything they were told was called PRISM, and they
happily acknowledged the existence of APIs to service warrants.

------
neom
As someone on a slack server I use pointed out: a server wouldn't need to
phone home, could have a planned failure and request an RMA, even if the
system was wiped when it came back, it could have data stored somewhere
secretly, and why you may not find anything in an audit.

~~~
reitanqild
AFAIK that wouldn't work against Amazon as IIRC hardware doesn't leave their
datacenters without being shredded first.

~~~
NullPrefix
So no hardware warranty repairs?

~~~
reitanqild
AFAIK, no.

It was presented as one of many things Amazon does to keep AWS safe at an AWS
day I attended or something.

------
baq
there's a comment on r/sysadmin that's quite chilling, implying that the state
of affairs is far worse than what the report describes:

[https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloom...](https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloomberg_how_china_used_a_tiny_chip_to/e75rwc8)

    
    
        I did a penetration test and security assessment for a major electronics manufacturer
        whose parts are likely in every smartphone and laptop. I identified almost certain compromise
        by the Chinese government with full access to modify the manufacturing specs using the
        access paths I identified.
    
        They chose to bury my findings as it would cause a huge stock hit. Sadly, NDA.
     
        I'm not surprised in the slightest.

~~~
wyldfire
Hard to give an anonymous comment any weight without the slightest
verification. Since we know that adversaries of freedom use social media as a
disinformation vector, the only thing you can do is ignore them or encourage
them to find a way to legitimately disclose the information protected under
NDA -- perhaps to the press or a legislator who could help make it possible to
invalidate NDAs that keep secrets that make us vulnerable.

~~~
baq
I agree that panic is ill advised and trust should be given carefully, but if
Bloomberg is to be believed, then it happened at least once - and if it did
happen once, it's very likely that it happened more than once, because why
not? Most organizations don't have resources to find a backdoor like that, nor
they have a reason to search for one (had no reason until today I should say)
and in this case we only heard about it because of multiple leaks to the press
from government officials.

------
peignoir
A lot more coming from this news for sure, but kudos to whoever found these,
that's solid tech due diligence!

~~~
Taniwha
I'm still waiting for technically literate description - as an electrical
engineer I don't know what a "signal conditioning coupler" is - searching
DigiKey doesn't find anything under that name - it's not a "common" part, at
least in my experience. The part looks like an RF filter look-alike of some
sort. But that would normally be hooked into a power rail (and ground) - kind
of hard to insert signals into a data line unless it was wired strangely

So tell us what the part is pretending to be and how exactly it it was wired
(and what it was connected to - is this another Intel ME backdoor?)

------
_trampeltier
Still no statement from supermicro. Nothing on Twitter. Nothing on the
website. That's a kind of strange after the stock price droped so much.

~~~
excalibur
Yeah there is:

[https://www.bloomberg.com/news/articles/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-
amazon-apple-supermicro-and-beijing-respond)

~~~
_trampeltier
True, but still, I would like to see a statement on the website or on twitter.

------
space00
It remind me another security risk "Google's new hardware security key was
made by a Chinese company"

[https://www.cnbc.com/2018/08/30/google-titan-made-by-
chinese...](https://www.cnbc.com/2018/08/30/google-titan-made-by-chinese-
company-feitian.html)

------
wyldfire
> not much bigger than a grain of rice, that wasn’t part of the boards’
> original design.

'original design' is hard to verify without help from SuperMicro.

> Amazon reported the discovery to U.S. authorities, sending a shudder through
> the intelligence community.

... especially for Amazon. Unless Bloomberg claims here that Amazon got parts
of this SKU and compared them to newer parts of the SKU and found differences?

Is it legit to revise your design in terms of changes to passives without
rev'ing the part and notifying downstream supply chain. Could the grain-of-
rice 'microchip' be a different or new resistor/cap? Could it be logic
masquerading as a passive?

> In one case, the malicious chips were thin enough that they’d been embedded
> between the layers of fiberglass onto which the other components were
> attached, according to one person who saw pictures of the chips.

Like a passive in a blind/buried via?

Anyways, for everyone who claims "it can't be done", "this is implausible" \--
you're probably being a little naive. US intelligence agencies do shipment
interdiction and adulterate products for this purpose, [1] why
couldn't/shouldn't China do the same?

[1] [https://www.theguardian.com/books/2014/may/12/glenn-
greenwal...](https://www.theguardian.com/books/2014/may/12/glenn-greenwald-
nsa-tampers-us-internet-routers-snowden)

------
jaclaz
As a side note/question.

I have no idea how things actually work in companies like Apple or Amazon, but
would it be "normal" (at their size/scale and given their surely advanced
knowledge in technology) to have inspections on the hardware they use
(inspections at hardware level of the kind capable of showing these
modifications)?

I mean do they routinely do these checks?

~~~
laurentl
I had the same question when I read the article. It sure was lucky that both
Apple _and_ Amazon checked the hardware and found the impossible to find chip.

Or, assuming the rest of the article is true, someone somewhere found out
about the chip and had a quiet word with a few select customers of Supermicro
(but decided to otherwise keep mum about the whole thing). Which would go some
way to explain the company denials, because the truth might be a bit
embarrassing —or gagged by a three-letter agency.

------
Animats
OK. So what's the minimum remote system management capability needed in a
modern data center? The major cloud sysadmin people should figure this out,
write a spec, and insist that's _all_ that goes in. If Amazon AWS and Google
wrote a spec for this, the manufacturers would fall into line.

Boards are shipping with way too much remote access capability. It's not like
you need to look at system busses via the network. You're not going to debug a
broken board remotely, you're going to turn it off and replace it. Now that
this is an identified problem, it's time to put IPMI and its ilk back in its
cage.

------
rajekas
Amazon bids for CIA contracts; Elemental bids for Amazon's contracts; Super
Micro bids for Elemental's contracts and the PLA "bids" for Super Micro's
contracts. Looks like Ouroboros, the global supply chain episode.

------
an0n404
A little suspect when they say " giving them access to the most sensitive code
even on machines that have crashed or are turned off."

When it is turned off? A magical chip that works without electricity is much
more valuable than any data which could be exfiltrated from servers. A chip
that works without power changes the world more than the iPhone... in the
iPhone Age

------
morpheuskafka
The US needs to make electronics supply chain sovereignty the number one
priority of the federal defense budget. Why are we fighting useless drone wars
when our country is being attacked on a daily basis in the cyber realm?

If this story is true, it will be an escalation clearly and willfully by the
PRC from mere state-sanctioned economic espionage to an act of war against the
US.

------
swsieber
This reaction from Amazon / Apple is reasonable if Bloomberg is lying.

This reaction also seems reasonable if Bloomberg was telling the true.

------
vthallam
> The White House requested periodic updates as information came in, the
> person familiar with the discussions says.

You can't make this up, at least not in Bloomberg, that too in a front page or
cover page write up. Truth could be somewhere in between and I am sure we will
have more information in the coming weeks.

------
getcollc
[https://www.reuters.com/article/us-china-cyber-dhs-
idUSKCN1M...](https://www.reuters.com/article/us-china-cyber-dhs-
idUSKCN1MH00Y)

So what happens to the reporter then?

------
throwaway5752
Certainly not saying this didn't happen, or creating a false dichotomy that a
nation-state has to be a friend or foe.. but evaluate this in the context of:
[https://talkingpointsmemo.com/edblog/white-house-begins-
the-...](https://talkingpointsmemo.com/edblog/white-house-begins-the-china-
counter-narrative) (in case people get the feeling that this story is being
pushed, or feel like it's coming out of nowhere suspiciously close to an
election)

~~~
rwc
Stories like this take many, many months of research -- particularly a story
as detailed as the Bloomberg. I don't think it's fair to the journalists to
simply explain the story away as a government-led conspiracy to mislead voters
in the midterm elections.

------
getcollc
Is this the tiny chip bloomberg is refering to? Maybe they shorted a bunch of
aapl and amzn, I'd suggest SEC to investigate the editor who wrote up this
crap

[https://www.digikey.jp/product-detail/en/tdk-
corporation/HHM...](https://www.digikey.jp/product-detail/en/tdk-
corporation/HHM1522B1/445-3987-1-ND/1955584?cur=JPY&lang=en)

------
directtt
Now with the annoucement from the Homeland Security Department of the US, it's
very clear, even to a kid, that Bloomberg is lying. The question is should the
authors, Jordan Robertson and Michael Riley, get punishment for creating this
fake story? They have already created damages. And they may also have some,
let's try some conspiracy, motivation in favor of the Trump's gov... since the
report is "very conveniently" appeared on the same day of Pence's speech.

------
xte
For me it's IRRELEVANT in the report is true or not, the very relevant part is
that today's hw is complex enough to be a black box for 99% of buyers so it
pose a super-serious security risk. Strong article from publicly known sources
are a way to shed the light on such ignored enormous security threat.

The sole solution IMVHO is IMPOSE open hardware and free software by law. We
simply can't have our society "nervous system" run on black boxes. It doesn't
matter if the rouge in charge is China, USA, a specific vendor or someone
else. Our banks, our states, our hospitals, ... relay on such connected black
boxes. We also need to re-transfer knowledge from big corporate to PUBLIC,
well founded, universities to AVOID dangerous evolution paths like actual IT
evolution.

~~~
CSEThrowaway
You are welcome to write the software for systems you describe and release it
for free. Unfortunately, the economic model of the real world is a little more
complicated than you seem to realize.

~~~
tobltobs
Tell this Linus.

~~~
CSEThrowaway
I love open source software and use it every day. I also donate to
organizations who contribute to that community. My issue with OP's proposal,
rather, was that those who create software and hardware should be _forced_ to
release it for free.

~~~
bubblethink
A more practical approach would be to at least be forced to provide the
ability to run your own software on the hardware you buy, which is also not a
thing for the vast majority of hardware produced. At least not for end users
anyway.

------
40acres
This back and forth between Bloomberg and Amazon/Apple reminds me of the
recent allegations against Supreme Court nominee Kavanaugh.

In both situations it's difficult to identify who is telling the truth and is
probably impossible to know the truth from the outside looking in.

In both cases I've taken a "who has incentive to lie?" view on the allegations
and denials, now this does not mean my hunch is correct at all, but it seems
way more likely to me that it's in the best interest of Apple and Amazon to
deny this story strongly as it makes both parties look bad and the
investigation may be ongoing. I don't believe that Bloomberg made this story
up but its understandable to question that maybe their sourcing was iffy,
based on Bloomberg's history and the level of sourcing that they have cited
I'm leaning towards believing the story as accurate.

~~~
braythwayt
I personally don't think this is much like the Kavanaugh situation.

For example, Dr. Ford stepped forward publicly, in full knowledge that a
massive machine would swing into action to besmirch and defame her, whereas
the allegations about espionage were made in confidence. The six(?)
individuals making these allegations are risking nothing at this point.

Also, Dr. Ford's allegations reach back into time and are difficult to
conclusively prove,* while this is a situation where physical evidence does
exist somewhere. It may not be presented to us at the moment, but if the
allegations are true, there were physical motherboards that could have been
examined to demonstrate the exploits.

What the two have most in common is that we the public are unlikely to be
provided with a full and transparent investigation. The FBI's "investigation"
into Dr. Ford's allegations did not involve speaking to her or others who
could corroborate aspects of her story. Others from Yale approached the FBI
and did not get interviewed.

The same is probably true of this story. Even if the allegations are true, for
diplomatic or other strategic reasons, the government is unlikely to shine a
spotlight on the details.

* Note that "Conclusively prove," is not the same thing as "Obtain enough confidence based on other reports to make a decision about job fitness."

~~~
40acres
I think similarly to how Dr. Ford knew she would be questioned and pushed into
the spotlight when making these allegations public, Bloomberg similarly
would've had to calculate the risk that Apple, Amazon, and China would push
back on this story vigorously and try to discredit Bloomberg.

I guess my basic premise is that I'm more liable to believe that Bloomberg is
operating in good faith in reporting this story. Now that does not mean that
their sources may not be credible, but I'm more likely to believe they
published this story in good faith rather than Amazon and Apple denying the
story in good faith.

~~~
braythwayt
I agree that Bloomberg is reporting this in good faith, and that they
corroborated as much as they could have given the information they received.

The credibility of their unnamed sources, their motives, and so forth, this is
all unclear at this point. It may turn out that they are courageous patriots
blowing the whistle on something very, very big. One can imagine a cover-up at
the highest level, with these six individuals risking their careers and
possibly their lives to reveal the truth.

Or one can imagine that there are some trade negotiations coming up, and a
coördinated effort to plant a story so that politicians can take credit for
swooping in and enacting regulations around the security of technology
manufactured outside of the USA.

Who knows? I don't.

