
The Big Hack: Statements From Amazon, Apple, Supermicro, Chinese Government - okket
https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond
======
Illniyar
There's denial and there's vehement to the point complete denial from multiple
different companies.

It's either a giant conspiracy by the FBI and multiple mega-corporations to
blatantly lie, on public record, about a matter that if happened would most
likely come up in the future again. Furthermore if apple and amazon were both
notified for comments, there is good reason to suspect that the FBI would hear
of the article and try to censor such an article for national security
reasons, especially so if they made apple and amazon lie about it.

Or ... Bloomberg didn't do their due diligence and were too eager to be duped
by agents who wanted to push an agenda to move manufacturing away from china
or something similar.

~~~
smsm42
> It's either a giant conspiracy by the FBI and multiple mega-corporations to
> blatantly lie, on public record, about a matter

Like, say, the matter of secret surveillance on the mass scale? I mean, the
track record here is not exactly pristine.

> Bloomberg didn't do their due diligence and were too eager

That is a distinct possibility too. But I think we are now beyond the point
where we could say "major tech companies would never lie together with the US
security apparatus on a matter of public importance". They would, if they
think it's worth it.

~~~
abalone
_> But I think we are now beyond the point where we could say "major tech
companies would never lie together with the US security apparatus on a matter
of public importance". They would, if they think it's worth it._

This is just conspiracy theory thinking. You offer no evidence for this
incredible assertion. _Some_ companies have previously collaborated with the
government, generally without explicitly lying, but we cannot jump to the
conclusion that all companies would voluntarily lie in a coverup conspiracy --
which, by the way, opens them up to investor lawsuits and risks destroying
their branding, for no good reason. We also do know that the government cannot
legally compel companies to lie, only to remain silent.

~~~
smsm42
It's not "theory", it's publicly known facts that companies participating in
mass surveillance denied it, and US official lied under oath to Congress (and
weren't punished for it) to conceal it. There's no "theory" here.

> You offer no evidence for this incredible assertion.

The evidence to the above is publicly available and has been discussed to
death. If you somehow managed to miss all of it, start with
[https://en.wikipedia.org/wiki/Mass_surveillance_in_the_Unite...](https://en.wikipedia.org/wiki/Mass_surveillance_in_the_United_States)
and go on the links from there, it will take you some time.

> Some companies have previously collaborated with the government, generally
> without explicitly lying,

Yes, saying "we do not conduct this particular kind of surveillance ordered by
this particular person" while knowing they conduct a slightly different kind
of surveillance, ordered by different set of persons - is not explicitly
lying. Just like saying "we don't have surveillance technology installed by
FBI" if it's installed by NSA instead. There are many ways of lying without
"explicitly lying".

> but we cannot jump to the conclusion that all companies would voluntarily
> lie in a coverup conspiracy

We can not and we do not. We do not know whether any specific company would
lie - we just know this option is now very much on the table.

> which, by the way, opens them up to investor lawsuits and risks destroying
> their branding, for no good reason.

Being on good terms with somebody as powerful as US federal government is a
very, very good reason. And I don't see anybody's branding being destroyed so
far by the revelation of mass surveillance. We know about
[https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)
and
[https://en.wikipedia.org/wiki/Hemisphere_Project](https://en.wikipedia.org/wiki/Hemisphere_Project)
\- has AT&T brand been destroyed? Not in the least. And the government granted
them immunity from lawsuits related to this.

~~~
lern_too_spel
Your whole post is filled with unsubstantiated conspiracy theories. The only
program that named these companies was PRISM, which ingests data from targeted
electronic wiretaps conducted by the FBI. None of the companies lied about
that.

------
hendzen
Apple and Amazon probably have a small set of TS/SCI cleared employees who
dealt with this mess. It’s likely 99.99% of the employees at those firms had
no idea what was going on. The switching out of thousands of compromised
servers was probably made to look like routine maintenance or upgrades and the
whole affair was kept secret. That is, until some high level government
employees intentionally leaked it to the media, probably under direction of
the White House to garner support for a more aggressive stance on China - the
trade war in particular. Read between the lines.

~~~
simias
That's a _lot_ of unsubstantiated assumptions. It's also explicitly denied by
both Amazon:

>It’s untrue that AWS knew about a supply chain compromise

and Apple:

>we have conducted rigorous internal investigations based on their inquiries
and each time we have found absolutely no evidence to support any of them.

There's no in "between the lines" or ambiguous wording there, they flat out
deny it. Unless this small set of "TS/SCI cleared employees" worked completely
on their own without reporting to anybody else in the company this means that
they are lying in these statements.

It's possible that they do just that but it's a bit strange to me that they
wouldn't find an easier way to deflect the issue without using such a strong
and explicit language. Something vague like "we've been working with the
authorities and have no reason to believe that any sensitive information has
been leaked etc..." would be easier to spin if it turns out that somebody can
prove that these attacks took place.

Surely if the scale of the attack was as large as reported by Bloomberg in
their article it should be possible to find one of these backdoored boards in
the wild? Or at least have testimonies by employees in these company that
could testify that batches of motherboards were suddenly replaced for no
obvious reasons?

And if trade war is the reason why deny it now? What do they have to gain from
that, they're the victims in this story as far as I can tell.

~~~
imglorp
I found this wording interesting:

> We did not uncover any unusual vulnerabilities in the servers we purchased
> from Super Micro when we updated the firmware and software according to our
> standard procedures.

Does that mean they did uncover some "usual" vulnerabilities?

~~~
runlevel1
Most of the out-of-band management systems (aka IPMI/DRAC/LOM) that server
vendors use are built by the same 2-3 companies and rebranded.

These companies churn out some truly horrible software with little
consideration for security.

It's often difficult to automate firmware updates, so they tend to stay
vulnerable.

It's a similar situation to webcams:
[https://youtu.be/B8DjTcANBx0](https://youtu.be/B8DjTcANBx0)

------
uptown
"Apple has never found malicious chips, “hardware manipulations” or
vulnerabilities purposely planted in any server. Apple never had any contact
with the FBI or any other agency about such an incident. We are not aware of
any investigation by the FBI, nor are our contacts in law enforcement."

Bloomberg's article and Apple's statement can't both be right.

~~~
baq
expected given that we're talking about is something that could be the plot of
a jason bourne movie.

~~~
cm2187
Or if Apple is bound by some kind of non disclosure order.

~~~
uptown
Even if they were bound by a non-disclosure order, this response goes beyond
what's necessary to refute the story and conveys material information which
would be used against Apple by shareholders if it is later found to be
factually inaccurate.

------
jnbiche
A lot of people are unaware of how anonymous sources in a serious news
organization work. Here, it means that the multiple high-level intelligence
officials described in the article are known to and vetted by Bloomberg.
They've looked at their resumes and bona fides, and confirmed their
backgrounds. They're just not revealing their names to us.

So which is more likely: that multiple intelligence officials are making this
up, or that Apple/Amazon/Supermicro feel obligated to lie because this is an
ongoing classified counterintel investigation?

~~~
lolc
Plenty of claims by "senior intelligence officials" have proven to be
factually incorrect. Same goes for press statements.

It's simply too early to tell who's telling the truth, who's mistaken, and
who's lying here.

------
nickelcitymario
Best line from an otherwise serious and and important piece of reporting:

"Two of Elemental’s biggest early clients were the Mormon church, which used
the technology to beam sermons to congregations around the world, and the
adult film industry, which did not."

...which did not.

------
tSheoghi2
These denials remind me of the vehement denials the big tech companies gave
when the Snowden leaks came out. Did they turn out to be false?

[https://googleblog.blogspot.com/2013/06/what.html](https://googleblog.blogspot.com/2013/06/what.html)

~~~
clubm8
> _These denials remind me of the vehement denials the big tech companies gave
> when the Snowden leaks came out. Did they turn out to be false?_

At what point do such denials constitute a deceptive trade practive, enabling
the Federal Trade Commission to bring action?

You can't lie in a privacy policy, or a TV commercial. Where is the line?

~~~
wstuartcl
...

It almost makes you wonder if there is a process for ensuring companies comply
with secret investigations and are forced to act publicly and privately as if
they have never happened.

[https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...](https://en.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court)

------
kevlar1818
To me, this is perhaps the most worrying part of the story.

Did Bloomberg, a widely renowned and distributed news outlet with immense
resources, sacrifice hard evidence for sensationalism and clicks?

Or are these companies, all widely renowned with immense resources, bound to
silence due to any multitude of shady reasons?

No matter the facts behind the story and these denials, this whole thing reeks
of FUD.

------
nvahalik
Regardless about how you feel about the hack, outsourcing the vast majority of
our technology to another country just doesn't seem like the smartest idea.
Why would we put our most trusted technology into someone else's hands—just
because it's going to save a few bucks? Wouldn't it be worth it to just do
these things ourselves?

~~~
rajataghi
Its not just 'a few bucks', it is a considerable amount of money in the long
term.

~~~
bashallah
You sure that’s adequate cost evaluation if the bloomberg accusations are
true?

~~~
sct202
I'm sure that companies will start to re-shore production on to friendlier
countries or at least require sub-contracting restrictions. From the article
it sounds like a sub-sub-sub-contractor (SuperMicro->Main Chinese
Contractor->Compromised Chinese Contractor) was the weak link.

------
ksec
>China is a resolute defender of cybersecurity

It is missing the word "offender" somewhere.

I think the question simply comes down to this; Can the Chinese Government be
trusted?

~~~
ElBarto
> Can the Chinese Government be trusted?

The US Government can trust the Chinese Government as much as the Chinese
Government can trust the US Government. ;)

The spy game has been played for 4,000 years...

------
crunchlibrarian
This is interesting in how vehemently all the companies are denying
everything. I am pretty clueless about how the feds work so I'll ask: is it
possible they would be violating secrecy laws or leaking classified info if
they acknowledge this really happened? Could they already be under NDAs or
whatever the equivalent is in the national security world?

Or is it simply a matter of their shareholders having lofty expectations about
tapping the biggest market in the world (China) and saying anything that
angers China is the worst thing you could possibly do from a PR perspective?

~~~
Eridrus
I don't think there needs to be a conspiracy for vehement denials to make
sense. Security is a hugely important reputational good for both Apple & AWS.

~~~
ISL
That reputation would be sullied substantially if the BusinessWeek article is
correct, and the companies' first public statements state otherwise.

------
neximo64
Well, this explains why all those chipmaker acquisitions failed/were rejected
on National Security Grounds.

It would be whole lot harder to find these modifications if this was on the
silicon itself.

~~~
phkahler
>> It would be whole lot harder to find these modifications if this was on the
silicon itself.

Intel ME is well know to be on the chip itself. When you're really good you
hide exploits in plain sight.

------
setquk
Nothing adds up here. Supermicro boards have "designed in the USA" proudly
stamped all over them. This means that either:

(a) the design process was infiltrated, which would have been done US side
thus the nationality of the actors is debatable.

(b) the manufacturing process was infiltrated, which SHOULD have been picked
up during design validation and production sampling.

(c) this whole thing is a load of rubbish.

Lots of questions here. This is not a tinfoil hat measure as well; genuine
questions from someone who HAS worked in the EE side of things.

I wonder if this is a bunch of pre-emptive finger pointing and ass covering
for an implant closer to home?

I don't trust either side of the fence if I'm honest.

~~~
abvdasker
I think the article is implying that the implanted chips might not have been
detected due to their low profile design. Given that the bad manufacturers
were subcontractors it's likely only a fraction of all the boards manufactured
were compromised. It's either that or someone at Supermicro was in on it.

~~~
setquk
See my comment here:
[https://news.ycombinator.com/item?id=18139739](https://news.ycombinator.com/item?id=18139739)

I don't find that method feasible.

Infiltration of supermicro IS but then you have to ask the question: who
really did it as they are on US soil.

------
ggm
I've always wondered if that passive device in the great crest at the US
embassy in Moscow had equivalents which got hooked up to consumer devices with
high voltage parts (to make people reluctant to play inside)

Remember the furore when Zenith was the last domestic manufacturer of TVs in
the USA? We've come a long way since then..

~~~
astrange
It's not that dangerous to open electronics ever since they stopped using
electron guns. But, consumer devices don't need to secretly spy on you when
consumers buy them literally for the purpose of being spied on.

------
cyphunk
I have not seen anything that indicates how installing this chip would do
anything at all without also modifying the trace design and fabrication of the
PCB itself.

Also does anyone have information about the "baseboard management controller"
mentioned? I would like to understand the complexity required to MiTM a ROM or
FLASH memory read by such a controller before concluding the feasibility and
number of players in manufacturing chain required for it to work.

~~~
adrian_b
The BMC is an ARM microcontroller that has complete access to everything,
exactly like the Intel Management Engine, but the server vendors prefer a
separate chip for the same job. What is described in the article is very easy
to do by inserting a microcontroller on the SPI link that connects the BMC
with the flash memory containing the BMC programs, which are copied from there
to a RAM at boot. However, such a microcontroller would need 8 pins for at
least 7 signals: ground, power, SPI clock and 4 SPI data, to and from BMC and
flash memory. Nonetheless, 8-pin packages can be very small, e.g. 0.8 mm by
1.35 mm, i.e. only slightly larger than 1 square millimeter. So from a
technical point of view, all is easily feasible.

The problem is that it would require compromised people in several places at
the subcontractors, because the design files for the PCB must be replaced
wherever the PCB is made and in another place, at the PCB assembly, the pick &
place document must be replaced and an extra reel with the backdoor component
must be mounted on the equipment and that reel must come from somewhere else
than from the normal suppliers of the assembly line without raising
suspicions.

It can be done, but many accomplices are required. Because most of the time
the backdoor component will pass the SPI data signals transparently, it will
not be detected at any electrical testing and the usual optical inspections
are unlikely to detect such a small change.

I am using many Supermicro motherboards, so I am wondering if this story is
true. If it were true, it would not be much of a surprise, because they did
not do something really novel but they just matched what USA also did, e.g. in
the Cisco case.

~~~
cyphunk
But also if a modification of the SPI bus can result in exploitation then the
manufacturer will just patch the memory in place. Cheaper, easier, less
detectable. So... the manufacturer is not the source of attack.

------
tsuru
Has this story from 2014 ever been discredited [1] ? If not, I don't see how
any supply chain is safe from G8 powers

[1] [https://arstechnica.com/tech-policy/2014/05/photos-of-an-
nsa...](https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-
factory-show-cisco-router-getting-implant/)

------
21
They allege 30 companies were compromised with implanted Supermicro servers.

Surely one of them could show the world/researchers an infected motherboard?

~~~
simplecomplex
This. Where did all the infected motherboards go??

------
DeusExMachina
Out of curiosity: why do they use the word "untrue" instead of "false"? Are
there some legal nuances I am not aware of?

~~~
brohee
PR rep deliberately using newspeak to convey he's ordered to lie?

------
leonroy
One thing I don't see much about in the article is the supposed chip which
allegedly compromised Super Micro servers. The cover image of the piece shows
a surface mount component with three solder pads balanced on a finger tip.
Looks like a very simple SMD part to me.

On top of that Apple, Amazon and Super Micro are flat out denying this - I
suspect Bloomberg messed up here.

------
nasseri
Can someone point me to the exact laws or cases with precedent that would make
Apple and Amazon's statements illegal if the Bloomberg article were true? I'm
not doubting it, but I can't find much about this online. Pump and dump
schemes, ponzi schemes, and insider trading are illegal, and those aspects of
securities fraud are well documented online, but this more general "lying to
the public" that these companies may or may not be doing has proved trickier
to find precedence for. I am not a lawyer, and my only resource here is
google, but I think there is an assumption at play that this form of lying is
illegal, and I'm honestly not sure that it is.

I could be wayy of base here, but if its not illegal, than it would be pretty
obvious what is going on.

------
Ditiris
The oldest tactic in the book: DENY DENY DENY
[https://youtu.be/yN2gU0XU5FU](https://youtu.be/yN2gU0XU5FU)

------
fmajid
The cost of manufacture is a very small part of the cost of assembling
computers. For the iPhone, for instance, it represents less than $10 out of
the $240 or so total cost. Thus shifting production to other another locale
with higher costs would not significantly increase the price of the product,
and in any case labor costs are going up in China as well.

Thus you wonder why more production isn't being shifted from China to, say,
Thailand, Indonesia or India. Steve Jobs once said the industrial capacity to
do the work simply isn't available outside of China, in terms of skilled
people and supply chains, and that may be a big reason why.

------
upofadown
Why wouldn't the current US government be shouting this from the rooftops if
it was actually true?

~~~
Palptine
Because at least half of the people wouldn't believe them and consider this
another conspiracy theory. Even when Bloomberg pushed this there are still so
many deniers in this thread.

~~~
pixl97
Hell, how many conspiracy theory's have come out as true in since 2001.

Remember the "The telephone companies are routing all our calls and internet
data to the NSA" conspiracy.

A bunch of people said that was fake, and there was no way this could happen.

Then more evidence came out, and the same people said there was no way that
could happen, its too big of conspiracy and it would have leaked way before
then.

Then the government gave the telco's retroactive immunity for spying on the
public.

------
sounds
"In the three years since the briefing in McLean, no commercially viable way
to detect attacks like the one on Supermicro’s motherboards has emerged—or has
looked likely to emerge."

I know a commercially viable way to detect hardware attacks.

Standardized hardware designs, such as the "x86 standard," ARM IP licenses,
and more recently, RISC-V, decentralizes manufacturing and drives the cost to
commodity levels. I'm specifically proposing that the U.S. Government
appropriate the patents on whichever hardware design and declare them a
National Security asset, and then guarantee royalty-free licenses to any
company that wants to use them.

When it's no longer a big profit center, China is no longer as interested in
owning a monopoly on it.

And presto: there's no longer a monopoly on the hardware. Thus it's no longer
a guarantee that your hardware is being bent to the will of a single nation-
state. Hardware attacks can be detected as variations between the hardware
made by one nation vs. the hardware made by another nation.

The downside is that Apple can't have the same profit margins that come from
closed, proprietary hardware.

The upside is that manufacturing and process innovation (such as Intel used to
do) becomes extremely desirable. It becomes so valuable that we saw Intel
reluctant to offshore their best processes.

There: economic solution and political points, to boot!

~~~
21
Have you read the article?

Apparently one of the big factors in Supermicro success is that it has over
900 different motherboard designs, and hundreds of hardware specialists which
can customize them further to client wishes.

~~~
sounds
Your post is a non-sequitur. I'm not disagreeing that Supermicro makes
customized designs.

~~~
21
It is a sequitur in the sense that a few open-hardware motherboard designs
will not be commercially successful.

~~~
sounds
Your original post states:

    
    
       Apparently one of the big factors in Supermicro success is that it has over 900 different motherboard designs, and hundreds of hardware specialists which can customize them further to client wishes.
    

How is that a sequitur arguing about "a few open-hardware motherboard designs"
and the projections of them being "commercially succesful"?

------
PascLeRasc
Could anyone find a list of the server models that had this compromise? I
haven't seen much if any technical info in either of the Bloomberg articles.

~~~
baybal2
microblade 6128

~~~
justtopost
Source?

~~~
baybal2
Just look at the photo, and the motherboard model.

It is only used in 1 server. And given Elemental is mentioned, it must be a
blade

So that matches nicely.

------
maerF0x0
This has been posted several times and there are tons of comments:

[1]:
[https://news.ycombinator.com/item?id=18146438](https://news.ycombinator.com/item?id=18146438)
[2]:
[https://news.ycombinator.com/item?id=18138328](https://news.ycombinator.com/item?id=18138328)
[3]:
[https://news.ycombinator.com/item?id=18145645](https://news.ycombinator.com/item?id=18145645)
[4]:
[https://news.ycombinator.com/item?id=18138990](https://news.ycombinator.com/item?id=18138990)
[5]:
[https://news.ycombinator.com/item?id=18141328](https://news.ycombinator.com/item?id=18141328)

------
dannyw
I think it's now time for Bloomberg to respond. This almost feels like a
repeat "Newsweek Reveals Satoshi".

~~~
supertiger
This response piece was published one hour after the original report.
Bloomberg obviously had both ready. It would have responded already if intend
to.

------
andrewstuart2
For what it's worth, this exact article is linked directly from within the
original article, and is addressed.

> Read: Statements from Amazon, Apple, Supermicro and Beijing

> The companies’ denials are countered by six current and former senior
> national security officials, who—in conversations that began during the
> Obama administration and continued under the Trump administration—detailed
> the discovery of the chips and the government’s investigation.

------
bogomipz
The article states:

>"Somewhere in the Linux operating system, which runs in many servers, is code
that authorizes a user by verifying a typed password against a stored
encrypted one. An implanted chip can alter part of that code so the server
won’t check for a password—and presto! A secure machine is open to any and all
users."

I realize the intended audience for this article is not a technical crowd but
can someone walk me through in practical terms how such a chip might subvert
the /bin/login binary?

~~~
stordoff
[https://jhalderm.com/pub/papers/ipmi-
woot13.pdf](https://jhalderm.com/pub/papers/ipmi-woot13.pdf) suggests one
(fairly visible) way:

> Another common feature is virtual USB disk media, which can be used to
> infiltrate or exfiltrate files or to provide new boot media. The combination
> of these capabilities and remote power cycling would allow an attacker to
> seize control of most common server configurations. For instance, they could
> restart the system and boot from a virtual live CD, then directly copy or
> modify data on the host’s storage devices

If the IPMI has write access to disc and/or main memory, you can do it more
directly - drop a new /bin/login on the disc, or patch it in memory (similar
to the LoJax attack:
[https://news.ycombinator.com/item?id=18090651](https://news.ycombinator.com/item?id=18090651))

~~~
bogomipz
Thanks. So presumably these backdoor chips are attached to either the I2C or
SMBus. That's the part I was missing. Cheers.

------
DenisM
It's interesting that the numerous sources all decided to confine in
Bloomberg. How did they all know to go to the same paper? And if some of them
went to other papers then we should hear about it tomorrow, I guess.

------
oropolo
Are there facilities anywhere in the US where motherboards can be
manufactured?

------
bovermyer
You know, I would watch a movie with a plot like this.

~~~
RRRA
two hours of chip decaping and circuit reversing and firmware dumping! :P

~~~
kalleboo
I would watch a YouTube channel of that

------
bnjmn
This sentence, wow:

> Two of Elemental’s biggest early clients were the Mormon church, which used
> the technology to beam sermons to congregations around the world, and the
> adult film industry, which did not.

~~~
fredley
That sentence is not in this article. Perhaps you meant to post it here:
[https://news.ycombinator.com/item?id=18138328](https://news.ycombinator.com/item?id=18138328)

~~~
bnjmn
Ah you’re right!

