
The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple - Osiris30
https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies
======
dang
For those interested, there are additional threads discussing Apple's and
Amazon's denials:

[https://news.ycombinator.com/item?id=18142277](https://news.ycombinator.com/item?id=18142277)

[https://news.ycombinator.com/item?id=18138990](https://news.ycombinator.com/item?id=18138990)

[https://news.ycombinator.com/item?id=18143569](https://news.ycombinator.com/item?id=18143569)

------
lmilcin
I have worked in card payment industry. We would be getting products from
China with added boards to beam credit card information. This wasn't state-
sponsored attack. Devices were modified while on production line (most likely
by bribed employees) as once they were closed they would have anti-tampering
mechanism activated so that later it would not be possible to open the device
without setting the tamper flag.

Once this was noticed we started weighing the terminals because we could not
open the devices (once opened they become useless).

They have learned of this so they started scraping non-essential plastic from
inside the device to offset the weight of the added board.

We have ended up measuring angular momentum on a special fixture. There are
very expensive laboratory tables to measure angular momentum. I have created a
fixture where the device could be placed in two separate positions. The theory
is that if the weight and all possible angular momentums match then the
devices have to be identical. We could not measure all possible angular
momentums but it was possible to measure one or two that would not be known to
the attacker.

~~~
zelon88
Wait a minute... So your company has a Chinese equipment supplier, finds out
that the supplier is tampering with your purchased equipment, and your
solution is to add criteria to the incoming inspection?

No wonder China keeps screwing with you guys. You aren't supposed to eat that
cost! Write a PO with tons of fine print that says "We will disassembly units
at random for compliance inspection. Non compliant products will be returned
at the suppliers expense." And then add a clause that says ">3 non-compliance
events in under # months will result in the entire PO (10 or 20 units) being
returned and all contracts cancelled."

I cannot believe you are getting screwed by a company you choose to do
business with and yet you eat cost to ensure they aren't screwing you. Just
get a new supplier! Do on-site inspections at their facility. This is nuts.

~~~
lmilcin
There were other considerations like the fact we were actually buing it from
large reputable company and what happened was that some employees were doing
it with no involvement of the company.

The fact is, doing any kind of hardware production in China, you have to be
aware Chineese have different value system and you would not be suited doing
any business if you throw tantrum at any sign of apparent dishonesty (assuming
the company was involved which they could not have been as they have been the
ones damaged the most).

If the company does screw you (like replacing components for something
cheaper) they typically will not be thinking they are doing anything wrong.
They are just testing if you notice and if you do not they will say it makes
no difference for you but saves them costs.

The way to work is then verify everything and politely point it out. If you
notice they will correct apparent mistake.

~~~
hinkley
This still sounds nuts to me.

Two thoughts:

> they typically will not be thinking they are doing anything wrong. They are
> just testing if you notice and if you do not they will say it makes no
> difference for you but saves them costs.

This is how children behave. Still feels like you’re rewarding bad behavior.
You’re enabling them.

I think it’s more that you’ve valued the low per unit price over having a
healthy contract. Your vendor is incentivized to make all of their money off
of externalities and your company think it’s cheaper to outwit them than to
demand QC on their end. Whose brand will be sullied if you miss some of these
units and a customer finds the spyware? Not theirs.

And second thought, shouldn’t serial numbers be coming off the line in
ascending order? The kind of work they are doing would require taking parts
off the line and putting them back later so odd lots of SNs are the ones you
need to verify.

You could also be mandating how many boards are allowed or that the SNs go on
early in the build process.

~~~
chaostheory
This is just how low trust societies function for hundreds if not thousands of
years. This is also why families tend to be much stronger in these places. The
American idea of a strong family is different from what I'm describing, since
it just centers on immediate family. My definition of strong family bonds,
implies both closeness and dependency not just on your immediate family, but
even on your distant cousins on both sides. It's close if not the same to how
Mediterranean and Hispanic cultures view families. I guess you can call it a
clan mentality. I'm now wary of countries where families are really strong ie
clans. It's not always the case, but it tends to mean the rule of law is weak
and corruption is crazy.

When you can't even trust that food is real, and when you have to bribe even
low level provincial government employees (can you imagine needing to bribe
DMW workers or even police?) - what else is an average person to do except to
treat it as the norm in order to survive? Unless you're an elite, the only
other option is to leave, and not everyone has that choice. Very little is
considered wrong over there, aside from criticizing the powers that be; as
long as it helps your clan. It doesn't help when Western companies and
governments look the other way, as we see in past news stories and even the
comments here.

I'm not condoning the behavior; just explaining.

~~~
ddoolin
Not much to add, but want to note that you're spot on. I've encountered this
SO many times over the years and this is always the explanation I'm given. At
first it just made me indignant, too, and to some degree I guess it still
does, but now I just try to let it go.

------
ThePhysicist
Is there an article that describes a bit more in detail what the chips
actually did (or were capable of doing)? They only say "the microchip altered
the operating system’s core so it could accept modifications.", which I might
interpret as circumventing signature checks to allow installing modified
firmware on the systems? But how does the chip connect to the network and how
does it receive commands?

That said, it's pretty scary that you can hide so much malicious functionality
in such a small device, makes me wonder what might be hidden in my Lenovo. In
any case it speaks highly of the auditing firm that they were able to locate
this. I wonder if they performed an x-ray analysis of the board, as given the
size of these chips it should be possible to embed such devices in one of the
internal layers of the board as well, making them essentially invisible to
optical inspection.

~~~
RL_Quine
SuperMicro hardware has very extensive IPMI integration into the mothebroard,
which amongst other things can take over and inject frames into the network
interface, emulate a VGA device, talk to the CPUs serial lines directly, flash
firmware, control the state of a number of physical devices- and this is what
it supports just from the web interface it presents by default with the
password "ADMIN:ADMIN". My money, based on experience attempting to harden
their devices, is that any modification were injected into the IPMI hardware
where most of this was already supported.

This stuff ends up being extremely difficult to disable. The naive approach
would be to not connect to the dedicated NIC that's indicated on the back and
in the instruction manual, but if you do this it masquerades onto the main NIC
invisibly to the OS and DHCPs on its own to open up an administration port,
web interface, and some assorted call homes. You have to explicitly tell it to
use the non-connected port, change credentials, and modify it so that it is
not accessible within operating system as well. Hopefully while the machine is
offline to prevented any automated scanning finding it within your network.

The number of times I'd end up nmaping our local networks and being able to
remotely access production hardware with an interface that allowed me to reach
this interface was maddening. The system is basically designed to be as
insecure as possible by default, and allow for the maximum possible persistent
threats with BIOS flashing, IPMI flashing, and other completely nu-
authenticated avenues exposed. The course of action was always just to write
off the hardware and bin it, because god knows what impact you could actually
have using that interface.

~~~
wtfstatists
But without the IPMI kernel modules loaded, IPMI is harmless, right ?

~~~
0xfeba
No, it boots prior/separate to the board itself. It's basically a mini-PC
embedded in the board that has it's own CPU/Memory and tentacles attached to
everything in the mainboard.

------
erostrate
They attacked the Base Management Controller. There's an article by Bruce
Schneier from 2013 warning about exactly this attack. Quoting:

"Basically, it's a perfect spying platform. You can't control it. You can't
patch it. It can completely control your computer's hardware and software. And
its purpose is remote monitoring. At the very least, we need to be able to
look into these devices and see what's running on them."

[https://www.schneier.com/blog/archives/2013/01/the_eavesdrop...](https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html)

~~~
craftyguy
> You can't patch it.

Sure you can. OEMs regularly release patches for platform BMCs.

~~~
erichocean
Not sure about you, but I'm not an "OEM".

~~~
craftyguy
Well, _you_ cannot patch the vast majority of the software in your computer
(assuming you are like the vast majority of users using proprietary crap for
everything). That does not mean it is all unpatchable. If supermicro care,
they could release a BMC update, for example.

------
blackrock
This reminds me of that old story about the Xerox copy machines that the
Soviet Union bought.

Where each unit was planted with a image recorder. And for years, the American
spy agencies had a great laugh, that they were able to intercept all the
documents that the Russians made a copy of.

Back then, this was an off-network infiltration. Where the copied images, were
retrieved during regular servicing intervals by a Xerox technician.

~~~
y04nn
Or the IBM Selectric Typewriter implant.

[http://www.cryptomuseum.com/covert/bugs/selectric/](http://www.cryptomuseum.com/covert/bugs/selectric/)

~~~
whydoineedthis
One of my personal favorites.

------
sandebert
"Two of Elemental’s biggest early clients were the Mormon church, which used
the technology to beam sermons to congregations around the world, and the
adult film industry, which did not."

Well played, Bloomberg. Well played.

~~~
airstrike
To be fair, removing the last three words could make it sound like the Mormon
church was beaming sermons to the adult film industry which would probably be
even worse...

------
dhx
Interesting about how some of the trojan chips were hidden in PCB substrate
layers to avoid optical detection. For a much stealthier approach again, it
was shown in 2013 that slighting changing the dopant mask for a few gates on
Intel Ivy Bridge chips could render RNGs insecure[1][2]. Mask inspection
systems[3] are used to detect mask manufacturing defects, but the question
then is, do those inspection systems use Super Micro motherboards?

[1] Paper:
[http://www.emsec.rub.de/media/crypto/veroeffentlichungen/201...](http://www.emsec.rub.de/media/crypto/veroeffentlichungen/2015/03/19/beckerStealthyExtended.pdf)

[2] Presentation:
[https://www.iacr.org/workshops/ches/ches2013/presentations/C...](https://www.iacr.org/workshops/ches/ches2013/presentations/CHES2013_Session4_3.pdf)

[3] Example system:
[https://www.lasertec.co.jp/en/products/semiconductor/mask_se...](https://www.lasertec.co.jp/en/products/semiconductor/mask_semicon/matrics_x8ultra.html)

------
unethical_ban
We need to get the fuck out of China. It is becoming less credible to throw
our hands up and say "China has all the silicon manufacturing, guess we have
to put up with it!" \- this is national security, both directly via hardware
in the DoD and through our economic stability.

Saying "Well the Chinese companies are different" or "It's just rogue
employees" or "We just have to accept it" is not good enough.

We need a little bit more economic nationalism and realize that this shit
matters. How is it so shocking? How did a Canadian security firm find
something before the CIA/DoD? How is this not a NatSec program to be
inspecting this stuff through shell company orders and honeypots, like we have
on the Internet?

~~~
nemo44x
How do you know that the DoD/CIA isn't behind this?

~~~
hart_russell
Because it came from Chinese manufacturers. Did you read the article?

~~~
nemo44x
Why does that matter at all? How do you know the CIA didn't use Chinese spies
to do this? Although from a different agency, the motto "Nothing Is Beyond Our
Reach" is telling.

------
awayenberg
So the chip shown in the article looks like a typical SMD balun, it is a type
of transformer used to adapt impedance between two transmission line. It’s
designed to replace a series a lumped element (capacitor, inductors,
resistors) normally used for impedance adaptation (in a T or Pi network). The
most common used for the device is directly between an antenna an a RF front-
end to serve as an antenna tuner.

Technically you could embed and power an RF front-end inside a “flavored”
balun to intercept or alter any communication passing through that front-end
or even use the antenna to communicate during the down time. So this literally
would hack Wifi / Bluetooth at low level and inject code and at the same time
create a mesh network of malicious devices to relay information. Welcome to
IoT Cyberwarfare.

But this clever hack is probably not limited to RF and is likely to also be
embedded in transformers used for isolating Ethernet lines. Common mode chokes
(some SMD chokes also look like the chip they are showing) or even some
integrated ESD protection solution would be an ideal target as they are
inserted in series of the signal.

~~~
07d046
I don't know enough about this, but isn't the article saying that the
appearance is deceptive:

 _The chips on Elemental servers were designed to be as inconspicuous as
possible, according to one person who saw a detailed report prepared for
Amazon by its third-party security contractor, as well as a second person who
saw digital photos and X-ray images of the chips incorporated into a later
report prepared by Amazon’s security team. Gray or off-white in color, they
looked more like signal conditioning couplers, another common motherboard
component, than microchips, and so they were unlikely to be detectable without
specialized equipment._

~~~
rasz
If the photo in the article is real you wouldnt be able to identify this
component as compromised "just" by visual (even xray augmented) inspection.
TVS Diode Array looks the same from the outside, whats more its build in same
way with silicon die embedded in tis structure. Other than signal analysis it
would take decapping every single component of a motherboard to find this
implant.

------
kjullien
It's been a few years I've given up on the idea of privacy with technology.
The number of security flaws that get discovered daily is only the tip of the
iceberg. I'm pretty sure some governments (or organizations) have had
backdoors, be they hardware or software, in place for more than 20 years. We
simply don't know about it yet (and probably never will). Would that actually
be that far-fetched? I think not sadly. Even the Intel Spectre et Meltdown
fiascos are a sign that we have no idea how to actually secure this stuff. And
that's normal, the very definition of IT security is that nothing can be
secure. Take the whole antiquated concept of processor rings for instance, we
are adding a new level every other year now it feels like... I find it way
more interesting (even if it is ultimately "worse") to adapt to the mentality
that "nothing is secure" than "let's try and make it secure", which as stated
is in itself a fallacy...

~~~
bootsz
The more I understand software the less I trust it (given the current state of
engineering practices). Meanwhile all my friends/family are scrambling to
install all the latest new "smart home" gadgets and I just look like a
paranoid kook trying to talk them out of it.

~~~
dullroar
This. I've even had an in-law say, "My brother works for the Defense
Intelligence Agency, and he uses smart devices in his home, so they must be
safe!", with no consideration that tech may not be his specialty, or he
doesn't follow the daily IoT fiascos, or maybe he just thinks he won't get
hacked. Dunno. Meanwhile, my year-old thermostat still wants me to connect it
to wi-fi, and that will never happen.

~~~
acct1771
Wait, can you use a Nest without connecting it to the internet?

~~~
dullroar
Dunno - my thermostat is not a Nest. Works great without the internet.

------
EwanToo
Amazon are going all out on the denial

[https://aws.amazon.com/blogs/security/setting-the-record-
str...](https://aws.amazon.com/blogs/security/setting-the-record-straight-on-
bloomberg-businessweeks-erroneous-article/)

~~~
teddyfrozevelt
Apple is very strongly denying it as well

[https://www.bloomberg.com/news/articles/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-
amazon-apple-supermicro-and-beijing-respond)

~~~
odorousrex
What is more telling for me is that Apple has cut off all contracts with the
company involved and will have nothing more to do with them - despite their
denial.

~~~
Atlas26
They cut off contact for a separate reason - 2000 (IIRC) supermicro boards
which _did_ have issues.

~~~
jessaustin
If they wanted to cut ties for a different reason, execs could have just said
"find a problem with some supermicro boards, no matter how much it costs to do
so, and never repeat this conversation". They are complicated human-designed
and -made artifacts; of course _some_ problem exists. This is known in other
situations as "parallel construction".

------
RL_Quine
Fascinating. A company I used to work for dismissed this sort of situation as
a problem and ended up using SuperMicro boards in a security crucial product.
Their hardware has always been notably very crappy, with the IPMI interface
defaulting to world-open unsafe parameters, but I'd not expected it to be this
cleverly hardware backdoored. It's possible to neuter Intel ME, but that's
only a small comfort with these motherboards.

------
vezycash
>Amazon’s security team conducted its own investigation into AWS’s Beijing
facilities and found altered motherboards there as well, including more
sophisticated designs than they’d previously encountered.

>...the malicious chips were thin enough that they’d been embedded between the
layers of fiberglass onto which the other components were attached

>...that generation of chips was smaller than a sharpened pencil tip, the
person says.

I hope this corrects the mistaken believe that China can't home grow
sophisticated tech.

~~~
adventured
> I hope this corrects the mistaken believe that China can't home grow
> sophisticated tech.

There's nothing particularly sophisticated about what they did, especially
given what China has access to as one of the central hubs of tech
manufacturing. There are two dozen nations (or more) that could do this from a
strictly technical standpoint (few have the kind of required supply chain
access to pull it off at scale in actuality). It's the audacity that is
primarily impressive. China is encouraging all of their richest trading
partners to further isolate them when it comes to supply chain and advanced
tech. They're confirming for the 107th time what everyone already believes.

~~~
vezycash
You're right.

One question though. Would a politically correct, by-the-book US president
have had the balls to sanction China? Considering such sanctions could affect
the US economy.

~~~
Eridrus
> Would a politically correct, by-the-book US president have had the balls to
> sanction China?

Does Trump even have a goal in mind with his tariffs? It seems like he just
wants to score political points, rather than achieve any actual outcomes.

~~~
forapurpose
The goal, explicitly stated and often acted on in many contexts, is to disrupt
international order and cooperation (including trade), and promote
nationalistic competition by all countries. Recently his UN speech, for
example, advocated it.

------
jtbayly
This seems either incredibly suspicious to me or indicates that Apple is _way_
more protective of security and distrustful of the USGov than I would have
thought: “Because Apple didn’t, according to a U.S. official, provide
government investigators with access to its facilities or the tampered
hardware, the extent of the attack there remained outside their view.”

Why wouldn’t the FBI be able to compel Apple to turn over such key evidence in
such an important investigation? Or why wouldn't Apple be willing to do it
even without compulsion?

~~~
jandrese
Apple seems cagey about its reputation with the Chinese government. Their
balls are directly in the Chinese government's hands, they don't want to do
anything that might make the government upset.

Beyond having their manufacturing there, it's also the biggest untapped market
in the world.

------
blackrock
I don't know which is more disturbing here.

That the Chinese military is technically competent enough to pull off such a
thing.

Or that they are incompetent enough, to not secure their own back doors and
networks, and allowed the FBI, NSA, and other American government
organizations, the ability to counter-hack them, and monitor all their
internal communications.

The truth is somewhere in between.

So, this article is basically saying: China got clever, and spied on us. We
spied back on them, and we also hacked all of their internal communications
too, so basically, we own them.

Well, at least America is finally admitting, that they too, actively spy on
other countries. And can easily infiltrate any other country that they want.

~~~
gkanai
> America is finally admitting, that they too, actively spy on other
> countries.

The Snowden revelations made that very clear to all years ago, no?

~~~
matthewwiese
One doesn't even need Snowden to know this. Given even a little reading of the
IC's (intelligence community), and more specifically CIA's, history. What else
do you think they do all day, if not for spying on other countries?

As mercutio2 mentioned in a reply to your same parent, the controversy w/r/t
Snowden is _domestic_ spying.

~~~
freeflight
> As mercutio2 mentioned in a reply to your same parent, the controversy w/r/t
> Snowden is domestic spying.

In the US maybe, but most of the US's supposed "allies", and their citizens,
would probably disagree with that interpretation.

------
zaroth
I hate leaving a brief unsubstantiated comment, but this is the new Cold War
and I think it needs to be acknowledged as such and brought out of the dark.

I don’t want my intelligence agencies fighting a war we aren’t even publically
acknowledging and subjecting to public congressional oversight. I don’t trust
Congress, but I trust the NSA/CIA far, far less.

The economic sanctions against China are disproportionality minuscule if they
are meant as a response to this level of attack.

If this is true, it should be proven in a court of law and trade sanctions
should be a positive integer trillions of dollars.

------
mfer
Anyone remember the hack of Apple via Supermicro firmware....
[https://arstechnica.com/information-
technology/2017/02/apple...](https://arstechnica.com/information-
technology/2017/02/apple-axed-supermicro-servers-from-datacenters-because-of-
bad-firmware-update/)

~~~
clear_dg
This is what Bloomberg's article refers to as "unrelated reasons" for Apple
cutting ties with Supermicro in 2016.

> Three senior insiders at Apple say that in the summer of 2015, it, too,
> found malicious chips on Supermicro motherboards. Apple severed ties with
> Supermicro the following year, for what it described as unrelated reasons.

And then as an "unrelated and relatively minor security incident" later on.

------
TheSpiceIsLife
> One country in particular has an advantage executing this kind of attack:
> China, which by some estimates makes 75 percent of the world’s mobile phones
> and 90 percent of its PCs.

Intel and AMD are both USA based companies.

Is it conceivable their processors contain backdoors in a similar vein?

~~~
chx
No, it is totally inconceivable AMD and Intel CPUs are backdoored this way.
Inserting a microcontroller somewhere on the Ethernet traces and then using
IPMI is not as sophisticated as this article wants to describe. Sophistication
is necessary from the payload to be stealthy but not the hardware.

There are at least two problems with China messing with CPUs: a) they are not
made there. TSMC is Taiwan, the Asian parts of Global Foundries are in
Singapore, Intel is manufacturing CPUs mostly in the USA with some in Ireland
and Israel, only 3DNAND / 3DXPoint ICs are made in China. b) the hardware
knowledge to change here, well, think of it, the microcontroller on the
motherboard is on the few millimetres scale, the parts here are on the few ten
nanometres scale so as a rough but not unjust quantifications would say it's
100 000 times harder.

Whether the platforms are hackable / hacked, that's harder to say thanks Intel
ME and such but the CPUs aren't.

~~~
simias
I don't think the parent was talking about this specific type of attacks but
other types of hardware backdoors. I think the answer to that is very
obviously yes, and given what we know about intelligence agencies I'm even
willing to go as far as saying that it is likely (or at least, if you have
reasons to be worried about Uncle Sam getting to your stuff you should
consider it a very real possibility).

Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into
the behemoth that's a modern CPU or embedded SoC would be almost trivial.
They'd also have great plausible deniability in case they're found, if
something like SPECTRE was an intentional backdoor how would you ever prove it
for instance?

~~~
wepple
> Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into
> the behemoth that's a modern CPU or embedded SoC would be almost trivial.

I suspect putting in a backdoor would be difficult _because_ they are complex.
Wouldn’t it be far too easy for the backdoor to inadvertently cause
reliability or performance issues? And the bug would have to be useful enough
to warrant potentially destroying the semiconductor business of a nation, not
just some difficult to trigger side channel.

------
flyinghamster
It makes me wonder how compromised the prosumer/enthusiast motherboards are.
All of my important home systems are homebuilds using off-the-shelf
motherboards purchased in person, and lack any IPMI access. Sure there's AMD's
PSP (grumble, grumble) in my Ryzen box, but I can't help wonder what might be
lurking on the motherboards, or for that matter in my network switch, router,
access point, etc.

I've sometimes thought about picking up some type of used rackmount server,
but hacks like this give me cold feet - aside from the usual issue of 1U/2U
boxes sounding like jets taking off.

------
yourapostasy
The denials by Amazon, Apple, Supermicro and the Chinese Ministry of Foreign
Affairs [1] are relatively _pro forma_ , both directed by respective nation
states involved in this matter. One of the reporters interviewed on Bloomberg
noted Amazon and Apple could be directed by US national security interests to
deny to protect the ongoing US investigation. Supermicro could similarly be
directed by Chinese national security interests to protect plausible
deniability.

There was a sense of _realpolitik_ by one UK guest commentator on Bloomberg,
comments along the lines of "hey, spying happens since time immemorial, put on
some big boy pants, yes there is shock but not horror when the Snowden
revelations came out, the US does it, too, _etc._ ". I disagree with this
sentiment, as while the attack was quite targeted, it puts into question a
quite large supply chain network.

Kudos to Bloomberg putting in the 12+ month investigative journalism to pull
off this scoop. Yet another validation of the reasons I subscribe to
listening/watching them.

[1] [https://www.bloomberg.com/news/articles/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-
amazon-apple-supermicro-and-beijing-respond)

~~~
adventured
> Supermicro could similarly be directed by Chinese national security
> interests to protect plausible deniability.

Supermicro is an American corporation, headquartered in San Jose. They're not
directed by Chinese national security interests, they'll do _anything_ the US
Government tells them to do when it comes to US national security.

~~~
yourapostasy
I stand corrected, thanks.

Supermicro's vendors they use in their Chinese manufacutring are the likely
vectors for the implantation.

------
bhauer
As someone who has used a few SuperMicro boards in custom builds, I would like
to know which motherboard models have been affected. It's not clear to me from
the article whether that is even known at this time.

I'd also like to know of a recommended work-around. I have always had IPMI
"disabled" as much as I can; assigned a non-routable IP address in the BIOS in
case it flips to a failsafe mode by piggybacking on the main NICs. But is that
sufficient? Is BMC ~= IPMI?

Has anyone found more technical details yet for system administrators who
don't work at the thirty companies contacted?

~~~
parliament32
Technically BMC is the hardware and IPMI is the software, but yes they're the
same thing.

It's good to change the network port settings to "dedicated" so it doesn't
failover... but if it was malicious, and it clearly has the ability to
piggyback on your production interface, why wouldn't it? There's honestly no
way to secure this without cutting off your hardware from the internet
completely (or firewalling it off similarly).

------
lisper
This is just the hack that was discovered because there was macroscopic
evidence of it. All it would take to pull off a similar hack that was
undetectable is one well placed mole in the company that designed a key piece
of silicon or software.

~~~
jiveturkey
no way, an intentional design-level plant would have to pass through many
eyeballs. a single mole wouldn't be enough.

~~~
lisper
I have worked in both chip design and security so I feel well qualified to
make this assessment: a single mole in the right place who knew what they were
doing would definitely be enough. Security holes get past design review all
the time _by accident_. A skilled hacker could easily insert an intentionally
obfuscated one that would escape detection.

~~~
glbrew
Yup. And that single well placed mole could have 100 or 1,000 security experts
working on their behalf. It is not as though the mole would have to design the
hack, only pass necessary documents to a military grade hacking group then
implement their modifications.

------
Narkov
Why aren't these attacks constrained by normal corporate firewalls? How does a
random server on a navy ship start contacting baddie.china.com without raising
red flags?

~~~
RL_Quine
Effectively nothing can be constrained by a whitelisting firewall if you have
a sufficiently bored actor. You can smuggle data through a variety of benign
looking protocols, things that wouldn't matter in the least generally. Your
average server contacts hundreds of different public NTP servers, binary
repositories, domain name servers every day. If the keys to the kingdom are a
32 byte ECDSA private key, you've lost if you think you can protect this from
reaching the outside world.

A method that wouldn't show up on any firewall in the world is simply to delay
or drop certain SYN packets. Even if you only intended to transmit a bit at a
time through this, any unauthenticated host on the internet could use this
without raising any suspicion or even printing log lines in most environments.
As soon are you're making an assumption that you're trying to prevent what's
inside from getting out things become substantially closer to impossible than
anybody would like.

~~~
Narkov
Firewalls in high security environments aren't just port/protocol based. You
lock everything down - source ip/port and destination ip/port. You should know
where it is coming from and where it is going to.

Navy ships don't upload via Dropbox.

~~~
RL_Quine
In the parent I described a system which would be able to communicate through
those restrictions to another compromised host (remember we're assuming
everything is compromised for the sake of this article, which actually seems
like a good assumption now).

~~~
FooHentai
Networks where security is taken seriously implement data diodes, and this
attack vector is mitigated.

------
dustinmoorenet
We need open source hardware designs that can be built locally (where ever
your local might be). This black box hardware crap has to stop. Smart people
who know how all this works need to dump all their knowledge in to a design
and a process. Trade secrets are keeping us not only limited in choices but
exposed to bad actors who can control a link in the supply chain.

~~~
MrEfficiency
This sounds beyond complex to manufacture. Especially at Fortune 500 levels.

DIY? Well its going to be larger, hotter, more failures, and probably more
expensive.

There is a reason companies specialize.

------
ryanmarsh
My gut feel about this for many years has been outsourcing chip manufacturing
to another country is a serious security risk. This was a hardware device,
hard to detect as it was, how much code is in the chips we already expect on a
motherboard?

It seems flat out foolish for one country to own the world’s computer
manufacturing.

------
ghobs91
This is the logical conclusion of companies giving in to shareholders frothing
at the mouth for increased profitability by any means necessary. Manufacturing
crucial components like motherboards in China is just absurd.

------
zwaps
When will this stuff finally have consequences for China?

Their behavior, not their communication, has been overtly hostile for a while.
Yet, very few politicians openly adress the issue.

~~~
baybal2
>When will this stuff finally have consequences for China?

Never, unless hardware manufacturing will take off somewhere else.

~~~
blake_himself
Somewhere poor, where labor is cheap and people are still susceptible to
bribes?

~~~
baybal2
Rather not, China's economic rise started in Jiang's era was both due to it
being cheap, _AND_ due to Jiang's era officials being more competent and
business friendly than those of an average bantustan (really sorry having to
use the term)

------
hef19898
I am sincerly impressed by the amount of supply chain analysis and operative
supply chain management that went into that hack. And once the Chinese
identified a distribution node in that particular supply chain, supermicro,
they opted for a brute force attack by seeding these backdoor chips into
supermicros servers and wating where they ended up. That was one hell of a
hack.

It also gives you pause. Did that happen only once? For how long? Where esle
did these chips and servers end up in the end?

~~~
anonu
Im impressed as well. But I'm sure its not as impressive as what the NSA is
capable of. If you're reading an article about this - its because the US
government wants you to know about it. The US has been doing this kind of
stuff well before the Chinese or anyone else... And yes, totally agree that
this is probably fairly common place. Some comments have shown that this
happens with run-of-the-mill hardware like credit card scanners.

This stuff also reminds me of the Snowden revelations of hard-disk firmware
hacking. Very similar conceptually to what is described in this article -
albeit without custom hardware.

~~~
hef19898
Every major major nation can do this stuff one way or the other. But somehow I
am used to software hacks by now. That someone is basically managing aphysical
supply chain (with suppliers, production and all of that) within an existing
one is a first for me. The only thing similar are stories of the Italian mafia
getting stuff by customs by duplicating shipping containers, still what the
Chinese did is different level. But the, being a supply chain guy, I have an
easier time understanding this as compared to what NSA and GCHQ are doing.

------
blackdogie
I’d expect that this type of parasite hardware attack isn’t unique. Securing
the supply chain is even more important than ever.

Apple’s response isn’t great IMHO.

------
octosphere
Just as a sidenote: X-raying PCBs and then diffing them against clean PCBs is
a worthwhile thing to do if you're concerned about hardware backdoors, or
'interdiction' of hardware in a supply chain. I do this sometimes when
ordering super-critical equipment like Thinkpads from the U.S as you never
know what lurks on the motherboard (keyloggers, etc).

I have a clean Thinkpad that I use to compare against potential backdoored
devices. So far I haven't spotted any differences in the PCBs. I guess the
intelligence agencies have not marked me as important enough to target. That
being said, I imagine there are people working in the cryptocurrency space who
have a lot to hide (if you own their boxes, you could be looking at thefts
worth millions of dollars, or whatever the equivalent is in the cryptocurrency
they are developing).

~~~
xur17
> X-raying PCBs and then diffing them against clean PCBs is a worthwhile thing
> to do if you're concerned about hardware backdoors, or 'interdiction' of
> hardware in a supply chain. I do this sometimes when ordering super-critical
> equipment like Thinkpads from the U.S as you never know what lurks on the
> motherboard (keyloggers, etc).

I'm curious - how does one go about buying or getting access to an x-ray
machine (and how much does that cost)?

~~~
cedivad
I would go to a local SMD assembly house and ask to use theirs.

------
ahmedalsudani
"In one case, the malicious chips were thin enough that they’d been embedded
between the layers of fiberglass onto which the other components were
attached, according to one person who saw pictures of the chips."

Wow.

------
HillaryBriss
> _That left the decision about where to build commercial systems resting
> largely on where capacity was greatest and cheapest ... “You can have less
> supply than you want and guarantee it’s secure, or you can have the supply
> you need, but there will be risk. Every organization has accepted the second
> proposition.”_

Once the critical mass of electronics manufacturing moved outside the US and
manufacturers outsourced the bulk of work, there was no longer any
alternative.

Economists say this is ok or even really good for global welfare.

But do _any_ economic models accurately account for this type of economic
damage, this type of cost to government, consumers, companies like Supermicro,
Apple, Amazon, etc?

------
ThomPete
And this is one of the major reasons why I support that the US is going after
China.

It's a disgrace that previous politicians from both the EU and the US have
just let China steal from the west through technology transfer and not
actually enforcing IP.

------
wslh
Side note: this is why private blockchains for supply chains don't have any
sense because there is no way to probe that you are delivering what you say.

------
sschueller
This sounds like something out of a movie. I would like some technical details
how this is supposed to work with only a 6 lead chip.

~~~
orbifold
Most likely it is using SPI
[https://en.wikipedia.org/wiki/Serial_Peripheral_Interface](https://en.wikipedia.org/wiki/Serial_Peripheral_Interface),
that requires four pins and the two remaining ones are power and ground. SPI
what is used to access EEPROMs and flash memory, so an attack that you can do
is daisy chain such a device in the path to the EEPROM the board management
controller uses as its firmware storage. Then you can very easily insert your
own instructions and get the board management controller to execute whatever
you want.

------
calebm
So it’s sounding like the Chinese government may have invested a large portion
of its war chest on subsidizing computer manufacturing in order to gain this
type of strategic position. It feels like chess, and seems brilliantly scary.

------
ksec
I wonder if SuperMicro being delisted from Stock market has anything to do
with this? Where their "Accounting Errata" were merely cover up.

~~~
gomox
I'm not sure exactly what "delisting" entails, but if my billion dollar
company was about to be in the news for having its supply chain compromised by
a foreign state actor, I would be happy to not see the stock crash triggered
by a public order book.

------
crwalker
Stories like these make me think there is a real need for electronic devices
made entirely in the US (or your preferred trusted country).

Products like the Yubikey are apparently made in the US / Sweden, but I wonder
how many components are actually made elsewhere and just assembled in these
countries. Are there any good examples of consumer devices actually made
entirely in the US down to the microchip?

------
ianamartin
Like data breaches, this will be big news that _some_ devices were compromised
at some time. Then followed by a slow dribble here and there, that, oh yeah,
well more than we initially thought. Then a few months later it will be,
"Yeah, it's really a lot more than we thought." Maybe a year from now it will
basically turn out to be everything. But numbers get so big so fast people
don't really know how to effectively parse the difference between 20 million
devices and 20 billion devices. So it won't matter to most people.

And really, it shouldn't matter to most people. There's nothing the vast
majority of people can do about this. It's not like we can just go buy stuff
made in the U.S.

------
xtrapolate
Which companies performed the security audits?

------
tmaly
Are those chips still sitting in functioning DoD servers?

How have they mitigated this issue, and where will they source new hardware
from?

------
RobertSmith
China is not a friend to the US alone, many countries hate them for their
unethical practices

~~~
i_am_nomad
The more I speak to scientist and engineers from sub-Saharan Africa, the more
hate I hear for China. It’s hard to comprehend.

------
User23
Building in China may be cheaper, but that’s a stupid consideration. After all
the Greeks charged the Trojans absolutely nothing for a fabulous wooden horse.

------
jiveturkey
what they describe isn't possible. which is perhaps why Apple et al. are
easily able to refute it.

what actually occurred must be something a bit different, that they didn't
understand and/or aren't conveying accurately or didn't receive accurate info
due to it being classified. or you know, because the general press is still at
movie technology ... the hacker is in the building!

A chip the size of a pencil point can't be inserted anywhere useful or do
anything meaningful. Something that small amounts to a discrete component, not
a "chip", if packaged. If bare die, it still can't be vastly complex and needs
to be mounted in a way that is very obvious (epoxy blob).

So, it must be an additional discrete that inhibits the burning of a W/O fuse
in the BMC or some other management function. Thus preventing the disabling of
some debug function.

There is no way it has communication capability, etc, on its own.

EDIT: ah, @baybal2 perhaps figured it out. it's likely flash memory and the
bmc already has provision to read it as the "recovery" flash.

------
amykhar
Seems like a bit of Karma to me. Outsource to try to get cheap labor rather
than pay workers a living wage, and this is what you get.

------
amai
The Bloomberg article might well be the better explanation for the recent
problems of Supermicro:
[https://www.theregister.co.uk/2018/03/14/supermicro_praying_...](https://www.theregister.co.uk/2018/03/14/supermicro_praying_for_nasdaq_time/)

------
slim

      The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.” 
    
    

Essentially China ils saying "it was not me". Plausible

~~~
rocqua
I Read that as "The US is also attacking our hardware supply chains". That is,
the statement concerned supply chain attacks in general, not this specific
one.

~~~
slim
China is sourcing virtually nothing from US. If US is attacking the supply
chain of China, it's doing it on Chinese soil. Which brings us back to my
interpretation

------
maerF0x0
This has been posted several times and there are tons of comments:

[1]:
[https://news.ycombinator.com/item?id=18146438](https://news.ycombinator.com/item?id=18146438)
[2]:
[https://news.ycombinator.com/item?id=18138328](https://news.ycombinator.com/item?id=18138328)
[3]:
[https://news.ycombinator.com/item?id=18145645](https://news.ycombinator.com/item?id=18145645)
[4]:
[https://news.ycombinator.com/item?id=18138990](https://news.ycombinator.com/item?id=18138990)
[5]:
[https://news.ycombinator.com/item?id=18141328](https://news.ycombinator.com/item?id=18141328)

------
gvb
If you scroll all the way to the bottom of the article you will find the
disclaimer

 _Bloomberg LP has been a Supermicro customer. According to a Bloomberg LP
spokesperson, the company has found no evidence to suggest that it has been
affected by the hardware issues raised in the article._

Why did Bloomberg not buy a bunch of SuperMicros (new and used) and find the
chip? That would be difficult but would turn this into a HUGE story. Even if
they didn't have the technology to do so in-house, there are many companies
they could hire to do a forensic investigation. The weakness of all bugging is
that it has to communicate to the "outside world" at some point to be useful
and that communications is discoverable. Even Stuxnet, which was much more
narrowly targeted than this, was eventually discovered.

~~~
addicted
It’s describing events from 2015. Presumably this issue has been resolved.

~~~
gvb
There are tons of old SuperMicro computers on eBay for very reasonable prices.
They could have bought lots of them of the appropriate vintage (2015 and
prior).

~~~
394549
> There are tons of old SuperMicro computers on eBay for very reasonable
> prices. They could have bought lots of them of the appropriate vintage (2015
> and prior).

I think buying a bunch of old servers and looking in them for implants is so
far outside Bloomberg News' core competencies that it's unsurprising they
didn't do it.

------
DarkWiiPlayer
I will certainly show this to everybody who tells me "google won't get hacked"
when privacy comes up :)

Talking about things like this
[https://news.ycombinator.com/item?id=18074097](https://news.ycombinator.com/item?id=18074097)

~~~
ryanmarsh
I take a different view of this. Some 30 companies got this hardware, two are
named as having the resources to find the offending hardware. By this logic
you’re still safer using a company such as Amazon.

~~~
DarkWiiPlayer
My point is that you should care about privacy, even if you're using, for
example, apple services. Thinking that nobody has the resources to hack a
company that big isn't an argument and has just been disproven.

In other words, if you have plans to ever be a politician that china may not
want in any position of power, don't store your nudes on icloud/dropbox/google
drive, or they may suddenly get leaked when you least expect it and ruin your
career in favor of a more... Shall we say "convenient"? ...alternative.

~~~
ryanmarsh
I’m not saying _nobody has the resources to hack a company_. I’m saying few
companies have the resources to detect such a sophisticated hack. Therefore my
infra is probably safer with, say, Amazon vs my local colo.

You are correct that individuals seeking personal infosec against state actors
must be eternally vigilant.

------
blackrock
Doesn't the printed circuit board also need to be redesigned, to allow for the
spy chip to be inserted?

So, this means that China forced multiple companies to modify their processes,
in order to pull this off.

1\. The PCB designer

2\. The PCB printer, if it's a separate company

3\. The company assembling the final product

And if they forced the PCB to be redesigned, then wouldn't this be a immediate
red flag? But this means that the customer, Supermicro, would have to audit
the PCB results as well.

~~~
daddylonglegs
They wouldn't need the PCB designer. They would need to suborn (employees of)
the PCB manufacturer and the assembly house. These will be supplied with the
PCB layout (in Gerber or ODB format, for making the PCB) and the bill of
materials and pick and place file (for assembling it). With these files you
can reverse engineer the schematic and carry out the modifications described.
The PCB manufacturer might well be a sub-contractor to the assembly house.

Most electronic engineers don't have the software tools for or experience of
sophisticated reverse engineering but there aren't any major conceptual
barriers. They would have to go from the geometry of the tracks and pads to a
connectivity graph (very automateable), then collect the pads into footprints
for components (probably partly automated), then identify those components and
the functions of their pins (easy with a complete BOM) and then workout the
circuit function (should be straightforward for standard parts and circuits).

There might not be automated tools for making the desired changes, in which
case they will have to manually draw the new track geometry on the Gerbers,
add the parts to the BOM and pick and place files and change or nobble the
test criteria / files. Hard work but quite straightforward.

These skills will be developed by people doing legitimate industrial reverse
engineering as well as espionage / intelligence. I would think there are also
unfortunate cases where firms have to reverse engineer their own products
after losing the original files.

------
baybal2
>Supermicro subcontractor

There is only one I know of: Soyo motherboard

------
hokkos
> In order to get further down the trail, U.S. spy agencies drew on the
> prodigious tools at their disposal. They sifted through communications
> intercepts, tapped informants in Taiwan and China, even tracked key
> individuals through their phones, according to the person briefed on
> evidence gathered during the probe.

So this is still possible ? And this is slipped so lightly in the article.

------
placebo
"Two of Elemental’s biggest early clients were the Mormon church, which used
the technology to beam sermons to congregations around the world, and the
adult film industry, which did not."

Amused to see some element of humor added to an article on a subject that will
have extremely serious consequences, the likes and extent of which are hard to
estimate at the moment

------
TheRealPomax
Where is the _actual_ explanation of how the chip works? I assume it's been
dissected to bits, and that's way more interesting than just "it's been found
on lots of boards". What does it actually contain that apparently lets it do
things that apparently no one else has managed to achieve at that scale?

------
ummonk
Interesting that Apple didn't give the FBI access to its datacenters for the
investigation. Didn't trust them?

------
dboreham
Do we know how this attack actually worked? From TFA it seems to involve the
BMC (which afaik everyone already assumed was untrustable), and also involved
the capability to "phone home" (also notable since in security-conscious
deployments the BMC NIC would never be allowed public Internet connectivity).

------
emeraldd
This reminds me of Anne McCaffery's PartnerShip:
[https://www.goodreads.com/book/show/410927.PartnerShip](https://www.goodreads.com/book/show/410927.PartnerShip)

One of the plot lines involves a similar attack vector.

------
raarts
In retrospect it seems that the decision to move all manufacturing to China
was ill-advised.

Western companies ultimately will have no choice than to move it all back.
(And Trump will want to take credit for that.)

I can understand all the big guys denying this. It's very hard to fix and very
bad for business.

------
crankylinuxuser
So, what _was_ the SMT part supposed to be, and what was put in there instead?
I would love identifying markings, or you know, a datasheet.

Nothing so far I've read includes part numbers. Sure would like to go through
my supply to see if I have any of the offending parts.

~~~
rasz
something like this [http://www.littelfuse.com/products/tvs-diode-
arrays/ultra-lo...](http://www.littelfuse.com/products/tvs-diode-arrays/ultra-
low-capacitance/ultra-low-capacitance-diode-arrays.aspx) not only looks the
part, but actually belongs on the data lines

------
e12e
"The resulting software dramatically reduced the time it took to process large
video files. Elemental then loaded the software onto custom-built servers
emblazoned with its leprechaun-green logos."

I didn't realize Pied Piper was based on a real company...

------
kuon
I'm very curious at what is in that chip. I wonder how much we can reverse
engineer it.

------
totalperspectiv
If I wanted to buy devices (laptops / cellphones) that, which are manufactured
and assembled in America, where would I even look?

(Yes I know that this does not necessarily make the devices any more secure,
better the enemy you know, etc etc)

------
isostatic
It's very odd that this was released the same day as claims against Russia.

[https://www.bbc.co.uk/news/world-
europe-45746837](https://www.bbc.co.uk/news/world-europe-45746837)

------
FrankDixon
Is this as big as it sounds?

~~~
yborg
Probably much bigger. It's naive to assume Supermicro was the only vendor
compromised in this way. Similar implants probably exist in equipment from
every major vendor of information-processing equipment. Routers, mobile
network equipment, you name it.

This is the leverage you give another nation-state when you let them control
your supply chain.

------
Robotbeat
Why didn’t they ask any of their sources to point out one of these chips on a
specific (common) Supermicro board? That’d allow independent verification by
everyone and way less vague speculation.

------
zouhair
Well it's nice to the see the specifics on how they did it, but anyone was
really surprised by this? Most electronics in the World are made there what
people are expecting gonna happen?

------
aj7
Where are Vcc and ground for the chip? These leads would show up on a simple
micrograph of the board. How does this chip make connections that enable
spying on data in the server?

Something is fishy.

------
sidcool
As long as you are powerful, you can get away with almost anything. This goes
from individuals to countries. Had China been weaker, it would have been
invaded already .

------
shambolicfroli
Maybe it's worth talking about how we can better secure the chain of custody
in shipping? I asked about this a year or two ago and my question got the
thumbs-down.

------
vadiml
I wonder, given the fact the C&C were identified, wouldn't be external
firewall configuration much cheaper mitigation method than motherboard
replacement?

------
shambolicfroli
Maybe it's worth talking about how we can better secure the chain of custody
in shipping? I asked about this a year or two ago, here, and it got squelched.

------
qwerty456127
Isn't this or alike chip put all the Chinese hardware other people and
enterprises buy too? Or at least in all the SuperMicro mainboards (I use these
too!)?

------
ayyyyylmao2000
As a nation, we need to unite against the external threat of China. Russia and
Israel are fine. China is the number one problem sowing discord in this
country.

~~~
doubt_me
Ok why not all of them?

~~~
brynjolf
You know the answer to that. Russian bots target Hacker news as well as other
sites.

~~~
taysic
Any evidence of that?

~~~
doubt_me
[https://www.justice.gov/opa/pr/us-charges-russian-gru-
office...](https://www.justice.gov/opa/pr/us-charges-russian-gru-officers-
international-hacking-and-related-influence-and)

Have fun googling what the rest of the GRU does

------
shambolicfroli
Maybe it's worth talking about how we can better secure the chain of custody
in shipping? I asked about this a year or two ago and it got squelched.

------
bitxbit
This article frightens me considering that you can almost always identify the
endusers for a lot of the personal devices shipping directly from China.

------
moftz
The image at the top says it's a Supermicro B1DRi. I wonder if this is an
actual infected board or if it's just for illustration purposes.

------
cfquaglia
We can agree this news is suitable for Hacker News.

------
dantheman
The graphics - the opening animation and the penny are some of the most
informative illustrations i've seen in an article.

------
galadran
> The illicit chips could do all this because they were connected to the
> baseboard management controller, a kind of superchip that administrators use
> to remotely log in to problematic servers, giving them access to the most
> sensitive code even on machines that have crashed or are turned off.

Obviously this would still be possible without IntelME, but having an always
on, highly privileged and remotely accessible baseband definitely makes the
modifications easier and smaller...

------
KillerRAK
What's the old saying about sleeping with dogs?

Yeah. That.

------
kuon
So. What now. Any consequences? I mean, will this trigger big changes, like
some production moving somewhere else?

~~~
akerro
>like some production moving somewhere else?

Sure, to US, UK, New Zealand, Australia, Canada, because we trust these
countries not to backdoor software and hardware.

------
yellowapple
Elemental servers sold for as much as $100,000 each, at profit margins of as
high as 70 percent, according to a former adviser to the company. "Two of
Elemental’s biggest early clients were the Mormon church, which used the
technology to beam sermons to congregations around the world, and the adult
film industry, which did not."

Church would certainly be more interesting if they did, though.

------
pmorici
So what is the alternative to SuperMicro if you aren't large enough design
your own board?

~~~
detaro
Other motherboard vendors (e.g. Tyan, Asrock, Gigabyte make server boards), or
buying entire servers from Dell, HP, ... And hoping they don't have issues
like this, despite probably all manufacturing partly in China.

~~~
pmorici
None of those guys offer the same variety of form factors and features that
Super Micro does. If you are an OEM that needs an Intel Xeon-D board in mini-
itx form factor for example.

------
beaner
I feel like there is a really large opportunity here for some other
international supplier...

------
__exit__
Holy molly! This sounds similar to what I thought a couple of days ago: China
is the major electronics supplier to most technological companies, what if
China added some sort of sniffing or malware on the chips to spy on their
customers?

------
JumpCrisscross
Is anything around the Intel Management Engine made in China?

------
partingshots
As something I heard: In China, business a fight for survival that is brutal
to the extreme. You will be cut down, cheated, extorted, and broken with zero
hesitation.

Doing business in the West is comparatively like a walk in the park.

------
knodi
Couldn't the same be possible on routes?

------
jiveturkey
the presentation is very good, at least. white text on black background, very
dramatic.

------
camdenlock
... huh. For the first time, I'm considering the possibility that the Trump
administration's tariffs on China might actually result in a positive outcome:
forcing US companies to look elsewhere for hardware, possibly creating new
supply chains with other (hopefully more trustworthy) partners out of sheer
necessity.

------
client4
Loose chips sink ships

------
SubiculumCode
1) I wonder how wide spread this is in electronics produced in China. Say, in
those cheap usb blutooth radios, for example. 2) I wonder if this is spurring
the current trade war push, besides just Trump. 3) Seems time for more
transparency on supply chain integrity and security.

------
superkuh
Anyone have a mirror of the article? I can't read it behind Bloomberg's
computational paywall.

------
thenetadmin
This is crazy

------
zjfroot
Statements from Amazon, Apple, Supermicro and Chinese government.

[https://www.bloomberg.com/news/articles/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-
amazon-apple-supermicro-and-beijing-respond)

From Apple:

"Over the course of the past year, Bloomberg has contacted us multiple times
with claims, sometimes vague and sometimes elaborate, of an alleged security
incident at Apple. Each time, we have conducted rigorous internal
investigations based on their inquiries and each time we have found absolutely
no evidence to support any of them. We have repeatedly and consistently
offered factual responses, on the record, refuting virtually every aspect of
Bloomberg’s story relating to Apple."

~~~
MrBingley
What liars. Apple has done this before as well, when they said they had "never
heard" of PRISM, despite a Snowden leak showing the exact opposite.

[https://www.theguardian.com/world/2013/jun/06/us-tech-
giants...](https://www.theguardian.com/world/2013/jun/06/us-tech-giants-nsa-
data)

~~~
swebs
Regarding both cases:

[https://en.wikipedia.org/wiki/Gag_order](https://en.wikipedia.org/wiki/Gag_order)

~~~
LolNoGenerics
I wonder how legit it would be for an US company to set up an internal gag
order? Running on compromised hardware is practically a nuclear meltdown (pun
intended) and publicly admitting to it as well.

------
tomglynch
Who else found the design of the page made the article difficult to read? I
know darkness, spies and hacking go hand-in-hand but it's a bit too much imho.

~~~
akuji1993
This is because, contrary to what a lot of people say about dark themes (even
though it's mostly a meme at this point), dark letters on white background are
better to read than white letters on black background.

~~~
Avamander
"People with astigmatism (approximately 50% of the population) find it harder
to read white text on black than black text on white. Part of this has to do
with light levels: with a bright display (white background) the iris closes a
bit more, decreasing the effect of the "deformed" lens; with a dark display
(black background) the iris opens to receive more light and the deformation of
the lens creates a much fuzzier focus at the eye." \- Jason Harrison – Post
Doctoral Fellow, Imager Lab Manager – Sensory Perception and Interaction
Research Group, University of British Columbia

This quote describes the situation well I think, why the opinions differ and
why neither dominates UIs really.

~~~
akuji1993
I actually have astigmatism so that might be why I think it's even harder to
read on black/dark themes. So yeah, definitely true in my case.

~~~
bpye
It's interesting to finally have some explanation as to why I have always
found dark themes worse, also having astigmatism.

------
ElBarto
How common is it for a seemingly standard security audit to inspect
motherboards with such level of detail or at all?

They likely needed to have the exact official schematic of the motherboard to
compare every single detail of the hardware with.

~~~
PeterisP
While you may find a particular attack if you're looking for, in general, it's
impossible for even the most thorough audits to check for the whole class of
such attacks. You're not going to look into the chips. Well, you can, but
that's prohibitively expensive and destructive - even if you could check that
this chip was okay, then you still have to throw it out after analysis and
plug in a different one.

The only feasible thing to do is thorough audits of all the supply chain for
every component in your system, ensuring that your supply chain does not
include even a single chip from an "untrustworthy" supplier, and even then it
reduces the chances of an attack but does not eliminate it.

~~~
snaky
Zilog Z80, 1976

> Faggin: Yes, we were concerned about others copying the Z80. So I was trying
> to figure what we could do that that would be effective, and that’s when I
> came across an idea that if we use the depletion load the mask that doesn’t
> leave any trace, then I could create depletion load devices that look like
> enhancement mode devices. And by doing that we could trick the customer into
> believing that a certain logic was implemented, when it was not. Then I told
> Shima, “Shima, this is the idea how to implement traps. Put traps, you know,
> figure out how to do the worst possible traps that you can imagine,” and
> then Shima with his mind, that was steel mind, was able to actually figure
> out a bunch of traps that he could talk about.

> Slater: You want to tell us a little about that Shima?

> Shima: I didn’t count [on] talking about that mostly. I placed six traps for
> stopping the copy of the layout by the copy maker. And one transistor was
> added to existing enhancement transistors. And I added a transistor looks
> like an enhancement transistor. But if transistors are set to be always on
> state by the ion implantations, it has a drastic effect on very much. I
> heard from NEC later the copy maker delayed the announcement of Z80
> compatible product for about six months.

[http://archive.computerhistory.org/resources/text/Oral_Histo...](http://archive.computerhistory.org/resources/text/Oral_History/Zilog_Z80/102658073.05.01.pdf)

------
ackfoo
The logical extension of this is to embed a sleeper chip between board layers
that can be activated by a radio signal, then try to disperse it widely within
the DND.

Any serious conflict with China would then look like the first day of the
second Cylon war...

Of course, they would probably have to get a Chinese boat really close to a US
warship a few times beforehand to test the system. All they would need to do
is to receive some sort of a ping back to confirm receipt of the signal. Could
be simple as an innocuous visit to a particular page of a website.

It would probably look like brinksmanship or posturing, or a navigation error
of some sort.

------
Dowwie
"Two of Elemental’s biggest early clients were the Mormon church, which used
the technology to beam sermons to congregations around the world, and the
adult film industry, which did not."

~~~
crescentfresh
Why does this keep getting quoted in the comments. Yes we read the article
too.

~~~
roel_v
Because it's so awesome. This must be one of this journalist's career
highlights, to be able to put something like this in a mainstream serious
reporting piece.

~~~
umichguy
So damn true! It's one of those things which you wait all your life to slip
into an article/ convo.

------
mabbo
> Two of Elemental’s biggest early clients were the Mormon church, which used
> the technology to beam sermons to congregations around the world, and the
> adult film industry, which did not.

------
housingpost
Looks like SuperMicro was delisted by Nasdaq just over a month ago. Wonder if
it has anything to do with this under their pretty poor surface excuses for
not being in compliance.

------
jordache
white text on 0,0,0 black background... ugh my eyes do not thank thee

~~~
randie63
In my OLED phone screen, my eyes do thank me (although I have anyway the black
mode in Kiwi browser all the time)

------
teknologist
Imagine if this had happened the other way around. Death threats and calls of
oppression and foul play would only have been the start.

------
phyller
Wow, the Obama administration _knew_ this was happening, even before it
happened, and did nothing to stop it?

------
Dowwie
This story could easily be interpreted as anti-China propaganda. Could you
think of any other sovereign power who would want a backdoor into servers used
by billions of people across the internet? Hardware comes from China. This
doesn't mean that the Chinese government orchestrated the attack. The United
States government is having a trade war with China. This article's publication
isn't just coincidence.

Further, those with insight about how to use the backdoor definitely made use
of them. This is one of the central issues with government-mandated backdoors.
They are introduced for one group but used by others.

~~~
nithinm
your comment looks like an pro-china propaganda. In china the state has heavy
control on all the companies. The hardware is manufactured in china. I don't
think something like this is happen without the state's knowledge.

~~~
ethbro
It seems like many people take communist-like propaganda at face value.

The Chinese government would like to have almost-complete control. The Chinese
government publicly says it has high levels of control.

But when they can't effectively regulate their medical (mass HIV infection
from blood plasma needle reuse), food (tainted milk), or chemical (unlicensed
mass CFC production) industries... reality seems to differ.

~~~
teknologist
So, suppose you're Supermicro. When some CPC official comes around to tell you
to make your technology a little easier for the intelligence department to
access, you're going to do it.

Companies in China (especially those in the tech sector) have to keep close
ties to the government, and most of their leaders are members are of the
party. You don't GET to be a multi-billion dollar tech company* in China
without toeing the party line [1] [2] [3]

The issues around regulating food safety and vaccines that you mentioned are
irrelevant.

* Foreign companies must operate Chinese subsidiaries to run their operations in China.

[1] [https://www.wsj.com/articles/beijing-pushes-for-a-direct-
han...](https://www.wsj.com/articles/beijing-pushes-for-a-direct-hand-in-
chinas-big-tech-firms-1507758314)

[2] [https://qz.com/1102948/chinas-communist-party-is-all-in-
on-t...](https://qz.com/1102948/chinas-communist-party-is-all-in-on-the-power-
of-technology-and-thats-tricky-for-its-tech-giants)

[3] [http://chinamediaproject.org/2018/05/02/tech-firms-tilt-
towa...](http://chinamediaproject.org/2018/05/02/tech-firms-tilt-toward-the-
party)

~~~
ethbro
I see your *, so I assume you do know SuperMicro is a US company? And that the
breaches reportedly happened at local Chinese subcontractors?

Everything you say seems to be valid for Chinese companies. Which
SuperMicro... isn't.

If they want to start auditing their incoming supply from China more closely,
or even shift production elsewhere, there's nothing except cost stopping them
from doing so.

And the rub is that any competitor also using Chinese supply (for cost
savings) is vulnerable to the same attack. SuperMicro was presumably targeted
because of their size and global customers.

~~~
teknologist
Apple is not Apple Inc. in China, it is Apple Computer Trading (Shanghai) Co.,
Ltd. SuperMicro will have its own subsidiary/subcontractors to handle their
operations in China.

As having worked for a subsidiary of a large multinational company in China, I
can tell you that a lot goes on that you might find surprising. Middle
management and those involved with establishing the deals with subcontractors
would often have arrangements to make money through various means, such as
through IP transfer or property theft. The subcontractors themselves are
Chinese companies.

At our place a lot of shady things were going on and were, thankfully, found
after a number of years. The global HQ had to step in and fire around 1/3 of
employees working at the China branch.

Now, there is no reason to suggest that this was related to any government
policy or request. But it should be clear by now that the CPC are not exactly
opposed to shady things happening to foreign companies. My point is that this
is an an environment where something like this can happen quite easily,
especially when a lot of technology companies have close ties to the state.

------
tomglynch
> Nested on the servers’ motherboards, the testers found a tiny microchip, not
> much bigger than a grain of rice, that wasn’t part of the boards’ original
> design.

> During the ensuing top-secret probe, which remains open more than three
> years later, investigators determined that the chips allowed the attackers
> to create a stealth doorway into any network that included the altered
> machines. Multiple people familiar with the matter say investigators found
> that the chips had been inserted at factories run by manufacturing
> subcontractors in China.

------
TheForumTroll
The only interesting thing about this is left out. Who planted it is clear
(someone told to do so) but not a single time is it questioned who they
planted it for. Smells like false flag to me.

We _think_ China does X Y and Z but we _know_ the US does X Y Z and the rest
of the alphabet. So unless something specific is leaked that shows who
actually ordered this, logic would point at the US.

~~~
yourapostasy
> ...but not a single time is it questioned who they planted it for...

The article and some accompanying reporting on Bloomberg audio/video segments
says the attack seems targeted relatively specifically towards nearly 30
organizations (only US-based organizations were mentioned as targets, unknown
if the list included organizations based in other nations). One known vector
was through four subcontractors in China that built the boards for
Supermicro's main Shanghai factory, specifically by bribing and/or coercing
managers of those subcontractors' factories to go along with accepting the
chip shipments and to make changes to the plant floor from the design to
perform the chip insertions.

Designing and building a chip like this and then mounting the logistical
effort to performing the insertions costing some non-trivial funds, coupled
with the known targets, (Amazon didn't seem specifically targeted, Elemental a
company they acquired was, who notably has US national security clients), form
the circumstantial allegation that a PLA spy unit was behind the attack. You
are correct that this doesn't entirely rule out a false flag possibility, but
until we get more details about this, we're operating in the dark.

A false flag is an interesting supposition, but how would the US benefit from
successfully convincing the world of the false flag's cover story?

~~~
lolc
The US has a few things to gain from this story: Economically, because Chinese
products are perceived as compromised. Politically: because the Chinese
government is seen in the offense.

The US has something to loose too: Being perceived as dependent on Chinese
manufacturing and potentially compromised down to military hardware. (The
first everybody knows, the second would be devastating for trust.)

All in all it would be a weird angle for a false-flag attack.

