
Making sense of the alleged Supermicro motherboard attack - bellinom
https://www.lightbluetouchpaper.org/2018/10/05/making-sense-of-the-supermicro-motherboard-attack/
======
ChuckMcM
Ok, that finally makes a bit of sense about "if" this is true, how it might be
carried out. And I agree with the author that the simplest action for a chip
on the SPI bus would be to hold the MISO line low during power on to suggest
to the BMC chip that its QSPI flash isn't programmed (note that QSPI starts up
as 'regular' SPI and then switches over[1]). I would guess that the next thing
the BMC would do is assume its on the factory floor and hasn't had firmware
loaded yet and so it would attempt to boot via TFTP from some server to start
the firmware loading process.

FWIW at NetApp the firmware engineers called the BMC the 'BiteMeChip' because
they always caused issues when bringing up a new filer motherboard. They were
finicky, were often hard to update, and when misbehaving could completely
screw up the system.

So I can see this as a vulnerability but really I'd want to stick some probes
on that 'mystery' chip and make sure it wasn't just some LC filter or
something which is shunting off noise on the data lines.

[1]
[https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi](https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi)

~~~
amelius
One question: how do you hold an existing line low without drawing lots of
current, and without cutting that line first?

~~~
dbcurtis
It's not like the MISO line is tied to a power rail. You just need to sink
more current than the output of the chip you are trying to override. So the
driver is likely in the single digit milliamp range, and in any case, you
don't care if you toast the output driver on that chip. In fact, if you do,
bonus!

~~~
amelius
Ok, but i suppose that in the future chip output pads can have circuitry which
can detect a forced output (by measuring current). When this happens, the chip
could short-circuit power lines, or superimpose a signal on the power-lines to
notify the rest of the system.

~~~
ispiansclsda
> When this happens, the chip could short-circuit power lines

No chip design would ever do this. The chip would incinerate itself.

> superimpose a signal on the power-lines to notify the rest of the system

The rest of the system i.e. the BMC already know that something weird is up.

------
alexandercrohde
In another thread Walterbell points out:

From 2016, [https://arstechnica.com/information-
technology/2016/03/repor...](https://arstechnica.com/information-
technology/2016/03/repor..).

> Apple has begun designing its own servers partly because of suspicions that
> hardware is being intercepted before it gets delivered to Apple, according
> to a report yesterday from The Information. "Apple has long suspected that
> servers it ordered from the traditional supply chain were intercepted during
> shipping, with additional chips and firmware added to them by unknown third
> parties in order to make them vulnerable to infiltration, according to a
> person familiar with the matter," the report said. "At one point, Apple even
> assigned people to take photographs of motherboards and annotate the
> function of each chip, explaining why it was supposed to be there. Building
> its own servers with motherboards it designed would be the most surefire way
> for Apple to prevent unauthorized snooping via extra chips."

~~~
Tempest1981
The industry was aware of intercepted/modified hardware shipments long before
this, also giving reason to be suspicious:

From 2013: “NSA reportedly intercepting laptops purchased online to install
spy malware”

[https://www.theverge.com/2013/12/29/5253226/nsa-cia-fbi-
lapt...](https://www.theverge.com/2013/12/29/5253226/nsa-cia-fbi-laptop-usb-
plant-spy)

Also [https://www.wired.com/2015/02/nsa-firmware-
hacking/](https://www.wired.com/2015/02/nsa-firmware-hacking/)

------
hugelgupf
I think the board that is pictured by Bloomberg is just an illustration.

For example, Elemental never used that board. See my tweets here for the
actual specs:
[https://twitter.com/hugelgupf/status/1048160794565861377?s=1...](https://twitter.com/hugelgupf/status/1048160794565861377?s=19)

Elemental boards come with GPUs. The blade pictured has neither GPUs nor PCIe.

------
uep
I think the attacks are real.

A year ago, Google announced their Titan firmware security chip[1], which
would limit these kinds of attacks. I don't believe they designed and built
this chip, and surrounding infrastructure, because of purely theoretical
attacks.

Besides that, over the last couple years there has also been a lot of work
trying to neuter the Intel ME, because of how dangerous it is. Another example
is that Google had also done work to replace the EFI/Intel ME with a Linux
kernel [2]. This has limited usefulness against the most recent attack, but it
is related (since it's still a chip that has more privileges than the primary
CPU).

I suspect that there are a very small number of people in these big companies
who are aware of these attacks. It's hard for me to guess why the companies
involved would deny that these exist.

1\. [https://www.zdnet.com/article/google-opens-up-on-titan-
secur...](https://www.zdnet.com/article/google-opens-up-on-titan-security-
heres-how-chip-combats-hardware-backdoors/)

2\.
[https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...](https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20with%20Linux.pdf)

~~~
wjnc
First guess: not being allowed to admit it due to national security reasons
and it being an ongoing investigation. On the same day several Russians were
exposed trying to attack OPCW. They were exposed by Dutch military
intelligence. At the press briefing the UK ambassador was there. Same day US
indicts several Russian spies. This to show that these are major,
international events and that proper disclosure towards investors is probably
on the back burner as long as operations are continuing. It must be maddening
for those involved not to be allowed to discuss it, or even having to actively
lie about it while government officials leak freely in the game of thrones.

~~~
lern_too_spel
If they are under a gag order, they would simply not comment on it. Lying
about it is never required and puts them at risk for shareholder lawsuits.

~~~
stephenr
Apple _specifically_ states that they are _not_ under any form of
gag/confidentiality order/conditions:

> Finally, in response to questions we have received from other news
> organisations since Businessweek published its story, we are not under any
> kind of gag order or other confidentiality obligations.

~~~
bsder
Apple cannot do anything bad to China or their manufacturing stops.

Apple would have to move tens of thousands of highly specific CNC machines
from China to somewhere else, set them up, and get their line moving again.

This is why I find the idea that Apple phones are "secure" to laughable on its
face. China could kill _years_ of Apple revenue if they ever did something
truly offensive to the ruling party.

Apple would roll over post haste if China demanded it.

~~~
Yoyoyou
China could cause huge harm to Apple, but the reverse is also true - how would
the US government, other corporations, consumers and investors react to such a
news?

The ultimate source of most Chinese factory equipment is Europe and US anyway,
under extreme political/consumer pressure electronics factories can be setup
in a matter of months in US.

~~~
bsder
> China could cause huge harm to Apple, but the reverse is also true - how
> would the US government, other corporations, consumers and investors react
> to such a news?

They would react as usual: lots of whingeing, no action, and roll over for a
belly scratch afterward.

> under extreme political/consumer pressure electronics factories can be setup
> in a matter of months in US.

It takes 9 months to make a baby no matter how many women you get pregnant ...

Many of these equipment manufacturers have entire facilities dedicated to
producing equipment solely for Apple. They simply cannot replace that amount
of equipment in any timely fashion.

Edit (external reference): [http://www.iphonehacks.com/2016/10/next-iphone-
probably-wont...](http://www.iphonehacks.com/2016/10/next-iphone-probably-
wont..).

> Koenig writes. “Apple is such a huge buyer of a particular kind of mill
> (BT30 spindle drill-tap centers) that Fanuc, Brother and DMG Mori each have
> factories dedicated to building machines exclusively for Apple.”

------
lovelearning
It was reported that the security auditor used during Elemental's acquisition
detected this compromise. I assume they found it in a randomly selected board.
Either they were very lucky, or hundreds of boards were compromised.

Now, I think all motherboard manufacturers - and especially high end server
manufacturers like SM - use sophisticated automated tests and quality control
on boards. Under what circumstances is it possible that SM's QC missed this
out on so many boards? Won't it affect things like the power budget, weight,
and latency? What do professional EE people here think?

~~~
crispyambulance

        > ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards.
    

MFG test guy here (not supermicro!)

Not necessarily. Automated production tests are there to configure and
exercise the system and confirm it works as specified. Such tests are good at
things like finding bad solder joints, pick-and-place mishaps,
misconfiguration of firmware and weeding out product that fails functional
test. That, by itself, is hard enough.

Such tests are not _really_ able to detect things which no one is expecting,
or worse, things which an adversary has specifically designed to avoid
detection. Sure, if something gets "discovered" during a failure analysis, a
test or process can be created to specifically address THAT problem in the
future.

To find "unknown unknowns", you need an audit of some kind and that is
definitely different from production test.

~~~
lovelearning
Thank you for the insight. I was rather naively imagining that every board is
subject to some kind of x-ray image matching with the original PCB design to
find differences. Not that simple, I see.

~~~
crispyambulance
Some manufacturers do X-ray boards.

Typically this is done to check that BGA devices
([https://en.wikipedia.org/wiki/Ball_grid_array](https://en.wikipedia.org/wiki/Ball_grid_array))
are soldered properly. Usually not done on every board, but only if there's a
problem suspected. When it happens they tend to focus only on particular BGA
components or suspect copper traces rather than the whole PCB.

~~~
scoggs
I'm guessing the answer is obviously yes that this type of x-ray would easily
be able to discover / distinguish something the size of what's been described?
(1mm x 2mm from what I've seen thus far?)

~~~
PeterisP
The x-ray picture would contain information sufficient to detect that thing,
but it's quite plausible that the process/procedure of analyzing that x-ray
would not find it. If they're looking for extra hardware, they're going to
detect it, but if they're looking for bad solder joints, they're going to
detect bad solder joints but not extra harware.

~~~
scoggs
Right. It doesn't exactly align with what a normal auditor would be looking
out for, I suppose. Thanks, I appreciate it.

------
mindslight
> _Let’s assume an implant was added to the motherboard at manufacture time.
> This needed modification of both the board design, and the robotic component
> installation process. It intercepts the SPI lines between the flash and the
> BMC controller_

Circuit wise, you don't really need to "intercept" (place in series with) the
SPI lines. Two parallel drivers [0] will generally fight it out quietly, and
you can guarantee yours will win by designing your implant to have the beefier
driver (and perhaps only changing 1's to 0's, as P-channel FETs are generally
weaker).

If there already was a footprint for an extra chip (debugging, part
flexibility, etc), then it would not require modification of the board design.
And an extra component can easily be hand soldered afterwards outside of the
standard robotic component placing.

[0] The standard topology of SPI, but in proper operation only one is active
due to CS lines.

~~~
theomarkettos
Post author here. That was what I was thinking of - simply overdriving the
lines to drag them high or low in opposition to the 'official' driver. That's
pretty much all you can do with 6 pins. I was trying not to overcomplicate the
explanation, since the material is already complicated enough!

Fair point about hand soldering, however it would be more obvious to the
manufacturing employees for an entire production batch to be diverted to have
parts added manually than to simply sneak in another reel to the hundred on
the pick and place machine.

~~~
mindslight
I just thought that was phrased as if it were a necessary requirement for the
attack, rather than a description of a likely approach. I could also see a
scenario where only the occasional unit was bugged, and then an agent at the
distributor made sure the right customers received the bugged units.

Thank you for writing your article! I'm currently a bit out of the security
headspace and reading the Bloomberg article had me scratching my head like
what are the actual details here.

------
zaroth
The fact that cursory examination finds the attack is not only entirely
feasible, and completely undefended against, but that the hardware shown in
the Bloomberg animation is _precisely_ the hardware which would be required to
pull off the attack is quite astonishing.

Whose “law” is it that the closer you are to an event the more you can see
that the reporting on the event is desperately flawed? This has almost always
been my own personal experience.

To have the article hit the nail on the head with a entirely plausible attack
means one of two things;

1) This is exactly what happened

2) A nation state wanted us to think this is exactly what happened.

Either way the NSA is involved. The people on the ground who knew it happened
would have to be NSL’d from telling their superiors that it happened, so they
leaked it.

Perhaps the denials from above are entirely sincere because the people who
discovered it can’t pass it up the chain.

The only thing I’m having trouble with is that would have required either a
low-level plant to have discovered it and someone else at a low-level found
out and was sworn to secrecy, or a middle-manager plant intercepted the
message as it went up the line.

The alternative, that this is a false flag, is pretty fascinating in its own
right. TAO would have had to conceptualize the attack vector, and then someone
planted the story.

Perhaps this is equally likely. In fact, this is actually an attack I assume
TAO is using in the field, which would burn this attack vector presumably
because they have an even better one.

Even more interesting, the idea TAO was actively using this attack vector, saw
evidence in the field their attack was discovered (devices going dark) and so
preemptively planted a story to blame the opposition before they accused the
US of the same thing.

~~~
kromem
Or, it could be both.

It could have been a real attack vector that didn't actually infiltrate the
companies in question due to having been caught upstream.

But as "attempted attack" sounds far less sexy than "actual attack," the
sources may have exaggerated the impact/discovery to further trade negotiation
objectives with the spin.

In fact, the BS discovery part of the story might be an attempt to parallel
construct a reason it was discovered to cover for a program that might do
independent testing of hardware which, if revealed, would give attackers some
sense of sampling methodology to be able to counter.

That would explain the vehement denials from the companies allegedly
compromised and yet the realistic attack vector published along with the US
entity bans on using Chinese hardware around that time.

------
DenisM
A decade ago when I worked at Microsoft I shopped around the idea of using
XBox as a basis for secure computing.

XBox was designed to function in the hands of the adversary, to be robust
against peripheral attacks and even motherboard mods. Even the main memory was
encrypted by the on-CPU controller. Obviously, no open JTAGs. A lot of
expertise there.

In my fantasies it would form the basis of the DoD infrastructure and then
trickle down to finance. It was deemed to be not solving any practical
problems, so it didn't go anywhere. Oh well, I found other fun thins to do in
my life.

~~~
joshstrange
Are you referring to the original Xbox? Modchips hit the scene for that in ~ 2
years after release. It's not exactly what I would consider a basis for secure
computing. Or maybe you are just referencing the procedures "main memory was
encrypted by the on-CPU controller. Obviously, no open JTAGs" should be
ratified to create a basis for secure computing, a checklist of things to
do/prevent before considering a device "Secure".

~~~
dawnerd
Didn't even need a modchip. There was a technique where you could softmod by
simply unplugging the ide harddrive right after boot and hot plugging it into
a desktop.

[https://www.reddit.com/r/originalxbox/wiki/hotswap](https://www.reddit.com/r/originalxbox/wiki/hotswap)

~~~
jacob019
Ah the old softmod, with the 007 game as I recall. Made a great party trick.

------
fhood
Can someone outline some reasons for me why nobody has come up with an actual
physical example of a compromised board? I'm not trying to make a point, I
just want to get a more complete picture of the issue, and the biggest thing
that stands out to me is the lack of physical evidence.

~~~
rando444
Well according to the original article, the attack was targeted, implying the
only way to get such a thing would be from one of the compromised customers,
who are probably involved in an investigation into the matter, don't have all
of the info themselves, and aren't super eager to release details to the
public before a full picture emerges and mitigation procedures are in place.

Further, going back to the original article, the majority of the information
comes from alleged government sources, so not the people directly impacted,
but rather those just helping deal with the fallout and coordination.

Assuming there is merit to the story, it will likely be some time before more
details emerge, unless having the story out there now helps accelerate that
process.

~~~
est
> the attack was targeted,

Supermicro assemble their products in US/Taiwan.

So in theory China need to accurately predict the pattern of how non-China
factories install batches of the motherboard. I think it's extremely difficult
to pull this off.

------
cmurf
This article is worth skimming/reading in order to fill in the technical gaps
in the Bloomberg article; but also describes the complexity of modern
computers in a way I didn't previously appreciate.

What's still missing in the overall story, is what Apple and Amazon denials
mean. The Bloomberg article says Apple and Amazon independently discovered
these chips (Amazon via a 3rd party) in 2015; but the blanket denials by both
companies appear to deny this discovery and reporting to U.S. authorities. I'd
argue it's unethical for a company to apply _falsus in uno, falsus in omnibus_
to a PR statement, but it's plausible there is a sufficiently misleading or
false claim in the Bloomberg article that they feel it's legitimate to dismiss
the entire article. In the meantime, the company denials have to be treated as
conjecture.

------
ivoras
While it's practically certain that hacking attempts do happen from both
state- and non-state actors, this particular instance is so "alleged" that
it's practically theoretical.

Where's the actual hardware? Why didn't someone decap the tiny chip and probe
it? Its design should be well within today's reverse engineering labs'
capabilities.

~~~
jonathankoren
What makes you think that people aren't going through they're POs and server
farms and looking today? Just because a couple huge players told prior to
publication, doesn't mean everyone was.

These things literally take time, and there's no indication of how widespread
the targeting was.

------
bayindirh
There's a problem with exfiltrate via BMC network theory.

In a sane setup, your BMC connection cannot access internet. You should build
an isolated intranet for it (including VLAN or hardware isolation, not just
subnet/IP), and put a VPN in the front gate. As a result, you login to your
data center, or go to there if you like metaphors. If nobody’s there via VPN,
BMC network is a silent and dark place. No connection to outside, no unknown
traffic, just silence. Only exception may be the discovery packets of some
BMCs, which can find similar servers and form federations for easier
management. Even this needs some setup beforehand.

~~~
EvanAnderson
If the BMC has write access to host memory it could surely use that access to
create a side channel using the host's network interfaces.

Having said that, it would be nice if networks were segmented in the way you
describe. I've been appalled at the lack of segmentation I've seen in
companies of all sizes that I've had gigs for.

~~~
bayindirh
Today's network cards are small computers of their own. So it'd be very hard
to inject packages with all this kernel-hardware integration at the module
level IMHO. The card would probably throw a tantrum if you try to access it
directly.

TBH, while I'm knowledgeable about hardware, I'm a total beginner in attack
side of cybersecurity.

At least, we are segmenting our networks like that.

------
simplecomplex
Where did all the boards in question go? Why wouldn’t a company notice any of
the outbound traffic using firewalls?

Two pieces of the story that don’t add up for me.

~~~
radicalbyte
There wouldn't be any weird outbound traffic; the attacker would use another
cloud instance as a controller. They would use a US-based account (and IPs via
VPNs) to control the controller.

~~~
simplecomplex
The traffic has to travel over wire as TCP/IP, regardless of what device
generated it. It has to travel over wire somewhere! The device can’t magically
communicate with China. Any outbound firewall could detect that, especially
traffic going to ports that aren’t even open/used.

~~~
Filligree
Unless, just to provide a nightmare scenario, the routers that would detect it
are also compromised.

~~~
paulie_a
The NSA did that, they literally intercepted routers en route. I'm guessing
other agencies have too.

Or you can just use the default credentials Cisco is addicted to leaving in
their code.

------
agumonkey
Few years ago [https://hackaday.com/2013/08/02/sprite_tm-ohm2013-talk-
hacki...](https://hackaday.com/2013/08/02/sprite_tm-ohm2013-talk-hacking-hard-
drive-controller-chips/) popped up, basically hdd have many arm co-
controllers. Nowadays everything is a processor, everything is thus virtual
and fakeable. Enjoy the paranoia :)

------
qualsiasi
I'm no expert, but wouldn't be more efficient to completely replace the BMC
with a malicious one? That way it should be virtually impossible to detect

~~~
creeble
Or even why two SPI roms?

Why not just one with special code? It's not like anyone is routinely reverse
engineering the BMC boot code.

It seems like an awful lot of provable trouble to go through (note that there
is no physical evidence in the public eye yet) when you could do the same
thing, at the factory, with just software.

~~~
paulmd
it's pretty easy to dump an SPI chip and some vendors/customers routinely do
so.

In this model, the implant basically contains a binary patch that is injected
at boot time over a segment of the BMC binary - counter overflows, chip cuts
out the SPI and transmits its payload instead. After the payload is injected
the implant goes dormant again/resumes passing through the SPI, so dumping the
ROM after boot, or even physically clipping onto the SPI chip will not reveal
the wu-tang secret.

~~~
jgalentine007
Wu-tang.

------
ezVoodoo
Technical possibility is one thing; proving the story has actually happened is
another thing. Until now, what we get is a categorical denial of the story
from all related parties. And all the evidence Bloomberg can provide so far is
just vague anonymous sources.

Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the
story is just a blunt lie fabricated by Bloomberg serving as a propaganda to
bash China amid the Sino-America trade war.

------
tomc1985
I really hope that this is the straw that breaks the back of all these
"management engines"

Like seriously, why does my hobby consumer motherboard need that feature? Corp
IT only ever deploys to large fleets of OEM machines.

~~~
contingencies
Take a look at ASpeed (BMC supplier) stock movement:
[https://finance.yahoo.com/quote/5274.TWO/chart?p=5274.TWO](https://finance.yahoo.com/quote/5274.TWO/chart?p=5274.TWO)

~~~
duskwuff
That movement seems to have started about two weeks ago. I don't think it's
related to this news.

~~~
contingencies
I'm sure it didn't help!

------
KaiserPro
BMCs have always been an Achilles heel.

HP's iLo could be got into if you used `curl -H "Connection:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAA"`

Supermicro's horrid BMCs, of which there are many, all were horrendously lax
in security, long before this chip was, or was not inserted. You didn't buy
supermicro if you were worried about security, you bought them if you needed
to stack stuff high, cheap and dense.

There is a reason why its best practice to put them on separate networks, with
as much stuff between it and anyone else. BMCs are massive backdoors, and
should be treated with caution.

------
beiller
Does this mean I can get some great deals on Supermicro boards now? :)

------
devy
Regardless this attack is real or not, this exposure and assessments (just
like Charlie Miller and Chris Valasek did for exposing the car hacks) raised
good awareness of how vulnerable BMCs are and hopefully the baseboard hardware
security and firmware security will be securer in the future when they evolve.

------
dsdsdsds2
The Register's article on the matter seems also good:
[https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...](https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/)

------
wahern
I don't understand why everybody assumes it must be pulling data over the
network autonomously. It could simply compromise the host OS to augment other,
targeted attacks. Say, by reintroducing a buffer overflow or race condition.
It would be incredibly short-sighted to go to such lengths to compromise these
machines just to naively pull from a command+control server, virtually
advertising its presence.

Defenders simply do not think like attackers. If you're a defender it's
_lethal_ to try to think like an attacker. "I can't imagine how this would be
useful therefore it must not be useful" are the famous last words of everybody
who has cast doubt on a new and novel attack vector. As a defender you need to
first estimate the unknowns and unknown unknowns, which over the last year
alone have exploded (e.g. actual and potential Intel microcode
vulnerabilities). If you needed this article to convince you of feasibility or
even just practicality, please don't pretend to be capable of assessing the
security posture of any complex system.

And let's be clear: this is _not_ a new and novel attack vector. There are
companies that have existed for quite some time researching and selling
products to deal with this sort of attack vector. On the spectrum of hardware
based attacks feasible today, this chip isn't at the complex end of the
spectrum but rather the simple end of the spectrum. The complex end of the
spectrum involves hiding logic deep within existing ICs, and there's ample
literature to demonstrate feasibility of both implementation and detection.

The difficulty in pulling off these attacks lies not in the software or
hardware, but the political, intelligence, and military apparatus of attacking
countries. The economic costs of detection are huge precisely because, at the
end of the day, fundamental security relies on trust, not technological
hurdles per se.

------
dphack
In the security world, there is always a new attack vector. With the advent of
Spectre and Meltdown, I am expecting to see more hardware-based attacks in the
future.

~~~
abledon
and subsidies to enrol more students in EE to build up local national talent?

------
exabrial
Regardless if someone of the details turn out to be true, there's a couple of
really important network security takeaways here.

First you need to run an air gapped network or at least switch enforced vnet
for your bmcs. Three firmware on these is only updated every few years anyway,
so they're likely full of security holes anyway.

Second, in general, outbound connections need to be monitored everywhere in
your data center.

------
docker_up
Is this really that hard to imagine?

I am willing to bet that there are teams of spies who have infiltrated Google,
Facebook, Amazon, etc. Spies from US, Russia, China, Britain, Israel, Germany,
etc, must have dozens of spies working as engineers are getting access to all
that data as we speak.

To think that they aren't would be rather naive, in my opinion. If I were head
of the spy agencies in any one of those countries, it would definitely be the
first thing I would do.

~~~
briatx
It's not at all hard to imagine, especially from a country who is so paranoid
about backdoors from other country's operating systems that they have written
their own:

[https://en.wikipedia.org/wiki/Kylin_(operating_system)](https://en.wikipedia.org/wiki/Kylin_\(operating_system\))

~~~
sho_hn
This is a Linux distro, not "their own OS". Many countries all over the world
have had government-subsidized Linux distro projects, at national, state and
city government levels. It's something that happens pretty quickly whenever
and whereever a large deployment happens (e.g. LiMux).

For that matter, many a CS faculty at many a university have _actually_
created their own OS. Nothing too sensational either.

~~~
briatx
Pedantry aside, it is the reasoning behind building their own OS distro that
is relevant here:

> In 2009, a report presented to the US-China Economic and Security Review
> Commission stated that the purpose of Kylin is to make Chinese computers
> impenetrable to competing countries in the cyberwarfare arena. The
> Washington Post reported that:

> China has developed more secure operating software for its tens of millions
> of computers and is already installing it on government and military
> systems, hoping to make Beijing’s networks impenetrable to U.S. military and
> intelligence agencies.

~~~
sho_hn
Sounds perfectly sensible, rather than "paranoid".

------
TickleSteve
QSPI has 4 _data_ lines, a clock and a chip-select line.

In order to intercept a QSPI bus, you would therefore need 4+1+1+2xpower lines
= 8 pins.

This is not sufficient for QSPI.

Single SPI on the other hand requires Clock, Data, ChipSelect, Ground & Power
= 5 lines, so that would be plausible.

(Yes, I know you can potentially get rid of the CS line, but that depends on
what else is on the bus).

~~~
mindslight
You need two lines - data/ground, or data/Vdd. Probably whichever gives you
voltage when idle.

You harvest power when data is idle. CS is irrelevant. The rough clock speed
is fixed, and you can match the precise timing from the data line. QSPI
actually gets you access to data in _both_ directions with the tradeoff of
only getting one quarter of the bits.

Logically, you likely only need to recognize a few patterns that are each say
30 bits long. If you could shrink this down, it would look exactly like a
boring pullup.

~~~
TickleSteve
CS isn't irrelevant if there are multiple devices on the bus, we havent seen a
schematic.

~~~
mindslight
Yes it is - a tap doesn't need to be 100% sure that a given chip is selected
like how we think when designing a circuit - it just needs to key off patterns
that look like memory reads of interest. If you've got ambiguity you don't
need to pull in more parallel bits, just record a longer serial history.

In fact I would think the most stealthy and robust way of bugging code would
be to just ignore addresses and look for a context-free stream of instructions
that does some security-sensitive initialization (either of the platform or of
the application code). Turn off some flag that's supposed to be on, causing
the code to continue running as normal but silently skipping certain checks.

Now that you mention other devices on the bus - those in fact would be a great
chip to subvert with such a backdoor. Why bother fitting everything into a two
terminal hack package when you could just have a malicious temperature sensor
sourced from the "lowest bidder" that does the attack and jumps to its own
ROM? (I think those are usually I2C but you get the point). Bonus points for
only being triggered through some sidechannel - recruit a low level employee
at the target, while they retain plausible deniability.

------
wyldfire
> There are few facts, and much supposition.

Can't we watch it in action? Probe it and run the system, watch for activity
on the SPI lines.

Can we dissect it and look for how it might interact with the world? Does it
have anything like an RF front end, or does it wait for some specific
BMC/network activity before triggering?

~~~
creeble
We could do all of those things if anyone were to come forward with actual,
physical evidence (i.e., a board).

But so far no one has.

------
jt3
Would be bad if a DoS attack could occur by killing the chip at a certain date
or on command. Imagine electronic devices going down, and only way to fix is
to get new hardware...ouch.

Maybe this will open up more manufacturing jobs in the US.

------
osterwood
If this attack vector is true, it seems overly complicated to me. Why not have
the assembly house load the normal SPI flash with firmware you've altered to
control the main CPU the way you want? That is WAY easier than hardware
changes to the PCB design and installing a part which isn't on the approved
Bill of Materials.

------
bobbob1921
every supermicro motherboard that I have dealt with (several 100s) always has
a jumper which can enable(shorted)/disable the bmc/ipmi. i’ve been surprised
at how many we have bought used off eBay/surplus that had the jumper open (ie
bmc disabled). wouldn’t having this disabled physically (jumper open), thwart
this attack?

(i’m asking not suggesting)

and I do realize the majority of people do have the BMC enabled, including me)

also the fact that supermicro still offers this jumper to disable the BMC
might speak to the fact that they (sm) are not in nor had any knowledge of
this issue (ie they were not “In on it”)

------
dboreham
So the point of the attack was to subvert the BMC firmware in a way that was
resistant to firmware re-flashing, but not resistant to firmware integrity
checks. But there are no firmware integrity checks?

~~~
baybal2
What is likely: they connected their own flash in place of recovery flash,
that was not soldered on on that particular board version, that for what that
soic8 or tsop8 pad is on photos; the chip will boot from the recovery, but if
someone were to read the main flash, it would output the correct content. Only
if somebody specifically poked the recovery, would the bug be revealed.

------
mirekrusin
If it is false, can/will/should Apple/Amazon sue Bloomberg?

------
VCinvestorSF
This story is ridiculous. I believe the arguments made int eh article didn’t
happen. Apple, amazon, and SMCI have all rebuffed. The Bloomberg journalist
was duped. SMCI did nothing wrong and their business remains strong as we
speaker. They will shortly: 1) Issue a detailed rebuttal with QC procedures
and 2) become current with their audited financials by the end of October. As
a result, the stock will revisit $30 per share within 3-6 months which is a
triple from today’s price. Who remembers the Johnson and Johnson Tylenol scar?
Someone posiomed Tylenol and the stock cratered. This is no different, except
the “poisoning” never happened. And even if it did, it was 3 years ago and was
discovered.

------
aussieguy1234
Smartphones, both iPhone and Android are manufactured in China

------
alexnewman
These types of attacks are distractions from the real attacks. Doping attacks
are the big attack vector now, and they can't be picked up with electron
microscopes

------
blitmap
Protecting tech in transit would be a new business. Tamper-evident packaging
and such.

~~~
GaryNumanVevo
obviously we need block-chain now, I'll start up the ICO generator script

------
cimi_
> But there’s another trick a bad BMC can do — it can simply read and write
> main memory once the machine is booted.

Doesn't ASLR[0] mitigate this?

[0]
[https://en.wikipedia.org/wiki/Address_space_layout_randomiza...](https://en.wikipedia.org/wiki/Address_space_layout_randomization)

~~~
ATsch
I don't see how it would. ASLR only really helps making it harder for
attackers to gain full control when they manage to execute some instructions
in your process via memory corruption. It relies on the memory layout being
hard to guess, however, the BMC can already just read from arbitrary memory,
so it can just look it up. What would help here is isolating PCIe devices with
the IOMMU, but this is currently rarely enabled, only for virtualization,
apparently due to it's relatively high overhead.

------
PaulHoule
The NSA has been caught doing this kind of stuff to Cisco routers so I can
believe it.

------
novaRom
This chip looks for me like a simple fuse. This is not for spying, but rather
a doomsday switch.

If activated, it will short the circuit destroying full system operations. I
don't even think it contains any sophisticated digital logic inside.

------
shawwn
Test comment, please ignore.

------
romed
I’m just going to throw this out there: BMCs and ILOMs are for tiny shops
where “the IT guy” might have to do something from the beach at Cannes on
their vacation. If you are a large operator like Apple you absolutely do not
need BMCs.

~~~
acdha
Not just tiny – any place where you don't want work to be rate-limited by the
time it takes to get a body in front of a server. The only scale where it's
not a big win is when you're so large that you have rock-solid auto-recovery
in your software stack and hardware failures are handled by disabling a server
and leaving it in the rack until it's periodically reaped at a regular
interval.

Most businesses are in that gap – definitely not tiny, but definitely not able
to just ignore remote management – especially if they have lots of physical
locations and it's not practical to have 24x7 IT staffing at every branch
office.

~~~
romed
Remote office management can be handled by a KVM with remote access. These can
be security hazards in their own right, but it’s a lot safer than having some
vendor junk literally hard-wired to your PCI bus.

~~~
acdha
Now you have a second device to buy, secure and support, and it can't do
everything that an ILOM can (most importantly, remote power management).
Having had security updates for KVMs, I'm also not sure your assumption that
it's safer is true in any meaningful sense.

Management engine vulnerabilities have gotten a lot of hype over the last year
or two but most of it has been marketing for security companies rather than a
serious risk (e.g. what percentage of bugs required local network access,
which should not be easy to get on your management VLAN). During a similar
period, we've also had Spectre/Meltdown and the usual slew of software
vulnerabilities so I have trouble with the conclusion that the answer is to
stop using a useful tool rather than continuing the industry-wide effort to
improve the level of security competency among development teams.

~~~
romed
For many BMCs the second network port is a figment of your imagination. The
BMC is capable of intercepting and injecting network frames on the host's
interfaces. Just because you have the management port wired to a different
VLAN or even a separate physical LAN means nothing.

~~~
dboreham
BMC NIC is generally capable of being bridged to one of the main NICs,
although it is possible to disable that feature, but also possible to override
the disabling if you control the hardware.

