
Supermicro boards were so bug ridden, why would hackers ever need implants? - drewg123
https://arstechnica.com/information-technology/2018/10/supermicro-boards-were-so-bug-ridden-why-would-hackers-ever-need-implants/
======
rayiner
I feel like this article reflects some significant technical confusion. The
BMC is supposed to be on a trusted network inaccessible from the outside. I've
always viewed authentication on the BMC as being like the numeric lock on
luggage--it's designed to keep honest people honest, not for real security.
Being able to bypass the BMC security is really not a big deal. What the
Bloomberg article says about the hardware exploit is much worse:

> > The rogue instructions, Bloomberg reported, caused the BMCs to download
> malicious code from attacker-controlled computers and have it executed by
> the server’s operating system.

It's using the fact that the BMC has unfettered access to the rest of the
machine to compromise the code running on the server itself. That's valuable
even if the BMC itself is on a private network inaccessible to the attacker.

~~~
stephengillie
BMCs like DRAC or iLO are invaluable when you have hundreds or thousands of
fresh servers with no OS. The BMC lets you mount an OS or hypervisor ISO in a
way reminiscent of DaemonTools _et al._ , and update bios and other firmware
from a shared network folder. I'm pretty sure there's even an API to develop
against.

~~~
rayiner
BMC's are great--all my home builds have them because I'm too old to be
fiddling around trying to figure out why a computer won't boot an installer
from a USB key. But even on my home network the BMC's are on a separate switch
on a subnet that doesn't have internet access except through a VPN gateway.

~~~
jacquesm
> except

There's your problem.

~~~
xyzzy123
I think the parent meant, you have them on a subnet with no default route, but
have a vpn / management system with one interface pointing in to the
management network. You can get in via the vpn but they can’t get out.

It’s a pretty common configuration.

------
tannhaeuser
If Supermicro boards are bug-ridden, then I'd expect other manufacturer's
boards to be equally bad or worse. I don't have a reason to defend Supermicro
or some such, but where else in retail do you get specialized server boards
like Supermicro sells? When Opteron was relatively new in 2004, I bought a
two-socket board from Supermicro as the alternatives from well-known Taiwan
manufacturers (ASUS, MSI, etc.) weren't as sophisticated (hadn't the PSUs and
PSU connectors, power ratings, and rack-mount/tower convertible enclosures).

------
toss1
>Supermicro boards were so bug ridden, why would hackers ever need implants?

Ummm, because if you need your hack to be reliable, you can't rely on someone
else's bugs to be there when you need them. You never know when they'll be
fixed, or just replaced by new bugs.

A long time ago when setting up computers and networks was driver version
hell, we had a short list of manufacturers' computers that we'd do setup
included in the price instead of on-the-clock. This came about when a shipment
of about 20 Dell computers, all supposedly of the exact same model# and
revision, required about about 11 different setups, because the various chips
on the board were different. They were clearly just using the chip-of-the-
week>from whatever supplier was cheapest -- great for their price points, but
every variant required a different driver for some subsystem. So the list was
created and Dell was not on it (it was IBM, Compaq, HP, DEC, to show when this
was).

That's solved now by hiding it with the much more automated OS and networking
setups, but it is easy to see how the Chinese spies would be in the same
situation -- some buggy boards are wonderfully exploitable, but how do you
tell that the version going to your target wasn't changed by some revision
that wasn't even noted in the Rev- listings? Better to insert your own bug if
you want to actually get the job done.

------
mhjas
I don't really see why everyone is calling this implausible. Modchips have
been around for at least 15 years. The idea of the clipper chip is 25 years
old. At every hacker conference there are people "hacking" devices by various
buses or interfaces.

If there is anything working against the Bloomberg story it is that it is too
plausible. Often reality clashes with imagination, but the Bloomberg story
contains almost everything you could imagine happening.

~~~
gvb
It isn't implausible because of it being difficult and expensive, its
implausible because there _already exist_ much easier, cheaper, and (arguably)
harder to detect ways of subverting SuperMicro motherboards.

As a bonus, subverting the BMC firmware is much harder to trace to the source
since it could be injected by in so many ways by so many different people.

Why use a thermonuclear device when a hand grenade accomplishes the goal?

~~~
mhjas
I just don't think the relationship between those two things you are
describing exists. If the Chinese government approaches a Chinese manufacturer
with the goal of compromising US software companies adding some sort of chip
that reconfigured the hardware would be the most straight forward thing for
them to do.

If anything I think the idea that a Chinese manufacturer with complete access
to the hardware having to execute some exploit towards the web interface to
get access is far fetched. So is that you could pretend to update the firmware
(surely no one is going to notice that the new version doesn't have the
features you wanted?) and that dumping the firmware would be inconvenient (it
would be the first thing you did if you suspected something).

~~~
gvb
The "chip that reconfigured the hardware" is already built in; it's the BMC.

All the Chinese government has to do is go to the factory and tell them "flash
the BMC firmware with this image" where the image is subverted (but
operationally indistinguishable) BMC firmware. It doesn't get much more
straight forward than that.

~~~
vel0city
There are attacks where flashing a malicious firmware on to the device
prevents real firmware flashing (just updates version numbers, re-infects the
flashing payload on write, etc). However, those attacks can be mitigated by
physically connecting to the flash module and writing to the device directly
through SPI. If you've got a chip between the BMC and the flash memory as the
report suggests, it can re-infect the memory even when you're done. You could
even read the contents of the flash memory directly and find no evidence of
the attacker, as the attack code might never actually write to the memory and
may only load when the BMC boots and attempts to read from the flash memory.

------
throwaway290342
Okay, crazy tinfoil hat time: what if this story is a plant from a particular
part of the Chinese government (like PLA Unit 61398), designed to give the
impression of the ability to disrupt global supply chains and to build respect
through fear?

If all of these unnamed sources are unnamed because they were adversarial
members impersonating government officials, then that would make a little more
sense why current government bodies are not just staying mum, but actually
denying knowledge of the story.

With the software attacks being much more feasible as the Ars article points
out than a hardware attack, then it would also make it so that the vehement
denials from affected companies would be true as well. The whole thing could
be a large disinformation campaign to strike at the very core of what many
would otherwise consider reasonable security.

~~~
freeflight
No need for that much tinfoil, this came in parts straight from the Pentagon
[0] and Bloomberg's "specialist", Tavis Ormandy, turned out to have a vested
interest in selling "cyber security" related products aimed at supposedly
fixing exactly these kinds of supply chain problems [1].

Imho The Register also points out some interesting details about this whole
thing [2]

It's not really that surprising, fits perfectly into Trump's narrative of
"They took our manufacturing, it's time to take it back to the US!". Gotta
start somewhere, telling everybody China is selling a lot of bad apples seems
like a simple enough start.

[0]
[https://s3.amazonaws.com/static.militarytimes.com/assets/eo-...](https://s3.amazonaws.com/static.militarytimes.com/assets/eo-13806-report-
final.pdf)

[1]
[https://web.archive.org/web/20170721190725/http://www.sepio....](https://web.archive.org/web/20170721190725/http://www.sepio.systems/assets/Sepio_Data_Sheet.pdf)

[2]
[https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...](https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/)

~~~
gjm11
Do you mean someone else rather than Tavis Ormandy? As someone else has
already pointed out, he's at Google Project Zero (which isn't in the business
you describe) and I don't think he's ever worked for the company whose
brochure you linked to, and so far as I can see he's been pretty rude about
the Bloomberg story.

------
HillaryBriss
One researcher criticizes this type of hardware attack, saying:

 _Once discovered, such an attack would be burned for every affected board as
people would replace them._

But this article also points out a case where, even after SuperMicro had
published a patch to a serious BMC firmware vulnerability, 32,000 servers in
the wild had not been updated _a year later._

So, if software updates aren't always speedily/reliably deployed in the wild
by customers, can we really expect hardware to be speedily replaced?

------
mannykannot
While I don't think Bloomberg's story looks very plausible, perhaps one
motivation for cryptic hardware modification at a time when firmware
weaknesses were being discovered might be precisely because the easier-to-
exploit firmware weaknesses were being discovered, and so might not be
exploitable much longer? It might not have seemed plausible that the
vulnerabilities would be discovered but then not fixed to the extent that, it
turns out, they were not.

------
ryanmarsh
_“There are so many far easier ways to do the same job. It makes no sense—from
a capability, cost, complexity, reliability, repudiability perspective—to do
it as described in the article.”_

Considering the US went to the trouble of wiring the North Atlantic for sound
to catch Russian submarines during the cold war, and tapped undersea cables
using divers and submarines, this is so implausible for a nation state? Large
state actors specialize in activities for national defense that make "no
sense—from a capability, cost, complexity, reliability, repudiability[sic]
perspective".

~~~
crististm
There are even more recent examples with software: Stuxnet

------
ThenAsNow
While it's ultimately going to help to shame vendors regarding their poor
security practices, it's really irritating and unfortunate this is all being
framed as a Supermicro issue. How about the other companies in the same market
space, like Tyan, that I'm sure are no better? For that matter how about the
"Tier 1" OEMs like Dell and HP - how well-written are their BMC firmwares?

------
zelon88
Not saying I believe in one side or the other, but from a standpoint of
avoiding detection I think firmware hacking goes out the window.

A deep-pocketed attacker isn't going to risk flashing the firmware with a non-
oem one on a brand new board leaving the factory. That probably gets quality
inspected somehow later on anyway whereas a visual inspection is just a rubber
stamp (IE: OK if the box isn't crushed or wet).

Not to mention a customer in the field who experiences problems is likely to
report their firmware version to Supermicro support, whose poking around could
expose the entire project.

There was an article recently about how hardware is "magic" and the IT world
mostly takes it for granted. Putting an extra chip on the board but making it
completely transparent to software debugging techniques is the best way to go.
The board is almost certainly going to be flashed at least once and probably
audited several times in it's lifespan by IT, but the hardware is never going
to get more than some compressed air blown on it. Nobody repairs these things
at the component level on a scale that matches how frequently firmware gets
flashed or checked out.

~~~
eridius
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed
inspect the hardware to make sure it hasn't been tampered with. I know less
about Amazon in this regard but I would expect Amazon to do at least some
level of hardware inspection to detect tampering as well.

~~~
wahern
They also do extensive code reviews, including of imported open source, yet
their software is hardly bug free. Bugs can linger for years, often fixed only
because someone stumbled upon odd behavior.

Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of
other chips? That doesn't do anything unless it's tickled in just the right
way? If you believe Amazon and Apple are even remotely capable of protecting
against that....

The solution to these problems is to put critical code and critical secrets on
discrete, _simple_ SoCs where you actually have a chance of defending both
hardware and software attacks. Apple and Amazon understand this because they
already do it. The difficulty is building your software systems (firmware,
kernel, etc) to make use of these secure elements, not to mention making them
available for ad hoc application software. It's an extremely difficult
integration problem, and even when you succeed you haven't.

For example, AFAIU Amazon's servers have secure elements to perform
attestation of the box; it's utilized by their hypervisors to authenticate VMs
for things like KMS. But it can't actually protect the data in the VM itself,
such as the secrets obtained by dint of the attestation. It can't even prevent
taking control of the hypervisor. All it does is help Amazon define a fixed
security parameter--that you can't impersonate their hardware nodes on the
network. That's extremely useful, but ultimately extremely limited.

~~~
eridius
Code reviews for bugs is very different than security reviews to look for
malicious tampering. Apple may have bugs that linger for years, but I'm not
aware of any documented case of someone managing to slip a backdoor into Apple
software via an open source package.

> _Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch
> of other chips? That doesn 't do anything unless it's tickled in just the
> right way? If you believe Amazon and Apple are even remotely capable of
> protecting against that...._

Why not? X-raying every board, inspecting every single component, making sure
it matches up with the documented specs and perhaps with a proven-good
board... if you're replacing a component with a different one, or adding a
component that wasn't there before (which is the case in this alleged attack),
even if the component _looks_ harmless and even if it's tiny, it can still be
revealed by a detailed comparison of the board against specs. A component
that's the size of a grain of rice is still a component that can be detected.

~~~
wahern
So are you saying that they x-ray every board now?

------
jiveturkey
Very fair article. Raises doubt in a very productive way, not the he-said she-
said of previous rebuttals.

I'd go further to say it isn't just about the accuracy of the bloomberg piece,
but implies bad things about their journalistic integrity. I mean, get real,
Ars doesn't have an investigative journalism team. The one-sidedness of the
bloomberg article becomes very apparent.

------
rbanffy
It depends on what you want to do. If you want to extract information from a
specific network. maybe custom firmware is a good option.

If you want to just disable a very large number of machines to create economic
damage or cripple infrastructure, a hardware implant would do just fine. And
you wouldn't need to be very careful as to where it ends - if you make enough
of them, they'll be everywhere.

If 1% of all MacBooks have a similar backdoor, there are about a dozen at my
building.

------
csours
BMC bug story time: I was working on automating health checks, and I needed
some information from a BMC. The information was provided in XML format...
fixed width. It's like something produced the document, and then output it to
console, then copied from console to web service output.

------
qaq
I would guess that large companies are refreshing with known good firmware
before deploying servers? So while described approach is easier prob will not
get attacker as much.

~~~
vel0city
Most BMC updates are handled in software on the BMC. You're giving the BMC a
new image file to write and trusting the BMC to actually write it. Who's to
say the BMC is dutifully writing that image to the flash memory? Who's to say
it doesn't re-infect the image before writing?

Even if you do directly connect to the flash module and directly write to it
through SPI, if the attack is being loaded by an additional module between the
flash memory and the BMC, it could still inject additional data into the BMC's
boot. If you're not physically listening to the SPI data being transmitted or
knew what to look for in the final environment of the BMC, you wouldn't know
it had happened.

------
xkcd-sucks
Okay, crazier lead-foil hat time: what if this story is a crappy hoax intended
to discredit/prevent from publication a real story with similar details?

~~~
bdamm
I don't follow. If anything it makes the parallel story easier to publish as a
sort of "me too" (no disrespect.)

Maybe it takes away from the firmware hacking version of the story because now
folks are looking at components as being the source of hacks and not the
firmware on the components, leading to a false sense of security when they
invest mightily in analyzing components with X-rays? I could see that outcome
as being plausible. If the ultimate outcome is simply to change corporate
priority towards futile component verification and away from firmware
verification then indeed the firmware verification vector remains safe for the
attacker.

~~~
xkcd-sucks
More like "Oh look, another story about extra components inserted by a big
state actor. Bloomberg just got burned for this, I'm not going to risk my/my
organization's reputation on the slight chance that this one is real"

Or, "Extra chips on the motherboard? You're about a month behind the news
cycle and didn't you hear it was all BS anyway"

But the false sense of security interpretation is plausible too.

I wonder who holds conferences on the cutting edge research of manipulating
media

------
zbentley
> a binary file that stored administrator passwords in plaintext.

I understand what they mean, but that sentence still hurts to read.

------
alexeiz
The purpose of this whole thing was to manipulate the market. Super Micro
stock fell 50% and still has not recovered since October 4th. Before the
report its trading volume was invisible. After the report the volume
experienced almost 2 orders of magnitude increase.

