
Supermicro hardware weaknesses let researchers backdoor an IBM cloud server - bauc
https://arstechnica.com/information-technology/2019/02/supermicro-hardware-weaknesses-let-researchers-backdoor-an-ibm-cloud-server/
======
Twirrim
There's a good talk on securing bare metal cloud servers against such attacks,
that was given at B-Sides in Portland, by two respected security people who
worked on securing bare metal cloud infrastructure for one cloud provider:
[https://www.youtube.com/watch?v=PEVVRkd-
wPM](https://www.youtube.com/watch?v=PEVVRkd-wPM)

Boiled down it comes to: Trust absolutely nothing. When a customer finishes
with a server, wipe absolutely everything, re-flash every single bit of
firmware on every single device in the machine, and _don 't_ use the standard
flashing mechanisms to do so. It's worth a listen/watch.

------
jlgaddis
> _This indicated that the servers ' BMC firmware was not re-flashed during
> the server reclamation process._

I'm not surprised. Unless you pay for their "enterprisey" datacenter
management products (which are still relatively new), it's a PITA to perform
BIOS and BMC firmware updates. Additionally, Supermicro specifically
recommends that you _DO NOT_ flash the firmware unless you are experiencing
issues that a new version is suppose to fix -- unlike pretty much every other
vendor (like Dell, who makes it fairly easy to do so).

~~~
greglindahl
Flashing the Supermicro BMC is so easy and reliable that I used to do it
frequently.

Flashing the Supermicro BIOS, yeah, that's a disaster.

------
cmurf
Could someone speak to the write endurance of BMC flash, vs whatever flash
BIOS/UEFI lives in, vs NVRAM where now certain boot parameters are stored for
persistence? Is it a total non-concern? It's not a user replaceable part.

------
rbanffy
It's a bit shocking that the people who develop the firmware for these
embedded computers is not more concerned with their security. I understand you
are not supposed to attach them to untrusted networks, but can we really call
any network trusted these days?

Do people still teach about the Maginot Line in schools?

~~~
dsfyu404ed
>Do people still teach about the Maginot Line in schools?

No.

There are plenty of other opportunities to teach the value of making your
adversary (be that adversary man made or natural) work every step of the way
and they are missed too. Nobody (for large values of "nobody") gets taught
about the value of layered or redundant systems until college and those that
do get taught it in college usually only touch on it in their mandatory ethics
elective.

FWIW I named the last firewall I configured "Little Maginot".

~~~
abbracadabbra
Another (perhaps more) common use of the term Maginot Line is as a metaphor
for expensive efforts that offer a false sense of security.

------
Zenst
This highlights how the sterilization process many cloud providers (IBM in
this instance) have is not cleaning out every nook and cranny.

They should, audit every bit of firmware (indeed it's odd how the researchers
changed one bit in the BMC firmware and no checksum flagged it up on boot) and
whilst this is daunting, it isn't that hard as they just have to compare and
verify it is the same as the known safe image. Sure they could blindly
reflash, but then they would miss any attempted expliotations and equally
shorted the life of the hardware by increasing the odds of the flash memory
failure.

Whilst people see BMC's as one avenue, a server/pc has many components, all
with their own firmware and in many cases, own CPU. Be that a network card,
graphics card and even keyboards and mice (though the later, not so much a
factor in server environments, still a consideration).

Security is and always will be a mindset. You need to think like somebody who
wants to break into your environment, and then counter those ways. But so many
avenues. Imagine your sat at your desk as an administrator and one morning you
get a nice shiny, cool top of the range keyboard sent, dressed up as a gift.
How many would think, cool, plug it in and feel all fuzzy? How many would
audit the firmware on that keyboard? How many would question the random gift
at every level?

I'm sure IBM are not the only ones who would fall foul of this avenue of BMC
exploitation, but I'm disappointed that for me, basic sanity checks in their
sanitisation process to decommission and recommission a server are being
overlooked.

Still, when you hire a car - do they audit the cars management engine
firmware? Do they erase previous BT and WIFI connections stored on the radio?
Well, from my experience - they don't.

Remember - you can pay an expert all the money in the World, but do check
their work.

~~~
jacquesm
> Do they erase previous BT and WIFI connections stored on the radio?

I've pulled many contact details from cached data on rental vehicles. Always
worth checking what the stuff you pair your phone with asks for and keeps.

------
jfindley
Is there any chance they have stumbled upon the same weakness that Bloomberg
tried to report[0]?

The industry at large has been pretty sceptical of Bloomberg's claims, and
rightly so, but what if they just got the details wrong and it was this (or
similar) vulnerability in the BMC software, rather than a dedicated spy chip,
that they meant to write about?

0: [https://www.bloomberg.com/news/features/2018-10-04/the-
big-h...](https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-
china-used-a-tiny-chip-to-infiltrate-america-s-top-companies)

~~~
mtgx
I think it's possible Bloomberg's "tiny chip" was a BMC, and that BMC was
found with a backdoor in it, but the backdoor may not have necessarily come
"from factory," but was later added by someone else.

Either way, this looks terrible for SuperMicro: "Yeah, our servers don't come
with built-in backdoors - it's just SuperEasy™ for attackers to add one once
shipped -- please buy more now."

------
londons_explore
The article looks written to hurt supermicro here.

It looks like the blame squarely lies with IBM (for not correctly resetting
the BMC between users of the machine), or Intel (for a poor design which
allows this in the first place)

------
paulfurtado
I'm a little curious why the BMC hardware is even exposed to the customer at
all. Is it provided as a feature or is this just an oversight? Can't they just
hide all of this hardware from the customer's OS?

~~~
toast0
Bare metal customers get access to the whole machine. We get to access the BMC
for remote management (console, power management, etc). Apparently, you can
configure the BMC via tools running in the host OS, without needing to
authenticate to the BMC; I've used this to fix BMC provisioning errors, that
would otherwise require a SL tech to fix via physical console in the bios
screen.

~~~
paulfurtado
Ah, interesting. I wrongly assumed the SoftLayer bare metal servers might be
closer to EC2's bare metal offerings where, from the outside, you're given the
same set of management tools as a VM, and from the inside, a fairly limited
hardware surface area is exposed to you.

------
wyldfire
Does this vindicate Bloomberg or is it a different BMC vulnerability?

~~~
Twirrim
Totally different. This is a compromise via a known firmware issue with the
standard components on a motherboard.

The Bloomberg article was about a hidden chip being installed on a motherboard
that provided a backdoor.

~~~
wyldfire
Strikes me as odd that they didn't acknowledge the recent history of reporting
on Supermicro BMC issues (in order to clearly distinguish them).

~~~
jlgaddis
If you read about both, it becomes pretty clear (at least, it did to me) that
they aren't related.

Bloomberg's story was about a (supposed) backdoored chip installed into the
server at the factory, this is about flipping a bit in the firmware installed
on the existing BMC.

