
What Happened with Supermicro? - equalunique
https://hackaday.com/2019/05/14/what-happened-with-supermicro/
======
lsc
Huh. so shortly after the story broke, I bought a bunch of SMCI. (It's
actually the first OTC stock I've ever bought. It was easier than I thought,
too... I had to call vanguard to enable OTC trading, but after that? super
easy.) I sold it months later, at a pretty good profit, mostly 'cause while I
think the bloomburg accusations are clearly false, there _are_ huge accounting
irregularities surrounding supermicro; getting delisted is no joke, and I'm
super not qualified to evaluate that sort of risk.

but, my reasoning? my reasoning was that if it was a widespread physical hack,
someone would discover it fairly quickly; I mean, people were tearing these
boards apart, looking for anything. There's no way they could keep that quiet,
even if they were willing to disappear people.

I can tell you that within my peer group, Bloomberg has taken a pretty huge
reputational hit. I mean, sure, we all still read 'money stuff' because it's
"the daily show" for financial news. It's great entertainment, and moderately
edifying. But, you know, it's one of those things where after a newspaper
writes about something in your field, you start wondering if they are that
ridiculously uninformed about the things that aren't in your field.

So, I felt really pretty safe with buying in the dip.

~~~
cantrevealname
> _I mean, people were tearing these boards apart, looking for anything. There
> 's no way they could keep that quiet, even if they were willing to disappear
> people._

Without commenting on whether or not I believe the Bloomberg story, I very
much doubt that "people were tearing these boards apart", doing reverse
hardware-level engineering looking for an unknown. Even in the case of open-
source software, with _full source code and documentation available_ , the
number of people who analyze code is miniscule.

Doing the same thing with ICs and PCBs, without engineering diagrams and
documentation, and needing very specialized skills and tools, must be 1000x
more difficult. Maybe Supermicro and some intelligence agencies did internal
studies, but they aren't going to tell you anything. Perhaps Apple and Amazon
asked around in their engineering departments to see if anyone saw something.
Possibly a couple security researchers took a look to see if they saw
something obvious. But I don't think teams of experts were mobilized all
around the world trying to pinpoint a deeply hidden hardware vulnerability.

I'd love to be proven wrong. Hacker News is read by ~50,000 engineers and
programmers and related professionals a day. Admittedly my comment won't reach
50,000 people, but I'll ask anyway: Is there anyone reading here who
personally analyzed a Supermicro board in any meaningful way, or knows anyone
who personally did so?

~~~
derefr
I mean, it's not _that_ expensive to just take some hi-res X-ray photos of old
and new boards of the same make, and compare the trace paths near each
component. Maybe you can even take a visual diff of the X-ray negatives, if
you can get everything lined up tightly enough.

You can parallelize this search, because the people doing it don't need to
know much about _what_ they're looking for—they're just hunting for visual
differences. Unless/until one of them actually finds anything different. At
that point, you need better resources to figure out _what 's_ different; but
at that point, all hell also breaks loose, and you'll get all the resources
and help you need to do better analyses.

~~~
cantrevealname
> _Unless /until one of them actually finds anything different ... at that
> point, all hell also breaks loose_

Boards bought weeks or months apart might have lots of differences in traces
and components due to engineering modifications and normal changes in
suppliers. Do you really think that all hell would break loose if you showed
that there were differences between a Supermicro board made in March 2018 and
one made in September 2018? Other than _maybe_ Supermicro or intelligence
agencies (who aren't talking), I doubt that anyone has done what you've
suggested. People are busy and it's a lot of work based on an allegation and
no exact place to look.

~~~
errantspark
Is that really true? Granted I'm working on low volume stuff (1000's of units)
in a niche B2B setting, but the times for rolling out a new rev of a board can
sometimes be measured in quarters. Changing traces/components can have
unforeseen effects for even fairly minor things, you can't just have a Tier1/2
CM tool up without a round of DVT. I doubt boards with the same PN change very
often if at all, but I'd be curious to know if i'm wrong.

------
clay_the_ripper
Somewhat unrelated: we used to do a lot of business with supermicro. In a
meeting one time I asked one of their guys why their billboards are so weird
looking (anyone who’s driven down 880 past their building knows what I’m
taking about). They have these billboards that have weird slogans and super
amateur looking graphics that look like they were made in MS paint.

So I asked the guy about them cause I thought it was funny that such a big
company had such cheap looking marketing. And he said: “yeah our CEO couldn’t
believe how much money some agency was going to charge us to design a
marketing campaign, so he just did it himself on his computer in like 2 days”

I thought that was hilarious. But good on the CEO for realizing no one really
cares what a server billboard looks like, and decided to save the $2MM or
whatever someone was going to charge him for design. Super funny.

~~~
blululu
The best photo I could find: [http://meritage-partners.com/site/wp-
content/uploads/2016/05...](http://meritage-partners.com/site/wp-
content/uploads/2016/05/www.wired_.comwp-
contentuploads201605GG3A0045-0a9ed03799d59508b280c14de62cf6867b2ab505.jpg)

I love that old school Silicon Valley aesthetic.

~~~
jhallenworld
That's funny, I had no idea they had billboards.

Are those the $1M Silicon Valley homes I hear about?

~~~
closeparen
Those look like mobile/manufactured, so in the $400-600k range.

~~~
chipotle_coyote
Well, around $200-300K, based on checking prices on Zillow for what sure
appear to be spaces in that mobile home park (around Oakland Road in San Jose,
ZIP code 95131). I don't think I've ever seen a mobile home listed for a half-
million even in this area, although I've seen ones the mid-$300s. (Granted,
this is still nuts, given that in Sacramento or Tampa, the other places I've
idly priced homes, these would be under $100K.)

~~~
twblalock
What's amazing to me is that people are willing to pay high prices for a
manufactured home in Silicon Valley even though the likelihood of the mobile
home park's land being sold to developers is higher than anywhere else.

This has already happened in a few places in the area. In Sunnyvale there are
massive developments of million-dollar condos literally next door to mobile
home parks. How much longer can those places survive?

One would think that the risk would make people less likely to buy. There
aren't very many places in the area for people to move their homes to if the
land they are sitting on gets sold.

~~~
tru3_power
Why is there such a disparity in quality of life for tech workers in SV? It
seems like the housing crisis there will eventually stifle productivity (no
one can afford to move there for entry level roles without giving up a ton).
Are there some special dynamics that cause these types of scenarios except for
the fact that a lot of people got rich quickly in one place?

~~~
closeparen
Entry level roles are for fresh graduates. A converted living room is a step
up from college and compatible with aggressive savings on a big-co wage. Entry
level engineers willing to run smaller savings rates can even afford to live
alone, though it can be a stretch.

------
redwards510
Felt pretty bad for SuperMicro after they took the huge financial hit from the
story which offered no physical proof. Glad to see they mostly recovered from
it.

The story was extremely interesting, but as people dug into it, it seemed like
the reporters had a bunch of conversations like this:

Security Researcher: So, in theory, you can do a lot of crazy stuff! Embed a
tiny chip on a motherboard, stick it in an Amazon datacenter, sniff all sorts
of things...

Reporter: Are you winking right now? I swear you are winking! So you are
saying this is true? OMG. Wow, what a bombshell!

Security Researcher: I was not winking.

Reporter: Suuuuuuuuuuuuure you weren't.

~~~
PixyMisa
The only on-the-record source said that was exactly what happened to him. He
assumed at first that Bloomberg had confirmed his hypothetical scenario with
another source, but it now seems that the story was hypotheticals and winks
all the way down.

------
mtw
There's an ongoing PR push to paint anything from China as unsecure, cheap or
untrustworthy. Now I see regular pieces on Bloomberg or popular sites like
theVerge, and all those pieces have no substantial facts in them. I know
reporters are hungry for stories, and they would eat up a semi-prepared file
with all the "facts" easily laid out for them.

This is not new. There was something very similar in the late 80s and 90s,
with everyone saying that Japan was going to take over, that they copy
everything by sending teams in North America to take pictures of everything.

~~~
azinman2
Japan did make giant leaps... but perhaps their own success (rising wages) and
failures (Sony is a shadow of its former self) got in the way. We can’t use it
as a model for how it’ll work out for China, with a totally different
government structure that so far has been very effective and competent.

~~~
iknowordidthat
> a totally different government structure that so far has been very effective
> and competent.

Authoritarian regimes can look appealing (from the outside, anyway) until
circumstances change. Then they falls apart spectacularly because the
commercial and governmental institutions are either absent, very weak, or are
unable to adapt to the change. For example, the Soviet Union.

------
abrookewood
What I can't fathom is why Supermicro haven't sued Bloomberg? Their stock took
a massive hit and so far no one has been able to independently verify
Bloomberg's claims. Seems like a court case would be fairly easy to win for
Supermicro.

~~~
CamperBob2
Could be a weird variant of insider trading. Pay some "security consultants"
to call up reporters and throw your own company under a fictitious bus. Then,
when the stock dives, buy.

Problem with that idea is that it wouldn't just chase investors away
temporarily, it would chase customers away who are a lot harder to get back.

------
rb808
from [https://www.engadget.com/2019/05/02/super-micro-move-chip-
pr...](https://www.engadget.com/2019/05/02/super-micro-move-chip-production-
china-spy-claims/?yptr=yahoo)

> Server maker Super Micro is moving production out of China in a bid to allay
> US customer's concerns about spying, even though independent tests have
> shown no evidence of cyber espionage. The company has also announced its
> plans to expand its own in-house manufacturing facilities to help mitigate
> any perceived risks. A spokesperson for the company said Super Micro wants
> to be more self-reliant "without depending only on those outsourcing
> partners whose production previously has mostly been in China."

~~~
nullc
LOL

NSA Guy: Damn, we could infiltrate that target if we could get in implant in
their boards but all their systems ship directly from china.

CIA Guy: Don't worry, I've got this.

------
hughw
For those dismissing the Bloomberg reporting, what exactly is the scenario you
find more believable? Did the reporters improperly extrapolate, either through
ignorance or eagerness for a story? Did they unknowingly succumb to CIA
disinformation? Knowingly? Something else?

[minor edit for readability]

~~~
PixyMisa
There's only one on-the-record source, and he only discussed hypothetical
scenarios with the reporters.

The other thing we know is that every time Apple, Amazon, and Supermicro told
the reporters there was no such chip on the motherboards, the reporters took
this as evidence of a huge coverup.

Basically, the reporters believed what they wanted to believe, spinning
conspiracy theories to explain the lack of actual evidence.

------
exabrial
Ok so let's say you're a small company... can you send parts to be xrayed by
an analysis lab? And the big conclusion here for me is outbound firewall rules
are almost universally ignored, but should be far more common.

------
geofft
Bloomberg lied to serve US political interests because the US is terrified
that China is legitimately competitive and is playing dirty. They're doing the
same thing with Huawei now.

~~~
WillPostForFood
Why do that to a company based in San Jose? Much more likely it is just bad
reporting, compounded by tech ignorance. The US government is very open about
its critique of Huawei, why would they secretly and illegitimately attack
Supermicro, Apple, and Amazon?

~~~
geofft
> _Why do that to a company based in San Jose?_

Because - as noted in the article - the practical effect of this story is that
now people are worried about supply chains involving China, not that people
are worried about Supermicro in particular. And selling this story on multiple
fronts, official and unofficial, seems most effective.

And allowing negative side effects for domestic companies as collateral damage
to harm other countries is standard practice; see e.g. visa policy. US
companies can't hire the best workers, but it's okay because it helps the US
government's political goals.

------
jasonhansel
> He managed to succeed in hacking the BMC with what was essentially a single
> component that could replace a resistor on the board, demonstrating with his
> proof of concept that it was plausible to do what Bloomberg’s reporting
> claimed was being done.

Whoa.

~~~
bluGill
Nobody is surprised. The accusation had enough technical details that
technical people generally accepted it could work on one motherboard. The
logistics of pulling it off in the real world were less believable. (even then
China probably could)

~~~
throwaway2048
Lots of people were decrying it as impossible due to pin count on the supposed
implant device, which looked like a passive component.

------
Causality1
I'm perfectly happy to believe Chinese hardware is riddled with backdoors.
What didn't make sense to me is why Bloomberg was so adamant about this claim
but never provided a single shred of evidence in support of it. If someone
proved this to them why don't they have a compromised board they can publish
pictures of and give to a third party for examination?

~~~
ssnistfajen
Because they don't have the evidence or it's unobtainable (either too
sensitive or doesn't exist).

If China really has a chip that can reprogram/modify server motherboard
instructions with only 3 pins all within the size of a rice grain (according
to Bloomberg's story), they've already won the tech war.

------
nineteen999
Is that the same Trammel Hudson of Canon "Magic Lantern" firmware fame? He's
one talented hardware hacker.

------
fulafel
A good thing that came out of this was raising awareness of BMC risks. Their
remote features are rarely if ever robust enough to connect to the net in good
conscience.

------
thomas
“So in conclusion, no conclusion.”

