Hacker News new | past | comments | ask | show | jobs | submit login
What Happened with Supermicro? (hackaday.com)
196 points by equalunique 8 days ago | hide | past | web | favorite | 84 comments

Huh. so shortly after the story broke, I bought a bunch of SMCI. (It's actually the first OTC stock I've ever bought. It was easier than I thought, too... I had to call vanguard to enable OTC trading, but after that? super easy.) I sold it months later, at a pretty good profit, mostly 'cause while I think the bloomburg accusations are clearly false, there are huge accounting irregularities surrounding supermicro; getting delisted is no joke, and I'm super not qualified to evaluate that sort of risk.

but, my reasoning? my reasoning was that if it was a widespread physical hack, someone would discover it fairly quickly; I mean, people were tearing these boards apart, looking for anything. There's no way they could keep that quiet, even if they were willing to disappear people.

I can tell you that within my peer group, Bloomberg has taken a pretty huge reputational hit. I mean, sure, we all still read 'money stuff' because it's "the daily show" for financial news. It's great entertainment, and moderately edifying. But, you know, it's one of those things where after a newspaper writes about something in your field, you start wondering if they are that ridiculously uninformed about the things that aren't in your field.

So, I felt really pretty safe with buying in the dip.

> I mean, people were tearing these boards apart, looking for anything. There's no way they could keep that quiet, even if they were willing to disappear people.

Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule.

Doing the same thing with ICs and PCBs, without engineering diagrams and documentation, and needing very specialized skills and tools, must be 1000x more difficult. Maybe Supermicro and some intelligence agencies did internal studies, but they aren't going to tell you anything. Perhaps Apple and Amazon asked around in their engineering departments to see if anyone saw something. Possibly a couple security researchers took a look to see if they saw something obvious. But I don't think teams of experts were mobilized all around the world trying to pinpoint a deeply hidden hardware vulnerability.

I'd love to be proven wrong. Hacker News is read by ~50,000 engineers and programmers and related professionals a day. Admittedly my comment won't reach 50,000 people, but I'll ask anyway: Is there anyone reading here who personally analyzed a Supermicro board in any meaningful way, or knows anyone who personally did so?

I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough.

You can parallelize this search, because the people doing it don't need to know much about what they're looking for—they're just hunting for visual differences. Unless/until one of them actually finds anything different. At that point, you need better resources to figure out what's different; but at that point, all hell also breaks loose, and you'll get all the resources and help you need to do better analyses.

> Unless/until one of them actually finds anything different ... at that point, all hell also breaks loose

Boards bought weeks or months apart might have lots of differences in traces and components due to engineering modifications and normal changes in suppliers. Do you really think that all hell would break loose if you showed that there were differences between a Supermicro board made in March 2018 and one made in September 2018? Other than maybe Supermicro or intelligence agencies (who aren't talking), I doubt that anyone has done what you've suggested. People are busy and it's a lot of work based on an allegation and no exact place to look.

Is that really true? Granted I'm working on low volume stuff (1000's of units) in a niche B2B setting, but the times for rolling out a new rev of a board can sometimes be measured in quarters. Changing traces/components can have unforeseen effects for even fairly minor things, you can't just have a Tier1/2 CM tool up without a round of DVT. I doubt boards with the same PN change very often if at all, but I'd be curious to know if i'm wrong.

I used to work for an electronics manufacturer. They did use xrays on a regular basis, but mostly to find cracked solder joints under ICs.

> I mean, it's not that expensive to just take some hi-res X-ray photos of old and new boards of the same make, and compare the trace paths near each component. Maybe you can even take a visual diff of the X-ray negatives, if you can get everything lined up tightly enough.

There are examples of attacks which cannot be detected by staring at X-rays of components[1]. Now, to be fair they are very advanced attacks but we know they exist.

[1]: https://link.springer.com/article/10.1007/s13389-013-0068-0

I know a cybersecurity professional who was contracted by a hedgefund to investigate the claims by reverse engineering and present a report so it’s not entirely unlikely.

>Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule.

I think... we might just live in different worlds? I'm a sysadmin and I maintain computers, but I am surrounded by serious programmers (I mean, kernel and system verilog types, not frontend people)

I have worked places where it was just me and the frontend folks, and in that case? yeah, It's kinda lonely when you need help on anything serious. But even then, with my social life being what it is now, I'd have people to ask if I was in that situation again at work.

I... usually don't have any trouble getting someone qualified to look at my backtraces when I have kernel panics, and I usually don't have trouble getting those people to give me patches. that requires analyzing the code.

I mean, sure, people don't spend a lot of time reading random code for no reason... but you give them a clue that something interesting is somewhere, and my experience? you can get plenty of help.

>Without commenting on whether or not I believe the Bloomberg story, I very much doubt that "people were tearing these boards apart", doing reverse hardware-level engineering looking for an unknown. Even in the case of open-source software, with full source code and documentation available, the number of people who analyze code is miniscule.

I can tell you people were literally tearing them apart; cutting open capacitors and the like (common place to hide extra components) pulling apart power supplies (as I recall, one of the bloomburg illustrations implied that something was in a power supply; I remember a long discussion of how you could compromise something over i2c from a chip actually in the psu) - I mean, sure, these things could be better hidden to the point where such techniques would be ineffective, but my reading of the bloomburg article was that they weren't particularly well-hidden, and that therefore they were discovered. Hiding that sort of thing from another EE who designs that sort of thing seems like it's probably pretty difficult.

(I mean, it is common to have compromised firmware; that's way easier. but that's also not what the bloomburg article was about)

But... yeah, I guess a lot of my assumption was based on my reading of the bloomburg article implying that this hardware was something tacked on in a detectible way; the sort of thing that could be done by some third-tier subcontractor. I acknowledge that if your attacker has the capability to produce ASICs that look just like the real thing and act just like the real thing out the same I/O pins, except in some rare case? yeah, that's not going to fall to this level of examination. but that wasn't my reading of the bloomburg story.

Furthermore, if an attacker went through that level of effort, they'd probably not just do it for supermicro. Most motherboards of the same generation are pretty similar; pretty heavily based on reference designs. If you were going to do something that sophisticated, that's where you'd attack.

If you were going to attack just one particular rev of one particular brand of a motherboard? well, something tacked on seems a lot more feasible.

so by my reading of the article, it was extra chips... as I recall, it looked like some of them were hidden in the power supply (totally possible, if you can find some exploit on the I2C interface, but also probably fairly easy to detect)

(the story was that it was a Chinese subcontractor. Remember that SuperMicro is a Taiwanese company; I'm pretty sure that SuperMicro has less Chinese made content than any of the other major manufacturers of commodity server kit. )

I did know people cutting open power supplies, cutting open capacitors (a place chips have been hidden in the past) and otherwise examining the systems at the macro level.

I also knew people who managed networks of these things... and one of the bloomburg stories, as I recall, claimed that the hack was discovered via looking at the network as the BMC tried to phone home.

I mean, yes, you could do something more clever than that, replace a chip with one that looks identical, which uses the same traces that did something different, but the article certainly implied it was a more simple sort of thing.

I mean, if I was trying to compromise a BMC in that way, I'd just flash it with firmware that did what I needed (and this has happened before... to many different vendors) - my understanding of the bloomburg story was that there was some bit of hardware to re-flash the firmware to the compromised version after the user upgraded to a non-compromised version.

> Huh. so shortly after the story broke, I bought a bunch of SMCI.

Likewise. Almost doubled my money before taxes.

My reasoning was that essentially no tech infrastructure company has significantly suffered financially due to security incompetence and it was unlikely that it would start with SMCI especially since it wasn't even being claimed that the company was in on it. Nothing even seemed to allege any particular error on SMCI's part, certainly not one that essentially every vendor isn't also making.

The idea that the report could be wrong wasn't a factor for me, only that the marketplace would continue to not care. (I find it hard to believe that it's entirely wrong: I am unwilling to believe that major state intelligence orgs are so incompetent that they are never backdooring hardware in that manner).

When Tim Cook came out against the story, I did the same thing. It just didn't seem plausible that he would be lying. Then add to that Joe Fitzpatrick's (one of the named sources) discomfort about the story, and something seemed wrong. Time seems to have validated this idea somewhat, but it is interesting that Bloomberg has stood by the story.

> it is interesting that Bloomberg has stood by the story

My guess as to why: one of their reporters was misled by a trusted source, or multiple trusted sources (who were maybe actually catspaws of some-group-or-other who would benefit from getting this story out.)

So now, although Bloomberg would like to speak truth by issuing a retraction, such a move would likely severely damage the original reporter's career; and they don't want to do that for something that wasn't that reporter's fault.

Having some corporation or government agency with tons of resources come along and force or trick a reporter into running a story, would be the journalism equivalent of an insurance agency's "act of God" waiver clause. I would guess that every major news outlet has a policy for protecting and defending reporters who—by hook or by crook—get used as patsies by forces out of the news outlet's control.

In that scenario, the obvious solution is to publish a new story and name names of all the people who lied to get the story published.

If you lie to a newspaper to get fake news published, you no longer have any right to remain off the record.

That's what a newspaper with integrity would do.

Two scenarios:

Cynical scenario: the trusted-source catspaws have been reliable sources in the past, and will seemingly continue to be reliable sources in the future. This is their “price” for giving up reliable information for free—the ability to be used as a mouthpiece by their organization. Bloomberg accepts this deal, because they’re so useful otherwise.

Noble scenario: the sources didn’t want to transmit this information any more than Bloomberg wanted to receive it, but they were forced to (by, perhaps, a National Security Letter; or just generic “we’ll hurt your family” coercion); Bloomberg knows this, and doesn’t want to hurt the sources either, when it isn’t their fault (and either don’t know who coerced them; or do know, but know that it’s an actor powerful enough that they know that reporting the source’s coercion would just mean the source’s untimely death. It’s not like a newspaper is a government that can offer witness protection.)

The flip side is that on a story this big, they shouldn't publish if they won't be able to burn their sources if it turns out they're being played. If a source is too valuable to challenge, they're not worth enough to publish.

Spin is one thing, outright lies is another.

Cynical version rings truer to me. But the number of different sources they claimed is surprising.

That requires having a degree of certainty supporting any attempt to name names. If you're not certain they lied to you, then your new story is potentially libel. For example, let's say they were given information they were certain was correct, and they handed it to a journalist in confidence. They're a trusted source. You both got duped. What's the value in burning them publicly, especially if you're going to do it by calling them a liar and exposing yourself to risk?

There are cases where the press really needs to take people to task but the amount of secrecy and ambiguity in this story makes it seem like a case where it can't (or shouldn't) be done.

They claimed to have something like over a dozen sources. Some of those were lying outright.

>If you're not certain they lied to you, then your new story is potentially libel.

No, you report the truth that they said X to you, and your opinion that it isn't true, along with any verifiable facts supporting that opinion.

I'm leaning towards that something did happen at SuperMicro, but the FBI agents who were interviewed by Bloomberg lacked the technical knowledge to understand what happened or how it happened and thus did not convey an accurate view of the incident(s). Bloomberg decides to give the go-ahead on publishing this story despite providing zero provable evidence, which is still a poor demonstration of journalism integrity on their part. However I don't believe Bloomberg will use completely fabricated stories to smear anyone despite how suspect such acts seem. If they realize that they've been misled, the proper reaction should be making a disclaimer regarding the story containing misinfo. Apple/Amazon/SuperMicro also could have sued Bloomberg for defamation but none of them did, which means they don't have a strong enough case on hand.

I have this view because FBI agents have acted upon information they could not fully understand or analyze before, only to be proven wrong later e.g. the arrest and subsequent exoneration of Dr. Xi Xiaoxing.

Tim Cook lies about there were no settlement talk between Qualcomm and Apple. Although one could argue he spoke of it in January and the Settlement were in April, so may be they did all that negotiation in three months. Although I highly doubt it, I am siding with Steve Mollenkopf on this one, out of all the evidence given in court I think Qualcomm were extremely professional.

There were bunch of other things where I caught Tim Cook lying. Steve Jobs uses to be very deceptive when he needs to, his wording weren't clear cut and leads you to wrong assumption. Tim Cook doesn't have that talent.

And once you have been caught lying, you lose trust.

While I do have doubt on Bloomberg story, I am not going to trust sold on Tim Cook's word.

There was a theory going around that the whole story was just and elaborate stock manipulation scheme.

But orchestrated by who? Surely not Bloomberg as they'd have too much to lose.

And if you have the means to do something like this, why not do it in a more subtle way that doesn't have people asking questions 6 months later?

I'm not suggesting you're wrong and I have no clue what happened.

what kind of keyboard is that?

That's a bloomberg terminal keyboard.

Who knows? Presumably whomever wrote the original story for Bloomburg and made the photos is involved somehow but beyond that it's anyone's guess.

What is Bloomburg going to lose here? Their reputation? They're already considered something of a gossip rag for financial news. FTC action? Unlikely.

> made the photos

uh, the pictures of a tiny rice-sized chip from the story were just photo illustrations. They were not supposed to be evidence of anything. It said so in the margins.

Friend tells me that it's the National Security Council - and specifically Matt Pottinger - feeding these stories to Bloomberg. The goal is to force US companies to shift their electronics supply chain away from China.

That's an interesting theory that I'd not heard before (and haven't found iterated anywhere within a few minutes of searching). Any additional details / references?

> But, you know, it's one of those things where after a newspaper writes about something in your field, you start wondering if they are that ridiculously uninformed about the things that aren't in your field.

Or maybe you keep reading because you believe the other things, see the Gell-Mann amnesia effect, https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

When you take this into account it makes me wonder what the motives could be... https://news.ycombinator.com/item?id=18162440 (bloomberg paying based on market movement)

Likewise, it's been good until the recent tariff war news.

I made similar money in the 2008 crash betting that certain banks would not fail.

After 9/11, American Airlines took a beating. I was in my 20s and had no clue about stocks. I went to my IRA guy, and asked to buy AA stock. I was told because it was down in the penny stocks they could not sell it. I had no idea of how else to buy stocks, and let it go. Regret that every time I think about it.

Cheer up kid, I went all in on Apple stock with the debut of the iPhone. A year or so later lost it all in the crash. Value approx $2.25 million today.

Somewhat unrelated: we used to do a lot of business with supermicro. In a meeting one time I asked one of their guys why their billboards are so weird looking (anyone who’s driven down 880 past their building knows what I’m taking about). They have these billboards that have weird slogans and super amateur looking graphics that look like they were made in MS paint.

So I asked the guy about them cause I thought it was funny that such a big company had such cheap looking marketing. And he said: “yeah our CEO couldn’t believe how much money some agency was going to charge us to design a marketing campaign, so he just did it himself on his computer in like 2 days”

I thought that was hilarious. But good on the CEO for realizing no one really cares what a server billboard looks like, and decided to save the $2MM or whatever someone was going to charge him for design. Super funny.

The best photo I could find: http://meritage-partners.com/site/wp-content/uploads/2016/05...

I love that old school Silicon Valley aesthetic.

That's funny, I had no idea they had billboards.

Are those the $1M Silicon Valley homes I hear about?

Those look like mobile/manufactured, so in the $400-600k range.

Well, around $200-300K, based on checking prices on Zillow for what sure appear to be spaces in that mobile home park (around Oakland Road in San Jose, ZIP code 95131). I don't think I've ever seen a mobile home listed for a half-million even in this area, although I've seen ones the mid-$300s. (Granted, this is still nuts, given that in Sacramento or Tampa, the other places I've idly priced homes, these would be under $100K.)

What's amazing to me is that people are willing to pay high prices for a manufactured home in Silicon Valley even though the likelihood of the mobile home park's land being sold to developers is higher than anywhere else.

This has already happened in a few places in the area. In Sunnyvale there are massive developments of million-dollar condos literally next door to mobile home parks. How much longer can those places survive?

One would think that the risk would make people less likely to buy. There aren't very many places in the area for people to move their homes to if the land they are sitting on gets sold.

Why is there such a disparity in quality of life for tech workers in SV? It seems like the housing crisis there will eventually stifle productivity (no one can afford to move there for entry level roles without giving up a ton). Are there some special dynamics that cause these types of scenarios except for the fact that a lot of people got rich quickly in one place?

Entry level roles are for fresh graduates. A converted living room is a step up from college and compatible with aggressive savings on a big-co wage. Entry level engineers willing to run smaller savings rates can even afford to live alone, though it can be a stretch.

density restrictions and human greed

i hope this is sarcasm...

Oh wow, that does give me mid-90s PC Magazine flashbacks.

There used to be a big billboard for Data Translation A/D converter cards on I-880 at San Jose city limits. I saw them right after moving to the Bay Area in 1990. It felt like driving through the entrance to paradise. :)

That reminds me of newspaper ads for someone selling beige box PCs.

Remember the old catalogs that were phone book sized of all of the mail order PC parts vendors? er, what's a phone book you ask? My first PC was built from one of these. It was a 486 after the Pentium line was introduced.

Computer Shopper was a strangely entertaining publication--the crude design, the suuuper homemade ads. Who needs a Hayes 9600 modem?

That would actually explain there website. Its awful in only the way a late 90's site can be, but they make great servers so you deal with it. I just wish they made it a bit easier to find everything.

I'm just impressed the CEO did something himself. I always feel I should be delegating more, but often DIY is easier than explaining what you want.

Their website is like that too. Everything just looks like it's from the 90s. But then again, if you're in particular industries like HPC, they make machines unlike any of their competition.

Felt pretty bad for SuperMicro after they took the huge financial hit from the story which offered no physical proof. Glad to see they mostly recovered from it.

The story was extremely interesting, but as people dug into it, it seemed like the reporters had a bunch of conversations like this:

Security Researcher: So, in theory, you can do a lot of crazy stuff! Embed a tiny chip on a motherboard, stick it in an Amazon datacenter, sniff all sorts of things...

Reporter: Are you winking right now? I swear you are winking! So you are saying this is true? OMG. Wow, what a bombshell!

Security Researcher: I was not winking.

Reporter: Suuuuuuuuuuuuure you weren't.

The only on-the-record source said that was exactly what happened to him. He assumed at first that Bloomberg had confirmed his hypothetical scenario with another source, but it now seems that the story was hypotheticals and winks all the way down.

There's an ongoing PR push to paint anything from China as unsecure, cheap or untrustworthy. Now I see regular pieces on Bloomberg or popular sites like theVerge, and all those pieces have no substantial facts in them. I know reporters are hungry for stories, and they would eat up a semi-prepared file with all the "facts" easily laid out for them.

This is not new. There was something very similar in the late 80s and 90s, with everyone saying that Japan was going to take over, that they copy everything by sending teams in North America to take pictures of everything.

Japan did make giant leaps... but perhaps their own success (rising wages) and failures (Sony is a shadow of its former self) got in the way. We can’t use it as a model for how it’ll work out for China, with a totally different government structure that so far has been very effective and competent.

> a totally different government structure that so far has been very effective and competent.

Authoritarian regimes can look appealing (from the outside, anyway) until circumstances change. Then they falls apart spectacularly because the commercial and governmental institutions are either absent, very weak, or are unable to adapt to the change. For example, the Soviet Union.

To me, the old Sony electronics was way ahead of anything US. Sony Beta/Betacam, cameras, etc. Also JVC's VHS. After RCA or Ampex, the US had really been left behind in the video/TV forefront.

What I can't fathom is why Supermicro haven't sued Bloomberg? Their stock took a massive hit and so far no one has been able to independently verify Bloomberg's claims. Seems like a court case would be fairly easy to win for Supermicro.

Could be a weird variant of insider trading. Pay some "security consultants" to call up reporters and throw your own company under a fictitious bus. Then, when the stock dives, buy.

Problem with that idea is that it wouldn't just chase investors away temporarily, it would chase customers away who are a lot harder to get back.

from https://www.engadget.com/2019/05/02/super-micro-move-chip-pr...

> Server maker Super Micro is moving production out of China in a bid to allay US customer's concerns about spying, even though independent tests have shown no evidence of cyber espionage. The company has also announced its plans to expand its own in-house manufacturing facilities to help mitigate any perceived risks. A spokesperson for the company said Super Micro wants to be more self-reliant "without depending only on those outsourcing partners whose production previously has mostly been in China."


NSA Guy: Damn, we could infiltrate that target if we could get in implant in their boards but all their systems ship directly from china.

CIA Guy: Don't worry, I've got this.

Typically not one for conspiracy, but I wonder if to some degree these kinds of things are intended to get US customers to buy US products and/or get companies to move their offshore facilities to the US. Kaspersky being hand-in-hand with Russia and <insert vendor here> with China and things like it seem pretty run of the mill in recent years.

I imagine it is nestled in some degree of truth, grant you, but I wonder if the goal is to force sentiment so heavily that vendors are essentially forced to go stateside. As in, did China suddenly want to exert state control over manufacturing done in China? Russia suddenly want data from Kaspersky?

For those dismissing the Bloomberg reporting, what exactly is the scenario you find more believable? Did the reporters improperly extrapolate, either through ignorance or eagerness for a story? Did they unknowingly succumb to CIA disinformation? Knowingly? Something else?

[minor edit for readability]

There's only one on-the-record source, and he only discussed hypothetical scenarios with the reporters.

The other thing we know is that every time Apple, Amazon, and Supermicro told the reporters there was no such chip on the motherboards, the reporters took this as evidence of a huge coverup.

Basically, the reporters believed what they wanted to believe, spinning conspiracy theories to explain the lack of actual evidence.

Ok so let's say you're a small company... can you send parts to be xrayed by an analysis lab? And the big conclusion here for me is outbound firewall rules are almost universally ignored, but should be far more common.

Bloomberg lied to serve US political interests because the US is terrified that China is legitimately competitive and is playing dirty. They're doing the same thing with Huawei now.

Why do that to a company based in San Jose? Much more likely it is just bad reporting, compounded by tech ignorance. The US government is very open about its critique of Huawei, why would they secretly and illegitimately attack Supermicro, Apple, and Amazon?

> Why do that to a company based in San Jose?

Because - as noted in the article - the practical effect of this story is that now people are worried about supply chains involving China, not that people are worried about Supermicro in particular. And selling this story on multiple fronts, official and unofficial, seems most effective.

And allowing negative side effects for domestic companies as collateral damage to harm other countries is standard practice; see e.g. visa policy. US companies can't hire the best workers, but it's okay because it helps the US government's political goals.

Strong claim supported only by conjecture.

> He managed to succeed in hacking the BMC with what was essentially a single component that could replace a resistor on the board, demonstrating with his proof of concept that it was plausible to do what Bloomberg’s reporting claimed was being done.


Nobody is surprised. The accusation had enough technical details that technical people generally accepted it could work on one motherboard. The logistics of pulling it off in the real world were less believable. (even then China probably could)

Lots of people were decrying it as impossible due to pin count on the supposed implant device, which looked like a passive component.

You should watch the talk on the CCC archives. It's super interesting.

or this annotation: https://trmm.net/Modchips

I'm perfectly happy to believe Chinese hardware is riddled with backdoors. What didn't make sense to me is why Bloomberg was so adamant about this claim but never provided a single shred of evidence in support of it. If someone proved this to them why don't they have a compromised board they can publish pictures of and give to a third party for examination?

Because they don't have the evidence or it's unobtainable (either too sensitive or doesn't exist).

If China really has a chip that can reprogram/modify server motherboard instructions with only 3 pins all within the size of a rice grain (according to Bloomberg's story), they've already won the tech war.

A good thing that came out of this was raising awareness of BMC risks. Their remote features are rarely if ever robust enough to connect to the net in good conscience.

Is that the same Trammel Hudson of Canon "Magic Lantern" firmware fame? He's one talented hardware hacker.


Hilariously enough it was proven as plausible and a proof of concept was built.


Critical, not analytical. The former is an emotional state of mind enabling the beginning of the analytical thought process.

“So in conclusion, no conclusion.”

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact