Hacker News new | comments | show | ask | jobs | submit login
New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom (bloomberg.com)
875 points by gorbachev 8 days ago | hide | past | web | favorite | 366 comments





Finally a named source, but still no photos and the alleged hacked board is still not in the hands of a public security researcher.

The "trojan ethernet connector" paragraph mentions similarity to an NSA implant, which appears to be this: https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS...

I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?


A named source, but not a named victim, in this case. I would not call this verification.

This is a really hard story to know what to think about. On the one hand, yes, hardware implants are a major risk. And having so many of our electronics manufactured in a country with massive state control over its economy and with which we have an adversarial political relationship is definitely a big concern.

On the other hand, the denials from the companies cited in the first article are remarkably strong. And again this article fails to give relevant details. It just cites a security contractor who says he had a client who had this issue.

But Bloomberg is a serious news organization and they are holding strong on this story as well. So what to think?

It strikes me that if your goal was to ramp up tension between the US and China at multiple levels, then planting this sort of story would be a great way to accomplish it. Politicians can cite national security. Wary consumers are triggered over privacy. Corporations become more and more gunshy of investing in China and partnering with Chinese manufacturers.

I hate to dream up conspiracy theories. And yet, we live in a world where many states, politicians, organized crime groups, political groups, and corporations are all intentionally spreading disinformation of all sorts all the time designed precisely to ratchet up tension and suspicion.

I don't really believe that's what's going on just yet. But I also don't believe it's as straightforward as the Bloomberg stories make it out to be, either. Something very strange is going on.


Yeah - when I add to it that, as a non-American, I can (annectodaly) observe a rise in different kinds of news that involve China in a negative context for the last 6m especially, it's hard to form an opinion.

In terms of security concerns also - come on, we know by now to which lengths the US goes in this area, and they're surely doing worse stuff than this, I'd expect no one would doubt it any more. So, either they are genuinely surprised by this, which would be silly (a politically adversarial nation using an obvious opportunity - cheaper stuff being produced there for decades - and doing the same), or it's a part of a broader narrative that's being built.

And, to be clear, I don't think we (the world outside China) shouldn't be a bit worried given in what position _we've_ put China and how strong they are now - it's just that this kind of mass-manipulation and propaganda is the most-detested way of doing it for me...


> I don't think we (the world outside China) shouldn't be a bit worried given in what position _we've_ put China and how strong they are now

At least China won't be as dangerous as the US or the Soviet Union in that it has absolutely no interest in enforcing its political ideals on other places or becoming a world police. I see no real reason why people should be worried about the rise of China as if this will turn the whole world Orweillian. China has its own way of organizing the vast and complex country and it's hard to come up with better practical solutions. Still if one doesn't like it they can just live somewhere else, and the authorities don't care, as long as it doesn't hurt Chinese business.


> .. no interest in enforcing its political ideals on other places or becoming a world police

Not yet. But becoming a banker for whole third-world is almost done. And when your debts are big, you lose sovereignty.


Still doesn’t mean you shouldn’t be concerned by China’s super position in the global supply chain...

You should, but China rarely cares about people beyond its own boarders. They don't have the power of the U.S. to reach for anyone across the globe, so I think NSA doing this is a tad more worrisome.

I think the main difference between the US and China, the Chinese have no mission to convert every country to their thinking. Not that they influence countries through their investments, see Greece as an example.

Ummm other than Silk Road pulling tons of countries into their sphere, using it to push new Chinese standards for people to switch towards, making HK/Taiwan/Tibet fully incorporated and culturally homogenous, taking islands away from other countries across SE Asia and calling it all china’s sea, etc etc.

They’re just more subtle, until they’re not. They have a very long term outlook on their efforts.


China does have ambitions, sure, but they're mostly local ones. I was mainly talking about cases like Kim Dotcom, where the U.S. reached all the way to NZ. China doesn't have Five Eyes, has not attacked a foreign country etc. It does not mean this cannot change, but I get the feeling China strives for a more local "sphere of influence" strategy, (hence the things like you mentioned), whereas the US strives to maintain "global dominance". As someone sitting in Western Europe, I don't think China has that much interest to dominate here.

China is not pushing countries to become 'communist' states.

You need to be a citizen of a nation neighbouring China to know the severe pressure that China exerts on borders, trade and geo-politics. You just don't hear about it much in the world-news as it doesn't affect the US.

China is not pushing countries to become 'communist' states.

"... don't ..."

They kidnap people from other places who are dissidents including from Hong Kong, Malaysia, and even the us.

Here's an article discussing this worldwide kidnapping. https://foreignpolicy.com/2018/03/29/the-disappeared-china-r...

This claim seems like a big dilemma for US white-hat security researchers:

1. As a white-hat security researcher, you have an ethical responsibility to publicly disclose vulnerabilities after doing the necessary due diligence (informing the affected parties privately, and giving them the necessary time to respond, investigate, and come up with an acceptable solution).

2. As a US citizen, you can't report attacks carried out by US intelligence agencies.

I can definitely see the responsibility that patriotic duty would entail, but a citizen with no links to their country's intelligence agency being held responsible for the said agency's failure in maintaining operational discretion doesn't seem sensible to me.

Edit: update formatting.


> As a US citizen, you can't report attacks carried out by US intelligence agencies.

Who says? Unless you've received a National Security Letter, a gag order from a court, or have a pre-existing relationship with the government that governs disclosure (e.g. security clearance), there's nothing preventing a researcher from disclosing lawfully obtained information. Stumbling upon a secret investigation doesn't make the information unlawfully obtained, even if you suspect it might be a secret investigation.


Are you sure that the Espionage Act (1917) doesn't cover this? In Australia we have many recent laws that completely restrict our ability to whistleblow on any government issue (though it's not illegal if we ensure that non-Australian nationals know about it -- which is obviously an impossible and stupid standard).

It might (particularly subsections (d) and (e))[1], but only because the wording is so broad. Whether such an application would be legal is another matter. I suspect it would not absent specific intent (i.e. you're deliberately seeking out secrets to share) or a duty (security clearance).

I suppose intent could be there if you share information about a device that says, "Warning: national defense injured if you disclose". But absent a duty I don't think a court would impute intent, especially considering the Free Speech issues (somewhat peculiar to the U.S.).

Notice that nobody has seriously suggested (AFAIK) that the journalists who assisted Snowden should be charged under the Espionage Act, even though their acts would seem to fit multiple provisions. I think that's because unlike Snowden they had no duty, which means the bar for the requisite intent and knowledge (i.e. whether something is really going to harm national defense) is incredibly high.

But who knows? It's a good point and it poses a ton of questions. Still, personally if I found a spying device on something I wouldn't hesitate to disclose it if it seemed noteworthy. I wouldn't feel chilled by the Espionage Act. The same law in some other country? Probably would think twice.

[1] https://www.law.cornell.edu/uscode/text/18/793


> As a US citizen, you can't report attacks carried out by US intelligence agencies.

Sure you can. Short of a gag order (and maybe not even then) you can report intrusions all you like.

In any event, how does one determine the nationality of hardware that shouldn't be there? It's not like there's going to be a snarky "Designed by the NSA in Fort Meade" logo on the chips in question.


If (IF!) you hold a civilian or military clearance, then you have a legal Duty to Report (DTR). That holds true whether its data in your clearance level or not.

You also abide by a whole slew of laws regarding sensitive, secret, top secret, or SCIF information. If I knowingly, or even suspect, some information if classified, and I transmit it to anyone else than my federal assigned contact, I'm breaking major federal laws.

A lot of security professionals in the US have such clearances. So finding a NSA implant or such proof makes it dangerous to talk about by default.

So yeah, a gag order by profession.


Are you really a white-hat if you have to disclose vulnerabilities to an organisation known to exploit (or at least hoard) them?

Vulnerabilities and weaponized implants are very different things.

Please quote me where I claimed to be a white hat.

You didn't, but the context was:

> This claim seems like a big dilemma for US white-hat security researchers

It seems like the two are mutually exclusive.



> But Bloomberg is a serious news organization and they are holding strong on this story as well. So what to think?

Are they? The authors of this story published an unverified and in corroborated story about Heartbleed a few years ago, claiming that the NSA knew about it and was exploiting it (https://www.washingtonpost.com/blogs/erik-wemple/wp/2014/04/...).


And here's someone else calling out Bloomberg for an "unethical hatchet job" when reporting on a technical issue: https://www.semiaccurate.com/2012/10/08/bloomberg-wrong-abou...

To be fair, Clover Trail had all sorts of driver issues that never got resolved. I personally had to deal with the shitty GPU drivers for work. I can't speak to the power management since we were using the chip in a place where power management didn't matter, but I can see those being shit too.

I think the issue is that they're a serious news organization about some subjects, and a dumpster fire for others. It's hard for the general public to recall which subjects they are authoritative on and which not.

It would not surprise me if the NSA did know about it but it's a shame they didn't have any proof.

It seems that the author of the research is not happy with Bloomberg's spin on the story: https://www.servethehome.com/yossi-appleboum-disagrees-bloom...

> But Bloomberg is a serious news organization and they are holding strong on this story as well. So what to think?

When the articles published by those "serious news organizations" concerns China, disinformation / lack of evidence are really common place if you carefully examine their source. I used to do so from time to time but grew tired of that


That's not a conspiracy theory. It's standard type of policy and it would be surprising otherwise.

If information, ideas, knowledge were shared openly we wouldnt have these kinds of ridoculous events. This kind of news is what keeps nations siloed and prevents collaboration. At the same time maybe this will also force us to abandon trust all together and move towards verifying.

Indeed. I saw a brilliant presentation in 2012 by Michael Mitzenmacher from Harvard on verifiable computing in the cloud. It was based on this paper:

https://arxiv.org/pdf/1202.1350v3.pdf


I think we should judge by facts,not by stereotypes. In my opinion, newspapers are all propaganda machines driven by some their benefit.

> I think we should judge by facts,not by stereotypes.

Cool. Let us test.

Error: Line 2:

> In my opinion, newspapers are all propaganda machines driven by some their benefit.


> I hate to dream up conspiracy theories.

No conspiracy theory needed, traditional media is dying, and in their last gasps of air they are destroying their credibility for the sake of clickbait articles without proper facts and coberation. They are being deceptive, because they know outrage and politically dividing stories are still working.

It's really said, but transparent. No way, Apple officially writes that rebuttle on their website if the story is true.


German telecom employee here. I've seen a number of sneaky backdoors and intercepting devices at all levels in my career. The most interesting thing was a server where TCP connections that were about to close (TCP FIN) were suddenly intercepted to dump additional (encrypted) data that was't part of the original flow. Obviously there was something out there that was seeing both sides of the flow and intercepted parts of it. We successfully confirmed the problem was on our (in)side by booting the affected server on a USB stick and made it generate controlled traffic to controlled destinations on the internet that were synchronized using a LFSR. The server was decommissioned and the issue was escalated above my paygrade with clear instructions not to talk about it. I won't give an exact year for this incident but it happened in this decade but before Snowden.

Personally I'm confident Bloomberg's reporting is accurate to a high degree. Based on prior experience with investigative journalism there's no way they would go all in with a story like this if they weren't standing on firm ground. Every single sentence would've been vetted. For each statement made there would be someone whose job would be to reject it unless you could back it up properly. This is also why you don't see these entities defending their story against random criticism that pops up. Most if not all decisions have already been made by the time it goes public.

The fact that there's now a second story on the same topic is a good sign. The reporting of these things are usually followed up by additional pieces to increase the impact (and revenue of course).

They claim they have 17 independent sources. That's pretty impressive in itself. It also means that they probably worked real hard verifying their sources' claims and inputs. I find it unlikely that they would've acquired all those sources unless the thing was real.


> Based on prior experience with investigative journalism there's no way they would go all in with a story like this if they weren't standing on firm ground. Every single sentence would've been vetted.

And based on my prior experience I would make the exact opposite conclusion. Technical writers are rarely technical, and they seem to be happy to make stuff up and mislead - even if unintentionally - so long as they make their deadlines.

Every tech article written about a subject I was directly involved in has not even gotten the spirit of the topic remotely accurate, much less minute technical details such as what chip is used where.

That said - I fully believe supply chains are entirely compromised. I just don't think in this case I'd really put much stock into this specific reporting - they've already been caught blatantly misleading their readers by putting up a photo of a stock Mouser part and not denoting it as such.


I agree that technical details sometimes gets misrepresented or come out plain wrong. That's my observation as well and it's annoying when you're knowledgeable in the subject and try to make sense of what you've read (or read between the lines).

I think a contributing factor is that it's generally hard to write about things you don't fully understand with the correct nomenclature. Especially when you might not be able to talk/ask for help about specifics with people more knowledgeable because of the secretive process.

Things could've been dumbed down, intentionally or unintentionally, by those involved. It wouldn't be hard to imagine a conversation like: "-So it was sort of a coupler thing? -You could say that, yes". Or what if the technical detail came from a Chinese source and Google translate mangled it?

The coupler thing is dumb and so is the picture (assuming it was a random product picture) but at least they might serve as a way of communicating the big picture: a hard to spot electronic "coupler" thing.


Not in this case. One of the sources was interviewed and said the technical details were only suggestive: https://risky.biz/RB517_feature/

Maybe there is something there, and/or there is a reason to talk/substitute in vague terms, but insofar as the explicit technical details are concerned, they don't appear credible. Then you're left with an empty allegation that you will have to decide to believe or not based on no other ground than potentiality.


Interesting experience and thank you for the first-hand perspective!

I do have to disagree about the competency of Bloomberg, though, they publish a lot of speculative, low-tech AI/ML scare articles that can be described at best as "inaccurate" and, more realistically, as "making stuff up". They used to have a good reputation, probably from their financial journalism, but their tech work is not good but any reasonable measure, in my personal opinion.


There is a big difference between claiming 17 sources, a claiming all 17 sources corroborate the full story. The Apple letter to Congress highlight that Bloomberg is relying on a single source for the specific claims about compromised servers being found at Apple.

I am not sure if you're just a German telecom employee based out of US or an actual native German working for this German telecom company in Germany. If latter, your english is extraordinarily above and beyond what I've generally exhibited with a lot of my personal German friends. Just a naive observation so please don't take this in any wrong or defamatory manner.

The lack of (publicly available) evidence is annoying, since there are a lot of people who'd love to check their own servers. As this is an attack directed at high profile targets it's unlikely the average size company will have ended up with one of those, but it's still a fun exercise IMO.

It would also be great to know what the attribution is based on. Just the fact that they're manufactured in China? Who else might get their hands on these devices in the shipping chain? What kind of traffic did they monitor? I guess just observing it's talking to a Chinese address doesn't tell much. I mean, just take an S3 bucket and dump your stuff in there. Setting up your own server in your home country pretty much screams "we're here!"

> I'm now wondering if someone found an NSA implant and misreported it as Chinese. We're going to end up in the stupid situation where people are afraid to report foreign intelligence attacks because it's illegal to report an attack by US intelligence agencies, aren't we?

That's pretty tinfoily, but it'd be a cool way to still report on it. "Whoopsie, I totally thought it wasn't you guys, sorry for disclosing"


"Who else might get their hands on these devices in the shipping chain?"

From the original Businessweek article:

"Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China."


"Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China."

That's interesting. As someone who has bought hundreds of thousands of dollars of gear from Supermicro (and has been a huge fan of their products and designs) I always thought their chassis were their core product.

Recently SM started to go down the "you can't buy our JBOD chassis without buying them full of our qualified drives" ... I knew that was the end of the golden age (of SM).

Luckily this coincided with the introduction of the 60bay HGST JBOD chassis. We haven't looked back.


> Luckily this coincided with the introduction of the 60bay HGST JBOD chassis. We haven't looked back.

Yes, these units are stellar and anyone buying Supermicro JBOD units should be looking into these as much better replacements. If you have volume they can be even more competitive than Supermicro if you push.


One very, very small gripe is that the HGST JBODs have no power switch. You power them on and off by inserting or yanking the power cables. Not my favorite SOP ...

Is that a real thing? Holy cow, I'm shocked (bad pun intended). What about adding an inline switch in the cord? Unless they expect everyone to be using a managed power system where each plug can be turned on/off, this just seems very odd decision to make.

<off to look up this beast>


As someone that flipped the power switch on a rack mounted machine by accident before, I could see how a power button or switch would be consitered a liability more than a benefit, especially when the solution (pull the power cable) is simple, foolproof, and doesn't happen that often to warrant optimizing!

Every storage vendor eventually goes full NetApp. The money just looks too good.

I have to assume we'll start to see a rise in American high tech manufacturing for security purposes alone. Some of these companies may want to manufacturer these critical components themselves, maybe even hand deliver them from their US factory to their customers in the US too.

I know that some refineries do direct delivery for some of their large customers, especially industrial lubricants and other by-products. If the order is big enough, or someone wants to pay the premium, then direct delivery could be very feasible for tech too.


It's odd to me to assume that people should trust US-based supply chains. We know that the NSA has done supply chain attacks in the past[1], while in this case we only have allegations of China doing the same (don't get me wrong, I wouldn't be surprised if China did this, I'm just saying we have more evidence for the NSA doing it).

Personally, as someone outside the US, I would gladly trust alleged Chinese malware over known NSA malware. Or even better, literally any other country outside the 5-eyes.

[1]: https://www.theguardian.com/books/2014/may/12/glenn-greenwal...


Is there any way to solve this problem without needing a "trusted manufacturer"?

I know it won't probably won't apply to general purpose motherboards or devices, but is there a way to design or build some components or devices in a way that you can verify that they can perform their purpose and nothing more?

If we start with that concept, and slowly build up "verifiably secure" components, they can be the islands of security that we can build off of without having to worry if the manufacturing plant left their door open one day and some random person was able to sneak in.


What happens when your attacker knows how your safeguards work and can route around your door though the windows?

For a motivated and well funded attacker who has an ability to manufacture a replacement chip with an additional coprocessor that can siphon or modify data from the main processors, network cards, and baseband modems, short of decapping every chip and component that comes through your assembly line your resources would be better spent on establishing trust mechanisms with your suppliers and the transportation couriers touching your devices before the end user acquires it.

https://en.wikipedia.org/wiki/Tailored_Access_Operations#Kno...

http://www.spiegel.de/international/world/the-nsa-uses-power...


My thought was it would be something that would get more secure the more knew about it, similar to math proofs or cryptography code.

A way to verify a chip is working as expected in a way that it can't be gamed without breaking multiple fundamental proofs, so that you won't need to worry as much about who makes it, just that it "passes the tests". (and you'd probably need a system to validate the validators, but splitting up the people involved means it is significantly harder to hack multiple products to all have them falsely verify each other)

Obviously I have no idea what I'm talking about and am just kind of musing at the idea, but trying to secure the whole supply chain from digging materials out of the ground all the way until it is in the hands of the consumer seems like an exercise in futility. You'll never be able to secure it in all cases, and like you said a truly motivated attacker is going to be able to break the chain (even if it means threatening a handful of people with death so you can get 5 minutes alone with a board).


What high technology manufacturing America does is in the security space, otherwise Japan is a trusted source.

>we'll start to see a rise in American high tech manufacturing for security purposes alone.

Already exists in the form of 'country of origin' procurement for high security applications.


I've been looking in detail at three different Supermicro motherboards but so far have not been able to spot anything. Even against a backlight there is no sign of tampering between the layers.

The most compelling explanation I've heard is that the BMC chip could be programmed by two distinct flash chips, one for factory programming and one for some other purpose. In some SKUs, the latter isn't populated but it has a higher priority than the first chip.

Since there are many flash chips fitting the same pin out, all it took was soldering a compromised flash chip (with firmware for the BMC chip) onto pads that are already part of the design to compromise the whole system without any obvious sign that the board was tampered with (because in some SKUs, both chips were populated).


The BMCs on the newest Supermicro servers are from ASPEED. The X10 models have the AST2400 [0] and the X11 models have the AST2500 [1]. They have ARM CPUs and run, basically, an embedded Linux.

If you wanted to "backdoor" motherboards that shipped with these BMCs, wouldn't it would be much easier to just install your own "customized" version of the firmware on them? It certainly seems that it'd be much more difficult to incorporate another device into the system.

[0]: https://www.aspeedtech.com/products.php?fPath=20&rId=376

[1]: https://www.aspeedtech.com/products.php?fPath=20&rId=440


If I'm right, that's exactly what they did. When the BMC chip boots, it checks two flash chips for firmware so the attacker just uploaded their own code to one of a million standard SPI flash chips and plopped it onto the board. They didn't have to incorporate another device into the system, the system was already designed for two flash memory chips. However, to save money on some SKUs, the manufacturer left one of the positions on the board open.

Normally this wouldn't be worth talking about because most active chips are too complicated and too design/supplier specific to carry out an attack like this, but SPI flash is about as standard a footprint/protocol as you can get in EE short of transistors so if you ship a product that could be reprogrammed from unpopulated pads, you're opening yourself up to a large attack surface.

Honestly, after I read the latest BMC chip theory I was like: "Oh, shit. Have I done that?"


If possible, it is better to have separate hardware that can continuously compromise the firmware. That way your exploit continues to exist even if valid firmware is flashed directly onto the memory module.

well companies like Apple and Amazon are reflashing/updating, so that wouldn't stick.

because all it takes for it to be discovered is someone checking the SPI flash contents

By explanation do you mean theory or is it coming from somebody who has special knowledge of the situation?

I'm not trying to be adversarial, even if it's only a theory it's an interesting one, but given the amount of conflicting information we have regarding this whole mess I think it's important to be clear about what's pure speculation and what's been reported by people supposedly in the know.


I looked at this back in 2013. Here's some slides from a talk I did after spending 48 hours with them[1].

The BMC back then was by a company called ATEN, who make KVMs. The modern BMC is by ASPEED - I don't know if they're related.

What's described in the article is exactly how the old ATEN firmware worked normally. It was a spectacularly poor product from a security perspective.

[1] - http://mandalorian.com/dl/himym.pdf


Maybe you are not a high value target?

That raises an interesting question about just how targeted this kind of attack could be. At manufacture time, do the folks on the assembly line (so to speak) know who a particular board is going to? If not, they would have to add the extra chip to all outgoing boards, which means there should be plenty of them in the wild, no?

If the motherboards were customized for a particular customer, you'd know exactly who they're going to. That would eliminate the problem of letting the exploit travel too widely as well.

Right, but does that happen? I honestly don't know. Clearly a company like Amazon or Apple buys in large enough volume that they could be asking for customized MB's, but does anybody know if that actually happens? If it does, then that would definitely moot the question I was posing above...

Or swap the boards out in transit.

Seems more problematic though. You'd have to manufacture the doctored boards, extract them from the normal shipping process, keep them hidden somewhere, then swap them out for the ones destined for the target customer(s). I guess it could be done, but it seems risky.

Couldn't it be done on-demand? Apple orders X hundred boards, motherboard manufacturer makes their small modification(s) to a line that is currently producing the same models of motherboard as Apple ordered, they produce a handful, then they revert and mix in a few of those modified boards into the real order. I don't really know the exact scale, so maybe they make a few hundred / the entire order with chips in them, but economic cost isn't a big deal for things like this, so even losing money making the modified boards wouldn't be the end of the world (and presumably they get a hefty sum of money for whoever is paying them to do this).

I thought China was famous for extremely short turnarounds for industrial engineering edits, so it seems plausible that they could manufacture the boards in a reactionary way and not need to do much in the way of logistics to get them to their targets.


No comment.

If I was a high value target (and knew about it) I would definitely not let you know, if I was a high value target and did not know about it I would not be able to tell if I was or if I wasn't. So any high value target and anybody else would not be able to tell you they were a high value target.

What about the variation where you're not a high profile target and you know it?

Same here. I have four different Supermicro motherboards purchased in May for servers in my home. I'm sure there exist people and organizations in the world capable of putting malicious hardware on one of these such that I can't detect them. But insofar as I've personally examined them and the available evidence from Bloomberg, color me skeptical...

Ok now try to patch the BMC, you can actually talk to it with openipmi on local host.

I'm familiar with OpenIPMI (I use it for remote fan control a lot) but I'm not clear on what exactly you want me to try?

I really want to see someone on here with access to one of their recent boards try and report on this. I'd try it, but I sold my last Supermicro board years ago.

Try what? Updating the firmware? I do it every time a new firmware version is released.

Back around 2014-2015 supermicro had this bug that would not let you flash the main firmware. Would not happen on every machine maybe 25%. Had to derack and send a number of machines back.

Are your Ethernet shells metal, as described in the article, or plastic which the article describes as normal?

This metal shells rubbish is a key indicator that the whole story is bogus. Metal is completely normal.

You purchased servers from 2013-2015 in May? As in used servers?

You would recognize what looks like an extra resistor?

The supposed infiltrated part is a six terminal RF device. Not something that would ordinarily show up on a server motherboard. In any case, Joe Fitzpatrick has already disclosed that he used the part merely as an example and Jordan Robertson expanded that into a work of fiction.

I hadn't seen this before, but searching for "Joe Fitzpatrick Jordan Robertson" finds https://appleinsider.com/articles/18/10/08/security-research... which seems to be what you were referring to?

The original source is Joe Fitzpatrick's interview with the Risky Business infosec podcast. Apple Insider is just summarizing some of the points from that interview:

https://risky.biz/RB517_feature/


Where is the 6-terminal claim from?


There were quite a few pictures of what is supposed to be the device in the Bloomberg article. Knowing what they say it looks like and knowing roughly where to look I'm 99.9% sure that none of the boards I have here have that device on them.

I don't have the reference handy but someone claimed to be a source and they pointed to a generic item on digikey / mouser as an example. I imagine that it got extrapolated by Bloomberg into that.

They really have no idea what they are talking about at this time and it's probably fluff.


I'm not sure why you're downvoted, except the lack of citation. Your recollection is correct, it's from the Joe Fitzpatrick interview with Risky Business, which was quoted by Apple Insider. (Fitzpatrick was named as a source in the original Bloomberg article.)

Long story short, that photo does not show the device involved.

"Robertson was unable to produce photographic evidence of the chips in question, saying they were described to him by protected sources. Indeed, Robertson in September asked Fitzpatrick what a "signal amplifier or coupler" looks like, suggesting the publication narrowed the attack package down to that particular component. Fitzpatrick sent Robertson a link to a very small signal coupler sold by Mouser Electronics. "Turns out that's the exact coupler in all the images in the story," Fitzpatrick said.

https://appleinsider.com/articles/18/10/08/security-research...


The image caption on the bloomberg story reads "Microchips found on altered motherboards in some cases looked like signal conditioning couplers". They didn't claim "that's the chip"

It has more terminals that a resistor, it's a pretty unusual package and it would stand out enough for me to spot it knowing that it is there. The area of the PCB that you could expect that thing to live in is about 5x5 cm square.

Isn't the idea that the boards weren't tampered with but manufactured by contractors including extras?

Well, that depends on your definition of tampering, but if you want to exclude manufacturing something that is not what was specced then I am fine with that but please do supply a new term.

I would definitely spot that device if it were on these boards because it was described in detail and there were some pictures of what it supposedly looked like.

A device like that is not on either side of the board and it isn't in between the outer board layers (where it would be much harder to spot, especially if the cavity would be covered by a ground plane on one side).

I am not saying it is impossible, it is just very hard to hide something like that once you know it is there. The only candidate spots left that I can not check without destruction is underneath some of the devices or inside some of the devices. That would be a different level of sophistication than the original article alluded to.


> I would definitely spot that device if it were on these boards because it was described in detail and there were some pictures of what it supposedly looked like.

In case you missed it, there is an article posted today [0] that has this quote from "Hardware security expert Joe Fitzpatrick", one of the Bloomberg sources, regarding "the supposed spy chip":

> In September when he asked me like, “Okay, hey, we think it looks like a signal amplifier or a coupler. What’s a coupler? What does it look like?” […] I sent him a link to Mouser, a catalog where you can buy a 0.006 x 0.003 inch coupler. Turns out that’s the exact coupler in all the images in the story.

[0]: https://9to5mac.com/2018/10/09/bloomberg/


Oh, that's interesting. So they basically took one guys hypothetical and turned that into a news item positively seeded with images of the hypothetical, rather than an actual device.

The original article has now dropped into the real of SF for me until they show a detailed shot of an actual board with a parasitic device on it. Until then this is a wild goose chase.

Thank you for pointing this out.


My understanding is that certain parts on the PCB were swapped out for malicous parts. If that's the case, it's probably not something that could be uncovered by a purley visual inspection. The 'spy' chips were likely made to look identical to the original parts.

That’s not what the BW/Bloomberg story claimed - it specifically called out a chip that wasn’t on the official BOM and had been added to the build.

The initial allegations from Bloomberg suggested ON the otherboard, not in, as I understand it.

There was mention of one being discovered buried inside the FR4 PCB material.

I don't think you'll find this in a board that doesn't otherwise normally have lots of other buried components ... The added cost of that extra process (using buried components) is so way higher than normal and such a board is going to look noticeably different from a normal board ... I'm tempted to think that someone told the Bloomberg guys that it was possible and the took it that it had happened

Yeah, particularly given it was against a US telecom company, the NSA would make sense as the source of the implant.

No that would make 0 sense. The NSA doesn't "attack" american companies with covert implants.

They get FISA court orders that force american companies to attach their equipment.



> ...doesn't "attack" american companies with covert implants.

Specific example aside, it's worth talking about why this does happen. "Black bag jobs" can mean "we didn't get a warrant", but they can also mean "we got a warrant and still aren't telling".

Even given a court order, there's still a possibility that employing surveillance by fiat will cause somebody to leak, or modify how they handle data, or simply reveal information about what sort of surveillance tools a given agency employs. Given that a FISA order can be obtained without a defendant, getting a court order and then doing the thing secretly anyway gives a sort of "bowling with bumpers" advantage where the project is approved if it gets revealed, but also done without revealing anything if it isn't.

More disturbingly, there's also substantial evidence that the NSA attacks companies covertly in places where they couldn't get a court order. Taking a specific device out of the supply chain and adding surveillance before it's shipped to the destination is a warrant-worthy project. Setting up systematic physical vulnerabilities with a use case of "turn it on some time in the future to get something interesting" isn't in the purview of a FISA order, so if the NSA did do that it would have to be without an order.


Wasn't PRISM all about attacking American companies with covert implants? For instance tapping into Google region to region data transfers, after which Google started encrypting everything.

I thought PRISM wasn't covert. Companies were compelled to allow them to install their sniffing hardware, it was all above-board. Snowden even leaked an internal slideshow with a nice timeline of when each tech company joined the program.

Parts of that whole expose were covert. In the case of Google we know by tapping fiber connections that they had between data centers (as sseth mentioned, using foreign intelligence peers to do an end run around legal protections), which was on Google owned fiber, theoretically entirely "in-house", so Google transferred it unencrypted. I believe they called this operation "Muscular". After the fiasco Google started assuming everything was hostile.

PRISM was an overt program (to the data stewards, not to the public) for processing FISA warrants and the like.

Other NSA programs that Snowden revealed were covert hacks. https://en.wikipedia.org/wiki/Global_surveillance_disclosure...


My understanding is that the google tapping was done in UK using British intelligence services, thus bypassing the legal constraints.

The program for tapping data center links had the internal code name MUSCULAR and was a partnership with the British GCHQ, who actually did the intercepting.

PRISM was at first reported as some sort of direct access to the servers of certain American companies, but it turned out to be the code name for a joint program with the FBI for using FISA warrants to request data from those companies.


I used to work in engineering at one of the big wireless telecoms. The impression that I got was that many of the outsourced services were compromised. For instance, we had zero control over our voice mail systems, they were outsourced to Amdocs.

You can see how this benefits the NSA; if the voice mail is outsourced to a foreign company, and the NSA buys intel from that company, it's technically not spying on US citizens, particularly if they're getting metadata.


You don't know that. We do know that the USG covertly intercepted fiber communications.

https://www.washingtonpost.com/news/the-switch/wp/2013/11/04...


The story literally quotes the general of the NSA, saying they go though the FBI to get a FISA court order to compel the company..

Additionally, the story quoted talks about how the UK obtained the data and gave it to the NSA.

Nowhere is the NSA installing covert implants. They just don't do that.

The CIA does that :)


> The story literally quotes the general of the NSA

Ah, so he pinky-promised? Well OK then!


What I meant was that you don’t know it isn’t done.

You are taking the word of a spy? Did he say it wittingly?


Sure! It was the least untruthful thing he could say.


"SSL added and removed here ;-)" doesn't sound like a FISA court order.

The NSA has "attacked" internet infrastructure for many years before it became sort of legal (but probably still unconstitutional).

but the NSA has been performing backdoors on hardware for years

PRISM

https://www.schneier.com/blog/archives/2018/08/backdoors_in_...

> Juniper has confirmed that an initial analysis of malware linked to the National Security Agency appears to affect its firewalls.

https://www.zdnet.com/article/juniper-confirms-leaked-nsa-ex...


China is just as interested in US' communications.

Huawei & ZTE have been alleged with these exact type of attacks, by the US government


> the NSA would make sense as the source of the implant.

That doesn't make sense based on the assumption that US telecom companies already cooperate extensively with US inteligence agencies.


"Extensively" is not 100%

The only US Telecom that did not allow NSA direct access to vacuum up transmissions was Qwest, and their CEO was sent to prison.

..for insider trading. You are implying that he went to prison because of the NSA. He went to prison because he sold $52 million in stock after the intelligence community said they would no longer consider Qwest for classified government contracts because of his refusal to cooperate with the NSA.

He went to prison because he sold stock based on insider information. Regardless of the reasons for his trade, it was still insider information.


Everyone breaks laws all the time. The question is whether the Government decides to focus on your activities in order to identify your crimes.

I find these attempts to distinguish between different state sponsored criminals to be a diversion and subterfuge.

Whether China or the US re-allocates your IP, you can expect a competing product made in China. That you might have a relationship with one of them probably doesn't change anything unless they actually think your firm is the best one for the job of maximizing the results on their tax base.

I mean maybe I'm wrong; would it make any sense that these Republics take private corporate property more seriously other parts of their Constitutions they have violated at least until caught?

The US makes the claim China does not and former president and CIA head George Bush floated corporate espionage as THE plan for handling the absurd costs of "intelligence" criminals after the cold war..

I'm always astounded that working in a competitive market seems to blind people to significant stated facts of the environment their market is operating in.

In nature, it might make sense to just outrun the weakest, after all, a bear has a limited appetite. But superpowers have unlimited apetites and will collapse like the USSR if they should ever expand slower than cancer.


Is it illegal to report an intelligence attack that is perceived to be foreign? If not, why not have all attack reports assume they are foreign to begin with? This would give the reporter credible deniability, and put the burden on the US government to argue otherwise. Regardless, the report is released without the reporter getting in hot water. Or am I missing something?

Incentive for a company to say "No" when the FBI offers to "fix" the problem quietly either by going up the chain of command internally to get answers and stop a blown attack on a US owned and operated business or use contacts within the US security infrastructure to stop the foreign criminal or state adversary.

Most of these attacks never leave the room at corporate HQ where they are discovered unless an engineer wants to permanently screw themselves out of a career.

I once tried to leave a linkedin recommendation for a friend I'd worked with on a high profile project where he discovered Chinese state actors performing corporate espionage and we stopped it. The FBI came in and carted off the servers, we switched data centers, re-deployed, and that was that. We would never have been the wiser if he weren't closely monitoring network characteristics. 3 years and 2 job changes later he messaged me back to say, "Thank you for the rec. but don't mention that shit!"


>it's illegal to report an attack by US intelligence agencies

Could you expand more on that?


IANAL, but New York Times Co. v. United States is a famous precedent for the first amendment protecting the press' right to publish classified government documents.

That's what I was kind of getting at, the press is free to publish pretty much anything they'd like AFAIK.

> because it's illegal to report an attack by US intelligence agencies

is it? and if you were witness to such an attack, how would you be able to attribute it to the USA vs some other actor?


> it's illegal to report an attack by US intelligence agencies

Is this true? I mean outside a specific gag order or working under a clearance, you could find an issue with a piece of equipment, publicly talk about it and then be arrested because it turned out it was the US government who caused the issue?


> The "trojan ethernet connector" paragraph mentions similarity to an NSA implant, which appears to be this: https://en.wikipedia.org/wiki/NSA_ANT_catalog#/media/File:NS....

Interesting. This one seems to rely on the presence of the USB connectors for powering. So, basically, any ethernet port with only Ethernet connectors should be safe from this kind of attack? The only powering option there should be PoE but I have yet to see someone buy PoE switches for a datacenter...


"any ethernet port with only Ethernet connectors should be safe from this kind of attack? "

I know that some IPMI implementations can control/use the on board ethernet. There might be some hole there. Similar for the built in management on some Intel processors and nics.


>but I have yet to see someone buy PoE switches for a datacenter

It was a small datacenter but we had a bunch for PoE KVMs.


IANASecurityExpert but one one the named sources of the original claim also came out to say that the 'original attack' was a conceptual idea he had but that doesn't even make sense to apply in practice (considering better alternatives) http://appleinsider.com/articles/18/10/08/security-researche...

> it's illegal to report an attack by US intelligence agencies

Can you provide a citation for this statement please


[flagged]


>> I'm now wondering

> You can't use uncertainty [...] as evidence for your own nuttery. Stop it.

To wonder aloud should not be so demonized much less demanded to cease.


If you diagram out that sentence, you'll find it wasn't the wondering I was "demanding to cease".

I mean, sure, it could be NSA. Let's wonder! The fact that no one in public has seen one of those things isn't supporting evidence for that theory. It's even less grounded than the Chinese one.


Who knows... but it is funny that healthy skepticism can transform into this kind of acceptance. First disbelief, then, well ok, maybe but if it is, it’s probably not the suspected entity, but an altogether different entity... it’s like multiplying probabilities but thinking it increases likelihood.

Conspiracy theories do love the combination of the whataboutism and the competitive debate strategy of "spreading" where you make lots of weak (or better yet, completely unsubstantiated) points, and if your opponent fails to refute everyone, you win.

~

Just about anyone on Google's Project Zero team, off the top of my head. They would probably be both competent and enthusiastic.

Beyond that Apple, Microsoft, Tesla and probably Amazon have sufficiently capable public (or past public) researchers working for them. But I doubt many would want the publicity one way or the other.


Most any security company would LOVE to get access to the purported hacks and study them.

I've seen several comments regarding whether or not Apple, Amazon etc. would deny the hacking if its true and if that is fraud or not. I work at Amazon now and previously was in the Navy, holding a TS/SCI. My firm belief is if such a hack happened, it would not be disclosed to anyone without a clearance, and the organizations that are denying it have no knowledge that it occurred. Furthermore if there truly was a compromise by a foreign nation it would be classified as a national security threat and subsequently classified and kept from public knowledge. Anyone who disclosed the truth would be at risk of loosing their clearance, job, and could end up getting the snowden treatment.

> the organizations that are denying it have no knowledge that it occurred

Are you saying that Steve Schmidt, the AWS chief infosec officer didn't know about the hack? Or that his article [0] was published to purposely hide it?

If only one person in Amazon knew about it, it would be Schmidt. And if Schmidt knew, I don't think he'd write an article so strongly claiming Amazon doesn't know anything about it. The only thing in my mind that lends credence to Schmidt covering it up purposely is that $10B contract the Pentagon is putting out- perhaps they've told him to play ball as part of getting the contract. But even then it seems a stretch.

[0]https://aws.amazon.com/blogs/security/setting-the-record-str...


CISO is not the most likely point of crossover, the most likely point is the general counsel's office. Companies don't talk to the Feds without a lawyer, and they also don't issue high profile statements without a lawyer. And unlike the CISO, conversations with your lawyer are privileged.

It's very possible Steve wouldn't know, both owing to past precedent (see SmokeyJ's comment on Alex Stamos) and owing to whether or not he's cleared.

He about has to be cleared if he's the security chief over govcloud.

Whoever directly oversees it and acts as the stakeholder for GovCloud should be, sure, but there's no reason for the person above the direct overseer to be cleared. Otherwise by that logic Bezos should be cleared as well.

I may be mistaken but I'm fairly confident govcloud is an unclassified network.

At least publicly, they're acknowledged to go up to Secret right now. https://aws.amazon.com/blogs/publicsector/announcing-the-new...


The Bloomberg article specifically claimed that Apple themselves discovered the chip in a random spot check. If an Apple employee discovered it, it would have been communicated all the way up to the executive level prior to notifying anyone outside the company (such as the FBI), which means you can't just chalk this up to a handful of lower-level Apple employees being covered by a gag order and the executives not knowing.

It also claimed Apple removed 7000 SuperMicro servers in a few weeks. That seems especially unlikely to happen without at least some explanations to upper management. Sure, they could lie to management about why but either way management can’t then claim no servers were removed without lying themselves.

Apple also said they didn’t even have 7000 SuperMicro servers to begin with.

unless the NSA or another intelligence agency has an insider that could catch that before it made it up high enough to cause trouble. conceivably, someone below the insider could leak to Bloomberg realizing that they have limited options.

That seems like a lot of work. What would be the point of that?

If Amazon is being spied on by foreign intelligence, wouldn't the NSA want Amazon to know about it? Particularly since government data is hosted on Amazon's servers.


Because now the NSA has a strategic foothold. If they acknowledge the hack, then the adversary will move on to something else. If they don't acknowledge it, they can secretly mitigate it, by feeding false data, for example, and waste the adversary's time.

> Furthermore if there truly was a compromise by a foreign nation it would be classified as a national security threat and subsequently classified and kept from public knowledge.

This is exactly what I think. Anyone confirming such a case publicly could cause a huge international confrontation between two largest economies in the world. It's not about tech or business – it's about national security and international politics.


I can see where the Navy/Military/Government could compartmentalize a hack like this. How could a company like Apple or Amazon keep this under wraps? How could they keep the knowledge of such a hack within the TS/SCI employees?

The cleared department is handled the same way as in the military in terms of security. Amazon has SCIF's etc. So unless a disgruntled employee steps forward who doesn't care about there life, I imagine its easily contained (and symptoms of an employee being disgruntled are highly monitored when they hold a clearance)

I’m thinking about the non cleared data center folk, the sys admins and developers who use the servers for their applications.

How do a bunch of Supermicro servers vanish wintout anyone noticing? I’d expect quite a few people would be involved that do not have any clearances. Apple is known for their secrecy but a few other companies named are not.


At the scale their datacenter are, they must be replacing a full rack of servers every single day, just to follow a standard 3 years depreciation policy.

Servers practically vanish every single day. Add a few more supermicro and it's not even noticeable. Business as usual.


Maybe they didn't remove them.

I knew a dozen people working on Amazon Go for like 4 years before it launched. Not one person leaked, even internally, what the hell they were building. Just that it was awesome and I should come join their team.

Somehow, Amazon is really good at keeping secrets.


1) Everybody involved has agreed to keep secrets.

2) You compartmentalize everyone so nobody has the complete picture.


If it is classified and a cleared employee at Amazon/Apple/etc. blabbled there would be life altering consequences for them.

Then there will be? As someone apparently/allegedly blabbed to Bloomberg?

I say again to Bloomberg: picture (x-Ray) or it didn't happen.



Except they didn't keep it wrapped, did they? And people all the way to the CEO knew about it.

Point being it started with the CEO.. At what point do you suspect the publicist of all people was clued in? Absolutely never.

What's the point of classifying national security threats?

When a threat is discovered it can be very helpful if the attacker does not know you've discovered the threat.

Now you can observe them and only intervene when absolutely necessary, thus giving you time to learn more about the attackers and their methods.


Right. So, if this hack is real, the attacker now knows we know.

The previously reported issue was alleged to take place in 2013-2015.

This issue in this thread is alleged to have taken place in August 2018.

In the intervening time, much could have happened.


They might actually know for much longer: if your spying devices suddenly stop communicating to you, that's likely you've been discovered.

If that story is true (and I personnaly think it has a high probability to be), what would a gov or a large org do? Investigate, confirm they have been compromised but then.... leave the hw in place and data flowing back to the alien mothership? Unlikely.


Yes, it was made public at this time for a reason. I have no idea about who made it public and why, but you can be sure there is a bigger game here.

Did the journalist and/or their friends and family make money on the massive drop in Supermicro stock?

Is the Trump administration asking to push this information out to earn favor in the trade war?

Are the investigators stumped and using this in an attempt to flush out new leads?

No idea.


If you know that something is compromised, you can use that knowledge to feed misinformation. You don't want them to know that you know.

I don't necessarily agree with the below, but one could argue that classification is necessary to prevent mass panic/prevent attempted vigilante justice/protect the government's image/buy the government time to investigate/respond appropriately.

Things get voted on and positions change so I have no idea what you're referring to with "the below," but it's much simpler than trying to protect "the government's image."

If you're attempting to hack me or steal data from me and I know you're trying (specifically as would be the case with this chip if the story holds up) then I'm in a much better position to try to figure out how, or provide misinformation, or try to turn someone in the chain of custody if anything needs to be physically handled. Or at the very least, if it's an espionage or military situation, it makes it easier to know who to kill.

All of that goes out the window if you immediately disclose every threat. Whoever is attacking you will simply use the means you haven't discovered yet and stop using the ones you have.


Perhaps I should've written "the following" - I just meant the list that I provided in the rest of the sentence.

I believe you covered more in-depth content that could be filed under "buy the government time to investigate/respond appropriately."


To not give away other nation that their capabilities are mitigated.

power

Ignorance is not a defense, especially for a director of security. Lying about knowing how the organization you lead operates is a bad as directly lying about how your organization operates.

OK so this is a different hack than Bloomberg reported before: ethernet jack piggyback instead of bmc. I'm not sure this adds credibility to the allegations in the other story.

The details that Bloomberg related previously are so different that this couldnt be what they originally were reporting on. This adds to the China hacking server board narrative, but it does nothing to prove the Bloomberg reporting actually true.

It does cast doubt on the denials made previously.

It seems this story isn't totally smoke and mirrors as Apple, Amazon and Super Micro seem to want us to believe.


Read it more carefully. The ethernet jack is a tactic used by US intelligence years ago. That was mentioned in the story to explain the history of supply chain attacks.

If it is well known... can't imagine others wouldn't follow if it works.

Hard to imagine it remains an exclusive method by one actor.


> subsequent physical inspection revealed an implant built into the server’s Ethernet connector

It looks pretty clear that this is the hack in question, not an example.


Do you think the NSA manufactured their implants in China?

Plausible deniability. Wouldn't surprise me if China is getting the blame for a LOT of US hacking.

With the current political climate, that might be the intention. If you undermine international trade though marketing you don't have to fight a tariff war.

Seems like all US conflicts are now an excuse to race to the bottom with whoever our "enemy" is. We imported torture from the middle east and now state run news and corporations from China.

I think it's totally plausible. See my other comment in this thread for my conspiracy theory :)

Note that BMCs can also piggyback on ethernet ports; I've seen some vendors use a shared ethernet port for OOB and ethernet.

Which is fun because you can accidentally put a super important insecure oob service on the same jack as an internet exposed web service.


IIRC... The DCMI spec actually requires that the BMC be capable of sharing an Ethernet port. A dedicated Ethernet port for the BMC is optional.

SM boards with dedi and shared phy for ipmi are usually defaulted to auto mode. I think first interface it can arp for the gateway on wins (or maybe dedi then shared).

This story is getting more incredible every day.

Bloomberg only has second hand sources, and all the exploit details are based on speculation from security researchers -- not from insiders.

It looks like Bloomberg heard several rumors of supply chain manipulations, mixed that up with plausible scenarios thought up by security researchers, added a few photos from random electronic parts, and voila you have a compelling story to tell.

This "new evidence" talks about a completely different type of attack than the original article. It corroborates nothing. It just shows how misleading the original story was.

I think the most damning part was the use of so many misleading photos and illustrations. All photos were pure speculation (this is what this chip might look like, this is where it would make sense to put the chip). But neither the captions nor the text made that clear.

The only thing I believe about the story is that they have a couple of sources who have vague, second hand rumors about supply chain manipulation.


The reporters in question apparently have a reputation for credulously repeating things they hear about cyber-attacks.

https://twitter.com/RobertMLee/status/1049617855396933632


Yup. Which is why I'm holding on to my discount bin supermicro shares.

The bloomberg story stinks to high heavens even if there are live examples of infiltrated supply chains


It seems like there are two possibilities to me:

1) Bloomberg has a number of sources that are mistaken/misinformed, but this is not necessarily a made-up story, or

2) Bloomberg is nearly correct (minus some technical details) but the US government is forcing these companies to respond as if the story is wrong - possibly because of diplomatic reasons.

What is the likelihood that #2 is correct?

(there are other alternatives, but I believe that the likelihood that this is 100% or at least majorly fabricated by Bloomberg is near enough to zero)


Taking the writer's statements as truth, 6 out of 17 sources for the original Bloomberg story were government. Presuming the information was classified, each one of those individuals is risking losing their job and going to prison for talking about it.

What kind of conclusions the writers made from talking to the other 11 sources, what seniority they had, or even what companies they were from besides 3 from Apple, are anyone's guess.

I always am suspect when a government official or employee releases classified information. In some cases extreme moral outrage seems very plausible (Manning, Snowden.) The information is so startling the leaker decides they can handle spending much of the rest of their life in prison. In a much larger number of cases it is for political reasons (Libby, countless other stories where the source is never identified.) Many times, however, I suspect, the information is made up and then no laws have been broken, as far as I understand it.


The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud.

But they probably can force a handful of engineers to tell nothing to their employer, who would be vehemently denying in good faith.

How would the government get in contact with the engineers who discovered the hack without going through their managers, the CISO, etc. ?

An LRAD "Voice of God" would certainly get their attention.

https://www.wired.com/2007/12/the-voice-of-go/


Not only would they have to force the engineers to tell nothing, they'd have to force all capable engineers to vehemently deny it to their bosses who ask or ask them to look again.

>The US cannot force those companies to lie.

When it comes to companies like ATT/Verizon it seems like no forcing is necessary. They are complicit.


>The US cannot force those companies to lie.

But they can. Maybe only in cases where silence is a canary, but that still sets a precedence for them being able to force you to lie.


You have it backwards. The government can't force them to put out "no comment" press releases. They can force them not to reveal certain information. Maybe that means the only logical thing to do is to say "no comment", but it certainly doesn't prevent a company from commenting as long as they don't reveal the gagged info.

> The US cannot force those companies to lie. They can force them to stay silent, in which case they'd just say "No comments". If those companies are lying, they are committing security fraud.

Would the US government have to force these companies to lie? It's quite possible that the denials were the result of voluntary cooperation.


A public company issuing such strongly-worded denials that turn out to be untrue would be leaving themselves at risk of an investigation by the SEC and/or a shareholder lawsuit.

It would be pretty extraordinary for the government to sue a company for cooperating with the government.

A company has to follow court orders. The government would have to sue itself.


The SEC prides itself on its independence; I don't think it would have any problem taking on a company that lied on behalf of some other agency of the federal government.

Now, a court order would indeed be something different, but I find it hard to conceive of a judge compelling a company to lie.


If they didn't have to lie and there was no legal order but voluntary cooperation, as the grandparent post suggests, then such voluntary misinformation can easily be a violation of SEC requirements; and one part of the government certainly can prosecute you for doing something that another part of the government suggested (but didn't/couldn't legally require), it wouldn't be the first time.

> The US cannot force those companies to lie

That's a nice government tender you're working on, looks very profitable! Would be a shame if someone rejected it because reasons.

While it isn't the direct influence described i imagine this scenario is highly effective and getting companies to stay in line.


What case law exists for not being able to force a company to lie? Particularly non-press.

You are right and you are wrong. Legally, you are right. Practically, there is evidence that various elements in the US Government have at times coerced people into making untrue statements.

Is it really so clear-cut what the US can do? Maybe this so significant (it may well be if the allegations are correct) that they're running a campaign to really try to make people believe that this story is false.

The companies may have a strong incentive to cooperate in this campaign too, both to save face and government relations.

Unfortunately all of Bloomberg's allegations don't hold much water either, even though it'd be so easy for them to make the story credible with some details.


It's interesting that everyone is assuming the only country interested in pressuring them to lie would be the US rather than China.

But if the story was entirely false, they'd have sued Bloomberg for libel already.

Well, the story just came out a few days ago. I wouldn't rule it out in the future, nor would I leap to the conclusion that an absence of such a move means there's truth to the story.

That might still be in the works, legal process moves slowly. If we don't hear anything in another week or so, this could be worth considering.

There could be other arrangements made with Bloomberg that would be jeopardized by that course of action. EG... you find a serious piece of news that can destroy me, so I pay you or make another arrangement to prevent that. If I later turn around and sue you, our backroom deal could blow up in my face. Too much is unknown right now.

or 3) Bloomberg is vaguely correct, but wrong in the specifics, including naming Apple and Amazon.

I.e., They get credible reports of SuperMicro servers having been compromised, they know that Apple and Amazon are customers. They find 1 over eager source willing to say Apple and or Amazon received compromised servers.


My take on this is that it's been fairly obvious for a long time that these kinds of attacks are possible (if not easy) with today's technology. One could design a microcontroller, for example, that was disguised as an 0805 capacitor and functioned like an 0805 capacitor, but also had other functionality.

So why is this suddenly breaking news? It bears resemblance to most of the propaganda stories we have seen in recent years:

- it is based on truth. Supply chain attacks are known to exist

- the US government has a goal of escalating with China over trade and IP practices.

- national security threats justify nearly any form of government action in today's world.

- the story turns out to have been leaked through foreign sources. This is typically the pattern we see because of concerns about propaganda coming directly from US government officials to US news outlets. There is typically a middle layer that is outside of the US where the allegations can originate from until they are broadly accepted as fact.

So Apple and Google are not really lying. Chances are there has not been any sort of major security breach in either of those companies due to supply chain attacks. It is possible that they have been barred from revealing information about it for national security reasons (in this case propaganda reasons).

So I think we can expect the following next steps:

- The story will continue to hover in this slow reveal format until enough laypeople come to understand the key concepts -- circuit assembly, components, trojan horse components, QC processes, subcontractors, etc. Once the stage has been set there will be more revelations and leaks from major companies that corroborate the story.

The goal is to make China the crisis in the buildup to the 2020 election. It's not a coincidence that this strategy is getting underway right after the midterm elections in the US.

Our president has already been attacking China with rhetoric and trade sanctions, and this story is meant to turn public opinion broadly against China.

The supply chain attacks do not have to have been significant (or successful) to make this happen. The very idea that "sneaky" Chinese intelligence agencies and firms would be able to slip this by US firms' quality control measures is enough to inflict paranoia on Americans and help them start to view China as a terrifying adversary that must be stopped.

China's military outnumbers the US military by 20:1 in terms of the number of active duty fighters, and China's economy is approaching first world standards in major cities far faster than the US had ever expected. China dominates scientific publications in the hard sciences, and its top universities are 10x more competitive (or more) than top US universities.

So hawks in the US realize that this may be the last opportunity for some sort of power projection or military driven containment of China's ambitions.

This is foolhardy, because China is led by a group of highly rational people whose policy responses to the administration's trade threats have been masterful and precise, and have conveyed with no uncertainty that China will not be bullied.

So what we're seeing is a short time horizon strategy by the US which is meant to have electoral consequences in 2020 and pave the way for some degree of hostile escalation with China. US weapons systems are still significantly more advanced, but China has likely weaponized many aspects of US infrastructure via these sorts of supply chain attacks. My guess is that the US Government is not aware of many of these, and will panic when they are discovered.

Fortunately for us all, China's leadership is calm and not prone to knee-jerk responses. China is rising to world prominence faster than expected, and the US will not take that lying down. However it is probably too late at this point, as China has a tremendous amount of soft power stemming from its importance to the US supply chain. Because of this there is still much hope for a peaceful, trade-driven equilibrium to emerge.


> Our president has already been attacking China with rhetoric and trade sanctions, and this story is meant to turn public opinion broadly against China.

Ah yes, Bloomberg, well-known for protectionist rhetoric and support for Trump.

> top universities are 10x more competitive (or more) than top US universities. A citation would be very helpful here. And no, number of paper's published isn

> This is foolhardy, because China is led by a group of highly rational people whose policy responses to the administration's trade threats have been masterful and precise, and have conveyed with no uncertainty that China will not be bullied.

Citation also needed. They're so rational, they just detained the president of Interpol, have committed mass internment of Muslim citizens, and can't go six months without trying to ram NATO ships in international waters.

> but China has likely weaponized many aspects of US infrastructure via these sorts of supply chain attacks

Wait, what? So doesn't that mean the hawks were right to be suspicious? And that this really should be a national security concern? "This is all just a pretense to make people afraid of the PRC! By the way, you should be afraid of the PRC."


> Ah yes, Bloomberg, well-known for protectionist rhetoric and support for Trump.

To be fair, many of Bloomberg's sources were government, so the government could easily have said "we'll leak this to Bloomberg." It doesn't require Bloomberg to have a protectionist agenda more than being a pawn.

I agree with you on the rest though; "masterful and precise" in particular seems pretty farfetched and ridiculous, and many parts of China's economy look like a house of cards.


You're conflating rationality with propriety. China wanted Meng Hongwei to make it easier to use Interpol to track down Chinese dissidents, directly in conflict with the Interpol charter; and merely two years later they kidnap, disappear him, and charge him with disloyalty to the Chinese Communist Party. He was supposed to be their stooge pigeon, but was clearly ineffective. Again from their point of view, entirely rational to have him removed because he wasn't doing what was expected. And that's what's so conspicuous, they openly admit China was his master, not Interpol.

China, Xi, and the Party are in a sense all one in the same thing. There is nothing more important than loyalty to the one Party, and the concept of one Party rule. And Uighurs are a threat to that, so they're labeled terrorists. If you accept the idea that one party rule is necessary, it's entirely rational to aggressively, perhaps even violently, ban all possible opposition. That's the nature of any autocracy.


>One could design a microcontroller, for example, that was disguised as an 0805 capacitor and functioned like an 0805 capacitor, but also had other functionality.

Don't you think it's going to be suspicious when you see a capacitor with 6 pins? Don't you think anyone that inspects the motherboard is going to wonder why a capacitor has 4 additional lines going to a critical flash chip? Seriously.... The entire article here is around replacing entire components. Adding new traces to a finished PCB is impossible without using wire that will be suspicious to anyone doing a visual inspection. Most sabotage will probably just program the flash chips with malicious software without changing the hardware or swap big components that use standardised footprints like QFN with a nearly identical part. Adding a small capacitor sized micro controller where it doesn't belong and on top of that connect it to existing chips in comparison is extremely hard. At that point you might as well design the backdoor into the PCB itself and embed it between the individual layers of the PCB.


It would not have to have more than two leads, depending on its use in the circuit. It was an example meant to illustrate how the dramatically different levels of miniaturization can make it hard to reason about attack vectors.

Consider what a state actor could do with access to modern microprocessor level fabrication.

I'd expect that we'd see features such as the following:

- sophisticated intra-chip communication

- long periods of total dormancy of the exploits

- circuitry capable of receiving a "it's safe to begin the attack" message

- surprising communications vectors for exfiltration

- technology to make malicious parts appear under x-ray to be normal

- fallback to awaiting the message to perform DoS if more sophisticated attack vectors are not possible

I agree with your suggestion about using the existing footprint, etc. There is likely some very sophisticated tech for making malicious parts x-ray and test as normal in every respect.

The network connector exploit described in the article would be easily detectable by temperature dissipation measurements. So distributed methodologies are likely in use.

I'd also estimate that a large number of mobile devices have built-in hardware compromises that are dormant and can be used if necessary. These would be the simplest attacks to carry out and would have extremely high yield. Things like:

- phones suddenly jamming the 4G and WiFi network simultaneously

- hardware implants to help detect whether a device is being used by a high value target. Such an attack could be created using a tiny bit of silicon and would be dormant in most cases.

The biggest risk to a state actor doing these kinds of attacks is being detected, so firmware based attacks are potentially more risky than hardware attacks, since we are better at detecting a checksum mismatch than we are at testing hardware across the spectrum of possible input conditions that might trigger unusual behavior.

So I think we'd see state actors dipping their toe in slowly to these kinds of attacks, first establishing the supply chain hacks without anything malicious going on, and then gradually phasing in actual malicious hardware once the relevant parameters for the attack are better understood.


   > China's leadership is calm and not prone to knee-jerk responses
Communist China is ruled by a genocidal mafia with a well-known penchant for sudden outbursts of violence. From its bloody inception, through the Great Leap Forward, the TianAnMen Massacre, the persecution of FaLunGong followers and recently Muslims — the regime has shown it's completely incapable of serving its people. When times get tough they invariably turn to intimidation and murder.

You're not refuting anything the post you're responding says.

Murdering people can be a completely non knee-jerk response to domestic issues. They ARE getting away with it, aren't they?


That's certainly one way to look at it.

I guess I'm projecting some semblance of humanity onto them. I'd assume even the most evil would usually want to hold off murder till nothing else works. Hopefully karma is about to catch up.


POTUS has also explicitly stated China is meddling in the 2018 election, against him and Republicans, and assigned a motive.

“They do not want me or us to win because I am the first president ever to challenge China on trade,”

And converse to the calm of China, he is impulsive and has floated the idea of selective defaults on Treasury securities. And he has ample experience with this himself. It is in the realm of ridiculous conspiracy theories, but POTUS is a walking pile of ridiculous conspiracies theories, so why is defaulting on only Chinese owned securities more ridiculous than nuking Pyongyong? (Of course, only one of those can actually be contained.)


What is the likelihood that #2 is correct?

I read a lot of (recent) American history books - #2 gets my vote.


That story is a bit odd, still -- normally behind the connector there is optionally magnetics, and at least a PHY... being able to integrate the magnetics in the connector exists allright, but adding the phy /as well/ must make it a marvel of integration regular manufacturers would dream of... especially at Gb speed!

Also, you can't really 'piggyback' ethernet easily, for the same reasons; you would need TWO phy in there to decode/reencode...

Even if you'd want to 'piggyback' on the link itself, it would be very, very difficult to say the least -- Gb ethernet is definitely not a gimme to synthesise, let alone piggyback.

So, color me dubious -- the SPI 'chip' of last week was a but dubious but doable (given not just a custom chip, but a custom PCB) but this ethernet story makes even less sense!


Elsewhere in this thread there is an actual link to an actual NSA device that does exactly this. I don't think it's in the realm of science fiction.

If you look at that illustration, you see that it's not just one ethernet connector, it's one of these massive connector stack with one ethernet and 2 USB, also, it adds quite a bit of depth to the connector; it must have been made with one particular brand/type of motherboard in mind.

Still, if these are in the wild, then perhaps our chinese friends might have reduced the footprint even more to the size of one connector.

I know the connectors with integrated magnetics are quite a bit 'longer' and 'beefier' than the passive ones.


And you don't think such a device could be made quite a bit smaller today, with better manufacturing support?

I'm thinking it's entirely plausible that such devices exist, and are broadly in the wild. Mostly targeted. That said, our own govt (US) is not innocent. Neither are China, Russia and many others. It's what government espionage actors do.

I think it's a fault of many that US mfg has fallen off as much as it has, and that critical infrastructure would allow foreign mfg in general. Or at least final inspection and assembly internally. Not just the US, but most countries.


That's also at least a five year old implant since Snowden leaked it back in 2013.

The wikipedia article on the device shows a date of 2008, so yeah, it's probably pretty outdated as far as spy tech is concerned.

> Also, you can't really 'piggyback' ethernet easily, for the same reasons; you would need TWO phy in there to decode/reencode...

Pretty much every BMC in existence (well, the ones that comply with the Data Center Manageability Interface, at least) can "piggyback" on top of an onboard Ethernet interface.


This works because the main PHY chip has a side channel (i.e., extra pins) for a connection to the BMC. There are not two PHYs for a single ethernet.
More

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: