Hacker News new | past | comments | ask | show | jobs | submit login
What Businessweek got wrong about Apple (apple.com)
267 points by rakkhi on Oct 5, 2018 | hide | past | favorite | 175 comments

Both Apple and Amazon have released VERY STRONG denial statements that bring the whole Bloomberg narrative into question. It's also convenient that no one has yet been able to verify or find any of these mysterious Chinese chips on any of the Supermicro servers in the wild.

So what is the real story here? Did Bloomberg reporters deliberately deceive everyone or were they deceived by the US IC ("intelligence community") as a way to scare technology companies from doing business in China?

Someone at the SEC should scrutinize SMCI shorts at the very least.

Remember when Clapper gave the "least untruthful answer possible" about domestic bulk collection? [0]

It's naive to think Apple and Amazon couldn't lie in response to the article, if the intelligence was highly-classified. They may be under extremely strict gag orders (e.g. "give no response whatsoever, including silence, other than denial") and protected by promises of indemnity, as telcos were in the wake of the NSA disclosures.

Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.) There's two possibilities, then: someone cajoled dozens of officials across several companies to lie to Bloomberg, or the tech companies have been cajoled into lying.

Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.

0. https://m.youtube.com/watch?v=Jkb5FKlETqY

> ...every hardware pentesting shop will be going after these boards like they're looking for golden tickets.

The way the original story was written, it suggested that four subcontractors were identified, and almost 30 targets selected, with the implied suggestion that either the boards were custom special order boards, or destined for a specific lot order made by a customer. If true, then it is unlikely these boards will be found in the wild except by slipping through those almost 30 targets into the used market.

Supermicro has about 600 board SKUs, so finding these needles in the haystack is likely more feasible by approaching data centers offering to help find the boards for free, in return for allowing the pentesting firms to take physical possession of such found boards.

The story did reveal that the original "tell" that gave away the chip was odd but not obviously malicious at first glance network traffic, and that the suspected intent was implementing an Advanced Persistent Threat model. The article also mentioned the chips were connected with the BMC, but it wasn't specified in the article whether or not the chips got out onto the Net.

So finding the boards in the wild probably will focus upon finding them in the same manner, over the network. Power down the server, let the BMC stay powered on, and watch for unwarranted network activity. Or boot a Linux on a stick that deliberately does as little as possible and premises networking allows it to do just enough routing out to the Net to capture traces of what unauthorized network traffic is trying to do, and watch for unwarranted network activity, in case the chip design is clever, and hides its activity until it detects the mainboard is already running before trying to inject its network payloads onto the mainboard's network interfaces.

What are others' thoughts on how to find these "golden ticket" boards?

That's the interesting bit. Having done so much to hide the exploit your most important aim is to hide it's presence. Anything that would just "connect to a Chinese server" would be discovered immediately. If the Bloomberg story is true, the network traffic scheme must have been extremely sophisticated to fool so many network security specialists at top companies for such a long time.

This, or the story is false, purposely or not.

I'd appreciate a detailed explanation of the steps needed to get from putting this chip on a motherboard to actionable intelligence useful to a hostile nation state.

If we're talking about being able to make changes at the OS level, surely those should be relatively easy to spot?

If we're talking about copying packets, wouldn't that be useless under most circumstances?

If we're talking about doing whatever on a mass scale, surely most of the data would be junk and a huge and very clever backend would be needed to sift out the useful elements?

Presumably none of this is impossible, but I haven't yet seen a good description of how it's all supposed to fit together.

Something like triggering a memory-read of a specific address the moment you see a TLS diffie helman exchange on the wire? The idea being that you can recover the session key, and thus break the encryption retroactively.

That requires more than just access to the network, but it is rather simple.

With just access to the network, perhaps one could do spoofed DNS responses that are never seen on the network? A very simple '1% of the time, set the gmail.com A record to the chinese gmail IP' might be enough.

One could create mayhem by sending false ARP or DHCP responses, but that is only mayhem. Perhaps if it is externally triggerable it is useful offensively as DoS.

Most places I know of isolate their OOB management network, requiring a vpn or jumpbox to access it. However, if someone did let their OOB network full outbound access, I could see this slipping through. I could imagine that simply going to a CDN or cloud provider like AWS/Cloudfront/cloudflare/akamai with a dns lookup along the lines of updates.supermicro.cdn-front.com wouldn't be too suspicious. At that point, you'd be looking for dns lookups and not firewall hits.

If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials.

But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.

> However, if someone did let their OOB network full outbound access, I could see this slipping through.

Quite, especially in small networks and inline ilos, entirely possible that people plug the ilo (ipmi etc) to a more open network. Sure, nothing in, but no block on stuff going out.

Sounds like many things would have to go right in the defender's court. Optimism is not a good defense strategy :)

Heh. One colleague has "Hope is not a valid deployment strategy" as a signature.

Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls.

However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.

That's my point. Security people were analyzing network traffic for decades trying to spot something that doesn't fit, host-wise, pattern-wise or even packet-wise (see The Museum of Broken Packets[0], for example). And someone managed to somehow hide all this traffic from security experts working for Amazon and Apple, for months or years? I'm very curious to see how.

[0] http://lcamtuf.coredump.cx/mobp/

I think you misread - "Unicorn jumping over a rainbow level rare" was about catching it, not missing it.

That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Or are these boards commercial-of-the-shelf things that are mass produced?

In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.

> That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).

I did read it, so if it was mentioned I missed the part where Bloomberg stated if it was targeted at specific orders and thus customers or if it was seeded into the supply chain to see what sticks.

So, if you know the answer, why not just providing the information? ;-)

I think the Command and Control traffic will be much harder to spot. In fact it might be dormant until an operative is on the same network and wakes it up.

This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately, as Apple and Amazon at this point would have to be for their denials to be false.

If Apple is lying here, they're lying because they want to, not because something is forcing them to. I believe them.

"Did you receive an NSL?"

You have been compel to lie, by the government, to answer "NO" if you've received one.

If you can be compelled to lie that you have received an NSL then I do not see why you can't be forced to lie more.

We already compelled speech, why not more?

I thought that you could simply not answer the NSL question (hence warrant canaries)

Iirc that's only in theory. I don't think there has been any precedent set. But I always see people say that a warrant canary is a false sense of security.

> This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately

Is there case-law to support this? I've heard the argument often, but I don't believe it has yet been tested.

It’s a well established doctrine that this is a direct consequence of First Amedment: https://mtsu.edu/first-amendment/article/933/compelled-speec...

I think there is no way they would claim they are not under any gag order if they were forced to never confirm such an order, if there also is the option of not addressing it at all.

  I think there is no way they would claim they are not
  under any gag order if they were forced to never confirm
  such an order
I've always thought such denials are pointless.

You think everyone at Google knows every secret gag order they're under? Of course not, that would get leaked within minutes.

If I was the NSA I'd find a "patriotic" mid-level or junior employee with the power I wanted to subvert, and give them a gagging order that stopped them telling their boss.

That way the Google PR can honestly say that the CEO and legal team haven't seen or heard of a gagging order.

I think it would be odd then if no one else in the company is aware of the issue that those mid/junior level employees would then leak the information to Bloomberg during an active investigation.

> Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.

From the story, this seems to have happened several years ago.

I wouldn't be surprised if house cleaning had been done for some time.

If the story is true then I doubt that the Chinese would have flooded the market with these boards. It must have been relatively targeted.

One thing that sticks out to me is that they are fairly persistent on using "Apple" rather than something like "Apple and its security partners and subcontractors", which would eliminate most of the potential for doubt.

> Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.)

They are very specific, yes. They show pictures of the alleged chips that are allegedly disguised, yes. They describe placement, yes.

However, photographic or video evidence of these chips on SuperMicro boards is conspicuously absent. The only visual of placement is an illustration, not a photo. I’m not suggesting Bloomberg is wrong. I find this story fascinating and incredible (if true). But I noticed there were no photos/video of the chips on boards.

The last paragraph of the press release on this hacker news post:

"Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations."

So you're basically implying that Apple is colluding with the US government to actively lie to discredit this story.

The funny part is that, based on your tone, you can't imagine a case where: 1) that's possible and 2) Apple may not be a willing colluding party.

I can't imagine a case where it's within the US's power to force Apple to falsely deny a story for them, but not within their power to feed a fake story to Bloomberg.

The proof of the pudding is in the eating. That makes sense. I ate the pudding it tasted good it proved to me its deliciousness. There is no proof in a pudding that is not eaten.

You clearly not an Agatha Christie Fan

The reason Bloomberg is so sure about this is because chips/'infected' Supermicro boards were originally found at Bloomberg. They noticed odd web traffic coming from a server, took a look, found nothing, looked closer, and finally found a hardware exploit.

What you're seeing in the Bloomberg piece is a bunch of half-truths backed by soild data. It is a BMC exploit, and they are doing it through the BMC EPROM, and even the position of the 'exploit' in the article's graphics is accurate. This is obfuscated because of an ongoing investigation. It's real, but there's purposeful misinformation, combined with a journalistic game of telephone or chinese whispers.

It's also about the worst possible way to drop and 0-day, but whatever.

Why would Apple and Amazon release such vehement denials, though? If it were an ongoing investigation, wouldn't they use more hedging and obfuscatory language? They could just as easily say "we're not aware of anything like this, but we take all allegations of this nature seriously and are looking into it". The whole situation is just odd.

Their reputations could take a huge hit from this story alone, so a strong denial is the first defense. If they are unaware of any investigation, as they claim, then that makes even more sense.

The part I don't understand is why Apple would take such a huge PR hit from the story. I way I read it is that they found out about the problem, diagnosed it and replaced all servers all within a very short time. A careful writeup about what happened, what kind of information that might have leaked, and what they do to avoid it in the future would to me be a net PR win at this point. The next question would be how all other customers has acted.

If Apple were to agree with and state that China is spying on them, then they might face repercussions from the Chinese government on their extensive business interests in China. They make all the stuff there. This is monetary threat to Apple, so lying is within character for a corporation to preserve their business.

If the story were true Apple cannot lie about it publicly like this; the SEC would have their heads.

They would have to hedge, "we are investigating Bloomberg's allegations and take security seriously."

I wonder why people on HN take these arguments seriously.

You either anger govt that is authoritarian & capable of confiscating your entire production (basically ending your current business)


You make some half-truth statement (not necessary to be lying, but that does not change the point), and - if everything will be revealed - one agency of democratic govt will be pissed and slap you with big fine.

Basically this is big enough to make SEC meaningless.

The idea that the Chinese gov't would seize Apple's "entire production" is up there with the possibility that the US invades China. It isn't going to happen. There are just too many consequences, not least of which is the withdrawal of most US companies from China.

The SEC on the other hand literally just got done fining Elon Musk $20m for a tweet. Fines for something like this could be in the hundreds of millions, that's not something you risk when you could just say "we are investigating the claims and take them seriously."

SEC fined him over something so much easier to verify that I wonder why is everybody bringing it up? Why do you think truthiness of Apple'a / Amazon's statements will be obvious in next year or two? Why do you think that accussation like "China is hacking our HW" from Apple would not escalate situation dramatically? (implicating they have to deny as hard as possible)

It's such obvious damage control in shit-has-publicly-hit-the-fan situation ... Why do you think SEC's fines (lol $20M, even for Tesla/Musk it's not a big deal, Apple would have to be hit with 100times that to even notice it) are even relevant now?

And "It isn't going to happen." could have been said about hardware attacks on USA from China last week. Situation has changed quite a bit, China is either starting to abuse its hw monopoly or someone is doing A LOT to prevent such scenario, so the stakes are too high for simple SEC fines to be worth discussing.

I certainly understand why they wouldn't want people to know about it. But lying outright in a public statement in a way that is direct and clear could expose them to serious legal risk. Trading PR risk for legal risk seems like a dangerous move that companies this large wouldn't take lightly.

Lying publicly and on the record about something that should have been disclosed in their financials as a risk is not something the SEC would take kindly to.

Publicly traded companies do not have the same freedom to lie because the truth or falsity of this claim has a direct impact on their stock price. Something we just saw with Musk less than a week ago.

Obviously to quiet their contractor and client fears. Shareholders know that replacing the hardware would be a major cost and PR damage intense.

It's still an odd strategic move though. They could get caught in the lie, in which case they would both be exposed to pretty serious legal and financial trouble.

Replacing the hardware would be cheap compared to the hit they could take getting caught in such an outright lie. Public companies have obligations to their shareholders not to lie about things like this. They're opening themselves up to lawsuits.

For me, the Apple piece is not as black and white.

> "Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple."

They deny Bloomberg's specifics, but basically admit that the general attack vector very much works.

That quote seems to be describing a software attack, not a hardware attack.

> "chinese whispers"

Does anyone have a better phrase for this?

A Chinese workmate called me out for using that phrase in his presence. Until then I had used the phrase since childhood without thinking that there were connotations and with nobody complaining. However, after that one fateful conversation I did see that my language could be improved. But how?

Does anyone have a concise alternative that conveys the same thing without implying anything about how Chinese people communicate in whispers, to get things wrong?

I had never even heard the phrase until this thread. A quick google/wikipedia shows me it's the British term for what we in the US call "telephone". I guess I learned something new today.


> As the game is popular among children worldwide, it is also known under various other names depending on locality, such as Russian scandal,[3] whisper down the lane, broken telephone, operator, grapevine, gossip, don't drink the milk, secret message, the messenger game, and pass the message among others.[1] In France, it is called téléphone arabe (Arabic telephone) or téléphone sans fil (wireless telephone).[4][better source needed] In Malaysia, this game is commonly referred to as telefon rosak, which translates to broken telephone. In the United States, the game is known under the name telephone – which in this use is never shortened to the colloquial and more common word phone.

Not a native speaker: Does it have additional meaning to the reference to the children's game where everyone whispers to their neighbor, passing a message along? If not, isn't that also known as the "telephone game"?

I was going to reply "I don't know, it's all Greek to me" but then decided that was not a good idea. So, [0] says

> Chinese whispers is the British term for what is known as the telephone game in the United States

and then lists several other names for the same game. I've also met people who when discussing confusing messages say "send three and fourpence, we're going to a dance", which is a result of 'Chinese whispers' being applied to the input "send reinforcements, we're going to advance". The reference to the pre-decimal coinage ('three and fourpence', i.e. three shillings and four pence, approx £0.17) in this old phrase shows how long the concept has been around in British English (we decimalised in 1971).

[0] https://en.wikipedia.org/wiki/Chinese_whispers

In France it is the "Arab telephone" !

"Pass the message" or just "whispering game" seems to be the same.

The Telephone Game is not common parlance in British English, just American English. So a better phrase is sought than that to convey the idea.

I suspect there is no common phrase. Chinese whispers isn't common parlance in american english.

This also illustrates why hardware implants do not work as a mass infiltration tool. The idea is not getting caught, otherwise risking the whole operation. Implementing a threat like this is counterproductive.

Intel ME seems pretty successful though.

This amazes me. Everyone freaks about the Chinese doing it, whereas we're surrounded by millions of computers doing the same thing - or at least able to - without us even knowing or being able to block it.

I suppose a big difference is that Intel ME is on the spec sheet, is known about, and is sold as a feature for sysadmins. It's not cool that it's not optional, but it can be switched off.

To be clear, there is a very real threat of China embedding chips that do "bad things" inside chips that are sold to the US (source: I once worked on a government contract and saw such chips - they were, to my knowledge, unrelated to this specific case). I'm not sure where that points, but there are strong motivations at play.

The key is ,,inside chips''. There's no reason to do those bad things on the die using an extra chip, as it's too easy to prove.

Not the least because since Snowden we know the NSA does it.

It's certainly beginning to look like Bloomberg got played. I don't know what the motivation would be for Bloomberg to deliberately deceive readers, but I can't help notice that the article came out at roughly the same time as Mike Pence was giving a speech whose central premise was that China is meddling in american politics, and all the attention currently focused on Russian hacking should be focused on Chinese hacking.

If this story actually is false the US government has plenty of incentive to have planted it.

Did you miss the other hacking news from yesterday?


But go read the Chinese response. It certainly looks like a non-denial to me. Pretty much says, "yeah, it was us, but US is doing the same to us".

The full statement can be found at the link below, but the take-away is pretty much this quote from the statement: "Supply chain safety in cyberspace is an issue of common concern, and China is also a victim."


My general response whenever there is something like this: Look at the past stories by the reporter. Bloomberg handily lists them for you. https://www.bloomberg.com/authors/AQrv1y2ieI0/jordan-roberts...

Take a look at those stories in the strong light of being able to see them in retrospect and decide for yourself if this reporter is prone to going to press without a full understanding of the situation.

I have my opinion, but it's best for people to form their own.

The headlines are somewhat over-the-top, but none of those articles show examples of them deliberately making up a story.

I wouldn't read it, because they seem to repeatedly overstate the possible impact of their stories. But if you work in the space, or are the victim of an attack, this thread-level-orange style of writing may feel entirely appropriate.

It's no different than, say, insinuating that something is wrong with <x>. But instead of clearly explaining the criticism, to "let smart people figure it out for themselves". That's the laziest conspiracy sales tactic, and it exploits peoples' insecurities by tying the carrot ("I'm one of the smart ones!") to the stick of accepting whatever theory is being peddled.

hmmm... definitely a trend of reporting on hacks. And perhaps with sensationalist clickbait headlines.

But they are confirmed hacks (Equifax, FB, Cambridge Analytica).

In this case, there's relatively little ground for misunderstanding or exaggeration. There's either a malicious chip or there's not. I suppose in the "worst case" scenario, there could be a malicious chip, but some other government than the Chinese put it there.

However, reading the statement from the Chinese government in response to this story, which never actually denies the incident and instead complains about being victim to such hardware hacking themselves, that's not the impression I get: https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

Another option is the attacker has those companies "by the balls". They knew that the supply chain was completely compromized, and has been for a while. Admitting this, after so much time, is financial suicide in the best case. Maybe the companies are even being blackmailed by the attackers.

Another twist, maybe someone in the government does not want this to come out officially, because they would be forced to take action against China.

On the other hand, maybe this is exactly what the government wants, so they "leak" this information - be it true or false - on purpose.

In the end, the world is so messed up currently that every conspiracy theory seems to be equally plausible.

Puts these somewhat irrational Trump claims from July into context? *This is speculation/conspiracy, but if they've been looking into this for years, I could see the Trump admin using this as leverage to bring Apple manufacturing and/or offshore cash back to the US in exchange for covering up this alleged national security issue.


I'm not convinced which story is strictly the correct one yet but I don't think the strength of the denial is an argument for either. It just shows that it's important that your narrative wins, truth or not.

The burden of proof should be on the side pushing the accusations.

In that sense, Apple’s denial puts it squarely back to Bloomberg to prove it’s not bulshit. Until then it would be fair to discard their narrative.

"before servers are put into production at Apple they are inspected for security vulnerabilities"

A chip smaller than a grain of rice nested between motherboard layers.

What tools could detect such a chip?

While I don't doubt Apple inspects, the zero-day threat always exists.

Xray inspection by a human specialist

Further in the article, Bloomberg mentions a chip, smaller than a pencil tip, sandwiched inside the PCB itself, under other traces. That seems bananas difficult to pull off, but could x-ray inspection find such a thing if it existed?

Yes, because the incident they reference was a mid-PCB-layer chip discovered by govt agencies using X-rays. The findings were presented at a private conference in Virginia. Sources who attended said "they've seen the xray pictures" etc.

I specifically mention non-automatic inspection as automatic one may well not have the "paranoid mode," unlike a human who specifically told to go over every individual square millimetre.

Minimum quantity orders for these servers are typically 10,000.

Even at 1,000, I don't see how humans could reliably detect this.

It is very odd. If there was any truth to the story, I wouldn't expect such strong denial from apple and amazon.

There has been a noticable uptick in anti-chinese news lately, particularly on business oriented news ( cnbc, bloomberg, fox business, et al ). Obviously some elements of the government or power structure has instructed them to do so. But why?

Also, if the IC wanted to put pressure on tech companies, wouldn't they just contact apple and amazon directly. Maybe there is a rift within the IC with one group being pro-china and another being anti-china?

Or is the IC and news companies done targeting social media tech companies and switching their attention to regular tech companies? Just have to wait and see.

Straight up some hardware/embedded nerd that posts on here would have one of these chips in the wild and already have posted a hardware break down blog post under a microscope of one.

They kinda seem like a Tom Clancy fantasy. But I dunno.

Why do you so blindly believe amzn/aapl over bloomberg? You do realize that reputation is a huge part of success, particularly in the security and hardware space. Of course amazon and apple will deny this story. For starters, they are probably under a gag order from the government to not leak anything regarding their findings since the investigation is still ongoing. Secondly, it is of no benefit to admit they failed in oversite on their side with this one.

> Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

That to me is the icing on the cake. Companies are usually not direct like that.

(Unless this is some kind of new NDA that allows/mandates itself to be denied)

In an appearance this morning on Bloomberg Television, reporter Jordan Robertson made further claims about the supposed discovery of malicious chips, saying, “In Apple’s case, our understanding is it was a random spot check of some problematic servers that led to this detection.” As we have previously informed Bloomberg, this is completely untrue. Apple has never found malicious chips in our servers. Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

It can’t get much clearer than that. The whole story is quite weird. I don’t know who should be believed but I’ve never read any such vehement denial. If Apple is lying they risk quite a bit of credibility here

Free get out of jail card: Three letter agency force me.

From 2016: Report: Apple designing its own servers to avoid snooping Apple suspects that servers are intercepted and modified during shipping.

"Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips." https://arstechnica.com/information-technology/2016/03/repor...

Eh? Instead of verifying an existing design they'll just make one themselves? That could be compromised too?

Because using a 3rd party design you have to ask or guess what each chip does, and they might not want to tell you everything for IP reasons. When you create your own board, you know what every little chip and trace does exactly, and it's a lot easier to detect a rogue chip.

The Norwegian National Security Authority (https://nsm.stat.no/english/) is quoted in a norwegian paper today saying they knew about problems with Super Micro since at least june. https://www.vg.no/nyheter/i/xRkLep/storavis-hevder-kina-inst...

What exactly did they know? Did they check anything themselves or did they hear the same story from the US IC that Bloomberg also heard?

Supermicro servers are extremely popular in data centers. Yet no one has noticed anything and no one has found any malicious chips in them. Unless the Chinese secret services also hacked all of the firewalls, someone would have picked up some outgoing packets that are going to Chinese C&C servers. And those would have been scrutinized and chased down. A company I interned at this past summer was scrutinizing every client on their internal network. They would have easily detected a machine that was connecting to an IP outside the subnet, for example.

This whole story is just fishy and every company mentioned in that Bloomberg piece is not only denying it, but strongly denying it ever happened. And Bloomberg reporters have zero evidence... except for some stories from "anonymous sources".

My translation of the part pertaining to the national security authorities in Norway. Direct quotes from the authorities are shown as such:

--- NSM is aware of the issues with Supermicro.

"- We know about this, but can neither deny nor confirm it is correct. We register that this is being denied by the companies", says Monica Strom Arnoy, communication director in NSM to VG.

NSM has, however, been aware that Supermicro may have been compromised, long before Bloomberg's article.

"- We have known about this since June", says Strom Arnoy, who does not wish to further explain from where they have this information. ---

My reading is that they suspect Supermicro. Further, that they are familiar with the claims about Apple and Amazon but won't confirm or deny whether they are correct. Damming for Supermicro, less clear for Apple and Amazon

The article suggested the chips were hidden under components. You would have to tear apart the motherboard and know what to expect exactly. The number of companies who do this kind of audit must be in the single digit if any.

Now the article may or may not be a hoax, but a few years ago the NSA was exposed doing exactly that. So at the very least it is credible.

As for detection, I’d rather expect this devices to be activated on demand, like creating a backdoor, rather than sending streams of data from all servers, most of which (per the article) will be owned by an adult website or a mormon church which aren’t exactly strategic activities.

well, the article said that amazon noticed pings to c&c servers in its beijing datacentre. it did also say that these were more sophisticated hardware attacks but didn’t make it clear if these chips were also found

Yes, sorry, agree. On stealthness, the article mentioned that “In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached” but also says that the chips came in different sizes so some could have been easier to detect.

If it's real, it could be part of a bigger persistent threat. Maybe those chips would react to some signals, start to heat and fry the motherboard.

In case of military war, it could DoS many servers at once from the inside.

It could also be used to prematurely break the motherboards, and sell new ones.

There could just be there to ping C&C upon activation in order to locate datacenters, including military ones maybe (information that could be useful in that case of war too).

We can think of a few use cases than just spying on network packets.

That's not how journalism works, "Anonymous sources" are not anonymous to the journalists and editor of Bloomberg. Evidence can be vetted but not published.

This is completely anecdotal, but I'm on several mailing list for bulk buyers of off-lease server equipment. Typically these list are 90% Dell, HP, IBM. About 6 months ago I started to notice a huge percentage of SuperMicros being sold. Sometimes the whole list was nothing but SM servers.

Obviously could be many factors, but does strike me as odd.

Any of those public or easy to get in to?

Would be nice to quantify this.

What they said in the interview was that they "knew" about the story, as in "We've heard it", but they also say that they can neither deny nor confirm it. So that's not much.

It is an interessting story indeed, espiacially with third country sources more or less confirming the general story but not the specific details. I personally have a hard time to believe Bloomberg went public with it without a thourough due dilligence. They should be totally aware of the consequences if they did.

At least for Amazon, having jepardiced servers, this goes strait to the core of their business namely customer trust. Until now Amazon did not have any data breaches of customer data which they did pride themselves a lot for. If the Bloomberg story is true, this ould be at risk. So, yeah, a strong denial is exactly what one could expect. I'm curious to see that story unfold.

I mean what was the alternative? To admit that your supply chain is compromised and blame directly the government of the country where you produce (and sell to a level) all of your hardware?

That would be a huge blow in the credibility of the company and would raise serious questions on why they did not move the manufacturing elsewhere.

If this were true I would have expected some corporate waffle like "Apple takes supply chain security very seriously and regularly audits suppliers. We cannot comment on internal security matters but customers should be assured that blah blah blah."

I wouldn't have expected this extremely strong denial which is one step short of outright calling Bloomberg liars.

No offense, but I think that your suggested statement would only lead to serious escalation for media enquiries, especially since this story came 2 days after the interview of the Apple CEO [1], where he underlined that privacy (and the relevant security measures) constitutes the core value of the company.

[1] https://www.youtube.com/watch?v=VD1cP8SK3Q0

If Apple or Amazon are found to have lied in their statements I think they could be in serious legal trouble with both their customers but also (sadly, more importantly) their stock holders.

More trouble than if they pissed China off enough to threaten Apple's manufacturing in China?

It's hard to say what the truth is here, but what I will say is if that Bloomberg reporter doesn't have substantial evidence to prove that claim he could be in serious trouble. SuperMicro's stock was down 50% straight after that articles release, and it's not looking so hot right now either. He could be looking down the barrel of an SEC investigation very soon.

> "As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections."

They do not have equipment to detect this kind of attack, period. It's not viable for each device, and it's not even viable for sampling a subset of devices from a given production batch. Some components are physically inaccessible and would require desoldering of other components to even access them in any way.

These kinds of attacks cannot be generically detected in any economically feasible way; it must be prevented by drastically clamping down the supply chain and the logistics chain.

Regardless of whether this particular case is true or not, given the crucial role of computer systems in so many key institutions, it seems to me extremely risky to trust Chinese suppliers not to try to compromise critical infrastructure.

Then again, I understand that it could be argued that, if this is confirmed, to me it would seem quite rash from the Chinese, given that they would have known all along that such a scheme would be discovered sooner or later. It is one thing to plant a device as part of a spy operation, quite another to consistently compromise a whole supply chain.

Whichever is the case, the national interest and commercial interests seem to be seriously incompatible with one another when it comes to outsourcing such critical infrastructure to China, this seems obvious to me, regardless of the China policy of who is in government in US.

> Despite numerous discussions across multiple teams and organisations, no one at Apple has ever heard of this investigation.

If this is some kind of ongoing national security issue with nondisclosure requirement authorized by the Director of the FBI, like this big breach could be, people involved are not allowed to talk about it even inside their company.

Of course it would be advisable to inform higher ups in the Apple so that they would not issue a denial.

Stupid legal question, could this end up becoming a defamation lawsuit?

Similar question, but should BW turn out to be correct and Apple was for lack of a better word, lying, aren't they on the hook as a public company?

Matt Levine likes to say that "everything is securities fraud".

Funnily enough, he actually covered this angle yesterday!

Regular readers know that a major theme of this newsletter is Everything Is Securities Fraud, so in that vein, let us consider a hypothetical. What if:

1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors.

2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it.

3. The companies are patriotically but falsely denying the hack.

If that is true—and I have no particular reason to think it is, it’s just the sort of hypothetical we like around here—then, obvious question: Is it securities fraud? (Assuming that the hack is potentially material to the companies’ business?) I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.


Pushing the conspiracy theory just for the sake of argument. You could imagine that some employees have been in contact with the NSA about these chips and been told that they cannot disclose anything to their employers. The management of Apple would deny the claim in good faith. It would be hard to make a claim that Apple meant to mislead investors.

It will definetly be bad for the business of these companies, especially for Amazon, as it undermines customer trust. In the case of Amazon also the most profitable division, AWS, which provided the majority of profits in 2017.

Apple would need to show that they had lost business as a result of the false story and hence claim damages. Businessweek would defend themselves based on it being in the public interest and they cannot reveal their confidential sources.

Probably hard for Apple to prove they have lost money because of the story and it would need to be enough to warrant the time and hassle of a court case. Not to mention they would probably rather avoid having the issue mentioned even more often by dragging it through court for a year or two.

Think you meant to reply to jdorfman (parent poster to the post you actually replied to).

Not if the US government has instructed or allowed Apple/Amazon/etc to do so on national security grounds.

Not a stupid question at all. Assuming the story is false (which I don't assume, and actually think it's probably correct) it would likely depend on the care and process that went into the reporter. Were the editors and reporters reckless and ignore what Apple and Amazon told them? For a cover story of this magnitude, almost certainly not. But it also wouldn't be out of the question for Apple and Amazon to sue if the story was truly completely off.

When I've been sued or threatened with defamation lawsuits, the first thing we go over is how we're going to show we did due diligence, how we came to the conclusions, how we reached out for comment and incorporated that comment into the process.

If Apple and Amazon can figure out the sources, there's also potential for them to sue the background sources, though that's usually tough and messy (because then you have to prove they not only defamed you, but also that they were the ones who were the original source so doesn't happen much).

Generally, for defamation to be actionable (that is, viable for a lawsuit) there needs to be either an intent to defame or a neglect of facts so severe that it is tantamount to intent.

I would say one specific detail (I haven't looked at it though) would challenge the truth of the rebuttal of both Amazon and Apple is that if it is confirmed that both have severed ties with Supermicro around the same time, the coincidence would really seem odd then.

DoD contracts for the military require the hardware to be sourced and made in the US to prevent compromise. I wonder if one day we will see the DoD require any Cloud contractor that has DoD datacenters to source from the US or NAFTA countries...and what impact that would have. I've heard ramblings about a lot of companies moving their manufacturing and sourcing from China to Vietnam already.

Companies don’t give vehement denials like this unless they’re telling the truth. People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers (I’m not saying they wouldn’t lie, just that they wouldn’t sign anything that would force them to do so).

That makes no sense. What do you think is the difference in "denial levels"? Kind of like Dragonball-Z power levels? Does a "vehement denial" cost the one making it any more than a "meek denial"? If making "vehement denials", coming at exactly the same cost as less strong denials, are more effective, you would just have made all PR companies/people extremely happy - they get a stringer weapon for free. All the have to do is issue "vehement denials" instead of just "denials" and a larger share of the population believes them (for no valid reason). Especially when the public has no way to get the "real truth" but can only watch a Dragonball-Z style "strong statement" vs "incredibly strong statement" showdown.

> People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers

Why do you think a "gag order" is something that has to be signed off on by the one getting it? That too does not make any sense. An order is issued by someone who has more power, here, the government. You don't have to sign anything for the order to exist. "Gag order" has "order" right in the name.

Maybe vehemence isn't the right measure. It would be better to say specificity. "The allegations are false" leaves more wiggle room then to say "we don't have a business relationship with Suoermicro and a Company called Apple doesn't even exist".

I saw figures that Bloomberg had 17 sources for this story. For that to in nay way be realistic, this must be a deliberate leak by the US government. Why would they put such a strong gag order on Apple/Amazon and then leak the information themselves? It makes no sense.

Then it would make no sense for it to not be true either, unless one assumes the government to spread lies to damage those companies, which also makes no sense.

Very strong denial. Frankly, if true and Apple is saying this kind of a "no" shareholders will sue.

Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret?

Bloomberg was a victim of a hoax, some nation state (huh huhm!) wants to target China for something so they need a story.

Based on what I've read here these past days, I'm leaning towards the second one. Apple can hire the best or all Infosec companies in the world if security was compromised. In other words, they'd know by now, even if they missed it originally. Cat and mouse and all...

Maybe someone is trying to trash Super Micro’s stock? Seems like this would be a good way to do it.

Though simplest explanation would be the journalist is confused and the sources inside the company aren’t very good.

SEC (and anyone else) can see the short positions, dumb move.

Plus, Bloomberg journos weren't born yesterday. This story must've been vetted to the highest levels...looks like they've been working for a year or so on it. It's not like they heard it at Starbucks while waiting in line. They are sources and sources.

Now a nation state can provide multiple sources from different alphabet agencies to Bloomberg.

I'm also leaning towards Bloomberg being wrong, but I can't help but feel that's partly wishful thinking.

Let's say Bloomberg was correct. Considering the tension between 'the West' and China, and considering how much it's been escalated recently, the consequences of all this are unpredictably catastrophic.

I don't find it difficult to believe that even just to maintain stability for now, various governments would instruct their corporations to lie in a way they haven't done previously.

It's important to keep in mind that for the first time in a while, there's serious tension between China and the West. On top of that, the idea of having China manufacture crucial infrastructure has probably been a concern for quite a while. Making assumptions based on recent history probably doesn't serve us well when predicting our current reality or the future.

So while I'm still inclined to think Bloomberg got things (mostly) wrong, many of my reasons for believing this are based on a 'status quo' that I don't think is the case currently.

Nobody sued over numerous public companies claiming "we've never heard of PRISM", despite it turning out to be a lie. You can't get sued for complying with a government order...

> Two possibilities: Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret?

I am sure the CEO must know both the hands, if the hands don't know each other. Also that Apple is going to release a strong denial must also have been known to Tim. I don't think that's the case here. Your second possibility looks more plausible.

If you talk to anyone with a TS/SCI clearance, you'll know that it is very common to be unable to discuss or divulge what you're working on to your superiors.

There are MANY material things that CEOs do not know about TS/SCI information.

But that's when your company has a contract with the government, right? Why would there be anyone on Apple's side with TS/SCI clearance? Didn't Bloomberg say that Apple found the chips? How would you be able to replace + change suppliers of 7000 servers without explaining why to senior leadership?

Can they say that "our servers /info" has been compromised? I mean, what's the point,Apple is not FBI /NSA owned. They have the right to know material things, even if specifics might be lacking.

Yes, looks like Apple dropped SuperMicro as their supplier in 2017. Why is that $64k question.

Assuming the story is true then acknowledging it would be more damaging that denying it (and they may have been directed to deny).

Remember also all the denials regarding PRISM.

If this is being correctly handled by the intelligence services (and you can have your doubts on that following these leaks) then nothing further will come out anyway.

For what it's worth, the Register mentions that Apple suddenly dropped Supermicro as a supplier.

The bottom line is that those denials don't really contribute much information.

Bloomberg needs to make a statement about all of this, either doubling down or issuing an apology. Either ways, we need a follow up and conclusion. Can we HN-ers tweet-request them (politely) to follow up?

This article sounded a bit weird to me from the technical level, but I just assumed it could be lack of clear understanding on the nitty-gritty from the journalist, or just me not knowing about hardware enough to know what's possible and how.

Given this is all getting a little fishy I'll share what had me thinking:

1. The article mentions "they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet..."

Servers tend to run on VPNs. This being a dormant backdoor is believable, but then the article mentions:

> "American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected."

Which makes me believe the devices were active and somehow circumvented corporate VPNs. I'm unsure how undetectable this could be using the system's network stack (or if it would be possible at all)-- would the claim then be that this tiny device shipped with a whole TCP/IP layer and some sort of very powerful wireless capability?

2. It continues with: "and preparing the device’s operating system to accept this new code"

Is this possible? Where would a device like this need to be wired to be able to write to memory with some arbitrary payload to do this? From the pictures it looks like it has 6 pins maximum-- could this do? If so, wouldn't this mean this device would need to do some next-level signal processing that would probably require advanced computation? Could said computation be done by a processing unit that fits the size of this chip?

Moreover, assuming it takes control of the OS independently would imply there's some decent amounts of memory in here, to hold the payload, etc. no? But if it's just a backdoor that doesn't take control of the OS, then how is it communicating over the internet with other machines like the article claims?

Again, I might be wrong and things that I don't think possible might. I'm mostly just curious to know if my intuition is too naive. Please comment below if you know more about these things than I do.

EDIT: I was really disappointed that the article itself didn't go into these technicalities, because IMO this would be an impressive feat and newsworthy by itself. The lack of alternative coverage in sources more close to technical expertise was weird to me.

About halfway down, the Bloomberg article mentions a BMC, a baseboard management controller. This is an existing part of mainboard design that has been around for just under a couple of decades. Read up about IPMI and BMCs, what they do and what their capabilities are. Then consider the threat, explained elsewhere on Hacker News several times, of simply supplying an extra ROM chip containing different firmware for that processor to run.

Focussing on the technical is focussing on the wrong thing. Bloomberg's point was not that BMCs can do this, nor that they can be made to do this. People have discussed these on-board systems and their problematic natures rather widely. It was that the supply chain is vulnerable, and that (on the assumption that Bloomberg is right) this problem with the supply chain is no longer a hypothetical case of what an attacker government could do, but is now a documented case of what one government has done, a few years ago.

If this is actually true (and I have doubts), the reporting will explode in the next few days/weeks with all the detail you could wish for.

A few pins can be enough to write to memory, if there's an interface for it (SPI, I2C, but I'd be surprised to see those unprotectedly lying around on a server board ...). It's be very slow, but I think networking could work as well. With efficient software the resources needed on the chip itself could be held very low and eventually offloaded to the main processor.

I don't think it has any wireless or DSP stuff.

I don't know enough about processors, memory, networking and low level software security to know how difficult it is to compromise a computer with such a chip on the main board, but the chip itself certainly seems feasible.

But: It must be really expensive to manufacture the compromised devices. It just seems stupid to manufacture every device with such a chip. 1/1000 seems reasonable. The larger companies have (had?) lots of them anyway.

FYI: "Britain’s national cyber security agency said on Friday it had no reason to doubt the assessments made by Apple and Amazon that refuted a Bloomberg story that their systems contained malicious computer chips inserted by Chinese intelligence. [...]"


Incredibly interesting story and discussion, this is why i come to this site.

Isn't this practice known, if not common, in the infosec/intelligence communities at the nation level? There's lots of stories of hardware exploits in copy machines, faxes, etc that took place during the Cold War.

It's a known practice, but security-sensitive organizations have supply chain management that's typically understood to prevent it from happening on a large scale.

The rice is indeed small, but it is not small on an IC chip. When people check the chip, they usually use a tool called microscope, like this https://goo.gl/1XK4YK.

And there is xray to detect what is inside a chip like this: https://www.youtube.com/watch?v=XXDsM3mUv3Y

These are quite mature and popular technique.

Rice is too big to be unnoticed....

No, you cant inspect inside chips with this Xray machine, its used for inspecting solder joints under the package.

The cynic in me wonders about the plausibility of all this.

Firstly, why would you add a new chip to a board, rather than alter an existing one? That would be essentially undetectable.

Secondly, why Bloomberg? It's an odd organisation to get a scoop on something like this.

Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret.

Finally, the timing is very suspicious - it comes with midterms approaching, and Trump and China arguing over trade tarrifs; it would serve the political narrative well for China to be painted as the 'bugbear de jour', and this also plays to the MAGA crowd.

«Firstly, why would you add a new chip to a board, rather than alter an existing one?»

Altering the flash chip would be too obvious. Looking at the flash image (dumping it) or chip (x-raying it) would be the first thing anyone would do if they suspected something fishy. Swapping a flash chip with a compromised one is a textbook 101 supply chain attack...

However a small rogue chip sitting on the SPI link (between the flash chip and the BMC) can be very sneaky: it can replace legit code with evil code ONLY when the BMC is booting up and loading code from flash. The rogue chip would not do that when the flash is read for verification (think dieselgate: a VW car disabled cheats when it detected lab testing conditions!)

Also Bloomberg talks about this rogue chip being sometimes hidden within(!) the fiberglass layer of the PCB. This is the ultimate stealthy attack. No one expects the bare PCB itself to be already compromised by a backdoor even before components are soldered on it...

Great insight. Thank you.

These are good points!

>Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret.

Well the bloomberg article said that intelligence sources knew that those boards were going to Apple and Amazon, so presumably (if the story is true) someone did speak, maybe

But why would anyone do this knowing that at some point they were practically guaranteed to get caught?

Because "getting caught"'s only consequence is that you have to use a new method. There is no punishment, no negative cost to be applied retroactively to compare it to the received benefits. The net effect still is highly positive since while it worked you got what you wanted. Same as in every spy operation, ever.

I'm not sure about that - this could really bolster the MAGA crowed, and I can imagine calls in the near future for manufacturing to be moved to the US. I doubt much will actually move, but it's a possibility.

I think respect for the feelings of the MAGA crowd is not high in the PLA's list of concerns.

The Trump doctrine is largely built on fictions anyways, and the international community seems to have settled on "distraction" as the only worthwhile strategy for dealing with him.

>Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret.

like offering cheaper, gray market source of capacitors to plant owners? that one got out what, 3 years after the fact? and only after hardware started dying en masse.

Can you delete one of your 2 nearly identical comments?

Done, and thanks for pointing this out. I wrote this on mobile, and thought the app (Materialistic) had lost what I'd written, hence I wrote it again.

Not targeting you with this, but it seems a bit harsh to get downvoted for what was obviously a mistake.

I think many people, myself included, will downvote duplicate comments without too much thought. It might seem harsh, and perhaps it shouldn't be done so thoughtlessly, but at least based on my personal experience it's not meant to be harsh.

I don't think Blomberg is politically motivated. Maybe their sources are. Bloomberg is traditionally anti-Trump.

Political mind games.

Right now, most of the tech industry, and a good portion of the news media are at odds with the executive branch of the government.

This article puts at least one popular news outlet against several tech industry giants. Divide.

What comes after divide? ...and who has the most to gain? I doubt it's actually our executive branch. I think they could be getting played just as much as Bloomberg and the Tech industry.

Consider that Apple also stated this a few years back:

>"We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order."

Their whole business is built around lying to customers.

My personal theory is this is a ploy to get people to believe Trump's anti-china narrative meant to distract from the Russia narrative.

As an outside observer of us politics it seems to me that the democrats want Russia as an external enemy whereas the republicans want China. Perhaps now it seems like the republicans have the upper hand but it is not obvious how it is going to end.

The Russia narrative makes no sense Russia and Saudi Arabia would surely have preferred Hillary who promised to outlaw fracking. Now the US is becoming the biggest Oil exporter. Russian GDP is like 40% from Oil, Saudi Arabia even more.

This will be interesting to follow, it's very unlikely there is not some truth to this. The fact that Apple and others are pushing so strongly against the story (very defensive) which makes me believe they are hiding something for sure.

They complain too much...

Apple apparently entirely dropped Supermicro as a supplier over a few weeks when they were planning a large order(source: theregister.co.uk).

The ones who should strongly deny such a story, if it is indeed incorrect, are Supermicro. Is there a statement from them?

Edit: yes, there is. They are "not aware of any investigation". That tells me all I need to know...

Yes saw that and edited.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact