So what is the real story here? Did Bloomberg reporters deliberately deceive everyone or were they deceived by the US IC ("intelligence community") as a way to scare technology companies from doing business in China?
Someone at the SEC should scrutinize SMCI shorts at the very least.
It's naive to think Apple and Amazon couldn't lie in response to the article, if the intelligence was highly-classified. They may be under extremely strict gag orders (e.g. "give no response whatsoever, including silence, other than denial") and protected by promises of indemnity, as telcos were in the wake of the NSA disclosures.
Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.) There's two possibilities, then: someone cajoled dozens of officials across several companies to lie to Bloomberg, or the tech companies have been cajoled into lying.
Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.
The way the original story was written, it suggested that four subcontractors were identified, and almost 30 targets selected, with the implied suggestion that either the boards were custom special order boards, or destined for a specific lot order made by a customer. If true, then it is unlikely these boards will be found in the wild except by slipping through those almost 30 targets into the used market.
Supermicro has about 600 board SKUs, so finding these needles in the haystack is likely more feasible by approaching data centers offering to help find the boards for free, in return for allowing the pentesting firms to take physical possession of such found boards.
The story did reveal that the original "tell" that gave away the chip was odd but not obviously malicious at first glance network traffic, and that the suspected intent was implementing an Advanced Persistent Threat model. The article also mentioned the chips were connected with the BMC, but it wasn't specified in the article whether or not the chips got out onto the Net.
So finding the boards in the wild probably will focus upon finding them in the same manner, over the network. Power down the server, let the BMC stay powered on, and watch for unwarranted network activity. Or boot a Linux on a stick that deliberately does as little as possible and premises networking allows it to do just enough routing out to the Net to capture traces of what unauthorized network traffic is trying to do, and watch for unwarranted network activity, in case the chip design is clever, and hides its activity until it detects the mainboard is already running before trying to inject its network payloads onto the mainboard's network interfaces.
What are others' thoughts on how to find these "golden ticket" boards?
This, or the story is false, purposely or not.
If we're talking about being able to make changes at the OS level, surely those should be relatively easy to spot?
If we're talking about copying packets, wouldn't that be useless under most circumstances?
If we're talking about doing whatever on a mass scale, surely most of the data would be junk and a huge and very clever backend would be needed to sift out the useful elements?
Presumably none of this is impossible, but I haven't yet seen a good description of how it's all supposed to fit together.
That requires more than just access to the network, but it is rather simple.
With just access to the network, perhaps one could do spoofed DNS responses that are never seen on the network? A very simple '1% of the time, set the gmail.com A record to the chinese gmail IP' might be enough.
One could create mayhem by sending false ARP or DHCP responses, but that is only mayhem. Perhaps if it is externally triggerable it is useful offensively as DoS.
If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials.
But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.
Quite, especially in small networks and inline ilos, entirely possible that people plug the ilo (ipmi etc) to a more open network. Sure, nothing in, but no block on stuff going out.
Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls.
However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.
Or are these boards commercial-of-the-shelf things that are mass produced?
In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.
Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).
So, if you know the answer, why not just providing the information? ;-)
If Apple is lying here, they're lying because they want to, not because something is forcing them to. I believe them.
You have been compel to lie, by the government, to answer "NO" if you've received one.
If you can be compelled to lie that you have received an NSL then I do not see why you can't be forced to lie more.
We already compelled speech, why not more?
Is there case-law to support this? I've heard the argument often, but I don't believe it has yet been tested.
I think there is no way they would claim they are not
under any gag order if they were forced to never confirm
such an order
You think everyone at Google knows every secret gag order they're under? Of course not, that would get leaked within minutes.
If I was the NSA I'd find a "patriotic" mid-level or junior employee with the power I wanted to subvert, and give them a gagging order that stopped them telling their boss.
That way the Google PR can honestly say that the CEO and legal team haven't seen or heard of a gagging order.
From the story, this seems to have happened several years ago.
I wouldn't be surprised if house cleaning had been done for some time.
If the story is true then I doubt that the Chinese would have flooded the market with these boards. It must have been relatively targeted.
They are very specific, yes. They show pictures of the alleged chips that are allegedly disguised, yes. They describe placement, yes.
However, photographic or video evidence of these chips on SuperMicro boards is conspicuously absent. The only visual of placement is an illustration, not a photo. I’m not suggesting Bloomberg is wrong. I find this story fascinating and incredible (if true). But I noticed there were no photos/video of the chips on boards.
"Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations."
So you're basically implying that Apple is colluding with the US government to actively lie to discredit this story.
What you're seeing in the Bloomberg piece is a bunch of half-truths backed by soild data. It is a BMC exploit, and they are doing it through the BMC EPROM, and even the position of the 'exploit' in the article's graphics is accurate. This is obfuscated because of an ongoing investigation. It's real, but there's purposeful misinformation, combined with a journalistic game of telephone or chinese whispers.
It's also about the worst possible way to drop and 0-day, but whatever.
They would have to hedge, "we are investigating Bloomberg's allegations and take security seriously."
You either anger govt that is authoritarian & capable of confiscating your entire production (basically ending your current business)
You make some half-truth statement (not necessary to be lying, but that does not change the point), and - if everything will be revealed - one agency of democratic govt will be pissed and slap you with big fine.
Basically this is big enough to make SEC meaningless.
The SEC on the other hand literally just got done fining Elon Musk $20m for a tweet. Fines for something like this could be in the hundreds of millions, that's not something you risk when you could just say "we are investigating the claims and take them seriously."
It's such obvious damage control in shit-has-publicly-hit-the-fan situation ... Why do you think SEC's fines (lol $20M, even for Tesla/Musk it's not a big deal, Apple would have to be hit with 100times that to even notice it) are even relevant now?
And "It isn't going to happen." could have been said about hardware attacks on USA from China last week. Situation has changed quite a bit, China is either starting to abuse its hw monopoly or someone is doing A LOT to prevent such scenario, so the stakes are too high for simple SEC fines to be worth discussing.
Publicly traded companies do not have the same freedom to lie because the truth or falsity of this claim has a direct impact on their stock price. Something we just saw with Musk less than a week ago.
> "Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple."
They deny Bloomberg's specifics, but basically admit that the general attack vector very much works.
Does anyone have a better phrase for this?
A Chinese workmate called me out for using that phrase in his presence. Until then I had used the phrase since childhood without thinking that there were connotations and with nobody complaining. However, after that one fateful conversation I did see that my language could be improved. But how?
Does anyone have a concise alternative that conveys the same thing without implying anything about how Chinese people communicate in whispers, to get things wrong?
> As the game is popular among children worldwide, it is also known under various other names depending on locality, such as Russian scandal, whisper down the lane, broken telephone, operator, grapevine, gossip, don't drink the milk, secret message, the messenger game, and pass the message among others. In France, it is called téléphone arabe (Arabic telephone) or téléphone sans fil (wireless telephone).[better source needed] In Malaysia, this game is commonly referred to as telefon rosak, which translates to broken telephone. In the United States, the game is known under the name telephone – which in this use is never shortened to the colloquial and more common word phone.
> Chinese whispers is the British term for what is known as the telephone game in the United States
and then lists several other names for the same game. I've also met people who when discussing confusing messages say "send three and fourpence, we're going to a dance", which is a result of 'Chinese whispers' being applied to the input "send reinforcements, we're going to advance". The reference to the pre-decimal coinage ('three and fourpence', i.e. three shillings and four pence, approx £0.17) in this old phrase shows how long the concept has been around in British English (we decimalised in 1971).
If this story actually is false the US government has plenty of incentive to have planted it.
The full statement can be found at the link below, but the take-away is pretty much this quote from the statement: "Supply chain safety in cyberspace is an issue of common concern, and China is also a victim."
Take a look at those stories in the strong light of being able to see them in retrospect and decide for yourself if this reporter is prone to going to press without a full understanding of the situation.
I have my opinion, but it's best for people to form their own.
I wouldn't read it, because they seem to repeatedly overstate the possible impact of their stories. But if you work in the space, or are the victim of an attack, this thread-level-orange style of writing may feel entirely appropriate.
It's no different than, say, insinuating that something is wrong with <x>. But instead of clearly explaining the criticism, to "let smart people figure it out for themselves". That's the laziest conspiracy sales tactic, and it exploits peoples' insecurities by tying the carrot ("I'm one of the smart ones!") to the stick of accepting whatever theory is being peddled.
But they are confirmed hacks (Equifax, FB, Cambridge Analytica).
However, reading the statement from the Chinese government in response to this story, which never actually denies the incident and instead complains about being victim to such hardware hacking themselves, that's not the impression I get: https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...
Another twist, maybe someone in the government does not want this to come out officially, because they would be forced to take action against China.
On the other hand, maybe this is exactly what the government wants, so they "leak" this information - be it true or false - on purpose.
In the end, the world is so messed up currently that every conspiracy theory seems to be equally plausible.
In that sense, Apple’s denial puts it squarely back to Bloomberg to prove it’s not bulshit. Until then it would be fair to discard their narrative.
A chip smaller than a grain of rice nested between motherboard layers.
What tools could detect such a chip?
While I don't doubt Apple inspects, the zero-day threat always exists.
Even at 1,000, I don't see how humans could reliably detect this.
There has been a noticable uptick in anti-chinese news lately, particularly on business oriented news ( cnbc, bloomberg, fox business, et al ). Obviously some elements of the government or power structure has instructed them to do so. But why?
Also, if the IC wanted to put pressure on tech companies, wouldn't they just contact apple and amazon directly. Maybe there is a rift within the IC with one group being pro-china and another being anti-china?
Or is the IC and news companies done targeting social media tech companies and switching their attention to regular tech companies? Just have to wait and see.
They kinda seem like a Tom Clancy fantasy. But I dunno.
That to me is the icing on the cake. Companies are usually not direct like that.
(Unless this is some kind of new NDA that allows/mandates itself to be denied)
It can’t get much clearer than that. The whole story is quite weird. I don’t know who should be believed but I’ve never read any such vehement denial. If Apple is lying they risk quite a bit of credibility here
"Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."
Supermicro servers are extremely popular in data centers. Yet no one has noticed anything and no one has found any malicious chips in them. Unless the Chinese secret services also hacked all of the firewalls, someone would have picked up some outgoing packets that are going to Chinese C&C servers. And those would have been scrutinized and chased down. A company I interned at this past summer was scrutinizing every client on their internal network. They would have easily detected a machine that was connecting to an IP outside the subnet, for example.
This whole story is just fishy and every company mentioned in that Bloomberg piece is not only denying it, but strongly denying it ever happened. And Bloomberg reporters have zero evidence... except for some stories from "anonymous sources".
NSM is aware of the issues with Supermicro.
"- We know about this, but can neither deny nor confirm it is correct. We register that this is being denied by the companies", says Monica Strom Arnoy, communication director in NSM to VG.
NSM has, however, been aware that Supermicro may have been compromised, long before Bloomberg's article.
"- We have known about this since June", says Strom Arnoy, who does not wish to further explain from where they have this information.
My reading is that they suspect Supermicro. Further, that they are familiar with the claims about Apple and Amazon but won't confirm or deny whether they are correct. Damming for Supermicro, less clear for Apple and Amazon
Now the article may or may not be a hoax, but a few years ago the NSA was exposed doing exactly that. So at the very least it is credible.
As for detection, I’d rather expect this devices to be activated on demand, like creating a backdoor, rather than sending streams of data from all servers, most of which (per the article) will be owned by an adult website or a mormon church which aren’t exactly strategic activities.
In case of military war, it could DoS many servers at once from the inside.
It could also be used to prematurely break the motherboards, and sell new ones.
There could just be there to ping C&C upon activation in order to locate datacenters, including military ones maybe (information that could be useful in that case of war too).
We can think of a few use cases than just spying on network packets.
Obviously could be many factors, but does strike me as odd.
Would be nice to quantify this.
At least for Amazon, having jepardiced servers, this goes strait to the core of their business namely customer trust. Until now Amazon did not have any data breaches of customer data which they did pride themselves a lot for. If the Bloomberg story is true, this ould be at risk. So, yeah, a strong denial is exactly what one could expect. I'm curious to see that story unfold.
That would be a huge blow in the credibility of the company and would raise serious questions on why they did not move the manufacturing elsewhere.
I wouldn't have expected this extremely strong denial which is one step short of outright calling Bloomberg liars.
They do not have equipment to detect this kind of attack, period. It's not viable for each device, and it's not even viable for sampling a subset of devices from a given production batch. Some components are physically inaccessible and would require desoldering of other components to even access them in any way.
These kinds of attacks cannot be generically detected in any economically feasible way; it must be prevented by drastically clamping down the supply chain and the logistics chain.
Then again, I understand that it could be argued that, if this is confirmed, to me it would seem quite rash from the Chinese, given that they would have known all along that such a scheme would be discovered sooner or later. It is one thing to plant a device as part of a spy operation, quite another to consistently compromise a whole supply chain.
Whichever is the case, the national interest and commercial interests seem to be seriously incompatible with one another when it comes to outsourcing such critical infrastructure to China, this seems obvious to me, regardless of the China policy of who is in government in US.
If this is some kind of ongoing national security issue with nondisclosure requirement authorized by the Director of the FBI, like this big breach could be, people involved are not allowed to talk about it even inside their company.
Of course it would be advisable to inform higher ups in the Apple so that they would not issue a denial.
Regular readers know that a major theme of this newsletter is Everything Is Securities Fraud, so in that vein, let us consider a hypothetical. What if:
1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors.
2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it.
3. The companies are patriotically but falsely denying the hack.
If that is true—and I have no particular reason to think it is, it’s just the sort of hypothetical we like around here—then, obvious question: Is it securities fraud? (Assuming that the hack is potentially material to the companies’ business?) I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.
Probably hard for Apple to prove they have lost money because of the story and it would need to be enough to warrant the time and hassle of a court case. Not to mention they would probably rather avoid having the issue mentioned even more often by dragging it through court for a year or two.
When I've been sued or threatened with defamation lawsuits, the first thing we go over is how we're going to show we did due diligence, how we came to the conclusions, how we reached out for comment and incorporated that comment into the process.
If Apple and Amazon can figure out the sources, there's also potential for them to sue the background sources, though that's usually tough and messy (because then you have to prove they not only defamed you, but also that they were the ones who were the original source so doesn't happen much).
> People claiming gag orders are crazy, mostly for thinking that Apple, or anyone else for that matter, would ever sign a document forcing them to lie to their customers
Why do you think a "gag order" is something that has to be signed off on by the one getting it? That too does not make any sense. An order is issued by someone who has more power, here, the government. You don't have to sign anything for the order to exist. "Gag order" has "order" right in the name.
Left hand doesn't know (or can't know) what the right hand is doing at Apple. Top secret?
Bloomberg was a victim of a hoax, some nation state (huh huhm!) wants to target China for something so they need a story.
Based on what I've read here these past days, I'm leaning towards the second one. Apple can hire the best or all Infosec companies in the world if security was compromised. In other words, they'd know by now, even if they missed it originally. Cat and mouse and all...
Though simplest explanation would be the journalist is confused and the sources inside the company aren’t very good.
Plus, Bloomberg journos weren't born yesterday. This story must've been vetted to the highest levels...looks like they've been working for a year or so on it. It's not like they heard it at Starbucks while waiting in line. They are sources and sources.
Now a nation state can provide multiple sources from different alphabet agencies to Bloomberg.
Let's say Bloomberg was correct. Considering the tension between 'the West' and China, and considering how much it's been escalated recently, the consequences of all this are unpredictably catastrophic.
I don't find it difficult to believe that even just to maintain stability for now, various governments would instruct their corporations to lie in a way they haven't done previously.
It's important to keep in mind that for the first time in a while, there's serious tension between China and the West. On top of that, the idea of having China manufacture crucial infrastructure has probably been a concern for quite a while. Making assumptions based on recent history probably doesn't serve us well when predicting our current reality or the future.
So while I'm still inclined to think Bloomberg got things (mostly) wrong, many of my reasons for believing this are based on a 'status quo' that I don't think is the case currently.
I am sure the CEO must know both the hands, if the hands don't know each other. Also that Apple is going to release a strong denial must also have been known to Tim. I don't think that's the case here. Your second possibility looks more plausible.
There are MANY material things that CEOs do not know about TS/SCI information.
Yes, looks like Apple dropped SuperMicro as their supplier in 2017. Why is that $64k question.
Remember also all the denials regarding PRISM.
If this is being correctly handled by the intelligence services (and you can have your doubts on that following these leaks) then nothing further will come out anyway.
For what it's worth, the Register mentions that Apple suddenly dropped Supermicro as a supplier.
The bottom line is that those denials don't really contribute much information.
Given this is all getting a little fishy I'll share what had me thinking:
1. The article mentions "they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet..."
Servers tend to run on VPNs. This being a dormant backdoor is believable, but then the article mentions:
> "American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected."
Which makes me believe the devices were active and somehow circumvented corporate VPNs. I'm unsure how undetectable this could be using the system's network stack (or if it would be possible at all)-- would the claim then be that this tiny device shipped with a whole TCP/IP layer and some sort of very powerful wireless capability?
2. It continues with: "and preparing the device’s operating system to accept this new code"
Is this possible? Where would a device like this need to be wired to be able to write to memory with some arbitrary payload to do this? From the pictures it looks like it has 6 pins maximum-- could this do? If so, wouldn't this mean this device would need to do some next-level signal processing that would probably require advanced computation? Could said computation be done by a processing unit that fits the size of this chip?
Moreover, assuming it takes control of the OS independently would imply there's some decent amounts of memory in here, to hold the payload, etc. no? But if it's just a backdoor that doesn't take control of the OS, then how is it communicating over the internet with other machines like the article claims?
Again, I might be wrong and things that I don't think possible might. I'm mostly just curious to know if my intuition is too naive. Please comment below if you know more about these things than I do.
EDIT: I was really disappointed that the article itself didn't go into these technicalities, because IMO this would be an impressive feat and newsworthy by itself. The lack of alternative coverage in sources more close to technical expertise was weird to me.
Focussing on the technical is focussing on the wrong thing. Bloomberg's point was not that BMCs can do this, nor that they can be made to do this. People have discussed these on-board systems and their problematic natures rather widely. It was that the supply chain is vulnerable, and that (on the assumption that Bloomberg is right) this problem with the supply chain is no longer a hypothetical case of what an attacker government could do, but is now a documented case of what one government has done, a few years ago.
A few pins can be enough to write to memory, if there's an interface for it (SPI, I2C, but I'd be surprised to see those unprotectedly lying around on a server board ...). It's be very slow, but I think networking could work as well. With efficient software the resources needed on the chip itself could be held very low and eventually offloaded to the main processor.
I don't think it has any wireless or DSP stuff.
I don't know enough about processors, memory, networking and low level software security to know how difficult it is to compromise a computer with such a chip on the main board, but the chip itself certainly seems feasible.
But: It must be really expensive to manufacture the compromised devices. It just seems stupid to manufacture every device with such a chip. 1/1000 seems reasonable. The larger companies have (had?) lots of them anyway.
Isn't this practice known, if not common, in the infosec/intelligence communities at the nation level? There's lots of stories of hardware exploits in copy machines, faxes, etc that took place during the Cold War.
These are quite mature and popular technique.
Rice is too big to be unnoticed....
Firstly, why would you add a new chip to a board, rather than alter an existing one? That would be essentially undetectable.
Secondly, why Bloomberg? It's an odd organisation to get a scoop on something like this.
Thirdly, they talk of the PLA approaching plant owners and such; to do all this, a lot of people would need to know about it, from the top to the bottom. I imagine that would be very difficult to keep secret.
Finally, the timing is very suspicious - it comes with midterms approaching, and Trump and China arguing over trade tarrifs; it would serve the political narrative well for China to be painted as the 'bugbear de jour', and this also plays to the MAGA crowd.
Altering the flash chip would be too obvious. Looking at the flash image (dumping it) or chip (x-raying it) would be the first thing anyone would do if they suspected something fishy. Swapping a flash chip with a compromised one is a textbook 101 supply chain attack...
However a small rogue chip sitting on the SPI link (between the flash chip and the BMC) can be very sneaky: it can replace legit code with evil code ONLY when the BMC is booting up and loading code from flash. The rogue chip would not do that when the flash is read for verification (think dieselgate: a VW car disabled cheats when it detected lab testing conditions!)
Also Bloomberg talks about this rogue chip being sometimes hidden within(!) the fiberglass layer of the PCB. This is the ultimate stealthy attack. No one expects the bare PCB itself to be already compromised by a backdoor even before components are soldered on it...
Well the bloomberg article said that intelligence sources knew that those boards were going to Apple and Amazon, so presumably (if the story is true) someone did speak, maybe
The Trump doctrine is largely built on fictions anyways, and the international community seems to have settled on "distraction" as the only worthwhile strategy for dealing with him.
like offering cheaper, gray market source of capacitors to plant owners? that one got out what, 3 years after the fact? and only after hardware started dying en masse.
Not targeting you with this, but it seems a bit harsh to get downvoted for what was obviously a mistake.
Right now, most of the tech industry, and a good portion of the news media are at odds with the executive branch of the government.
This article puts at least one popular news outlet against several tech industry giants. Divide.
What comes after divide? ...and who has the most to gain? I doubt it's actually our executive branch. I think they could be getting played just as much as Bloomberg and the Tech industry.
>"We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order."
Their whole business is built around lying to customers.
Apple apparently entirely dropped Supermicro as a supplier over a few weeks when they were planning a large order(source: theregister.co.uk).
The ones who should strongly deny such a story, if it is indeed incorrect, are Supermicro. Is there a statement from them?
Edit: yes, there is. They are "not aware of any investigation".
That tells me all I need to know...