> Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."
I have done this before, and we actually found an unspeced part! Thankfully, it was not from a malicious state actor, but just one supplier being creative and not telling anybody. Especially if you don't have an iron grip on your supply chain, you have to be vigilant. As a manufacturer, there are more problems to watch out for than espionage.
I also found parts that had been changed without notice (one of which had the potential to be tons of fun because it was a crystal oscillator with a far worse tolerance than the original).
When the supply chain itself, the management effort and the handling of the supply chain gets so large that it's done almost completely overseas, by a whole team of different people, under constant time pressure and in various degrees of partnerships with other companies (not just those who sell the supplies), these things can slip between the cracks surprisingly easy.
As for stupid changes, while I don't remember the details now, I definitely remember drafting at least one schematic that supported accessing the same peripheral in different ways (or something of this type?) because we couldn't figure out the best one (or the right one?) from the datasheet. It's definitely the kind of thing that I'd rather not see in a final design, and which I'd iron out in a subsequent revision, but I suppose if you work under the consumer industry's tight deadlines...
Oh, and of course, some PCB traces literally don't lead anywhere in the connection sense. E.g. guard rings aren't there to connect electronics together. I suppose it wouldn't be hard to mask some malicious connections that way.
Why make something easy to photograph when you can embed it an area that can only be seen in an x-ray?
That’s how I’d do it at least.
But... why put it on an unpopulated footprint. Why not just replace the original Quad SPI IC with a backdoored device?
It’s not like doing both is extra sure, it’s just weirdly difficult and more easily detectable to do it in the way described.
> to prevent
That being said, SVP level people did categorically deny it, and I can’t see them doing so unequivocally unless they really believe that will hold up on the court of public opinion for their entire tenure there.
Building a network tap into a CAT5 cable isn’t that hard you just need to essentially modulate the traffic into RF and have another implant near by that can intercept the radio signals.
We live in a day where we have demonstratable side channels attacks against RSA keys by listening to how a laptop squeeks when it’s under load and you think building a chip into a cable is a stretch?
Heck I have one cable like that atm which is a converter from a model M keyboard to a USB you can’t see the converter IC it’s built into the RJ45 connector the keyboard originally used without any additional bulk.
I’ve seen a demonstration of similar taps on VGA cables that transmit the entire image to a remove reciecer which is often implanted near by in a power socket, light fixture or anything else where you have a reliable power supply and enough wires to hide and even transmit a signal out side over the power lines.
All of these taps rely on external transceivers that will either record the traffic for later extraction or exfiltrate it through other means.
Also while PoE has more power it’s also more sensitive to voltage drop over the line which means the tap it self will be detected while normal Ethernet works from -+0.5v to -+2.0v without any issue the voltage range is to allow for voltage drop over longer cables so any drop would be ignored by both end points as they’ll just assume the cable is a few meters longer.
Again, as someone who worked directly w/ executive level people (and lawyers), the denials mean only that the statements were cleared by lawyers and probably corporate communications staff. (Obviously, Elon @ TSLA's an exception.)
After reading the story and the discussions here, I reviewed some of the statement language. There's lots of wiggle room.
All is further complicated if US gov't agencies etc., are concerned. Rules and regs that the average person thinks will apply won't. This is just how it works.
And not even then.
I suspect if someone did find something like this, they were told to bury it for plausible deniability.
"Okay, crap, you found a hardware security breach. We'll tell people inside to quit buying those servers, but we'll cough up some other reason. Don't breathe a word of this any further."
Is it possible that many of these sources received their information from others on that list of 17, propagating imperfect or inaccurate information? I'd assume it's standard practice for journalists to confirm that multiple sources aren't essentially from the same source, but this doesn't look great.
Terminal sales are the metric Bloomberg cares about.
SuperMicro was delisted from Nasdaq in August  after failing to meet its reporting reauirements “amid an ongoing audit committee investigation” . This is a name already receiving attention from the analytic parts of Wall Street.
Bloomberg has more to lose than BuzzFeed.
I do not believe it is.
Sure, you could see it, but to know it was wrong you'd have to have a non-compromised board to compare against. Or knowledge of every design and supplier decision, which Supermicro/Apple do not have.
(the whole vid is worth watching)
Bloombergs "chipgate" fails Occam's razor and this whole story is losing credibility by the hour.
If the story is not true, occam's razor would suggest the journalists just got it wrong, maybe by turning a molehill (couple of hacked servers or server firmware) into a mountain (industrial scale espionage) .
People should look what was deemed critical infrastructure and manufacturing during war time (e.g. WW I & WW II). It isn't particularly difficult to understand why certain companies were/are continually bailed out.
China has far more to lose here. Companies losing money to cycles spent auditing hardware is trivial compared to companies permanently losing business due to loss of trust as a part or the supply/manufacturing pipeline.
Jedi72: Posits possibility that the story could be fake and planted by the US as a propaganda tool for their trade war.
steve19: Posits reasons for why it's not likely to be a propaganda lie planted by the US.
inetknght: Posits that it could be China that planted the fake story for the sake of making US industry waste funds (again, this is all still a hypothetical conversation based on Jedi72's original contemplation of the possibility of the story being fake).
topmonk: Posits that if China planted the story, China could lose reputation and should be given the bill for the audits. It's possible that topmonk misunderstood inetknght and was in fact referring to the US. But I'm assuming that topmonk understood inetknght to be referring to China and so posited that China should be left with the bill if the story turned out to be fake.
Me: First, questions how the US would make China pay the bill. Second, notes reasons why it doesn't make sense for China to start spreading misinformation like this (again, going on Jedi72's contemplation of the possibility of the story being fake). Ironically, your reply actually bolsters my original contention against topmonk's comment. It makes no sense for China to want to plant a fake story like this.
I think you aren't understanding my comment, and you confused ra1n85 with your response to me.
It's strange that you think I believe that it would make sense that China would plant a fake story. As you wrote yourself, "topmonk: Posits that if China planted the story, China could lose reputation and should be given the bill for the audits."
I actually said "An argument could be made..." I meant that if there was a trade negotiation going on, the USA could bring this up as a way of trying to force China to give up something else.
But, back on topic. I agree with you, as you said, "It makes no sense for China to want to plant a fake story like this." Can you tell me where I suggested otherwise? I reread the whole thread and I can't see why you'd think I thought that China might have planted a fake story, or it would be a good idea for China to do so.
Ignoring ra1n85 (it seems the 3 of us are probably on the same page), my original question for you was how would you make China pay the bill. You say that the US could use this as leverage to force China to give up something else. I'm not confident that's great leverage by itself, but I just thought of something that may answer my own question.
IF China had the gall to plant a fake story, I would imagine it would be a sign of weakness. There's no reason to plant a fake story if their bargaining position is strong, so a fake story plant would be only the craziest of Hail Mary options, a poison pill that acknowledges, "hey, we're gonna lose, how can we cause the most damage we can before we go down completely?"
So if the US was able to confirm a fake story plant by China, they'd have more confidence in going full court press and getting everything they want because their bargaining position would just be that strong (discounting actual war).
I suppose discussing all the hypotheticals for what happens if China plants a fake story is getting out of hand and not worth all the typing....
Tomorrow: Oh, your servers /parts /smartphones are made in China?
I think it’s just a case of national-security actors sending out a message while leaving FAANG with enough plausible deniability to avoid tanking the whole market. As long as it’s semi-official, the only victim will be Supermicro, and everyone else will have received a message that they should pay more attention to their supply chain.
The problem with this theory is that Bloomberg says all the sources were from the American govt.
Bloomberg provides zero evidence this happaned, outside of their anonymous sources.
How do you know Bloomberg's sources don't? They're anonymous, and while they might know about the implants in detail, they may not have the authority to take examples on a public dog and pony show.
Also, if they want to keep their anonymity, they probably have to be careful about what gets released in order avoid exposing themselves. For example, if you have a limited-distribution report you want to leak info from, leaking a summary of the report is a lot safer than leaking the report text itself. At a minimum, the latter narrows down the leaker to someone who had physical access to a copy.
That tone is pretty uncalled for. The Bloomberg story may or may not be completely accurate, but it's fairly detailed and plausible. While you may categorically distrust anonymous sources, it's not stupid to think they may sometimes be right and that you can trust reputable journalists to vet what they say a fair amount of the time.
This story is still young. I wonder what other news organizations can find out about it (beyond the press release responses).
Again, how do you know this story hasn't been confirmed by multiple sources and isn't backed by solid facts? IIRC, Bloomberg claims they confirmed details with sources within the US Government, Apple, and Amazon. Apple and Amazon have issued denials, but its quite possible those denials may have been lies or the people who made them may not have had all the facts.
How do you fact-check anonymous? Plausible has nothing to do with it. Plenty of things can be plausible, but that doesn’t make them even slightly true.
Reuters as a counterexampke, doesn’t ship anonymous stories but Bloomberg has occasionally dipped into willful innacurracy in the past. Here is one example: https://www.newswire.ca/news-releases/bloomberg-continues-in...
Assuming credibility for an anonymously sourced story is a folly, especially when the allegations are both market-moving and completely unverified. It’s irresponsible. They should have held the story until they had verifiable info.
It doesn't mean there are no dishonest journalists and made up sources, but assuming a source is real it is never without any verification at all.
Certainly I might not believe "briandear" writing an article with only confidential sources (and really, that should be the term, not anonymous), but you don't make your living by being a reliable source of news.
Not to mention, if this hardware had been trying to phone home, it's safe to assume it would have set off some kind of an alert at at least one of these places.
"...let us consider a hypothetical. What if:
1. Everything in the Businessweek story is true, Chinese spies planted hardware backdoors in computers built and used by major American companies, and the FBI investigated along with those companies and discovered the backdoors.
2. It is a national-security secret and the companies were instructed by the FBI never to acknowledge it.
3. The companies are patriotically but falsely denying the hack."
But no, they went thermonuclear on the denial.
“In 2016, Apple informed Supermicro that it was severing their relationship entirely—a decision a spokesman for Apple ascribed in response to Businessweek’s questions to an unrelated and relatively minor security incident.”
Maybe at some big companies, but not anywhere I've worked. I hardly know anyone who audits outgoing traffic with dedicated hardware.
I wonder if there is some magical market cap boundary beyond which companies stop being grossly negligent. We know it's over 200B as Intel somehow never bothered fixing their products for decades, let's hope five times that is big enough.
It would most certainly be illegal.
Bloomberg probably ran this hoping that now that people are looking, some folks outside the circle of anonymous sources will find the chip so that they don't risk exposing their sources.
The story is so explosive that I find it very difficult to believe that Bloomberg isn't on very solid ground.
Nevertheless, getting hold of irrefutable physical evidence may be very difficult. By breaking the story, they now have lots of people now looking for that evidence.
In addition, they may now have enough cover to be able to actually present evidence in their possession and claim that it came from an outside source in order to protect their sources.
There is no good reason for Bloomberg to lie about this as it will significantly damage their reputation and bottom line if proven false.
Now, that doesn't mean that Bloomberg wasn't the target of an operation and was given planted, false information to trace leaks. However, as this has been in the playbook very recently, I would expect the press to be on guard for this.
How many people at Apple or Amazon have the ability to steal compromised hardware and surreptitiously hand it to a journalist? That seems like a pretty lofty expectation.
My guess is that Elemental was specifically targeted because the cost of doing so would be pretty small and with nearly a 100% chance of success. Back in 2015, Elemental was nearly guaranteed to be acquired by one of the greats (Apple, Google, Amazon, etc.) because they had grown too large to be acquired by smaller companies but were also unlikely to go public on their own. The company was doing very well, plus they had government clients.
Knowing that Elemental would likely be acquired and infecting their hardware beforehand would have been pretty sophisticated but also an easy thing for a malicious party to do. Even if the hackers didn't know/plan for Elemental's acquisition, they still would have been a great target based on their government work.
(I'm not trying to fault Elemental; I would expect the same thing to happen at basically any small company that employs maybe 10 hardware specialists)
And if the story were fake, why would Elemental even be mentioned? It's too small and obscure to be of note otherwise.
I don't think the acquisition potential had anything to do with it being a good target. It was all about the government clients.
I could see the acquisition potential as actually being a downside. Apple, Google, Amazon, etc. have histories of acquiring companies just to withdraw their products from the market.
E.g. if Apple contacted the FBI about this, then who at Apple did so (or at least what was their role), on what date, and what FBI office? Or how did Apple detect it in the first place, what happened next, etc. Even if sources can't provide technical details, they should certainly be able to provide names and dates.
I'm pretty sure there is an open line between Apple and the FBI for these exact risks. Else I don't see how these cases do not get investigated multiple times in parallel.
No, they have to provide evidence. Evidence is what distinguishes legitimate assertions from 'just claims' in everything from science to law to reason and rationality in general. Bloomberg assembled extensive evidence.
I've got an X10SAE and X9SRA I could check, but I would be surprised if the same thing targeted the consumer market. It seems like looking for redundant SPI flash and/or unpopulated/half-populated footprints would be a start. Although I've got to wonder if the implant was really using a redundant footprint for the flash, why it wasn't just in the appropriate package rather than the custom jobber Bloomberg implies.
Frankly I've got to reread that original article. It gave me a headache with the continual reiteration/illustration of just how small the implant was, and other anti-informative cruft. Wait until they find out about the size of transistors inside CPUs...
Agreed that article was painful.
I2C works closer to how you're thinking, but even there a hostile implant doesn't need to have an protocol-dictated address to corrupt someone else's traffic.
I suggest the skeptics keep an open mind, instead of categorically denying it could be true, just because a couple of for-profit companies don't want to see their stock plummet the way Supermicro did. Nothing reported so far is out of the realm of plausible, considering the value of a successful supply-chain attack against tech companies.
I would love to see some research on accounts and comments on HN similar to Twitter analysis post 16. Seems to me any time China is broached the HN thread gets more comments than average. Many posts read to me as strongly defensive or taking straw man/obfuscation type tactics.
But then again that could be personal bias I don't know the actual human composition of HN comments - which is why I would love to see some research on HN comments/accounts.
What would you call Iraq WMD stories?
Could be the case here as well. Might not be, but there certainly would be a motive for it, (trade war).
For a prior see https://en.wikipedia.org/wiki/BadBIOS
Times we live in...
I mean ask Joe Nacchio how going up against the NSA worked out.
* this all seems within reason, knowing the hardware
* the denials are unusually strong
If any of it is true or not I don't know, but the IPMI stuff is crappy, if not backdoored.
We’re apparently home free by using Azure, but I think the responses are justified if the story is fake.
If you're just ordering boards through a reseller, I wouldn't expect those to be infected, but, when you're ordering 10,000+ servers at a time, you'll get your own Part Numbers, your own specs, and your own build times/specs.
If they were lying about this why wouldn’t they lie about lying about it?
They managed to write 30 breathless paragraphs about how the UK police were inexplicably treating what they claimed was the Russian murder of a key scientist in the Litvinenko investigation as a suicide, and that the police had mysteriously testified that “no-one in his family seemed particularly surprised he had taken his own life" even though some of his relatives had suggested foul play, before they thought to mention that he'd been showing signs of depression for some time, his wife said he'd tried to kill himself the week before, and there was no signs of foul play and no evidence anyone else was present. They then argued this shouldn't have ruled out foul play because he could've been given some kind of secret Russian mind control chemicals. Seriously.
To make matters worse, from what I can tell he didn't even play the key role in the Litvinenko investigation that they claim he did. He merely recalculated Litvinenko's exposure in 2010 after it was discovered he was exposed twice rather than once - several years after all the announcements and further investigation the Buzzfeed story portrayed as a direct result of that work, and well after this had been clearly pinned on Russia. The original analysis was done by different scientists who are presumably still alive.
That's exactly his point. He's saying they always wanted to be legitimate news, but didn't have the money, so they built their war chest using clickbait.
Most of the 20th century! Have you seen their expense accounts?
Even if this attack actually didn't happen, you can be damn sure that the tech giants now massively will intensify efforts to prevent hardware hacks like this will ever happen to them.
The economic damage would be huge, and the value of the data they could gleam seems worthless by comparison.
If this is as common as it sounds someone will get a chip and do a teardown and the similarities to BadBIOS will be gone.
So if Apple found a batch of 7,000 manipulated boards a year earlier, why would that not cause them to drop Super Micro as a supplier?
A government gag order is plausible, but is a government keep-buying-malicious-hardware order a thing?
You don't want to tie the two events together. If the article is accurate, the hope would be that by waiting, the could garner support with comments like yours.
e.g. Let's wait 6 months before you do something in response to something today so you can say that this has nothing to do with the even 6 months ago.
There's so much attack surface at the motherboard "management" hardware level that some kind of attack wouldn't be all that hard.
1. US intelligence planted & played along for this story, for a long time.
2. The story is true on all fronts: i.e. those inside Apple with knowledge about this are lying to senior executives under immunity protection from US intelligence/law
Personally, I think #2 to be a lot more likely - US intelligence has managed to sneak in backdoors into tech forever, in cahoots with sympathizers who probably have immunity agreements if outed.
There is just too much lined up against that single article...
As it is, Bloomberg just kind of said, there is this issue that we're certain exists. So the industry is left to guess what the issue is in so many ways.
I think we'll all need to wait for the outside reporters and investigators to run some of this information down to get a better idea of what's going on. Because right now, even most of us are just guessing at what it could be.
They are legally required to tell the truth.
They've lied in the past to their shareholders.
Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the story is just a blunt lie fabricated by Bloomberg serving as a propaganda to bash China amid the Sino-America trade war.
Bloomberg is a journalism organization. They'll report and cross-check testimony that there's been a chemical weapons attack in Syria, but they're not going to go there to collect samples of the chemicals. They aren't going to have "the code/server/chip" to show you, and they shouldn't be expected to.
Furthermore, the people who talked to Bloomberg who may have access to the "code/server/chip" are anonymous and may not be able to have too many details released publicly without compromising their identities. Bloomberg may have more details than they have reported, but be unable to release them publicly while respecting their sources' confidence.
This is where you misunderstand. An anonymous source isn't "just some dude" who called in "saying so." Journalists, in the case Bloomberg, knows exactly who their "anonymous" sources are.
The question you have to ask yourself is whether you trust Bloomberg or not. If you trust them, then you trust that they did their due diligence. if you don't trust them, then nothing presented by them will get you to trust them.
I swear, it's like people don't know how investigative journalism works. Anonymous sources aren't anonymous to the journalists.
As for the companies denying this, this wouldn't be the first time they've lied in such a manner.
Tech companies don't want to be hacked. And if they are, they want to be able to say "we cleaned things up and everything is safe now," not "we were infiltrated several years ago and have no idea what the malware does or even which systems it impacts."
Does anyone more knowledgable know if this must be an at-the-factory thing, or if it's possible to do this afterwards, "interdiction" as the bloomberg story put it?
If such a small chip can steal info, why can't the remaining other large quantity of "normal" chips on the same board?
It's already creeping into business sections, just make it stop.
If it's important stand up and put your name behind a story as a source, everyone just cowering in the corner because they want to keep their careers safe is making things worse, not better.
Anonymous sources have been around as long as journalism. It’s not like what those sources say is taken as a given, they are heavily corroborated against other sources of information, often documents/records/etc.
Look into Softbank.
To know who is telling the truth you must know what the government policy really is related to NSA exploits and how bold NSA is when protecting secrets.
These things can be verified only when whistleblowers release documents. Snowden and other whistleblowers have revealed multiple lies, including that Director of National Intelligence James Clapper lied under oath.
That said, it's also possible that Riley & Co. rely on bad or unreliable sources.
It's evidence that US intelligence community sources can be unreliable. That's directly related to this story because Bloomberg claims that many of their sources are from that community.
1.The intelligence flaw is a high order lie in order to mislead public to believe the war is launched by wrong information. It's not. The intelligence flaw is irrelevant or is intentional. The war is predetermined and flaw intelligence has nothing to do with real cause of the war other than a cover afterwards.
2.The narrative that the Bush Administration launch Iraq war with flaw intelligence is another lie that the majority of public who support the war were not responsible because they were misled by their leaders. The war is launched by not only Bush administration but also UK. The main stream media were not orchestrated by government like a totalitarianism regime can do. For example, CNN interviewed a famous Iraq nuclear scientist again and again to sell the impression to the public that Iraq DO have nuclear weapon. CNN independently promote the war which happened to match the government agenda. US/UK are democratic countries. Both countries collectively(meaning enough portion of the people ) decide to over thrown Iraq regime
I hope you realize this was all an invention of the US intelligence community — they knew all along it was bullshit. It's well documented at this point.
In this case the media is doing it themselves from the start.
At that time I did not even have internet and most people knew it was bullshit.
The only thing that needed convincing was that the US would do it anyways, so better to have a smooth pathway.
They also made perfect sense for missiles/rockets.
The chemical weapon claims were believable. I mean, didn’t we help Iraq manufacture chemical weapons during the Iran/Iraq war? Wasn’t that long-suspected belief later confirmed?
So, yes. I knew at the time, and I said so. I wouldn’t fundamentally have had much of a problem with invading Iraq (one could have that argument), but the justification, timing, and prioritization didn’t really make sense. Watching Powell pitch that goat rodeo was pretty sad.
I don't know how much of the US population is aware of their governments actions throughout the 20th century and the impact to it's public image.
"Well, if crime fighters fight crime and fire fighters fight fire, what do freedom fighters fight? They never mention that part to us, do they?"
See the PNAC documentation, if nothing else. You don't need to be a card-carrying member of the Illuminati to understand the personal and political dynamics that existed between Saddam's administration and Bush 43's, or to foresee what was likely to happen.
The 2003 Iraq invasion had nothing to do with warnings from informants about state secrets.
The idea very idea that there was any concern about Iraq's capability to wage war is a joke. Iraq was pretty well softened up by no fly zones and sanctions, so as to be sufficiently anemic, and decapitating the incumbent dictator for life (literally) was mostly just sour grapes for him going off script, and besmirching the sanctity of Kuwait.
It was gloves off for Iraq, as soon as the 9/11 hijackings unfolded. Literally next month people were whispering about Iraq, even though Afghanistan was well understood as the official point of origin for the attacks.
Not only that, even though Afghanistan played an actual role, the majority of the hijackers were Saudi. People sort of mention that in passing and then go back to pretending it has no relevance.
Except for the large (15 of 19, plus OBL), percentage of Saudi nationals who perpetrated the attacks?
Nonetheless, I'd agree that waging war on Afghanistan, The Country would be just as silly as waging war on Saudi Arabia. It's like Canada waging war on both the United States and Italy, for something The Mafia perpetrated.
Meanwhile, war with Iraq was akin to Canada invading Norway for it's whale blubber, because the Norweigan king sunk a fleet of Danish whaling ships ten years prior, and was now suspected of hoarding a cache of illegal harpoons. Thus triggering a cascade of geopolitical events, whereby Canada stepped in to defend Denmark, thus angering a member of the Gambino family, who subsequently demolished the CN tower, for tampering with Denmark's sovereignty. As if to say that had Norway not attacked Denmark, the CN tower would not have been destroyed by a hijacked train derailment.
Made up, manufactured "evidence", not faulty sources.